Flash (storage) is deployed to provide data centre applications with high input/output rates. However, its capacity and bandwidth are often underutilised because servers aren’t readily designed to accommodate Flash, memory and CPU over time and across multiple applications.

This problem has become more acute as organisations are increasingly deploying hybrid cloud models, taking advantage of both on-prem data centres and public clouds — cloud architectures are leading to flash storage disaggregation.

“As cloud data centre architectures continue to evolve toward shared infrastructure and automatic orchestration, the tension between storage performance and exibility is becoming paramount,” confirms a 2017 whitepaper from Toshiba.

 What is Flash storage disaggregation?

Flash storage disaggregation combines both SSDs and HDDs (performance or capacity) to provide a high-performance aggregate more economically and flexibly than an SSD aggregate.

It helps organisations address the imbalance of different resource requirements and the usual over provision or underutilisation of data centre resources that follows.

“Flash capacity is often underutilised as servers are deployed with enough Flash to satisfy projected future de- mands. Deploying high-capacity Flash devices provides on-demand flexibility as application requirements vary or spike,” according to a 2016 Stanford publication.

 Market trends driving the need

There are a number of market trends driving the need for Flash storage aggregation.

1. Flash storage — the acceleration in Flash adoption, acting as storage medium that enhanced the storage of computers, has lead to its cost decreasing. The challenge now is address the performance and endurance issues with a disaggregated solution.

2. Networking technology — networking speeds are increasing and 100GbE is now viewed as a commodity. This enables the disaggregation.

3. Applications demand — storage is growing faster than compute and data centres are increasingly moving to cloud native architectures. This creates the need for storage solutions that have high elasticity and agility, with high availability and reliability.

4. Hybrid cloud — as mentioned, hybrid cloud is being adopted to improve operational efficiency. This needs solutions that enable hyperscaling in private clouds.

All these factors mean that traditional storage is now in decline:

Flash storage: transforming the storage industry

The storage industry is going through a massive transition, which is being caused by flash. This has allowed a number of disruptor brands to challenge the incumbency of legacy vendors which have been dominant for decades. Read here

Requirements of cloud native applications

The ever growing list of cloud native applications share the need for low latency, high bandwidth, consistent response times, deployed on local flash (NVMe) and are usually ‘bare-metal’.

Currently, these applications all suffer from poor local flash utilisation (in terms of capacity and performance), long recovery times when a node fails, which is detrimental to network, physical ties to application servers and flash endurance issues.

This leads to severe underutilisation of flash storage capabilities — between 50-85% of Flash expense is wasted.

Flash storage disaggregation with NVMe/TCP

Applications are increasingly data-driven, processing higher volumes of data at low latency. To meet this and the above challenges, cloud providers must continually innovate and refine storage technologies.

Providers, like Lightbits Labs — who Information Age spoke to as part of the most recent 34th edition of the IT Press Tour — helps these providers deliver hyperscale storage to private clouds by disaggregating storage and compute over standard networks.

Founded in 2016, Lightbits Labs pioneered NVMe/TCP or Non-Volatile Memory express (a protocol for accessing high-speed storage media) over Transmission Control Protocol (a main protocol of the internet).

NVMe/TCP storage is designed to deliver low latency, making cloud-native storage easier to deploy at scale.

NVMe opened the bottleneck for hyperscale storage

“Our hybrid cloud offering came about because clients wanted NVMe to run software on the public cloud as well as on-prem,” said Kam Eshghi, chief strategy officer at Lightbits Labs.

“We’re utilising the existing infrastructure that customers have on-prem, on the public cloud to deploy NVMe-as-a-Service. This creates better performance and more flexibility as a black box solution on the public cloud,” he added.

This idea of hyperscale and shared storage is nothing new. But, providing clustered NVMe/TCP shared storage that behaves like local storage (data centres or edge computing devices) in terms of latency and performance, is. Delivering that disaggregated storage will lead to Flash optimisation.

NVMe – a revolution in storage

NVMe is a new interface standard for storage, designed to optimise the higher performance of newer flash drives. Read here

Disaggregation in practice

The early customer experience with LightOS, Lightbits Labs’ software-defined storage solution, has been generally well-received.

One customer in the financial services industry said: “We have chosen NVMe/TCP for our cloud…Lightbits is the only vendor who was able to get a storage solution to work in our environment, integrating with OpenStack and k8s.”

Another infrastructure-as-a-service customer said: “With Lightbits technology, we can provide network attached NVMe devices with as good or better performance as local NVME over our standard 10/25G Ethernet network.”