Software driver standardization and performance increase for PCIe SSDs Jerome Gaysse IP-Maker

Explore IP-Maker IP here

Introduction

The PCI Express Solid-State Drives (PCIe SSDs) are the only storage devices which are able to sustain the increasing performance demand for cloud & financial computing, broadcasting or social network. This article describes how the NVM Express specification, based on standard driver, will help the storage players, including controllers, SSDs and servers manufacturers, to enable the broad adoption of PCIe-based SSDs.

Actual architecture and limitations

The PCIe SSDs suffer from a lack of standard driver. Each SSD manufacturer develops its own driver. That leads to a higher development cost, and on the server side, driver integration and qualification process takes a long time.

The typical processing architecture for the current PCIe SSDs is based on two kinds of controller: a main processor (a) and flash controllers (b) (figure 1). On the PCIe side, the main processor embeds a high performance core for NandFlash management software processing, such as wear leveling, back block management and garbage collection. On the NVM side, the NandFlash memories are connected to specific flash controllers. The 2 sides are linked through the SATA communication interface. The main processor embeds up to 8 SATA controllers, and each Flash controller embeds a single SATA interface. Some SSD manufacturers embed directly SATA SSDs, adding connectors and packaging space.

Figure 1 - Current PCIe SSD architecture

This architecture is powerful enough with PCIe Gen 1 devices. Limitations started to appear with the PCIe Gen2 (up to 6.4GB/s for a Gen2 x 16 device). For multiple reasons, that becomes not viable with PCIe Gen3. The first one is that it would require too many SATA links to ensure the throughput. For a Gen 3 x16 device, it reaches 12.8 GB/s. A SATA 3 link is 600MB/s, which leads to 21 SATA 3 communication channels in parallel on a single board. In addition, SATA can provide a limited number of commands (32). For an architecture based on 16 flash interfaces, with 8 chips per interface, a maximum of 25% of the flash memories may be addressed in parallel.

In addition, even if SSDs embed powerful processor, the data transfer control is managed by the host processor, which leads to a significant load for high throughputs. Finally, it is not cost optimized since it requires many controllers on the PCB.

The use of SATA in a PCIe-based SSD is definitely not the technology of the future.

NVM Express Specification

A workgroup composed of storage industry leaders has defined the NVM Express specification which aims to eliminate all these limitations and to provide a standard interface. The first version has been released on March 2011, and is now available under the 1.0b version. For more details on the specification: www.nvmexpress.org.

The NVM Express specification defines a controller interface for PCIe SSD used for Enterprise and Client applications. It is based on a queue mechanism with advanced register interface, command set and feature set including error logging, status, system monitoring (SMART, health), and firmware management).

Thanks to this specification, SSD controller manufacturer will be able to develop a single controller (figure 2) for the whole SSD. It will integrate the PCIe Interface, a NandFlash controller and will be compliant to NVM Express.

Figure 2 - High-performance PCIe SSD Controller Architecture

Storage players will benefit from:

• Driver standardization: an open source version is already available from the NVM Express web site
• Performance increase since the SATA bottleneck has been removed. With an unlimited number of commands, NVM Express is able benefit from all the NandFlash resources in parallel (only 32 with SATA).
• Significant reduced BOM cost
• Reduced power consumption

IPM-NVMe IP core

IP-Maker has developed its own hardware IP, fully NVM Express 1.0b compliant, to be integrated in the SSD Controllers. Using pre-validated NVMe IP core, allows to greatly reduce Time-To-Market for storages OEM which want to benefit from a powerful data transfer manager. The IP-Maker NVMe IP core is full featured, easy to use into FPGA and SoC designs.

Figure 3 - IPM-NVMe IP core architecture

The complete IPM-NVMe system is managed by a submission and completion administration queue (activation of others queues, arbitration rules, request of log, and attribution of events...). All queues are managed independently, so the system is very flexible, and allows user to attribute dedicated queue to a specific action (for instance a queue dedicated to write, and one to read). To add service quality, a set of arbitration is available: either a simple round robin or a weighted round robin. There is also the possibility to execute a configurable amount of command on a queue before requested a new arbitration.

All available memories (cache buffers of page size) are signaled to the system by the status memory bus and the memory dispatcher allocates dynamically buffers to submission queues. All the command executed by the different submission queues are sent to the CPU with the allocated buffer to allow an eventual interaction with a third part system.

The host configures the NVMe controller through the administration queue. It defines the system architecture, the number of submission queues (X), the number of completion queues (Y), the features and the log event. Then, the host may send two types of command: the submission command, based on a 64byte format, it includes the priority, the address of the data to read or write, the size of the packet (4k to 10MB), the command (read or write), and the completion command based on a 16byte format. Finally, the NVMe controller manages all the dataflow, therefore off-loading the host processor.

PCIe SSD integration

The IPM-NVMe IP core is fully NVM Express 1.0b compliant. It supports up to 64k submission and completion queues and a dedicated administration queue. Before silicon synthesis, numbers of parameters can be defined in order to get the best mix performance/gate count. The backend interface is RAM-like native, but wrapper for OCP 2.0 and AHB 3.0 may be adapted. For evaluation purpose, a free evaluation package is available.

Conclusion

The NVM Express specification is definitely the best way for the broad adoption of PCIe SSDs in the marketplace. With the IPM-NVMe IP core, IP-Maker is a technology provider for the next generation PCIe SSDs compliant to a standard interface. It brings software driver standardization and a drastic performance increase. This is the first step, with a primary focus for enterprise and client systems. With a second step focused on the form factor, the industry leaders may go further with the adoption of PCIe SSDs on all the storage markets.

Explore IP-Maker IP here

• IPM-NVMe
• BCH Encoder/Decoder IP Core
• Universal Nand Flash Controller

About the Author

Jerome Gaysse
IP-Maker

Jerome Gaysse is responsible for business development for IP-Maker, with a strong focus on SSDs. He held positions from hardware design at General Electric Healthcare to product marketing at Atmel. He used to present papers at Embedded System and IEEE international conferences