Hewlett Packard Enterprise Persistent Memory

Hewlett Packard Enterprise (HPE) has announced a new product that uses non-volatile DIMMs (NVDIMMs), which they are calling Persistent Memory. This short video gives a high level view of how it works, via a whiteboard time lapse. The initial product is an 8GB module which has 8GB of DRAM backed by 8GB of flash for $899.00, which is pretty pricey!


Figure 1: HPE 8GB NVDIMM

HPE has a blog post with some more details about their SQL Server testing using NVDIMMs.

They compared the OLTP database performance of a SQL Server database running on an HP DL360 Gen9 server in two scenarios. The details they provide are frustratingly incomplete. They don’t specify what version of SQL Server, what operating system, what type of SSDs, etc.

First Scenario:

  • Data file(s) on six 400GB SSDs
  • Log file on two mirrored SSDs (so two SSDs in RAID 1)
  • 970K transactions/minute
  • 373 µs log write latency

Second Scenario:

  • Data file(s) on six 400GB SSDs
  • Log file on two mirrored SSDs, with two mirrored NVDIMMs as a write-back cache in front of the SSDs
  • 1.08M transactions/minute
  • 181 µs log write latency



Figure 2: HPE SQL Server Testing with NVDIMMs


These results are actually not as impressive as I would expect on the surface, so I would be very curious to more details behind their testing. For example, was their workload previously limited by how fast it could write to the transaction log? After they started using NVDIMMs, did they run into a different bottleneck, such as CPU utilization?

I also want to know more details how this is implemented, with existing server models and existing operating systems. It looks like there is a driver that excludes the DIMM slots that are being used by NVDIMMs from being visible to the operating system as conventional memory, and instead makes them available as a write-back cache layer for an existing storage device. It looks like you can combine multiple NVDIMMs into a single, mirrored cache layer in front of a single storage device. This seems pretty similar in concept to the hardware memory cache in a RAID controller.

This might be pretty useful if you have a workload that is actually seeing bottlenecks writing to a transaction log (or perhaps you have multiple databases with log files on the same logical drive), and you don’t want to use the Delayed Durability feature in SQL Server 2014 and newer or the In-memory OLTP features in SQL Server 2014 Enterprise Edition and newer.

Upgrading a SATA III SSD

Back in April of 2012, I built an Intel Z77 workstation that used a 512GB OCZ Vertex 4 SATA III SSD, plugged into a SATA III port for it’s Windows 7 boot drive. Back then, this was arguably the fastest consumer SSD available, with an MSRP of $699.99. This drive has soldiered on for over three and a half years, with no problems. It was getting a little low on disk space though, so I decided it was time for an upgrade to a larger, faster SSD.

I ended up getting a 1TB Samsung 850 EVO SATA III SSD, for $329.99 at my local Micro Center. The 850 EVO line has been around for about a year now, and prices have come down quite a bit since they were introduced. It is pretty amazing to get double the size (and better performance) at less than half the price, compared to what was available back in 2012.

Before I cloned the existing drive, I ran CrystalDiskMark 5.0.2 on it with a 4GB test file. The results are shown in Figure 1.



Figure 1: 512GB OCZ Vertex 4 SATA III SSD Benchmark results


I used the free Samsung Data Migration software (which only works with Samsung SSDs as the cloning target) to clone the old OCZ drive to the new Samsung drive. I used an Apricorn SATA Wire 3.0 plugged into a front-panel USB 3.0 port to connect the new Samsung drive for the cloning process. I could have shut down the system, and plugged the new Samsung drive into a native SATA III port to get better copy performance, but I was too lazy to do that… As it was, I was seeing about 125MB/sec during the cloning copy process, which was fast enough. If you are cloning/upgrading a drive in a laptop, you pretty much have to use a USB port to do it.

After the cloning process was complete, I shut down the system and swapped the drives. Windows 7 booted up without any problems, although it wanted a reboot once it realized that the drive had been changed. I also noticed that Windows 7 had lost it’s recollection of ever checking for Windows and Microsoft Updates, but asking it to check for updates fixed that issue.

Next, I fired up the Samsung Magician 4.9 software, which informed me that the new Samsung 850 EVO needed a firmware update. Before I ran the firmware update, I ran CrystalDiskMark 5.0.2 with the same settings as the previous test. The results are shown in Figure 2.



Figure 2: 1TB Samsung 850 EVO SATA III SSD Benchmark results (before firmware update)


After the drive firmware update, Windows 7 booted up without any problems, although it wanted another reboot once it realized that the drive firmware had been updated. I ran CrystalDiskMark 5.0.2 once again with the same settings as the previous test. The results are shown in Figure 3.



Figure 3: 1TB Samsung 850 EVO SATA III SSD Benchmark results (after firmware update)

As you can see, the benchmark results improved after the firmware update. I have not found any release notes for the firmware update (and it is not even listed on their web page), but at least the latest version of Samsung Magician knew about it.



Figure 4: Samsung Magician 4.9

I have not enabled RAPID Mode on the drive yet, but I know from prior experience that it can have a nice positive effect on performance. It does make it harder to analyze your storage performance when SQL Server is running on your workstation though. All in all, a pretty easy, trouble-free installation.

Some Quick Comparative CrystalDiskMark Results

(New: we’ve published a range of SQL Server interview candidate screening assessments with our partner Kandio, so you can avoid hiring an ‘expert’ who ends up causing problems. Check them out here.)

A few weeks ago, I built a new Intel Skylake desktop system that I am going to start using as my primary workstation in the near future. I have some details about this system as described in Building a Z170 Desktop System with a Core i7-6700K Skylake Processor. By design, this system has several different types of storage devices, so I can take advantage of the extra PCIe bandwidth in the latest Intel Z170 Express chipset, and do some comparative testing.

The latest addition to the storage family is a brand new 512GB Samsung 950 PRO M.2 PCIe NVMe card that just arrived from Amazon yesterday afternoon. As of now, here is the available storage in this system:

  1. (2) 512GB Samsung 850 PRO SATA III SSDs in RAID 1 (using the chipset RAID controller)
  2. (1) 512GB Samsung 950 PRO M.2 PCIe 3.0 NVMe card in an Ultra M.2 PCIe 3.0 x4 slot
  3. (1) 400GB Intel 750 PCIe NVMe card in a PCIe 3.0 x16 slot
  4. (1) 6TB Western Digital SATA III hard drive in a SATA III port

Since I have an NVidia GeForce GTX 960 video card in one of the PCI 3.0 x16 slots, both that slot and the PCI 3.0 x16 slot that the Intel 750 is using will go down to x8 (which means 8 lanes instead of 16 lanes). The Intel Z170 Express chipset supports 26 PCIe 3.0 lanes, so you need to think about what devices you are trying to use. This system has Windows 10 Professional installed, so it has native NVMe drivers available from Microsoft.

I did some quick and dirty I/O testing today with CrystalDiskMark 5.02. The two NVMe devices are both using the native Microsoft NVMe drivers from Windows 10. As you can see below, both the Samsung 950 PRO and the Intel 750 PCIe NVMe cards have tremendous sequential and random I/O performance!

Device Sequential Reads Sequential Writes Random Reads Random Writes
512GB Samsung 950 Pro 2595 MB/s 1526 MB/s 171755.6 IOPS 104801.3 IOPS
400GB Intel 750 2369 MB/s 1081 MB/s 177938.0 IOPS 151642.1 IOPS
512GB Samsung 850 Pro 1104 MB/s 532 MB/s 100420.4 IOPS 60765.1 IOPS
6TB WD Red HD 176 MB/s 170 MB/s 386.7 IOPS 448.2 IOPS

Table 1: Sequential and Random Results (Queue Depth 32, 1 Thread)

Keep in mind that the two Samsung 850 PRO SSDs are using hardware RAID1, which seems to help their sequential read performance, and that the two NVMe devices are both using the native Microsoft NVMe drivers, which may be hurting their performance somewhat.


Figure 1: 512GB Samsung 950 Pro M.2 PCIe NVMe Results


Figure 2: 400GB Intel 750 PCIe NVMe Results


Figure 3: 512GB Samsung 850 Pro SATA 3 (RAID 1) Results


Figure 4: 6TB Western Digital Red Results