SAS disks shipped with write cache off: ×13 on writes with ZFS
A five-disk ZFS pool managed 95 fsync writes per second, and a four-disk mirror, 108. Several disks came with write cache disabled. Turning it on, without buying anything: 1,237 and 946.
- ZFS
- Linux (Debian, Alpine, Arch)
Check
sdparm shows each disk's WCE (write cache enable) bit. On several of mine it was 0: every write waited to reach the platter before being confirmed.
sdparm --get=WCE /dev/sdX
# Behind the SAS shelf, SATA disks don't accept the normal command
# («MODE SENSE command: Illegal request»): use the 6-byte one.
sdparm --six --get=WCE /dev/sdXIs it safe with power cuts?
With ZFS, yes. ZFS flushes the disk cache when closing each transaction group and on every synchronous write, so a cut doesn't leave the pool inconsistent. What I deliberately didn't touch, since there are power cuts and no UPS: sync=standard and the default transaction interval.
Enable it and make it stick
sdparm --set=WCE=1 --save /dev/sdX # SAS
sdparm --six --set=WCE=1 --save /dev/sdX # SATA behind the shelfMake it survive a power-off
On SATA disks behind an adapter the saved setting doesn't always survive a full power-off, so a systemd service enables it again on every boot, identifying disks by WWN. And a udev rule sets the «none» scheduler on the ZFS disks: ZFS already orders and groups I/O.
Measured with fio
- Five-disk RAIDZ1 pool: 16K random writes with fsync, from 95 to 1,237 IOPS (×13). Sequential was already bandwidth-bound.
- Four-disk SAS mirror: from 108 to 946 IOPS (×8.8) and sequential from 38 to 123 MB/s.
- Always on a temporary uncompressed dataset, deleted afterwards.