Dave - two zpool, or not two zpool, that is the question

New About Yours API Help
1.8 KB, Plain text
Happy FOSDEM folks!

Whether tis nobler in the mind to suffer the slings and arrows of outrageous performance ... etc[1].

I have a fast NVMe disk (yay) and a large slow (by comparison) zpool of striped mirrored SATA disks. Currently these are separate zpools. day-to-day work including src ports and jail stuff runs on the NVMe, and the archive pool is abused for longer-term storage and further databases, bhyve and jail torture. The system runs 12.0-RELEASE-p2 amd64  and has plenty of RAM.

Doing large zfs reads (zfs send > /dev/null) as a naive benchmark, the NVMe drive does a leisurely 1400 GiB/hr | 400MiB/s, and the disk zpool does around 700GiB/hour | 200MiB/s zfs send, roughly half of the throughput of the former. 

The NVMe drive is faster (it's already far faster than my day-to-day needs) by quite a margin, but it's not as fast as I'd expect. Am I missing some secret FreeBSD tunable or am I simply at a victim of lying deceitful vendor-marketdroids? 

Questions:

0.  Can we go any faster here - is this performance in line with other consumer grade NVME-based zpools?

1. How does having 2 zpools impact ARC and other zfs internals?

2. might there be any advantage to using some of the NVMe drive as a write-ahead cache for the larger disk-based zpool? I don't see any advantage in adding the whole 1 TB NVMe drive to the zpool it would be a waste.

3. How can I tell if the NVMe drive should use 4096 byte sectors or 512B ones? The drive reports 512B sectors like all lying storage devices of the modern age [2].

Thanks!

[1]: excuse the misquote from Hamlet I beg you good sirs - methinks A Tale of Two Pools would have been a better choice?
[2]: https://gist.github.com/dch/890ec336875663349a02cbe0b9b19171 has all the pciconf diskinfo nvmecontrol & sysctl output
Pasted 2 months, 3 weeks ago — Expires in 279 days
URL: http://dpaste.com/123AANV