Basic set up of dm-cache

I mostly found bcache to suite me better, but due to a discussion today I came to test dm-cache. I didn't find a great entry doc how to do so, so I thought I could as well document and share here - and by that make "searchable".

So how to actually set dm-cache up on Ubuntu?

1 Answer

I was starting on this info on get most out of your nvme, but also man lvmcache is a good resource.

I have (sorry) not more disks around:

/dev/sda2 (931G slow)
/dev//dev/nvme0n1 (372.6G fast)

Basic setup:

$ sudo pvcreate /dev/sda2 Physical volume "/dev/sda2" successfully created.
$ sudo pvcreate /dev/nvme0n1 Physical volume "/dev/nvme0n1" successfully created.
$ sudo vgcreate cache /dev/sda2 /dev/nvme0n1 Volume group "cache" successfully created
$ sudo lvcreate -L 200G -n origin_device cache /dev/sda2 Logical volume "origin_device" created
$ sudo lvcreate -L 60G -n cache_block cache /dev/nvme0n1 Logical volume "cache_block" created.
$ sudo lvcreate -L 2G -n cache_meta cache /dev/nvme0n1 Logical volume "cache_meta" created.
$ sudo lvconvert --type cache-pool /dev/cache/cache_block --poolmetadata /dev/cache/cache_meta WARNING: Converting logical volume cache/cache_block and cache/cache_meta to pool's data and metadata volumes. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert cache/cache_block and cache/cache_meta? [y/n]: y Converted cache/cache_block to cache pool.
$ sudo lvconvert --type cache /dev/cache/origin_device --cachepool /dev/cache/cache_block Logical volume cache/origin_device is now cached.

After that you can use that device "as usual". I also created a non cached device as reference for a basic test:

$ sudo lvcreate -L 200G -n origin_device_reference cache /dev/sda2 Logical volume "origin_device_reference" created.
$ sudo mkfs -t xfs /dev/cache/origin_device
$ sudo mkfs -t xfs /dev/cache/origin_device_reference

And mounted it

$ sudo mkdir /mnt/lv-xfs
$ sudo mkdir /mnt/lv-xfs-cached
$ sudo mount /dev/cache/origin_device_reference /mnt/lv-xfs
$ sudo mount /dev/cache/origin_device /mnt/lv-xfs-cached

After this my setup looked like this:

$ lsblk (filtered of other disks)
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
|-sda2 8:2 0 931G 0 part
| |-cache-origin_device_reference 252:4 0 200G 0 lvm /mnt/lv-xfs
| `-cache-origin_device_corig 252:3 0 200G 0 lvm
| `-cache-origin_device 252:0 0 200G 0 lvm /mnt/lv-xfs-cached
nvme0n1 259:0 0 372.6G 0 disk
|-cache-cache_block_cdata 252:1 0 60G 0 lvm
| `-cache-origin_device 252:0 0 200G 0 lvm /mnt/lv-xfs-cached
`-cache-cache_block_cmeta 252:2 0 2G 0 lvm `-cache-origin_device 252:0 0 200G 0 lvm /mnt/lv-xfs-cached
$ sudo dmsetup table
cache-cache_block_cdata: 0 125829120 linear 259:0 2048
cache-origin_device_reference: 0 419430400 linear 8:2 423626752
cache-cache_block_cmeta: 0 4194304 linear 259:0 125831168
cache-origin_device: 0 419430400 cache 252:2 252:1 252:3 128 1 writethrough smq 0
cache-origin_device_corig: 0 419430400 linear 8:2 2048

Please be aware that dm-cache has evolved a lot. There are still many guides that recommend to tune the cache with "dmsetup message ...", but that all is part of the old "mq" policy. See Kernel Doc. These days the Stochastic multiqueue (smq) is the default, considered superior and comes without any tuning knobs. That went as far as to drop "mq" since Kernel 4.6 and make is an alias to the smq policy.

So very basic benchmarking with two slow sync io sequential disk crawlers and two aio random hot spots (and not all of the crawlers fitting onto the cache, but the hotspots easily do). There are more details on the results if you want to look. Results are much better than without cache, but the testcase was in no way sophisticated enough to check that in any detail.

Uncached
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.10 0.20 259.95 126.45 1840.00 599.32 12.63 65.96 170.43 126.56 260.62 2.59 100.00
dm-4 0.00 0.00 260.05 126.65 1840.00 599.32 12.62 65.99 170.37 126.53 260.39 2.59 100.00 READ: io=1109.4MB, aggrb=1891KB/s WRITE: io=370212KB, aggrb=616KB/s
Cached
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.85 1.75 395.75 112.00 1679.40 9.01 33.18 83.44 82.97 83.44 2.52 100.00
nvme0n1 755.05 0.00 159339.95 0.25 873790.40 16.00 10.97 25.14 0.16 0.16 0.00 0.01 100.12
dm-0 0.00 0.00 156881.90 395.95 873903.00 1679.40 11.13 58.35 0.37 0.16 84.19 0.01 100.12
dm-1 0.00 0.00 160095.25 0.25 873791.00 16.00 10.92 25.41 0.16 0.16 0.00 0.01 100.10
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 1.75 396.60 112.00 1679.40 8.99 34.50 86.51 82.97 86.52 2.51 100.00 READ: io=415116MB, aggrb=708356KB/s WRITE: io=1004.8MB, aggrb=1714KB/s

This shall not become a discussion on bcache, dm-cache, ... as I stated at the beginning I usually prefer bcache as well but that is not the point. But if OTOH you have any recommendations for dm-cache to add please feel free to use the comment section.