To Dedup or Not to Dedup??
In additional to compression, ZFS offers deduplication functionality.
It (potentially) allows you to use less disk space to store the same amount of
data. Nothing comes for free, as the say, and in this case deduplication does
not either.
You need to provide ZFS with massive amounts of RAM to store its
deduplication tables in memory or you can kiss any kind of write performance
goodbye (and in worst cases make your fileserver unresponsive for significant
periods of time).
Even if you throw plenty of RAM at deduplication, it is still
best to treat it as a “backup” device like a tape drive. That is to say, you do
backups to it and:
DO NOT USE IT FOR VM STORAGE. LZJB is probably fine, but
deduplication will kill VMs responsiveness.
DO NOT USE IT FOR A PRODUCTION DATABASE. Testing databases where
response time/processing speed are not a factor could be a possible use.
There, I warned you.
Now, here are some (half-assed) numbers you can use as a rough
guide to
Pre-allocating your RAM for a deduplicating pool.
I’m testing with ZFS on Ubuntu. For ZFS on Solaris or
Openindiana, the procedure
to change the zfs_meta_limit is slightly different.
Testing Setup:
root@ubuntuzfs03:~# uname -a
Linux ubuntuzfs03 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5
17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu-zfs
spl-0.6.0-rc13
zfs-0.6.0-rc13
VM with 2 virtual CPUs and 8GB of RAM
Set our ARC max size to ~3Gb
Set our ARC meta limit to ~2Gb
Edit /etc/modprobe.d/zfs.conf, the number is # of bytes:
root@ubuntuzfs03:~#
cat /etc/modprobe.d/zfs.conf
options zfs
zfs_arc_max=3000000000 zfs_arc_meta_limit=2900000000
YOU MUST REBOOT for those changes to take effect.
I use a small script to check the status of my config and
running values:
zfs_show.sh
------------------
#!/bin/bash
echo
"config"
echo
"---------------------"
cat
/etc/modprobe.d/zfs.conf
echo "
"
echo "
"
echo
"runtime values"
echo
"---------------------"
grep -E
arc\|c_max /proc/spl/kstat/zfs/arcstats
now check that your settings (AFTER REBOOTING) have actually taken
effect:
root@ubuntuzfs03:~# ~/zfs_show.sh
config
---------------------
options zfs zfs_arc_max=3000000000
zfs_arc_meta_limit=2900000000
runtime values
---------------------
c_max
4 3000000000
arc_no_grow
4 0
arc_tempreserve
4 0
arc_loaned_bytes
4 0
arc_prune
4 0
arc_meta_used
4 132929416
arc_meta_limit
4 2900000000
arc_meta_max
4 1457235528
root@ubuntuzfs03:~#
The config options in /etc/modprobe.d/zfs.conf should match the
runtime values, or something went wrong.
Now, for our test, we explicitly set the recordsize to 4K:
root@ubuntuzfs03:~# zfs set recsize=4K zpool1/cifs/dedup_storage
root@ubuntuzfs03:~# zfs set dedup=on zpool1/cifs/dedup_storage
I’m copying the files over 100mbit LAN connection, so the 11.1
MB/s is reasonable:
Total size of files (same 2 large files copied multiple times)
Math
----
47698814976 bytes of data (files and folders included)
47698814976 / 4096 = 11,645,219 data blocks
1279193040 bytes arc_meta_used
1279193040 meta bytes / 11645219 data blocks =
109 meta bytes/per 4k data block
1279193040 meta bytes / 47698814976 bytes of data = 0.0268181304010097
bytes of meta /per bytes of data (2.7%)
So by that math, 2TB of disk space
(depending on how manufacturer calculates 1TB)
2,199,023,255,552 bytes of data would require AT LEAST 59,373,627,900
bytes or about 55.3 Gigabytes of RAM allocated to arc_meta_limit.
(owch)
Deleting files (for some reason) sends the arc_meta_used up even
higher.
I highlight all the files and delete them at once and
arc_meta_used goes up to
1457235528 bytes arc_meta_used
So to recalculate,
1457235528 meta bytes / 11645219 data blocks =
126 meta bytes/per 4k data block
1457235528 meta bytes / 47698814976 bytes of data = 0.0305507700502249
bytes of meta /per bytes of data (3.1%)
Just to be safe, if you are going to use 4k blocks, I would go
with 5%
2,199,023,255,552 bytes of data would require AT LEAST 109,951,162,777.6
bytes or about 102.4 Gigabytes of RAM allocated to arc_meta_limit.
(super owch)
Change the record size to 16K:
root@ubuntuzfs03:~# zfs set recsize=16K
zpool1/cifs/dedup_storage
After copying the files, dedup ratio and AVAIL space report the
same values, but that is to be expected if you are just
copying the same files over and over again.
same total bytes in files, but arc_meta_used comes out
different:
47698814976 bytes of data (files and folders included)
47698814976 / 4096 = 11,645,219 data blocks
bytes arc_meta_used
522642072 meta bytes / 11645219 data blocks =
45 meta bytes/per 16k data block
522642072 meta bytes / 47698814976 bytes of data = 0.0109571290662666
bytes of meta /per bytes of data (1.1%)
Interestingly, the arc_meta_used value did not spike as much
during the delete of the 16K blocks, it reported 524992920.
The math I’m using is a
little flawed, because it does not take into account how
much of the arc_meta_used is not related
to dedup. Probably I could turn off dedup, copy the same files
again, and use that as a baseline, but I feel fairly comfortable
saying:
TLDR summary:
- you need 5% of total storage (or more) for 4K block size
- you need 2.5% of total
storage (or more) for 16K block sizes
- deletes will always be slow, even if you have enough RAM. Imagine slow, and then multiply that times 10. Luckily it happens in the background, but if you are on a quad core or less machine it will slow everything to a crawl. If you don't have enough RAM it is slow times 100. NOTE: I have experienced complete system locks deleting multi GB files on Ubuntu. I believe this was supposed to be fixed on Solaris, but apparently hasn't made it into the Ubuntu ZFS yet, so be warned.
- 97% of the time you will be better off using ONLY compression
Your mileage will definitely vary, so you will have to
keep an eye on the reserved vs. runtime values and adjust accordingly.
ZFS supports larger block sizes, up to 128K, but the larger your
block size the lower dedup ratio you are
going to see. (Unless you are ONLY storing multiple copies of
the same large files, in which case just
go with 128K the max.)
<edit 2013-08-07> as David pointed out, you can see memory usage reported by ZFS by using 'zpool status -D'
