zfs on ubuntu locks up if you overcommit arc

Hey just a quick note.. be careful what you ask for.
(this applies to the zfsonlinux kernel mode zfs, not the fuse variety)
From a couple different servers, it seems you need to leave (at a rough guess) 4GB of memory free for ubuntu or your zpool scrub can hang your system if you manually set your zfs_arc_max parameter too high.

Safe config (has been working for me anyway) for a 10GB VM:

root@ubuntuzfs03:~# ~/zfs_show.sh
config
---------------------
options zfs zfs_arc_max=6000000000 zfs_arc_meta_limit=4900000000 zfs_arc_min=5900000000



runtime values
---------------------
c_min                           4    5900000000
c_max                           4    6000000000
size                            4    43729016
hdr_size                        4    1011296
data_size                       4    41999872
other_size                      4    717848
anon_size                       4    16384
mru_size                        4    14393344
mru_ghost_size                  4    0
mfu_size                        4    27590144
mfu_ghost_size                  4    16384
l2_size                         4    0
l2_hdr_size                     4    0
duplicate_buffers_size          4    0
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    36782200
arc_meta_limit                  4    4900000000
arc_meta_max                    4    36782200
root@ubuntuzfs03:~#

I was running with 8GB of ARC and my zpool scrub was crashing. Anyway, just wanted to share, as its not obvious why the zpool scrub was locking up the system, but it seems to be something to do with the kernel not being able to allocate memory.

Breaking up with your tape drive

Dear tape... it’s not you it’s me. I want to see other storage

If you’re like me, backing up to tape for small to medium sized businesses (SMB’s) just doesn’t make sense anymore. The high cost of a tape drive, even higher for a tape library, and the high cost per MB for each tape make backups an expensive (but necessary) job with $0 ROI.

                If you had a satellite office with a reasonably fast VPN connection between them, you could easily consider replicating your data (one way). This would have the advantage of having offsite backups and disaster recovery, but it is assuming that you have the money for 2 of every piece of hardware it takes to run your production systems. And two datacenters operating 24/7 is an additional expense.

                You can pay for cloud storage and replicate your VM’s offsite to Amazon or one of those file hosting services, but at the price you pay per MB that’s more expensive than tape, and doesn’t work well for large volumes of data anyway (SQL Server backups, mail backups, fileserver backups, etc).

                So, if you work in a SMB with one office and a limited budget, but you still want to be rid of the hassle of tapes, consider a removable disk storage alternative paired with a non removable drive. More specifically: any machine (desktop/server) with a PCI-e 1X slot and empty SATA drive bay will work, but gigabit NIC as close to the production server(s) you are backing up is a definite requirement. If you are backing up large quantities of data, you might go for option 2.

Backup Server O/S options:

Option 1: Ubuntu 12.04 with 1gb NIC

·         free

·         rsync

·         ZFS support for mirroring (www.zfsonlinux.org), send/receive replication, snapshots

·         supported by Veeam 6.5 B&R as a backup repository

Option 2: Hyper-V server 2012

·         free

·         NIC teaming in either switch independent or LACP modes with different brand NICs or even a mixture of plug-in NIC cards and MB NIC ports

·         can run Ubuntu 12.04 as a VM (option 1) to handle backups, plus other VM’s to get more use out of server grade hardware

·         also use as for VM replicas/disaster recovery

Option 3: Windows Server 2012

·         not free. Standard edition (only 1 VM included) will run you $900 or so.

·         NIC teaming in either switch independent or LACP modes with different brand NICs or even a mixture of plug-in NIC cards and MB NIC ports

·         Can run in parallel with existing tape backup jobs or as a supplement to tape backup jobs (if for some reason you are not able to replace all of them)

If you are at a SMB you probably would chose option 1 or option 2 as they are the most cost effective.

The strategy here is to run a backup job (say through Veeam B&R, or Backup Exec, or whatever backup software you are using) to the internal hard drive on the backup server. Then (at a later time) rsync or robocopy any changed blocks/files to the USB external storage. The external storage can be removed for storage in a safe, offsite/whatever. You can even swap out external storage like you do tapes. For our experiment, we went with a 4TB USB 3.0 drive paired with a PCI-e 1X controller card. Both are recognized with no problems in Ubuntu and seem to deliver reasonable speeds. We do a “monthend” backup where we take the 4TB drive out of the safe, do a reverse incremental backup with Veeam, and put the tape back in the safe. The total cost of this setup (considering we re-used a desktop PC as the Ubuntu server option 1) at the time of writing this blog:  

$159.99 for the 4TB seagate drive (as listed on newegg)

$26.99 for the StarTech 2 Port PCI Express SuperSpeed USB 3.0 Card Adapter Model PEXUSB3S2

$186.98

Even if you wanted to buy multiple 4TB drives and rotate them out on a weekly or daily basis to a safe or offsite location its still cheaper than buying a server, windows server license, backup exec (or other) license, and a tape drive or tape library. I will update this blog at a later date if we run into any issues with this setup.