32bit Solaris performance issues

[2011-05-27]
We have 3 OpenSolaris/OpenIndiana fileservers at work. One of them is an intel core 2 duo with 8gig of DDR2 ram (it has 64bit Opensolaris and works like a champ). The second is a really old Dell pentium 4 with 2.5 gig of memory, and the third is an old HP 4U server (I’ll note down the model number and a pic later). After multiple benchmarks and digging, I’ve come to the conclusion:
Never use 32bit solaris for an ESXi nfs target!!!!
Why you ask? It seems the 32bit kernel has been put to pasture in terms of feature updates (and rightly so) and will scarcely use any of the server’s memory for the ARC (ZFS memory cache). Write performance doesn’t seem to take a hit (that I’ve noticed) but read performance is awful. Even just doing a
echo ::memstat | mdb -k
you can’t even see the ZFS file cache memory allocation on a 32bit installation. Doing kstat monitoring, you see about 64meg to 128meg (pitiful) memory being used short term for the ARC and then cleared (yeah the data doesn’t even stay in memory).
so… if you are like us and severely constrained by budget, resist the temptation to use that old 32bit machine (even if it was an expensive 4U server at the time) for an nfs target for ESXi, your VM’s will thank you.
Incidentally, we use the pentium-4 with a RAIDZ array of 1TB drives connected via USB to store database backups to disk. You heard me right, a USB RAIDZ array. What kind of write performance do you get with such a beast? A whopping 4megabytes/sec. Yes 4. But that’s enough for our purposes to keep transaction/full dumps of our Sybase database for an entire year. (Done in addition to tape backups).
So… 32bit Solaris/OpenSolaris/OpenIndiana should be relegated to disk backups replacing a tape drive more or less. Just my 2 cents.
(the 4U HP server has 9 gigs of ram, of which a whopping 900 megabytes or so actually get used by the OS, what a waste)
[2011-07-16]
I read somewhere online that Oracle has chosen to drop 32bit support from subsequenct Solaris versions (11+). I'm guessing that the openindiana folks are keeping it because they're trying to appeal to desktop users in addition to the server crowd.