I have been happily using SmartOS, both at home and at work, for a couple of years now. However, it now seems to be time to move on.
At home, it has been in use in an (AMD Turion-based) HP Microserver N40L. As a file and media host, RAID-Z2 and ZFS Snapshotting for backups are amazing. Being able to boot the OS from a USB stick, is also a nice benefit.
At work, SmartOS was chosen not so much for data integrity (though that’s a nice bonus), but again because of cheap ZFS snapshots, and for zones. Zones, as lightweight containers, are an ideal match for many software engineering tasks — particularly Continuous Integration, where you want to be able to quickly tear-up and tear-down isolated environments to run your builds. Zones are a much more attractive option than full-on VMs (even if virtualisation’s memory ballooning worked better than advertised) — as they have a much lower overhead, and are scheduled and memory-managed as a part of the ‘parent’ OS.
Sadly, several things have happened that means it’s time to move away.
- When I started, the state of ZFS for Linux was pretty poor — the best option was via a FUSE driver. Today, however, there is a native kernel module, which is graded as production-ready. It requires some additional steps from a ‘default’ installation, but I was surprised at how easy it was to add to an ubuntu install — and how easily I could import my ZFS pools.
- The Linux equivalent of zones — Linux Containers (lxc) — was not very stable. Since then, not only has it improved, but docker happened. It’s now good enough for my use-cases, and under active use.
On their own, that’s not a reason to move away — but there are push factors too.
- The KVM system doesn’t do libvirt. That means it can’t be controlled easily by tooling like Jenkins.
- @Home, I want to run Plex. Plex doesn’t have an Illumos / SmartOS build, so I needed to run Linux in KVM. So, I upgraded to 16Gb RAM to begin the process. Being an AMD processor, this entails rebasing and compiling a separate branch of the KVM module code (which for some reason has yet to be upstreamed, despite being unchanged for some time). Once that was running, it works… but… specifying a KVM image above a certain size seems to lock the system on ‘imgadm create’. I gave up trying to figure out what the boundaries of that were.
- In the process of compiling the above, when I still only had 2Gb of RAM, I managed to — repeatedly — lock the system solid. Whilst waiting for the memory upgrade, I managed the exact same behaviour, repeatably, in VMWare (you can probably do the same — just follow the ‘compile SmartOS on SmartOS’ instructions). It’s not a CPU or swap issue — both end up being idle.
The problem with the lockups are it isn’t the first time I’d seen them. Our dev server had done the same (often under periods of load), and it has 64Gb of RAM — I’d put it down to a hardware incompatibility gremlin. Whatever else should happen (swapping, processes being killed) on an under-provisioned machine under load — locking solid requiring a hard reset is unacceptable - and really dented my confidence of using it in production.
It’s a shame, as there’s much to like about SmartOS (particularly the community, which is very helpful), and Illumos in general (ZFS, Zones, DTrace; sane command-line tooling) — it was clearly ahead of the alternatives. But I suspect there’s just not enough users or developers for it to remain a viable option, as the rest of the world caught up with ‘good enough’.
