This weekend I was playing around with FreeBSD in order to add support to Pupistry. Although I generally use Linux exclusively, it’s fun to play around with other platforms now and then, bit like going on vacation. Plus building support for other platforms ensures that I’m writing code that’s more portable.
FreeBSD is probably the most popular BSD in use and it’s the only one available for download from the Amazon Web Services (AWS) Marketplace and as a supported platform from Digital Ocean alongside their Linux offerings.
However as popular as FreeBSD is, it pales in comparison to Linux, which means that it doesn’t get as much love and things don’t work quite as seamlessly with these cloud providers. In my process of testing FreeBSD with both providers I ran into some interesting feature differences and annoyances.
FreeBSD on Digital Ocean
I started with Digital Ocean first, love them since they’re a nice simple, cheap cloud provider for personal stuff – not much need for the AWS enterprise feature set when I’m building personal machines and paying the price of a coffee for a month of compute sure is nice.
They provide a FreeBSD 10.1 image via the usual droplet creation screen, I have to give Digital Ocean credit for such a nice clean simple interface – limiting user selection does make it much more approachable for people, something Apple always understood with their products.
As always Digital Ocean is pretty speedy, bringing up a machine within a minute or so. Once ready, login as the freebsd user and you can just sudo to root.
Digital Ocean provides a pretty recent image with pkg already installed and ready to go, although you’ll want to run the update process to get the latest patches. You need to login initially as the freebsd user and then can sudo to acquire root powers.
Over all it’s great – so naturally there is a catch. Digital Ocean doesn’t yet support user data with their droplets. So whilst you can fill in the user data field, it won’t actually get executed.
This is pretty annoying for anyone wanting to automate large number of machines, since it now means you have to SSH to each of them to get them provisioned. I’ve raised a question on their community forum around this issue, but I wouldn’t expect a quick fix since the upstream bsd-cloudinit project they use hasn’t implemented support yet either.
It’s not going to be an end-of-the-world for most people, but it could be barrier if you’re wanting to roll out a fleet of BSD boxen.
The best feature from Digital Ocean is actually their documentation – with the launch of FreeBSD on their platform, they’ve produced some excellent tutorials and guides to their platform which can be found here and are useful to both Linux gurus and noobs alike.
Finally their native IPv6 support extends to FreeBSD, so your machines can join the internet of the 21st century from day one.
FreeBSD on Amazon Web Services (AWS)
Next I spun up an instance in Amazon Web Services (AWS) which is the granddaddy of cloud providers and provides an impressive array of functionality, although this comes at a cost premium over Digital Ocean’s tight pricing.
It’s actually been the first time in a long time that I’ve built a machine via the AWS web console, normally for work we just build all of our systems via Cloud Formation and it was an interesting experience to see the usability difference of AWS’s setup page vs that of Digital Ocean’s.
The fact that the launch wizard has 7 different screens says a lot and I suspect AWS is at risk of having it’s consumer user base eaten by the likes of Linode and Digital Ocean – but when a consumer user is paying $5.00 a month and an enterprise customer pays $300,000 a month, I suspect AWS isn’t going to be too worried.
Launching a FreeBSD instance is not really any different to that of a Linux one, you just need to search for “freebsd” in the AWS Market Place to find the AMI and launch as normal.
Once launched, things get more interesting. Digital Ocean’s FreeBSD instance came up in around 1 minute which is standard for their systems – but AWS took a whopping 8-10mins to launch the AMI to the level where I could login via SSH!
Digging into the startup log reveals why – it seems the AWS AMI (Amazon’s machine images/templates) for FreeBSD launches the instance, then runs a prolonged upgrade task (freebsd-update fetch install), before doing a subsequent reboot and finally starting SSH.
Whilst I appreciate the good default security posture this provides, there’s a few issues with it:
- It differs from most other AWS images which deal with patching by having new images built semi-frequently and leaving the patching in-between up to the admin’s choice.
- During the whole process, the admin can’t login which causes some confusion. I initially assumed the AMI images were broken after reviewing my security groups and seeing no reason why I shouldn’t be able to login immediately.
- You can’t trust the AMI images to be a solid unchanging base, which means you need to be vary wary if doing autoscaling. Not only is 10mins a bit too slow for autoscaling, having the potential risk of it not coming up due to app changes in the latest update is always something to watch out for. If doing autoscaling with these images, you’ll need to consider
- It caused me no end of frustration when trying to test user data since I had to wait 10mins each time to get a confirmation of my test!
The last point brings me to user data – the good news is that Amazon correctly supports user data for FreeBSD machines, so you can paste in your tcsh script (not bash remember!) and it will get invoked at launch time.
The downside is that the user data handling of FreeBSD is a lot more fragile than Linux images. Generally with Linux, the OS boots (including SSH) and then runs the user data. If the user data breaks or hangs or does anything other than expected, you can still login and debug. Whereas since FreeBSD runs the user data before starting up SSH, if something goes wrong you have no way to easily login and debug. And given the differences between tcsh and bash plus annoying commands that default to expecting user input on non-interactive ptys, changes are you’ll have more than one attempt that results in a machine getting stuck at launch.
The ultimate fix is that you’ll probably have to use Packer if using FreeBSD in any serious way on AWS to get the startup performance to an acceptable level.
Finally remember that on AWS, you need to login as the ec2-user and then su – to become root.
Which one?
If you’re interested in FreeBSD and want to pick a provider to play around with, the choice seems pretty simple to me – Digital Ocean. They’re got the better pricing (~ $5/month vs $15/month) and their ridiculously simple dashboard coupled with the excellent documentation they’ve assembled makes it really attractive for anyone new to the *.nix or cloud space. Plus they’ve bothered to invest in IPv6 which I appreciate.
However if you’re doing business/enterprise systems and want user data, autoscaling or the benefit of automating entire stacks with Cloud Formation, then you will probably find AWS the more attractive offering purely due to the additional functionality offered by that platform. Just be prepared to spend a bit of time baking your own AMI to allow you to skip the overhead of having to wait for updates to apply for each instance you bring up.
Neither provider has got their FreeBSD experience to be quite as slick as that of their Linux offerings, however hopefully they improve on these deficiencies over time – there’s not much needed to get the experience up to the same level as Linux distributions and it’s nice having a different type of unix to play with for a change.
