Tag Archives: geek

Anything IT related (which is most things I say) :-)

Puppet 3 & 4 on FreeBSD

I’ve been playing with FreeBSD recently and wanted to run Puppet on it to provide my machines with configuration.

Unfortunately I faced the following error when I first ran Puppet with a simple Puppet manifest that tries to install NTP via usage of the puppetlabs/ntp module:

Warning: Found multiple default providers for package: pip, gem; using pip
Error: Could not set 'present' on ensure: Could not locate the pip command. at modules/ntp/manifests/install.pp

Error: /Stage[main]/Ntp::Install/Package[net/ntp]/ensure: change from absent to present failed: Could not set 'present' on ensure: Could not locate the pip command. at modules/ntp/manifests/install.pp

This collection of errors is pretty interesting – we can see that Puppet seems to understand that it’s running on FreeBSD and that the package is called “net/ntp” but it’s trying to install it with pip, which is the Python package manager :-/

Turns out that Puppet versions older than and including 4.0.0 [1] lack support for PkgNg package manager which is used in FreeBSD 10.x and since it doesn’t know how to use it, defaults to trying one of the only two providers left over – pip & gem. Not really the smartest approach…

There’s two ways to fix right now:

  1. An independently packaged provider for PkgNg was written and is available at xaque208/puppet-pkgng. This can be included into any existing Puppet 3/4 deployment and you can define it as the default for all packages globally.
  2. Pull the commit from Puppet that provides PkgNg support (thanks to author of the above project). The downside is that you then have to build your own Gem and install it.

Longer term you can either:

  1. Wait for Puppet to release a stable version that actually works. I’m guessing it will be in 4.0.0 +1 (so 4.0.1?) but I’m not familiar with their approach to releasing new feature additions. See [1] for more information.
  2. Hope that the FreeBSD developers can backport the commit that adds PkgNg support to Puppet4. I’ve actually started chatting with one of the FreeBSD developers who’s been kind enough to have a look at upgrading their package from Puppet3 to Puppet4, but it’s not as simple as it should be thanks to the terrible way Puppetlabs has packaged Puppet4 (lots of hard coded /opt paths everywhere) so it may take some time.

 

Additionally there is an issue if you choose to install Puppet 4 via RubyGems and when you attempt to run it, you will get the following error:

Error: Cannot create /opt/puppetlabs/puppet/cache; parent directory /opt/puppetlabs/puppet does not exist
Error: /File[/var/log/puppetlabs/puppet]/ensure: change from absent to directory failed: Cannot create /var/log/puppetlabs/puppet; parent directory /var/log/puppetlabs does not exist

This is because the Gem is not properly packaged and doesn’t run nicely with FreeBSD’s filesystem structure. You can hack in a “fix” with:

mkdir -p /opt/puppetlabs/puppet/cache
mkdir -p /var/log/puppetlabs/puppet

It’s far from the right way of properly fixing this, the best solution if you value OS purity right now is to stick with the FreeBSD packaged Puppet 3 version and add the xaque208/puppet-pkgng module.

 

[1] So at time of writing, *no* stable Puppet version supports PkgNg yet, but given that it’s been merged into the master git branch for Puppet, I have to *presume* that it’s going into 4.0.1… I hope :-/

FreeBSD in the cloud

This weekend I was playing around with FreeBSD in order to add support to Pupistry. Although I generally use Linux exclusively, it’s fun to play around with other platforms now and then, bit like going on vacation. Plus building support for other platforms ensures that I’m writing code that’s more portable.

FreeBSD is probably the most popular BSD in use and it’s the only one available for download from the Amazon Web Services (AWS) Marketplace and as a supported platform from Digital Ocean alongside their Linux offerings.

However as popular as FreeBSD is, it pales in comparison to Linux, which means that it doesn’t get as much love and things don’t work quite as seamlessly with these cloud providers. In my process of testing FreeBSD with both providers I ran into some interesting feature differences and annoyances.

 

FreeBSD on Digital Ocean

I started with Digital Ocean first, love them since they’re a nice simple, cheap cloud provider for personal stuff – not much need for the AWS enterprise feature set when I’m building personal machines and paying the price of a coffee for a month of compute sure is nice.

They provide a FreeBSD 10.1 image via the usual droplet creation screen, I have to give Digital Ocean credit for such a nice clean simple interface – limiting user selection does make it much more approachable for people, something Apple always understood with their products.

Screen Shot 2015-04-18 at 23.30.50

As always Digital Ocean is pretty speedy, bringing up a machine within a minute or so. Once ready, login as the freebsd user and you can just sudo to root.

Digital Ocean provides a pretty recent image with pkg already installed and ready to go, although you’ll want to run the update process to get the latest patches. You need to login initially as the freebsd user and then can sudo to acquire root powers.

Over all it’s great – so naturally there is a catch. Digital Ocean doesn’t yet support user data with their droplets. So whilst you can fill in the user data field, it won’t actually get executed.

This is pretty annoying for anyone wanting to automate large number of machines, since it now means you have to SSH to each of them to get them provisioned. I’ve raised a question on their community forum around this issue, but I wouldn’t expect a quick fix since the upstream bsd-cloudinit project they use hasn’t implemented support yet either.

It’s not going to be an end-of-the-world for most people, but it could be barrier if you’re wanting to roll out a fleet of BSD boxen.

The best feature from Digital Ocean is actually their documentation – with the launch of FreeBSD on their platform, they’ve produced some excellent tutorials and guides to their platform which can be found here and are useful to both Linux gurus and noobs alike.

Finally their native IPv6 support extends to FreeBSD, so your machines can join the internet of the 21st century from day one.

 

FreeBSD on Amazon Web Services (AWS)

Next I spun up an instance in Amazon Web Services (AWS) which is the granddaddy of cloud providers and provides an impressive array of functionality, although this comes at a cost premium over Digital Ocean’s tight pricing.

It’s actually been the first time in a long time that I’ve built a machine via the AWS web console, normally for work we just build all of our systems via Cloud Formation and it was an interesting experience to see the usability difference of AWS’s setup page vs that of Digital Ocean’s.

The fact that the launch wizard has 7 different screens says a lot and I suspect AWS is at risk of having it’s consumer user base eaten by the likes of Linode and Digital Ocean – but when a consumer user is paying $5.00 a month and an enterprise customer pays $300,000 a month, I suspect AWS isn’t going to be too worried.

Launching a FreeBSD instance is not really any different to that of a Linux one, you just need to search for “freebsd” in the AWS Market Place to find the AMI and launch as normal.

Screen Shot 2015-04-18 at 23.43.18

 

Once launched, things get more interesting. Digital Ocean’s FreeBSD instance came up in around 1 minute which is standard for their systems – but AWS took a whopping 8-10mins to launch the AMI to the level where I could login via SSH!

Digging into the startup log reveals why – it seems the AWS AMI (Amazon’s machine images/templates) for FreeBSD launches the instance, then runs a prolonged upgrade task (freebsd-update fetch install), before doing a subsequent reboot and finally starting SSH.

Whilst I appreciate the good default security posture this provides, there’s a few issues with it:

  1. It differs from most other AWS images which deal with patching by having new images built semi-frequently and leaving the patching in-between up to the admin’s choice.
  2. During the whole process, the admin can’t login which causes some confusion. I initially assumed the AMI images were broken after reviewing my security groups and seeing no reason why I shouldn’t be able to login immediately.
  3. You can’t trust the AMI images to be a solid unchanging base, which means you need to be vary wary if doing autoscaling. Not only is 10mins a bit too slow for autoscaling, having the potential risk of it not coming up due to app changes in the latest update is always something to watch out for. If doing autoscaling with these images, you’ll need to consider
  4. It caused me no end of frustration when trying to test user data since I had to wait 10mins each time to get a confirmation of my test!

The last point brings me to user data – the good news is that Amazon correctly supports user data for FreeBSD machines, so you can paste in your tcsh script (not bash remember!) and it will get invoked at launch time.

The downside is that the user data handling of FreeBSD is a lot more fragile than Linux images. Generally with Linux, the OS boots (including SSH) and then runs the user data. If the user data breaks or hangs or does anything other than expected, you can still login and debug. Whereas since FreeBSD runs the user data before starting up SSH, if something goes wrong you have no way to easily login and debug. And given the differences between tcsh and bash plus annoying commands that default to expecting user input on non-interactive ptys, changes are you’ll have more than one attempt that results in a machine getting stuck at launch.

The ultimate fix is that you’ll probably have to use Packer if using FreeBSD in any serious way on AWS to get the startup performance to an acceptable level.

Finally remember that on AWS, you need to login as the ec2-user and then su –  to become root.

 

Which one?

If you’re interested in FreeBSD and want to pick a provider to play around with, the choice seems pretty simple to me – Digital Ocean. They’re got the better pricing (~ $5/month vs $15/month) and their ridiculously simple dashboard coupled with the excellent documentation they’ve assembled makes it really attractive for anyone new to the *.nix or cloud space. Plus they’ve bothered to invest in IPv6 which I appreciate.

However if you’re doing business/enterprise systems and want user data, autoscaling or the benefit of automating entire stacks with Cloud Formation, then you will probably find AWS the more attractive offering purely due to the additional functionality offered by that platform. Just be prepared to spend a bit of time baking your own AMI to allow you to skip the overhead of having to wait for updates to apply for each instance you bring up.

Neither provider has got their FreeBSD experience to be quite as slick as that of their Linux offerings, however hopefully they improve on these deficiencies over time –  there’s not much needed to get the experience up to the same level as Linux distributions and it’s nice having a different type of unix to play with for a change.

Puppet facts, json and max nesting

I use Puppet for both business and pleasure and my work often involves writing custom Puppet facts to expose various bits of information.

Recently a fact I had written that worked on the development machines started throwing errors when run on our production machines:

Could not retrieve jethros_awesome_fact: nesting of 20 is too deep

 After digging around it turns out this relates to how many nested levels are inside JSON responses. By default Ruby enforces a maximum level of nesting, I guess to avoid parsing bad JSON or JSON deliberately structured to cause infinite looping.

My fact involved pulling JSON from a local application API and then providing various bits of data from the feed. In the development environments this worked without an issue, but the production systems returned a lot more information via the API feed and broke it.

The fix is pretty easy, just need to add the :max_nesting => false parameter when parsing the JSON – or set it to a different number of levels if you prefer that approach.

json         = JSON.parse(response.body, :max_nesting => false)

AEM/CQ 5.6.1 package file parameter missing

Having the joy and “pleasure” of patching a bunch of AEM/CQ 5.6.1 instances at work recently, I ran into a weird issue where the hotfixes would refuse to install on a particular development machine.

Any attempt to install the packages would result in the following error, regardless of whether or not the upload is done via the CRX PackMgr UI, or via Curl.

{"success":false,"msg":"package file parameter missing"}

I would have assumed that it was a bad package if it wasn’t for the fact that it worked OK on other machines, so started looking for other factors.

$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdb        30G   29G   88K 100% /tmp

Turns out, the /tmp volume was full on this machine thanks to a lovely collection of heap dumps being stored there. This leads to CQ being unable to write to it’s configured temporary location (which might differ on your install, check for -Djava.io.tmpdir) which is used to store the file between uploading and installing.

Clearing the tmp volume resolves the issue. You might also get this error if your tmp volume can’t be written for any other reason, such as a permissions issue or broken filesystem/mount.

Ubiquiti Unifi UAP-AC in-home performance

Having completed the most important move-in task of wiring the house for ethernet and installing the Ubiquiti Unfi UAP-AC access point, I decided to run some benchmarks to see what sort of performance I could actually get.

Unifi UAP-AC just chilling out on the roof.

Unifi UAP-AC just chilling out on the roof.

Manufacturers always quote theoretical figures which can make buying hard, but even when relying on public third party tests, a particular device will perform differently in your own specific situation in a house with walls, interference, crappy consumer devices and other causes for them not to hit the theoretical maximum performance.

Ubiquiti claim that the UAP-AC can do a theoretical maximum of 1300 Mbits in 5Ghz and 450 Mbits in 2.4Ghz. Unfortunately public benchmarks for my testing laptop (Macbook Pro Retina 2013) show it capping out at 615 Mbits, so it’s not going to possible to truly test what’s possible with the UAP-AC past that speed.

Rather than a benchmark of the absolute maximum speed possible consider this benchmark a general consumer use case test in a suburban setting to see what the real world performance is going to be like with a common consumer client device.

 

Environment

The testing was done with the following equipment:

  • iperf2 as the network performance tool.
  • Ubiquity Unfi UAP-AC access point (duh)
  • Apple Macbook Pro Retina 15″ late 2013 (Running MacOS Mavericks).
  • Lenovo Thinkpad x201i laptop running Linux and connected via GigE to run the iperf server.
  • Mikrotik CRS226-24G-2S+RM switch, acting as a dumb switch to connect the AP and the Lenovo Thinkpad running iperf2.
  • House

The latest firmware was applied to the access point, all other settings like power modes are on their default values.

The Macbook Pro should be pretty typical of most high end laptops currently available and supports 802.11ac and both 2.4Ghz and 5Ghz bands, sadly I didn’t have any other 802.11ac client devices that I could run tests with – would be very keen to test with a client device that can deliver closer to the theoretical 1300 Mbits, as well as being able to test with multiple devices concurrently.

Because of where I am, there were no other WiFI access points detected (yay for elderly neighbours) so the testing was free from interference by other networks. Other sources of potential interference such as microwaves and wireless audio streaming was powered down.

Whilst they were not particular active, or being used or testing, the following other devices were also connected to the access point during the tests and could of course influence how well the AP performs.

  • Apple Iphone 5s (802.11n 2.4Ghz and 5Ghz capable)
  • Samsung Galaxy Note 2 (802.11n 2.4Ghz and 5Ghz capable)

The house itself also requires note – it’s an old single-story Wellington home with walls containing about 20mm of solid hardwood plus a layer of plasterboard. They’re thick walls and utterly punishing to 5Ghz WiFI which generally doesn’t like solid objects at the best of times.

Finally the AP is roof mounted in the rough middle of the house in the hallway – the downside is that every room gets it’s signal after passing through at least one wall, but it gives me all of house (and some garden) coverage from a single AP.

 

Test Results

I did two distinct tests – one where the client device transmits, then receives and another where the client device transmits and receives concurrently. Both tests were done using UDP.

Respectively, the following were the commands used:

  • iperf -u -c SERVERIP –tradeoff -b 1000M
  • iperf -u -c SERVERIP –dualtest -b 1000M

I performed each test a total of 3 times and the results provided are the averages result of those 3 tests. I’ve also included a column showing the general connection type that was established for most of the testing, however in some rooms it’s possible the laptop swapped between 5Ghz and 2.4Ghz throughout testing which would mess with results.

6 different locations were used:

  1. Wired GigE connection (For control purposes to verify the iperf setup and switch were not impacting performance).
  2. Hallway – This test was holding the laptop and standing directly underneath the roof mounted AP.
  3. Master Bedroom – 1 wall
  4. Office – 1 wall, but signal potentially impacted by house details.
  5. Lounge – 1-2 walls depending where the signal goes.
  6. Dining Room – 3 walls between AP and room.

 

Individual Tests Dual Tests Reported Mode
Location Transmit Receive Transmit Receive
WiredGigE 745.00 803.00 null [1] null [1]
Hallway 565.67 570.33 120.43 379.00 802.11ac 5ghz 64%
Master Bedroom 127.00 394.00 84.40 92.47 802.11ac 5ghz 12%
Office 150.00 341.33 93.37 109.13 802.11ac 5ghz 15%
Lounge 67.47 153.00 32.87 62.30 Mixed [2]
Dining Room 55.33 116.00 33.10 45.30 802.11n 2.4Ghz 47%

The locations in the results are ordered closest-furthest, so these results make sense. The fact that receive performance is often better than transmit makes sense when considering that the AP probably has far better antennas and transmission power than the laptop.

Some specific notes:

  • [1] The dual test of Wired GigE just wouldn’t work – I think this was a bug in iperf2 and it wasn’t vital to solve for the testing, hence no metrics.
  • [2] The laptop’s reported mode when in the lounge would vary. On some tests the AP would report 802.11n 2.4Ghz 50% and other times it would report 802.11ac 5Ghz 0.0% (sic), so I think the laptop was struggling to maintain 5ghz, yet could see it strongly enough to try to switch to it.

 

 

Conclusion?

So am I happy with my purchase? The answer is yes – some other access points might have scored higher performance at range but considering the punishment the walls of the house are delivering to the AP, I’m pretty happy with the speeds I’m getting in the rooms.

Considering my WAN connection is 40mbits down / 10mbits up maximum, even the worst location is still able to max the connection and if I need more capacity to push to the file server, I’ll go use one of the stronger signal rooms or just plug into one of the ethernet ports in every room of the house.

I also intentionally brought Ubiquiti’s hardware since in future it’s likely that I’ll add additional access points around the house to expand the coverage in the areas we end up spending the most time in. The software provided Ubiquiti makes configuration and management of multiple access points very easy which is a big advantage.

At just under $500 NZD, these units aren’t cheap, but compared to the time/cost of running ethernet everywhere, it wouldn’t be a bad investment to install a 3-pack spread around the house to provide maximum coverage if you really want to get as fast as possible without the pain of installing ethernet (well you still have to run it through the attic… but it’s easier than running inside the walls). You’re not going to get GigE speeds, but you should be able to get 500-600 Mbits within the same room as the AP.

Installing Cloud Pipes

One of the essential upgrades for the house has been the installation of computer network data cabling throughout the house. Whilst some helpful individual went to the effort of installing phone jacks in almost every room, an analogue phone line in every room isn’t that useful to me in 2014, so I decided to get the place upgraded with something a little more modern.

A few people have asked me why I didn’t go entirely WiFi instead – granted it’s a valid question given that most devices now come with WiFi and no wired ethernet (curses Apple Macbooks), but I figured there’s still some good reasons to install cables through the house:

  1. WiFi still needs some copper cables to backhaul data and even if mesh networking evolved to be good enough to eliminate the backhaul link requirements, there’s no wireless power yet, so POE is damn handy.
  2. With ports in every room, I can always plug in more APs in any room to get better coverage. Could be very handy if new tech like WiGig takes off, which may require an access point in each room due to poor performance through walls.
  3. The current WiFi tech is acceptable for transferring current HD video content over the network, but it’s not going to handle ultra-high-def content like 4K footage very well.
  4. The Cat6 cabling I’ve installed should be capable of up to 10Gbit speeds. It’s going to take us a while to get 10Gbit with wireless.

Only time will tell if I was as foolish as those who installed coaxial cabling for their 10mbit networks before they got bitten by the uptake of Cat5, but I suspect it should be useful for another 10-20 years at least. After that, who knows….

Since I’m putting the cabling into existing clad rooms, I lack the convenience of a newly built or renovated property where I can simply run cables at my leisure and put the plasterboard up afterwards. On the plus side, unlike a modern house, mine has a number of gaps from items like old brick chimneys that have been removed and a number of walls that lack the conventional 2×4 horizontal studs, which offers some places where cables can be dropped from ceiling to floor uninterrupted.

With this in mind, I gathered my tools, geared up with confidence, and set off on my cabling adventure.

OPen wide

“Trust me, I know what I’m doing”

Only one problem – one quick look up into the attic quickly confirmed for me that “yes, I do certainly dislike heights” and “yes, I also really do dislike confined spaces that are dark and smell funny”.

To overcome these problems, I recruited the services of the future-father-in-law who has a lot of experience running TV antenna cabling, so is pretty comfortable moving around in an attic and drilling holes into walls. Thankfully he agreed to assist me (thanks Neville!!) with climbing up and around the attic which allowed me to move to the next step – getting the cables in.

It's high up here :-/

3.5 metre ceilings mean you need to get comfortable with working at the top of a ladder

I decided that I wanted 4x cables into the lounge, 4x into the back bedroom/office, and then 2x into the other 3 bedrooms. I pondered just putting 4x everywhere, but had a whole bunch of 2x plates and I figure the bedrooms aren’t likely to host a small server farm any time soon. Again, one of those things where I might be cursing myself in the future, or might not ever be an issue.

The small bedroom was the easiest, being in part of the original house, there was no studs from floor to roof, so we could simply drop the cables right down and cut a hole at the bottom for the ports. Easy!

The older walls never needed studs, since the walls are lined in about 10mm of solid sarking timber made from native hardwood Rimu, these timber planks hold the building together tight and eliminate the need for the more modern practice of horizontal studs in the walls. The upside of this sarking is that the place is solid and you can put weight baring screws into almost any part of the wall, and screw the faceplates for power and data directly into the wall without needing flushboxes. The downside is that the hole saw takes some effort to cut through this hardwood native timber and it also means that WiFi penetration betweens the rooms isn’t as great. Infact when I had the WiFi access point in a cupboard before it got properly installed, I struggled to maintain connections to some locations due to the thick walls.

Hello data!

If you look carefully, can see the thickness of the walls with the plasterboard + sarking.

The back rooms – bedroom, office and lounge were a bit more challenging. Due to structural blockages like support beams and horizontal studs in the younger office and bedroom renovation, it wasn’t simply a case of dropping the cable down from the roof to each room.Instead we ran the cables through the roof and then down all together in a single bunch thought the space that was once occupied by the brick chimney, to get the cables down from the roof to under the house. Once under, we were able to run them to the required rooms and pop the cables back up into the rooms by drilling a hole into the middle of each wall from under the house.

Cloud Pipes!

Cloud Pipes!

To do the master bedroom took some creativity, but we found that the hall cupboard that backs onto the bedroom was an easy to target location and dropped the cables down there, before doing some “creative re-construction” of the cupboard walls to get the cables down and through to the other side in the bedroom.

Even this coat cupboard needs to be GigE connected

Even this coat cupboard needs to be GigE connected

Running the cables was a three-step task. First we ran a single cable being fed out of the reel until we got it to the desired location. This is required to determine the length of the cabling needed, although if we were willing to be wasteful with cable and coil the excess in the roof, we could have made estimations on the generous side and skipped this step and just have run a draw wire from the immediate start.

We then removed the cable, pulling a draw wire through after it. In my case, we use some old cat5 cable as the draw wire since it’s nice and tough. Once the original cable is recovered, we cut the required number of lengths exactly the same, then create bundles of cable as needed.

We used electrical table to bind them together (since it doesn’t add any thickness to the bundles, unlike cable ties, which get stuck on holes) and made sure to stagger the cables so that it wasn’t a single large flat end trying to get through the holes.

Once the bundles are ready, it’s just a case of attaching it to the draw wire and pulling the draw wire through, pulling the new bundle behind it. Make sure you’ve drilled your holes to be big enough to fit the whole bundle!

 

The easiest installation was that of the roof-mounted WiFi access point (Ubiquiti Unifi UAP-AC). Since it just needed a single cable which needed to be run along the attic to the patch panel, we simply drilled a hole up into the attic and fed the cable up straight from the reel, no need to mess around with draw wires.

I suspect we could have done this for all the rooms in the house just as easily, so if you decide you want to invest in WiFi APs in every room rather than wired ethernet ports, you would have a lot easier time putting the APs up.

Leave plenty of length so you don't have to crimp a cable above your head

Drill hole, feed cable, doesn’t get much simpler than this.

Rather than a socket, the roof cable is terminated with an RJ45 connector which plugs directly into the back of the AP which then fits snugly on the roof hiding all cabling.

The end result looks quite tidy, I was worried about the impact on the character ceilings and part of me did feel bad putting the drill through it so I took care to keep the holes to the absolute minimum to ensure it could be patched without too much grief in future.

All done!

The blue-sun god we worship for internet access.

The blue square on the AP is visible in the dark, but doesn’t light up the area. It’s bright enough that I’d think carefully about putting it in a bedroom unless there’s a way to turn it off in software.

 

Whilst the roof mount AP had an RJ45 port due to space constraints and aesthetics, all the room cables have been terminated at proper RJ45 jacks.

Hardware

PDL Cover, Grid (sold together), Third Party RJ45 keystone and PDL RJ45 clip.

My house is primarily PDL 500/600 series faceplates, which means I ended up sourcing the same PDL products for the data faceplates. The faceplates aren’t too outrageous at about $6 each (2-port / 4-port), but PDL charge a whopping $13.80 for their RJ45 data inserts. Given that I installed 14 sockets, that would cost $193.20 for the ports, which is crazy… I could buy several entire patch panels for that.

Fortunately the RJ45 inserts themselves (the keystones) are made by a number of vendors, you just need the PDL keystone clip to join any third party keystone with the PDL faceplates.

Hence I sourced 14x of the PDL clips at $1.38 each ($19.32 total) and then sourced 3x 5-pack of keystones on TradeMe from a seller “digitalera” for $9 per pack ($27). This brought my total spend for the RJ45 data inserts to $46.32… much much cheaper!

Result

The finished product.

All these cables terminate back in a small cupboard in the hallway, into a 24-port patch panel. There’s a bit of space space left over on it for future expansion as it occurs.

Cabling!

All patched and all ready to go

The patch panel is installed in a wall-mount 9 RU 19″ comms cabinet. I went for a 9RU and 30cm deep model, since it offers plenty of space for PDUs, cabling and also can fit lots of models of switches and routers with room to clear.

Data & Power sorted!

Data & Power sorted!

Cable management with this setup has been a little tricky – usually I’d prefer to have a cable management bar for the patch panel and a cable management bar for each switch, an arrangement that is space expensive, but the easiest and tidiest.

Unfortunately I quickly found that putting the cable management bars in the roof-height cabinet limits visibility too much, since it blocks the ability to see the switch ports or patch panel labels since you can’t look at it face on, but rather look at it upwards from the top of the ladder.

The approach I’ve ended up with is therefore a little unconventional, with a cable management bar at the very top of the rack, and cabling going up into it from the patch panel and then back down to the switch.

The downside of this approach is that cables cross the patch panel to get to the switch (arghgh I know!), but the upside is that I can still see all the other switch ports and patch panel ports and its still quite readable. I’ll understand if I’m kicked out of the data centre cabling perfection club however.

There’s still 5RU spare for some shelves for devices like VDSL model or a dedicated router, but given that the Mikrotik CRS226-24G-2S+RM RouterOS based switch can do almost everything I need including almost 200mbits routing capability, there’s no plan to add much more in there.

Currently the power and server data runs down to the floor, but next time I have an electrician on-site doing work, I’ll get a mains socket installed up by the cabinet to save a cable run and potentially a very shallow rackmount UPS to run the cabinet.

Finished! For now...

Cabling and equipment installed!

The final step was making sure everything actually worked – for that I used a $5 cable tester I picked up off Trademe – has nothing on a fancy brand like Fluke that can measure the length of cable runs and tell you the type of cabling pin out, but for a casual home installation it was great!

Remote control

Testing the cabling jobs – the meter runs through each wire in order so you can detect incorrectly punched cables or incorrect arrangements of the wires at either end.

 

I had most of the tools needed on hand already, if you’re tempted to do similar, you’re going to need the following:

  1. A decent electric drill.
  2. A hole saw (goes into your drill, makes big holes in walls). You need this to make the opening for your wall plates with enough room to sit all the RJ45 modules into the wall.
  3. Regular drill bits if you’re going up through the ceiling into the roof for WiFi APs – just need something large enough for a Cat6 cable and no more.
  4. An auger drill bit if you want to drill holes suitable for running bundles of cables through solid wood beams. Having a bit big enough to fit all your cables in your bundle a bit of slack is good.
  5. A punch down tool, this is what you use to connect each wire in the patch panel and RJ45 wall modules. Its worth buying a reasonable quality one, I had a very cheap (~$5) unit which barely survived till the end of the build since you tend to put quite a bit of force on them. The cheap tool’s cutter was so bad I ended up using a separate wire cutter to get the job done, so don’t make my mistake and get something good.
  6. A good quality crimping tool – this will allow you to terminate RJ45 (needed if you want to terminate to the plug, rather than socket for roof-mount access points), but they also tend to include a cable stripper perfectly aligned to strip the outer jacket of the cat5/6 cable. Again, don’t scrimp on this tool, I have a particular solid model which has served me really well.
  7. Needle nose pliers or wire cutters – you need something to cut out the solid plastic core of the Cat6 cable. You can do it in the crimping tool, but often the wire cutter or pliers are just easier to use.

And of course materials:

  1. A reel of Cat6 Ethernet. Generally comes in 305m boxes.
  2. A roll of black electrical tape, you’ll want to use this to attach guide cables, and to bundle cables together without adding size to the cabling runs.
  3. Cable ties are useful once you get cables into position and want tight permanent bundling of cables.
  4. RJ45 plugs if you are terminating to a plug.
  5. RJ45 modules and related wall plate hardware.
  6. Pipe/Saddle Clips can be useful for holding up cables in an orderly fashion under the house (since they’re designed for pipes, big enough to fit cable bundles) and they’re great to avoid leaving cables running across the dirt.

Note that whilst there are newer standards like Cat 6a and Cat7 for 10 GigE copper, Cat6 is readily available in NZ and is rated to do 10GigE to a max of 35-50m runs, generally well within the max length of any run you’ll be doing in a suburban house.

Settling In

This blog has been a little quiet lately, mostly thanks to Lisa and I being busy adjusting to the joys of home ownership with our new house we moved into in mid-September!

I'm a trust worthy reputable resident of Wadestown now!

I’m a trust worthy reputable resident of Wadestown now!

It’s been pretty flat-out and a number of weeks have already passed us by very quickly – we had anticipated the increase in expenditure that comes with owning a properly, but the amount of time it consumes as well is quite incredible, and given that the property hasn’t had a whole lot of love for the past 5 years or so, there’s certainly a backlog of tasks that need doing.

There’s also the unexpected “joys” that come with ownership, like the burst waterpipe on our first day in the new house, or the one hob on the cooker that appears to like leaking gas when it’s used, or the front door lock that has broken after a few weeks of use. For the first time ever, I almost miss having a landlord to complain to – however the enjoyment of putting a power drill through your first wall without requiring permission cannot be understated either.

 

Amusingly despite becoming home owners, it’s actually been the outdoors that’s been occupying most of my time, with large masses of plant life that has crept over the sheds, the paths and into roof gutters. I cleared 8 wheelbarrows of soil and plant material off the upper path the other day and it’s barely made a dent.

Rediscovering the lower pathway slowly...

Rediscovering the lower pathway slowly…

So far I’ve been mostly concerned about the low level plants, I haven’t even begun to look at the wall of trees and ferns around us – a lot of them are great and we will keep them, but a few certainly need some pruning back to make them a bit tamer and let a bit more light into the property.

Ferns in the mist. Pretty kiwi as bru.

Ferns in the mist. Pretty kiwi as bru.

I’ve been discovering the awesome range of power tools that exist these days – seems tools have come a long way from the days of my fathers wired drill, I’ve now got drills, sanders and even a weedeater/line cutter which all share the same cordless battery pack!

Got 99 problems but wires ain't one.

Got 99 problems but wires ain’t one. Cordless freedom baby!

I’ve had to learn some new skills like how to use a saw or how to set a post in the ground. Of course I cheated a bit by using ready-to-pour fastcrete, but hey, I’m lazy Gen Y-er who wants the fastest easiest way to make something work. ;-)

Hole digging

Harder than it looks. Stupid solid clay ground :-(

I also have two sheds that I need to do up – the first is in pretty good shape and just needs some minor fixes and paint job. It’s even got power already wired up so you can plug in your tools and go :-)

The second shed is in a far worse state and pretty much needs complete stripping down and repairing including a whole new floor and getting rid of almost a meter high pile of detritus that has collected around the back of it over the past 100 years. Helpfully some trees also decided to then plant themselves and grow right next to it as well.

The older shed, pretty but somewhat unusable without some hard work.

The older shed and upper pathway after tidying up the over growth.

 

The house is thankfully in a better state than the garden and sheds, although there is certainly a lot of work needed in the form of overdue maintenance and improvements. The house was built in 1914 (100 years birthday this year!), but thankfully despite the age of the property, the hardest and most essential modernisation has been done for us already.

There’s been a complete replacement of electrical systems with modern cabling and both the structure and interior is in good shape with the original Totara piles having been replaced and whatever scrim wall linings that previous existed having been replaced with plasterboard.

Most of the interior decor is playing it safe with neutral coloured walls, carpet and curtains and the native timber exposed on the doors and skirting. However there are a few garish items remaining from an earlier era where style wasn’t as important, like the lovely maroon tiled fireplace or the cork flooring in the kitchen :-/

The Lounge: Where 2014 meets with 1970 head on.

The Lounge: Where 2014 meets with 1970 head on.

 

Generally the property is nice and everyone who comes over describes it as lovely – but of course nobody tells you if your baby is ugly, so it’s entirely possible everyone is questioning our tastes behind our backs… But give it time, we have a lot of plans for this place that are yet to be actioned!

Our primary task right now is dragging our 20th century house into the 21st century with a few modern requirements like data cabling, heating and decent lighting.

Oddly enough I’ve already started on the data side of things, getting Cat6 ethernet cable run through the house to all the living spaces and roof mounting a WiFi AP and installing a proper comms cabinet. Priorities!

The next major issue is heating, the house has an old wood fire and old unflued gas heater, both of which look pretty dubious. We’ve left them alone and have been using a few recently installed panel heaters, but we need to consider a more powerful whole-house solution like a modern gas fireplace to handle the cold Wellington winters.

Power drills! Holes in walls! This is what home ownership is all about.

Power drills! Holes in walls! This is what home ownership is all about.

In addition to heaters, we also need to fix up the shocking lack of insulation that is common with New Zealand properties. Whilst we have roof insulation already, the floor needs insulating and at some point there is going to be a very expensive retro-fit double glazing cost we need to investigate as well.

 

Aside from these immediate priorities, there’s the question of changes to the layout. The biggest annoyance for us right now is that the kitchen/dining space and the lounge are two separate rooms with a bedroom in-between, which doesn’t really suit modern open plan living so we are pondering the cost of knocking out a wall and re-arranging things to create a single open plan living area.

Additionally we have a really small bathroom yet we have a massive laundry that’s about twice the size just through the wall. Considering the the laundry has almost nothing but a single lonely washing machine in it, it’s a prime candidate for being annexed for a new role as a massive new bathroom.

The tiny wooden cabin bathroom.

The tiny “wooden cabin” bathroom. If it wasn’t for the skylight and our character 12 foot ceilings, it would be really dark and tiny in there. :-/

We are also thinking about how we can improve the outdoor area which is a bit weirdly organised with a large patio area detached from the house and the back deck being a tiny strip that can’t really fit much. We’re already pondering extending the deck out further, then along the full length of the house, so we can join up with the lower patio and make it a nore usable space.

World's tiniest deck.

World’s tiniest deck, not exactly that useful…

Of course all these improvements require a fair bit of capital, which is one thing we don’t have much of right now thanks to the home loan, so its going to take some careful budgeting and time to get to where we want to be. For now, we are just enjoying having the place and plotting…..

 

Aside from the garden and sorting out house improvements, the other major time sink has been unpacking. We didn’t exactly have heaps of stuff given that we just had limited bits stored at each other’s parent’s houses, so it’s pretty scary at how much has emerged and arrived at our new house. I think everyone was kind of glad to get our junk out of their houses at long last, although I’m sure my parents will miss the file server buzzing away 24×7.

It’s been a bit of a discovery of lots of stuff we didn’t realise we had, I have literally a small data center worth of tech gear including rackmount PDUs, routers, switches and other items.

I know what you're thinking "Oh how typical of Jethro, boobs on a box" - but this one ISN'T MINE, it came out of Lisa's parents house... :-/

I know what you’re thinking “Oh how typical of Jethro, boobs on a box” – but this one ISN’T MINE, it came out of Lisa’s parents house which kind of disturbs me deeply.

This is probably the biggest negative of home ownership for me – I hate owning stuff. And owning a house is a sure way to accumulate stuff very, very quickly.

Owning a house means you have space to just “store that in the cupboard for now”. Being a couple in a large 4 bedroom home means there’s a lot of space and little pressure to use it, so it’s very easy for us to end up with piles of junk that actually doesn’t serve a purpose and not feel forced to clean it out.

I came back from AU with two suitcases and I could probably have culled that down to as little as one suitcase given the chance. There’s a huge amount of tech gear I’m considering offloading and Lisa has a massive pile of childhood stuff to make some hard decisions about, because as hard as it is to get rid of things, I think both of us are keen to avoid ending up in the same hording situation like our parents.

Of course some stuff can’t be avoided. I’ve spent a small fortune at Bunnings recently obtaining tools and materials to do repairs and other DIY for the house, so there’s a lot of additions to the “stuff I have to own but hate having to own” pile.

We also needed to purchase all new furniture since we had essentially nothing after returning from Sydney. I don’t mind buying a few quality pieces, but sadly it seems impossible to buy a house load of furniture without also obtaining an entire shed worth of cardboard and polystyrene packaging that we need to dispose of. Sorry environment! :-(

Trapped by packaging.

Trapped by packaging.

We’ve gotten through most of the unpacking, but there’s still a lot of sorting and finding homes for things left to do.

I’m looking forwards to getting to the point where I can just enjoy the house and the space we have. It should be fantastic during summer especially for entertaining guests with its large backyard, patio and sunny afternoons and I’m really looking forwards to having a proper home office setup again for my geeking needs

Oh how I've missed a home office!

Got my home office! If only I had money for computer upgrades left :-(

 

So that’s an update on where we are at for now. It’s going to be a busy year I think with a lot of time spent doing up the place, and I’ll have plenty more blog posts to come on the various adventures along the way. I suspect many of them are going to be quite low-tech compared to the usual content of this blog, but maybe I’ll wake up and suddenly decide that home automation is an immediate vital task I need to complete. ;-)

If you want some more pictures of the house, there’s a copy of all the real estate agent listing photos on my Pinterest account taken by an actual competent photographer, the plan is to try and take pictures along the way as we progress with our improvements to the property to see the progress we’ve been making.

First thoughts and tips on EL 7

Generally one RHEL/CentOS/Scientific Linux (aka EL) release isn’t radically different to another, however the introduction of EL 7 is a bit of a shake up, introducing systemd, which means new init system, new ways of reading logs plus dropping some older utilities you may rely on and introducing new defaults.

I’m going through and upgrading some of my machines currently so I’ve prepared a few tips for anyone familiar with EL 4/5/6 and getting started with the move to EL 7.

 

systemd

The big/scary change introduced by RHEL 7 is systemd – love it or hate it, either way it’s here to stay. The good news is that an existing RHEL admin can keep doing most of their old tricks and existing commands.

Red Hat’s “service” command still works and now hooks into either legacy init scripts or new systemd processes. And rather than forcing everyone to use the new binary logging format, RHEL 7 logs messages to both the traditional syslog plain text files, as well as the new binary log format that you can access via journalctl – so your existing scripts or grep recipes will work as expected.

Rather than write up a whole bunch about systemd, I recommend you check out this blog post by CertDepot which details some of the commands you’ll want to get familiar with. The Fedora wiki is also useful and details stuff like enabling/disabling services at startup time.

I found the transition pretty easy and some of the new tricks like better integration between output logs and init are nice changes that should make Linux easier to work with for new users longer term thanks to better visibility into what’s going on.

 

Packages to Install

The EL minimum install lacks a few packages that I’d consider key, you may also want to install them as part of your base installs:

  • vim-enhanced – No idea why this doesn’t ship as part of minimum install so as a vim user, it’s very, very frustrating not having it.
  • net-tools – this provides the traditional ifconfig/route/netstat family of network tools. Whilst EL has taken the path of trying to force people onto the newer iproute tools there are still times you may want the older tools, such as for running older shell scripts that haven’t been updated yet.
  • bind-utils – Like tools like host or nslookup? You’ll want this package.
  • mailx – Provides the handy mail command for when you’re debugging your outbound mail.

 

Networking

Firstly be aware that your devices might no longer be simple named ethX, as devices are now named based on their type and role. Generally this is an improvement, since the names should line up more with the hardware on big systems for easier identification, and you can still change the device names if you prefer something else.

Changing the hostname will cause some confusion for long time RHEL users, rather than a line in /etc/sysconfig/network, the hostname is now configured in /etc/hostname like other distributions.

The EL 7 minimum installation now includes NetworkManager as standard. Whilst I think NetworkManager is a fantastic application, it doesn’t really have any place on my servers where I tend to have statically configured addresses and sometimes a few static routes or other trickiness like bridges and tunnels.

You can disable network manager (and instead use the static “network” service) by running the following commands:

systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl restart network

Red Hat have documentation on doing static network configuration, although it is unfortunately weak on the IPv6 front.

Most stuff is the same as older versions, but the approach of configuring static routes bit me. On EL 5 you configured a /etc/sysconfig/network-scripts/route-ethX file to define IPv4 and IPv6 routes that should be created when that interface comes up. With EL7 you now need to split the IPv4 and IPv6 routes apart, otherwise you just get a weird error when you bring the interface up.

For example, previously on an EL 5 system I would have had something like:

# cat /etc/sysconfig/network-scripts/route-eth1
10.8.0.0/16 via 10.8.5.2 dev eth1
2001:db8:1::/48 via 2001:db8:5::2 dev eth1
#

Whereas you now need something like this:

# cat /etc/sysconfig/network-scripts/route-eth1
10.8.0.0/16 via 10.8.5.2 dev eth1
#

# cat /etc/sysconfig/network-scripts/route6-eth1
2001:db8:1::/48 via 2001:db8:5::2 dev eth1
#

Hopefully your environment is not creative enough to need static routes around the place, but hey, someone out there might always be as crazy as me.

 

Firewalls

EL 7 introduces FirewallD as the default firewall application – it offers some interesting sounding features for systems that frequently change networks such as mobile users, however I’m personally quite happy and familiar with iptables rulesets for my server systems which don’t ever change networks.

Fortunately the traditional raw iptables approach is still available, Red Hat dragged their existing iptables/ip6tables service scripts over into systemd, so you can still save your firewall rules into /etc/sysconfig/iptables and /etc/sysconfig/iptables respectively.

# Disable firewalld:
systemctl disable firewalld
systemctl stop firewalld

# Install iptables
yum install iptables-service
systemctl enable iptables
systemctl enable ip6tables
systemctl start iptables
systemctl start ip6tables

 

LAMP Stack

  • Apache has been upgraded from 2.2 to 2.4. Generally things are mostly the same, but some modules have been removed which might break some of your configuration if you take a lift+shift approach.
  • MySQL has been replaced by MariaDB (community developed fork) which means the package names and service have changed, however all the mysql command line tools still exist and work fine.
  • PHP has been upgraded to 5.4.16 which a little bit dated already – over the lifespan of EL 7 it’s going to feel very dated very quickly, so I hope Red Hat puts out some php55 or php56 packages in future releases for those whom want to take advantage of the latest features.

 

Other Resources

  1. If you haven’t already, check out Red Hat’s release notes,they detail heaps of new features and additions to the platform.
  2. To learn more about the changes from previous releases, check out Red Hat’s Migration Guide as a starter.
  3. My guide to running EL 7 on EL 5 as a Xen guest for those of you running older Xen hypervisors.

MacOS TTY limit

I’m currently trialling the use of MacOS as a primary workstation on my work laptop, I’m probably bit of a power user and MacOS isn’t all that happy with some of the things I throw at it.

Generally my activities tend to involve vast number of terminals – one day I suddenly started getting the following error when trying to create new sessions inside of iTerm2:

Unable to Fork iTerm cannot launch the program for this session.

Turns out I had managed to exhaust the number of tty sessions configured by default in the Darwin kernel (127 max). Thankfully as per this helpful error report it’s generally pretty easy to resolve:

# Change the current value for the running kernel
sudo sysctl -w kern.tty.ptmx_max=255

# Add the following to /etc/sysctl.conf to make it permanent:
kern.tty.ptmx_max=255

I am liking the fact that although some of what I do is a bit weird for MacOS, at least there is a UNIX underneath it that you can still poke to make things happen :-)

Create MacOS Mavericks Installer

Whilst Apple’s hardware has a clever feature where you can re-install the operating system directly from the internet (essentially netboot install from Apple’s servers), it’s not always suitable if you need to install a machine on an offline connection or via a slow/expensive connection.

Fortunately Apple provides Mavericks as a .dmg file download which you can get from the app store – whilst that .dmg itself isn’t bootable (sadly) you can use a binary tool Apple provides inside it to generate installer media onto a USB drive.

Firstly download this Mavericks installer from the Apple store:

Properitery Evil. Shiny shiny propietary evil.

Proprietary Evil. Shiny shiny proprietary evil.

Then format a USB drive (at least 8GB) to have a single partition of type “Mac OS Extended (Journaled)”, with a partition name of “InstallMe”.

Now you’ll either have a Mavericks installer inside your applications directory, or on your desktop as a dmg file. If on the desktop, mount the dmg. Once done, in your terminal you can run the installer application to generate an installer:

sudo /Applications/Install\ OS\ X\ Mavericks.app/Contents/Resources/createinstallmedia –volume /Volumes/InstallMe –applicationpath /Applications/Install\ OS\ X\ Mavericks.app –nointeraction

(Replace /Applications with the path to the mounted dmg if installing from inside that).

You’ll see some output as it writes to the USB stick, it can take a while if your USB stick isn’t that fast.

Erasing Disk: 0%... 10%... 20%... 100%...
Copying installer files to disk...
Copy complete.
Making disk bootable...
Copying boot files...
Copy complete.
Done.

Once done, you can reboot and by holding down option you can select the USB stick to install from.

Thanks to this forum post for posting the original answer – there are a lot of long convoluted processes mentioned on the web, this is the easiest one by far out of all the options I found.