Bit of a photo finish, but moved this webserver over from CentOS 7 to Ubuntu 24.04 LTS today. Here’s to another 5 years of long operational life span of this webserver!
I started my career (and before then even!) on RedHat Linux so it’s kind of sad moving away from an RH-derived distribution, but Ubuntu is kinda where it’s at for me right now in terms of predictable long term maintenance lifespan, reliable operation, and keeping access for the community.
It seems somewhere along the way I’ve lost my passion/time for updating this blog. I personally blame getting the bike as now I have yet another exciting way to spend rather limited amounts of available time, but it’s probably largely thanks to the massive renovations we undertook in 2020-2021 which has such sucked up any spare time.
When we brought our house in 2014, the roof had been flagged as a likely problem and due a replacement. Naturally we had zero money after buying the house, so managed to bodge the roof along for a few more years with some improper use of silicone sealants and managed to squeeze out another 6 years of life, but in 2019 it got to the stage where it was clear it wasn’t going to be possible to bodge any more and would need the long overdue full replacement as we were starting to experience leaks that could no longer be patched around.
So we bit the bullet in late 2020 and kicked off a replacement. Given we’d need scaffolding for this project, we decided we’d roll it into part of a wider renovation/upgrade project and do some other big ticket items on the house at the same time and wrap it into a small loan extension on the current mortgage whilst rates were sub 2.5%.
Before the roofing work started, we made the choice to remove the natural gas connection from the house. Ripping it out had multiple benefits:
We had no love for gas for cooking, it’s expensive to keep with line charges and is also a fossil fuel and having pipes full of explody gas in an earthquake zone has never filled us with confidence. So we were more than happy to rip it out and replace.
The first big challenge was that our old 170l gas cylinder was located inside the house. The hotwater cylinder installer really really wanted to install the new one on the outside of the house which would be very easy, but IMHO would have looked visually terrible given the layout of our property and not really having a hidden utility space where something big and ugly like that could be located.
To fix this, I ended up demolishing the entire cupboard that existed around the old hot water cylinder, providing a space for the new cylinder to go in and then rebuilding a new cupboard around the installed cylinder.
An electrical cylinder can never get more than 100% of theoretical efficiency using electrical resistive heating. So given my love for heat pumps, we paired this new cylinder with a heat pump hot water system to give us a lower running cost solution with potential efficiencies around 300%.
Essentially there’s a small outdoor heat pump unit (like you would have with a typical mini-split system). This pulls heat in from the environment and turns it into hot water. Unfortunately the technology still requires a sizeable cylinder to act as a reservoir hence why we still have a standard 300l cylinder inside, but I hope that at some point they evolve the technology to the point where it can run tankless and be a drop-in replacement for an instantaneous gas boiler.
We looked at some different brands/options. Some brands required their own special cylinders, this seemed to be the models which pumped refrigerant from the outside unit to a special coil inside an interior tank. Instead, we went for a Reclaim which pumps cold water directly to the outside unit, then pumps back hot water in/out of the “solar hot water” ports on most off-the-shelf cylinders. Should the cylinder or heat pump ever need replacing, there are a number of options that we could swap either out for, and not be tied to a single vendor.
We also kept the ability to be able to failover to the element inside the hotwater cylinder. The electrical circuit has a bypass switch to flip between either the heat pump, or the cylinder’s built in element if a fault ever occurred with the external heat pump.
So far this solution has been working really well, power bills are low and the reclaim unit is very quiet outside. In fact, with the move from gas to electric, our household energy bill decreased on average, despite electricity generally being quite expensive here in NZ.
We then had the scaffolding put up. We were going to need scaffolding edge protection for the roofing work, so decided we’d use the opportunity to also get the house painted which meant scaffolding and walkways around the house.
Due to some poor organisation with the roofers, the project was supposed to be finished before xmas but ran over into the new year which lead to the frustration of paying extra rent on the scaffold over the holiday period, but it did give a good opportunity for me to use the break to fix up various carpentry issues before the painters came and started.
Whilst it took much more time, unexpected water ingress and stress than it should have taken, the end result of the re-roof is excellent. Our roof line is particularly complex and we have plenty of gullys and also a large butynol area in the middle that needed a full replacement.
I found our contractor super tough to deal with due to poor project management and comms but they sure did know how to build a roof. Checked everything against the NZ metal roofing spec and it all looks like they’ve done everything perfectly to code. Crazily enough, replacing an entire roof in NZ? No building consent required! So the home owner really is reliant on trusting the contractor to do the job properly and/or being capable of researching and validating themselves.
Most our entire roof is sarked (covered in timber) which made it easy to get around and work on it. And we had some other good luck, all the timber surface was still in excellent condition and didn’t need any work – with the exception of the butynol section that had been installed with ply that was too thin and so we had it upgraded to the proper standard.
It looks like a lot of our roof was still original, with the roofers pulling off heaps of horsehair underlay that has probably been there for close to 100 years. The steel sheets were super thick as well – in many ways the sheets themselves were fine, the issue was with rust around all the nail points and where the sheets lap/join. Over the decades water had gotten in and caused some serious corrosion in points.
The new roof is all done in pre-painted steel so there’s no painting needed for the first 15 years of the roof. It actually looked so sharp I felt bad that I was intending to keep and re-paint the beat up old gutters, so I ripped them all down before the painters arrived and then installed new ones myself at the very end of the project.
I ended up using Marley Stormcloud spouting which conveniently comes in a range of pre-coloured options that matched our roof and wall colours. There’s also a heap of good materials from Marley on how to install it, with the house being fully scaffolded I was able to do it with some help from mum in the course of a weekend.
And like the roof, the spouting being pre-painted saves massively on the labour and hassle and looks far sharper than anything I could achieve with retrospective painting.
Speaking of painting labour – the amount of work that our painting contractor put in was incredible. The cost of painting a whole house in NZ is eye watering but at least I could see where my money was going with this crew, there was so much prep work fixing joins, rust spots, sanding, etc before the actual painting started. Took a crew of 2-3 people about 2 weeks to do the whole house. I used Graham’s Painters in Wellington and was really happy with them.
With the roof, gutters and paint being done, naturally I decided to squeeze in a few more projects for the summer.
Lisa, having got tired of having now 3 different bikes inside the house, gave me an ultimatum that I needed to fix the old shed before I could buy any more bikes. Having looked at it in detail the verdict was that the old wash house / front shed was too far gone to simply repair and I brought a new made-to-order kitset shed that matched the original’s dimensions exactly to replace it. The team at Sanders Cabins & Sheds was able to alter the dimensions of their off-the-shelf product slightly and position the window/doors at the specific spots I wanted which was excellent.
With the shed done, what else? Well we had a slight side quest to dig up and fix a segment of the sewer pipe. Like most activities that involve me digging, it took about 2 days of effort to get down 1.5 meters to where the pipe was located, but that sure bet the thousands it would cost to remotely re-surface the pipe.
And I had to learn how to plaster to fix the previous massive missing section of the kitchen roof that was damaged by the old roof leaking. Oh and you might notice that the kitchen has changed colour, that’s thanks to Lisa pulling off all the doors and painting them as well as fitting new handles.
We managed to complete the full set of projects (roof, scaffold, paint, path, shed, hot water, cooktop etc) for a bit under $100k NZD, but this was possible only by doing a bunch of stuff myself like the gutters, plastering, shed, digging and house carpentry. I had estimated that we’d run about 20% over budget with unexpected costs and that ended up being almost perfectly accurate. Stuff like higher than expected electrical costs, the decision to replace the gutters, other miscellaneous materials, etc.
Being smart about sequence of work events and shopping around for trades also helped massively. For more than one of the trades we received quotes that were twice as high as the ones we eventually selected. Sometimes you get what you pay for, but sometimes people are just taking the piss. Worth shopping around for these big ticket items.
My favourite was the painter who came to quote and turned up in a huff because he couldn’t park right outside the house (he just parked his ute on the yellow lines anyway like a classic tradie), proceeded to diss the house as “needing a lot of work” infront of me and then quoting twice as high as another firm. For some strange reason he didn’t get the job.
Finally, having finished up the bulk of this project I treated myself to a new car and made sure to get out a heap to make up for the prior summer being consumed entirely by the renovation chaos.
Aside from it’s bike carrying capabilities and gas thriftiness, one of the big aspects of buying the RAV4 2021 Hybrid was getting something with AWD for handling the mountains in winter – and then promptly only managed to spend 1 day up the mountain thanks to COVID-19 lockdowns. But I’m ready for 2022 season!
Where does this leave the house? In a pretty good state now. There’s always something more of course, I have some interior work to finish off and it’s looking increasingly likely that we’ll need to do a bathroom replacement in the next few years due to everything starting to reach end of life in there, but all critical stuff is sorted for now and we’ve gone through 2 winters now without water coming through the roof anymore which is nice.
And I got my bike shed at last! Which has been wonderful, finally cleaned all the junk out of the laundry and have a proper space for bike tools, parts and of course the bikes themselves.
I’ve even cleaned up the older tool shed so it’s actually possible to navigate and find things in there, so that makes two quite usable spaces which is really handy given we have a smallish house at around 140m2.
Back in October 2018, I finally did something I’d been meaning to do for a while and got myself an ebike.
Two main reasons for this decision – Firstly the public transport in Wellington is useless so rather than taking the bus, I would walk or drive to the office, but I really wanted something faster than the 1 hour walk to the office that didn’t involve sitting in my car, getting frustrated at traffic and paying for parking. Secondly I really enjoyed getting back on a bike again a few years back when I tried an eBike at a demo event in Wellington so it seemed like a good source of fun activity.
I decided right away that it would need to be an ebike. Wellington is known for its hills and going directly from not riding for almost a decade to immediately trying to climb some of those super steep hills would not be likely to go well. I wanted something that could get me up the hills, but also be fun both on and off road, so I ended up looking at bikes that were a cross between commuters and mountain bikes.
I ended up getting a Moustache Samedi 27 XRoad 3. This bike is made by a french company and uses a Bosch Performance-line motor and battery system. It’s a cross between a mountain bike (suspension forks, suspension seat post, chunky tyres) but includes commuter features (luggage rack, lights, chain guard, etc) making it ideal for around town, but also taking on gravel and dirt trails.
Since I brought it, I’ve done almost 2,500km on it, all around Wellington. Typically around 7-8km return for my daily commute, plus longer runs in weekends for brunch or random bike adventures riding around the Wellington region.
It really can go almost anywhere – I’ll ride it up super steep hills or light mountain bike trails just fine, or ride to town in my jeans for brunch just as happily. It’s certainly reduced my car usage, I’ll much rather take the bike for any trip where I don’t need to take passengers or pickup any heavy amounts of cargo.
Around town I can cruise at 30km/hr comfortably. My model is EU-spec, so the motor doesn’t provide any boost above 25km/hr, but it’s not hard to keep it at 30km/hr in most conditions with pure human power, sometimes I’ll even get it as high as 50km/hr on the flat if there’s not an energy-sapping headwind.
Uphill isn’t so fast, but I can average around 15km/hr on almost any hill with this bike. Fast enough that I’ll generally want to pass most over cyclists, but not so fast that you can leave the cars behind sadly. For context, this is about twice as fast as when I ride the manual bike.
The real measure of speed is to take a look at the end-to-end performance once traffic, shortcuts etc are factored in. I recently raced the car from our home in Wadestown to Kilbirnie on a weekend and it takes exactly the same amount of time – 30mins – to go by car, sitting in traffic queues, or by bike and having fun (and exercise) instead.
Speaking of exercise – it’s hard to measure the benefits exactly. I’ve not lost as much weight as I would have liked (~10kg over 1 year), but my fitness is significantly improved, to the level where I now find activities like snowboarding, yard work and running much much easier to complete and I’ve even become fit enough to buy and ride a regular non-ebike mountain bike for the more adventurous trails that my ebike isn’t well suited for.
Probably the closest measurement I have is that when I first got my ebike I would ride it uphill in turbo mode (275% boost). Now I’m able to ride it uphill just as quickly in eco (50% boost), which is a big improvement in my physical power output. My model is a pedelec, which means it only supplies power in response to ride input – there’s no throttle or ability to ride it like a motorcycle.
Really happy with the decision to get this bike, the level of fitness and enjoyment I’ve gained from it is worth every cent of the purchase price – which is really the only downside, these aren’t the cheapest bikes around at over $5k NZD which makes the barrier to entry a bit high for some people. That being said, you’d spend that in just over a year on carparking if paying casual day rates, so it’s a lot of money and yet at the same time, not a lot when you compare it to other forms of transport and it’s easy to justify return on investment with.
I’ve recently also converted Lisa over to the ebike life, so in the past of 12 months we’ve gone from being a 1-car household to being a 3-bike household. Lisa wrote about her experiences for ACC here, including her experiences with the excellent Pedal Ready course.
I didn’t have too many issues getting used to riding around town so I haven’t done the Pedal Ready course myself, but I’d recommend it strongly to any nervous riders based on the feedback I’ve had from people who have taken it.
On the whole I think that Wellington drivers are a lot better than they used to be and I’ve not had many close calls. Also not much abuse – a little bit here and there, usually when I have to take the lane (eg Molesworth St) for safety reasons but are going slow (eg because it’s uphill), but I’m not sure if the relatively lower levels of abuse are due to drivers being more conscious than when I was younger, vs the fact that they don’t want to pick a fight with a large white male. I sadly suspect it’s the latter contributing for the most part.
Douchebag drivers aside, I feel Wellington is right at the verge of being a great cycling city. There’s been enough investment over the past decade to make it no longer a daunting activity with changes like green cycle boxes on most intersections, cycle lanes in some places (still not enough!) and 30km/hr speed limit in city.
I’m really hoping the growth rate goes exponential and forces the council to really stop treating cycle infrastructure improvements as gradual side projects. I would particularly like to see proper segregated cycle lanes on all the uphill roads (eg Molesworth st) so that cyclists can ride at their comfortable speed without choosing between blocking cars or getting doored by people parked on the side of the road. I’m not a huge fan of the “shared” cycleways that the council has started putting in, I find they tend to end up with people randomly wandering around on the bike side ignoring the fact you’re wanting to go faster than walking speed but I guess they’re better than nothing. Ideally we need proper micromobility (bicycles + escooters + etc) lanes throughout the city at the same priority and space allocation as cars. Also really looking forwards to a proper bi-directional cyclelane on the proposed new sea wall heading out to Petone to open up the hutt valley to my weekend rides. The current “solution” of riding along the 100km/hr motorway edge isn’t something I want to try out anytime soon.
Given the combination of currently terrible public transport, affordable ebikes and goodish cycling infrastructure I think summer 2020 is going to be a record peak of bikes in Wellington. The cycle counters in Wellington have recorded the second highest number of cycle trips ever in August and that’s still full of cold wintery days when riding isn’t the most appealing idea.
Anyway this is a long overdue post regarding my ebike excitement, something that might actually be exceeding my interest level in tech right now. I’m also going to put together some notes about my recommendations if you’re looking to buy an ebike so will get those up on this blog soon hopefully the next post on here.
We recently experienced a weird and frustrating problem with storage reliability on our RabbitMQ cluster running on AWS c5-series EC2 instances with EBS storage and Ubuntu LTS 16.04.
The problem is super weird in that it occurs somewhat randomly – no obvious correlation to load, time of day – but it does tend to happen after the instance has been running for at least a few weeks. It also occurs on any of the three RabbitMQ instances, so it’s not something specifically weird about any one instance in the fleet.
The one thing we do know, is that the issue is storage related. Firstly nothing is persisted in the logs (RabbitMQ or system/kernel) from the time the issue occurs and secondly we can see a large spike in disk I/O wait time in our Datadog monitoring for the instance, showing that the instance is stuck with processes waiting for the disk to respond.
Why RabbitMQ is impacted in this manner is unclear. It makes sense that the cluster quorum and status negotiation wouldn’t require working disk to keep running, but in every test we made where we deliberately broke the storage, the RabbitMQ process would correctly detect something was wrong on the host and go into an unhealthy state, removing it from the cluster. Tested ripping out EBS storage whilst still mounted, corrupting with dd, force unmounting, etc… nothing could trigger the exact same behaviour.
Reviewing what differs about production was difficult since it didn’t persist any of the kernel or RabbitMQ logs, however we did manage to extract some information from the AWS instance console for one of the impacted systems before we restarted it:
Ubuntu 16.04.4 LTS localhost ttyS0
[349442.682614] Not tainted 4.4.0-1062-aws #71-Ubuntu
[349442.684363] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[349442.686890] INFO: myprocess:1956 blocked for more than 120 seconds.
Essentially the Linux kernel is proceeding to log a number of different processes (basically everything on the box that does anything) as being blocked for over 120 seconds, thanks to the storage failing and being unable to do anything about it to unblock the processes
Given we have been unable to identify the exact fault or reproduce the behaviour (could be something in the Linux Kernel, could be something in AWS c5 or EBS…), we needed a solution that would at least help us by terminating any instance that experienced this storage issue.
The solution is helpfully identified by the kernel log lines above. We can use the hung task panic feature in the Linux kernel to force a host to immediately reboot itself if processes are hung for too long.
We do this using two different sysctl configuration changes (note – you need to set these up in /etc/sysctl.conf to survive reboots):
# Panic if a hung task was found
sysctl kernel.hung_task_panic=1
# Reboot 5 seconds after panic
sysctl kernel.panic=5
The first instructs the kernel to panic if a hung task (any task blocked for more than 120 seconds) occurs. The second, instructs it to reboot shortly after this occurs. We set it to 5 seconds, to give time for any logging to persist or be delivered about the hung task before it’s reboot, although in this particular situation, with all storage being busted its of very limited benefit.
This has been in place for several months now and is working beautifully. Every so often an instance experiences this fault and instead of causing any disruption, it is quickly self-terminated and replaced. Because it terminates completely, the RabbitMQ cluster negotiation is successfully able to re-assign responsibilities to the other instances in the cluster.
In theory there is a 2-minute period where the unhealthy instance is still running, however reviewing the production metrics, it appears that when the fault occurs, RabbitMQ doesn’t immediately break – sometimes it continues to run for 15mins or more before jamming up the cluster. So having it run for 2 minutes has turned out to be just fine.
Ideally going forwards we need to setup a network logging endpoint for these hosts to see if we can capture anything like a stack trace from a kernel driver. It seems likely that it’s a Linux kernel bug rather than an AWS EBS bug, since the issue is resolved with a soft reboot, rather than a force stop-start of the instance via the AWS API, meaning it’s still running on the same hypervisor host, etc. But until then, this kernel configuration parameter means we are not going to disrupt operational services when the fault does occur.
I recently had the pleasure to speak at the 2018 Mobile Refresh conference held here in Wellington and did a talk introducing how we run some parts of the Sailthru Mobile platform, along with recommendations and advice to anyone else also building backends to support their mobile applications.
It’s more entry-level than some of my other infrastructure talks as it’s focused on people that are primarily mobile developers with maybe a limited set of infrastructure awareness.
I’ve recently been experimenting with Xiaomi Zigbee IoT/Smart Home devices. These devices are super affordable (approx $20-55NZD) and make it very easy and affordable to upgrade a home with useful smart devices.
My plan is to do a more detailed blog post at a later stage, but if you’re interested in learning more, I recorded a talk I made at the Wellington Home Automation Meetup a few weeks back that introduces these products and how I’m using them.
Having gone through three Wellington winters in our home, Lisa and I really didn’t want to make it a fourth this year. Whilst our property is fortunate enough to get a lot of sun during the day, it still can’t sustain a comfortable temperature in the middle of winter when we have weeks with day highs of 10c and lows of 5c. In the worst case, it’s gotten right down to 2c outside.
We insulated as much as possible shortly after we moved in, both the roof and floor, but unfortunately with house being very low to the ground, there were sections of the floor that simply couldn’t be insulated, so we ended up with approximately 60% coverage. We aren’t sure about the walls either, some of the ones we’ve had to cut open to do work have revealed insulation but we don’t know for sure if it is fully present in all walls of the house or not.
Insulation upgrade in progress
Most New Zealand homes are also only single glazed and ours is no exception, which means even with the roof (100%), floor (~ 60%) and walls (??%) insulated, we still have massive bay windows leaking heat like crazy. So regardless of the natural sun we get, a high capacity heat source is still critical to maintain a proper temperature at the property.
Which is also precisely what we lacked – when we moved in the house had a rusted wood fireplace in dubious condition and an unflued gas heater which are generally just a bad idea all-round. Neither of these were safe options, so we needed to replace them with something better.
We considered a few different options:
The gas pipe limitation also impacted our ability to drive other high consumption gas appliances, for example feeding large fireplaces could be tricky and we’d probably be unable to run more than a single instantaneous hot water system. Probably for the better anyway given that it’s a fossil fuel, but it’s so much more cost effective than electricity on a per kWh basis in NZ that it’s a compelling option if you have gas piped to your property.
Given the above issues, the only remaining option that made any sense was electric heat pumps. Heat pumps are generally 300-400% efficient, meaning although they use super expensive electricity, you get very good heat output compared to running traditional electric heaters.
What most NZders will think of when you say “heat pump”
These are becoming quite popular now in NZ as an affordable upgrade (a fully installed basic unit might cost as little as $2-3k NZD) for existing properties – the purchase of a mini-split system such as the above allows for easy retrofit installation and you’ll often see these added to lounges to ensure a reliable heat source in the main living space.
Their weakness is that you don’t get the heat spreading from the primary room out into other parts of the house, such as bedrooms. And they’re not cheap enough that you would install one in every room to get the coverage, but even if you did, you’d then have to deal with managing them all in a sensible fashion so they don’t end up fighting each other when maintaining temperatures.
To solve this problem, we went for a less common (in NZ residential homes) option of putting in a central ducted heat pump. These units use the same essential technology (using a refrigerant to shift heat from the outside unit through pipes to an inside unit), but rather than having an on-wall unit, we have a large unit in the attic, with ducting pushing air into every room of the house, to be recirculated back via a hallway intake.
Due to the amount of specialist skills and tools required for this project, plus a lot of time climbing around the attic which I just love doing, we instead contracted Wasabi Air to perform the complete installation for us. They did a really good job, so pretty happy with that.
Diagram from Mitsubishi’s website illustrating how a ducted heat pump setup looks.
We went with a Mitsubishi PEAD 140 unit based on Wasabi’s recommendation as an appropriately sized unit. This unit is capable of 16 kW heat output or 13 kW cooling output and can consume up to 4 kW of electricity to operate at peak.
The installation is pretty full on, it took 3 days for a team to get everything installed and set up. I took a selection of photos during the install since I’m a weirdo who finds installing things like this interesting.
Firstly the exterior unit needed installation. Whilst modern heat pumps are “quiet” they’re still audible when working hard so you want them away from bedrooms – thankfully we had a convenient space between two bay windows where it could sit below the floor level of the house. The only slight downside of this location, is that I’ve noticed the deck has the tendency to slightly amplify the vibration of the unit at times, whereas if we had gotten it mounted on a concrete pad elsewhere it may have been less of an issue.
Exterior unit. This thing is *big* and shovels a significant amount of air through itself when operating.
Vacuum pumping the refrigerant lines ready for the gas to go in.
Because the exterior unit needs to run copper piping for the refrigerant, power and controls up into the roof of the house, we needed to go from under the floor to the attic at some point. Rather than an ugly boxed conduit on the outside of the house, we had it installed through the middle of the house.
As mentioned earlier, our fireplace was wrecked, so I knocked it out entirely a few months back (post coming once I finally finish the project) which has left a messy hole an accessible space from the floor to the attic that was perfect for linking the internal and external units.
Whilst the piping is copper, it has a surprising degree of flexibility so the team was able to feed it through and bend it varying degrees, whereas I had assumed it would all have to be right angles and welds. The piping is wrapped in a insulation tube which does a pretty good job at avoiding heat loss – even when running on full, the insulation is just mildly warm to the touch.
Piping and cabling going in.
One of those big electrical cables had to go all the way back to the switch board to feed the unit with electricity. Unfortunately our switchboard is totally full, so the electrician ended up having to add a couple additional breakers off it to the side.
It’s not wonderful but it solved the problem – given future plans for the property (electrical hot water, induction cooking, car charging), I think we’ll end up doubling our switchboard and making it all tidy as a future project.
Not going to win any prettiness awards, but it’s safe and compliant.
Of course the exterior unit is just half the fun – we also needed the interior unit installed and all the ducting connected. We had what is considered a slim unit, but it’s still a very sizeable item to try and lift up into a roof space!
Indoor unit before going into the roof and having ducting connected.
Impromptu skylight
As there’s no way to get the unit into the roof via existing access ways, the team cut a massive hole in the hallway roof. They then swung the unit up into the roof with a team of guys who do a lot more physical activity than me and backfilled the space with the intake vents.
Intake vents going into the hallway to nicely fill the big hole that got made earlier.
Once fitted in the roof, it takes up even more space for all the ducting to be connected. A large amount of our spacious attic is now consumed with this unit and associated ducting. I’m just glad we’d already done insulation and most of the cabling work we’ll want to do in there for the foreseeable future as it’s a lot more difficult to navigate now.
Fully installed in the attic space
Despite being above the bedroom, the unit is really quiet when operating and we can barely hear it. I took some video of the installation to give an idea of it’s low level of background noise even when inside the attic.
We decided to install outlets into every single bedroom (x4), one in the lounge and one in the dining room for a total of x6 outlets. The hallway features the intake vents for recirculating the air.
Note that there are two key downsides to the ducted heat pump option vs radiators here:
Ducting getting prepped pre-installation
The above photo showing the ducting being prepared shows the built-in dampeners used to balance the system when it’s installed. These are used to restrict/open the flow of air to each outlet to handle the fact that outlets have different length ducting runs and sizes. It also makes it possible to do things such as setting the heat output in the bedrooms to be a bit lower than that of the living spaces by keeping their volume of air output a little lower.
Once installed, the outlets are unobtrusive in the corner of each room.
I had some concerns around how the vents would handle our high ceilings and we do get some degree of heat blanketing when the system first starts heating the house (warm head, cold toes), but once it brings up the whole house to temperature the high ceilings aren’t a major issue.
That being said, it does mean there’s a lot more air to heat than a property with lower ceiling heights and certainly on the coldest days we’ve found the system takes too long to bring the house house up to a comfortable temperature if we’ve let the house get very cold. The solution is pretty simple, we just leave the house on a lower temperature when away from home to ensure it’s only a few degrees off our comfortable temperature when we come home (typically found 20c when awake and 18c when sleeping is our preferred climate).
It works well – for the first time we’ve actually had a comfortable time in winter, rather than huddling for warmth in only a couple rooms with the rest of the house closed off trying to conserve heat.
Whilst the heat pump ensures the house can now always be at a comfortable temperature, it won’t prevent moisture build up. It can reduce window condensation on account of the warmth and even be put into a drying mode, but it’s ultimately not a ventilation system and doesn’t cycle out moist stale air if you fail to manually ventilate your house.
Our house wasn’t too bad with moisture (not compared to many other NZ properties) on account of the kitchen and bathrooms all featuring extractor fans which ensured the bulk of moisture produced in the house is vented out, but the simple act of living in the house is enough to create moisture and having weeping windows in the morning was still not uncommon for us, especially during winter when opening windows isn’t very compelling.
Given the fact that we have long and cold winters, we decided to invest a bit more and had Mitsubishi’s ventilation product (Lossnay) installed as part of this project. This unit integrates with the ducted heat pump and regularly cycles out stale air and sucks in fresh air, recovering most of the energy in the process so you’re not constantly discarding the energy you just spent heating (or cooling) the house.
Diagram from Mitsubishi’s website illustrating how a Lossnay connects to the ducted heat pump system.
Lossnay unit operating away quietly in the maze of ducting.
Because of the layout of the house, the two Lossnay vents ended up having to go through the roof – the egress near the existing bathroom fan vent and the intake a number of meters away on the other side to prevent just recirculating the same air, so we now have a few more mushrooms to add to the other unsightly appearance of our roof. Summer 2018-2019 has a painting project on my list to tidy it all up with a fresh coat of paint.
Lossnay vent (R) alongside existing bathroom vent (L)
The end result is that we have a completely moisture free property and I haven’t opened the windows once all June, since the system is constantly ensuring we have fresh, dry and warm air in the property. Even on the coldest days, the windows are now completely dry which is just fantastic.
To control the combined system, there’s a central thermostat and control interface installed underneath the return intake. It’s pretty basic, capable of setting modes, fan speeds, temperatures and timers, but does the fundamentals nicely enough.
Nice modern look but it’s not all that intelligent.
Ultimately I wanted proper IP control of the system to allow a range of integrations rather than the manual drudgery of having to touch actual buttons with my actual hands.
Surprisingly for 2018 this isn’t standard for all heat pumps like this, some vendors seem to lack options entirely and whilst Mitsubishi have their “WiFi Control” product, it’s still sold as an extra ~$250 add-on, rather than being natively built into the control interface as standard feature.
Essentially it works by connecting a small embedded computer to the heat pump via it’s onboard serial port (You can see the little white box on one of the earlier photos of the attic install with the trailing cable going to the unit’s control circuit). It then bidirectionally syncs with Mitsubishi’s cloud services, which then allows you to control it from anywhere via your phone app. The experience is well polished and Mitsubishi is even doing HTTPS for their connections which was a (positive) surprise. Plus the WiFi module was designed/built by a local New Zealand company on behalf of Mitsubishi, so that’s kinda cool.
WiFi module + app
Note that I mentioned serial before – this isn’t too tricky to interface with and it is possible to build your own module – I’m not great with hardware and didn’t want the hassle and warranty risks, but if that’s your kind of thing, take a look at this blog post on Nicegear.
Overall it’s a pretty decent Internet of Things product, the only issue I had with it is that it relies entirely on their cloud service. And whilst Mitsubishi’s cloud app is *probably* going to stick around for a while, I wanted local control to allow me to link it to Home Assistant to make it part of a holistic smart house, not just an independent appliance. Getting it into Home Assistant also makes it possible to then control via Apple HomeKit, something not possible with the off-the-shelf product.
I had a bit of fun figuring out how I could interface with it locally (see my talk at the inargural Wellington Home Automation Hackers if interested) but ultimately I discovered that the latest version of the IP module (MAC-568IF-E) supported a Japanese standard called ECHONET Lite. This standard allows for completely local network control of various smart home appliances, including heat pumps, which allowed me to then write a bridge application between Home Assistant (using MQTT) and the heat pump (using ECHONET Lite).
The source code for this bridge application is available at https://github.com/jethrocarr/echonetlite-hvac-mqtt-service including a Docker image for easy installation.
The only limitation is that I have not found a way to control the Lossnay via ECHONET Lite – given that the official Mitsubishi app also lacks control, I suspect they simply haven’t developed functionality into the WiFI control module to expose the options for the Lossnay, it’s probably not a massively popular addon
End result is that I now have the ability to control the mode and temperature via HomeKit from my iPhone/iPad – and since HomeKit links with the wider Apple Ecosystem and the fact we have an Apple TV, it’s possible for both Lisa and myself to control the heat pump (and all the other linked stuff) remotely from anywhere without a VPN needed. It also makes things super trivial to delegate and revoke access to guests when visiting.
Heating controls in HomeKit!
The other main drive to integrate locally is that I want to be able to setup smarter rules inside Home Assistant. Some of the stuff I’m planning is:
So aside from the remaining automation work, project complete! We are super happy with this solution, it’s completely changed the liveability of the property during winter and it’s an upgrade I would absolutely make to any other property that I ever buy in future or recommend to anyone else living in our climate here in NZ.
It’s not a cheap solution – expect to spend $15-20k NZD for a 3-4 bedroom home, although it will vary considerably based on sizing and installer effort needed. Like most home upgrades, labour is a huge component of the ultimate bill. Our final bill came in around $22k, but this also included the Lossnay ventilation unit.
There’s also the electrical cost to operate. This one is super subjective since it depends on your usage patterns, house, etc. Our first winter month bill at ~$450 NZD seemed super high, but when we compared with previous years, it was only 30% more and instead of having two rooms barely liveable, we got an entire house that was comfortable. In fact it’s efficient enough that we’re likely to still come in just on or under the 8,000 kWh yearly low user consumption, given that our hot water is currently gas. Once we go full electric with our hot water, we will probably surpass that and need to go to the standard plan (cheaper per kWh rate, higher line charges).
I’m expecting to get 15 years lifespan on the system, maybe longer. At 15 years, it’s essentially $1,460 per year deprecation for the asset, which isn’t too bad for the level of comfort it provides. If we get longer, or can refurbish to extend beyond that period, even better. It’s also worth noting that even if the mechanical components need replacing at their end of life, we have all the ducting, piping, electrical etc already installed which would be reusable, making a replacement project at it’s EOL considerably cheaper than a new installation given how much of the cost is the labour doing ducting runs, cutting holes, etc.
Whilst there are many other reputable brands out there, we went with the Mitsubishi offering for two main reasons.
Having seen the effort and complexity that went into this installation, I’m glad I got the pros to do it. It would have taken ages to do by myself over weekends, etc and there’s a number of specialist steps (eg filling the refrigerant lines with gas) which are non-trivial plus just the experience of knowing how to size and setup the ducting runs appropriately for the property. So I’d recommend staying away from a DIY project on this one, despite the temptation given the costs involved. A good friend went through a project to do most of the work on his own install himself and it ultimately didn’t save as much as expected and created a number of headaches along the way.
I’m slowly setting up more and more Internet-of-Things stuff in my house and because of my interest in this space I’ve started a Wellington Home Automation Hackers meetup that will meet monthly! If you’re in Wellington and are interested in this sort of thing, please do join the group and come along!
I’m aiming to have 2 different presenters lined up every month to ensure a range of topics, including a mix of presentations suitable for less technical beginners as well as more technical gurus (and everything in-between).
I kicked off the first month by talking about my adventures so far with Home Assistant, using it to integrate with my air conditioning, house alarm and how I’m using it in conjunction with Apple Homekit. If you missed it but this sounds fun, check it out on YouTube below!
I’ve recently been playing with the popular open source home automation software Home Assistant. One of the nice features of this platform is that it can export most of the devices it manages as HomeKit devices for easy use from iOS devices.
HomeKit isn’t perfect, it’s a generic management platform so it’ll never be as good at doing thing X compared to a native app from vendor X – it just can’t have all the same parameters and configurability.
Despite this, there’s some compelling features for a household that’s fully on the Apple ecosystem:
The only thing that’s a bit annoying, is that if you get your firewall rulesets wrong it can be tricky to debug.
I had opened up TCP port 51827 (used by HomeKit) and was able to pair my device successfully, but then had weird issues where the accessories would go into “No Response” state for prolonged periods and only occasionally update with the latest information.
Steve says you’re holding it wrong
The trick to finding this was to do some packet dumping. I ran a packet dump for all traffic from my phone to the server running the Home Assistant app to see what was coming across the wire and could see a pile of mDNS requests that weren’t being answered.
Wireshark never lies
mDNS is a tricky protocol – essentially it’s DNS, but instead of going to a name server for resolution, devices using mDNS send out a multicast packet to the network and wait to see who replies with the answer. Devices implementing mDNS need to listen to these packets and respond where appropriate. It’s most commonly implemented as Bonjour (Apple) and Avahi (Linux).
This means that we need to setup a firewall rule for UDP port 5353 to allow HomeKit clients to find the HomeKit accessory (in this case, Home Assistant). Without it, you get the “No Response” problem when lookups fail.
Why did it work at all without it? Not 100% sure, but I think HAP-Python might occasionally send out it’s own multicast messages advertising itself to iOS devices which allows them to find it for a period of time, but when the TTL expires and they try to re-resolve for connected accessories it’s nowhere to be found.
So the complete set of iptables rules you probably want (or something like them) is:
# mDNS iptables -A INPUT -p udp -m multiport --dports 5353 -j ACCEPT # Homekit Protocol iptables -A INPUT -p tcp -m multiport --dports 51827 -j ACCEPT # Home Assistant interface iptables -A INPUT -p tcp -m multiport --dports 8123 -j ACCEPT
I completed this project a while back and had the images saved up for a blog post – somehow almost a year has gone by since then in a blink of an eye. But anyway, enjoy this delayed update about the exciting world of bathroom fans!
If our house had one deficiency, it would be that the layout of the property has resulted in a rather small and interior only bathroom. Since this bathroom has no direct access to any outside walls for easy ventilation via windows, it instead had a pretty chunky fan already installed when we purchased the property to extract all the shower moisture and other undesirable elements.
By my estimation, the fan would be a good 20 years old and whilst it was doing a good job of extracting air from the bathroom it had two major issues:
The second issue above really started causing significant issues, particularly since the moisture was damaging the paint and plasterboard and with a small bathroom that makes it almost impossible to extend a ladder and reach the super high ceilings, mould was becoming an issue due to inability to access to tackle the moisture.
The situation required remedial work and one day the old fan decided to make the decision easy with the fan cover getting brittle enough that it suddenly fell out of the roof into the bath one evening without warning with a loud crash.
Unscheduled rapid disassembly
For this replacement project, I started with tackling the ventilation problem to make sure the air would actually get extracted out of the house, rather than into the attic (side note: pretty sure venting bathrooms into attics is now forbidden under the building codes for any new installs).
To do this, I brought a 150mm Simx/Manrose Thru Roof Cowl Kit – these can be found at consumer hardware stores and it’s a kit that consists of a plastic tube, the metal cowl ontop and a suitable rubber mounting boot and assorted hardware.
Going through the roof was the only option on this property. Not only is the bathroom in the middle of the house, even if I ran a long ducting run to the nearest walls, there’s no soffits or other tidy location to install the vent without damaging the character of the property.
Who cares about the iPhone, *this* is the unboxing you’ve been holding out for.
Installing this was… fun. I purchased my new most-favourite-thing-ever, a reciprocating saw (also called a sabre saw) in order to cut a hole in the roof. This tool has gone on to work hard in a large number of other projects and I consider it an extremely good investment given it’s versatility.
Cut all the things!
One of the quirks of our property being approximately 100 years old is that the roof is sarked with timbers which makes it extremely solid – a proper house, from a more refined age. They stopped building houses like this a long time ago, I think the whole non-sustainable forestry part become a slight issue…
In this situation the solid build both helped and hindered – trying to cut through corrugated iron is a lot easier when there’s not 20mm of native hardwood underneath it as the saw has the habit of picking up the iron and vibrating it like crazy whilst trying to cut the timbers.
But the upside is that it meant I could screw the roof vent directly into the timber and be assured of a tight and long-lasting fit, whereas if I had only floating iron sheets it would have been a lot tricker to get a really tight fit… If this had been the case like in a more modern property, I’d have probably brought some plywood sheets and run them across the rafters to provide a solid surface for screwing the vent into for a similar effect.
You can get an idea of how solid the house is from this photo inside the attic. In this photo the vent has been installed and ducting attached.
Once I cut the roof hole, I sealed the vent kit rubber boot thoroughly with silicone, with a layer between the boot and the iron sheets, as well as around the edge of the boot and the plastic tube.
The rubber boot has a metal strip allowing it to be bent to fit the corrugated iron snugly. Once the screws went in, it’s a very tight seal and the silicone makes 100% sure it’s sealed.
Note the diagonal placement – this is intentional since it ensures you don’t end up with a pool of water collecting at the top of the boot, which could happen if you placed it square.
The new vent next to the bathroom skylight. You can see the interior intake through the skylight.
You’ll notice that I’ve cut the hole overlapping two separate sheets. This… isn’t ideal, I’d much rather have cut into a single sheet (easier to seal, less to go wrong as the sheets expand and contract) but I was trying to utilise an existing hole that was already cut in the sarking timbers for what must have been a small vent (maybe an overflow pipe?) in the past to minimise the amount of cutting needed.
This placement also caused another small annoyance for me, which is that the vent is not quite 100% straight and you can notice it sometimes. It’s not an issue for water ingress in the rain, but it annoys me not being 100% perfect. That being said, I’ve had trades install other vents in the property (future project post coming!) and they didn’t get it properly straight either, so I feel somewhat vindicated.
Slightly wonky vent will annoy me everyday forever more…. but if I’ve learnt anything re DIY, don’t fuck with anything that ain’t broke.
To connect the fan to the vent, I brought some insulated ducting. It’s important to use insulated ducting for this, rather than the cheaper uninsulated stuff, since bathroom air is warm and moist – if the ducting is not insulated, you are likely to suffer condensation in the duct as the air cools when it transits through the cooler attic temperatures. By keeping it as warm as possible on it’s way out you can avoid this.
I was worried about some condensation occurring in the vent tube itself – I can’t insulate outside of the roof after all – but this fortunately hasn’t been a problem. Additionally there was some concern that the roof cowl wouldn’t be enough to keep out rain in Wellington winds, but this also hasn’t been an issue – it could be a different situation in more exposed areas of the city and there are cowl fittings that are more suited for unfriendly wind conditions if this was the case.
Next I had to sort out a new fan. I was tempted to keep the existing fan given it’s strong air throughput and the motor still running fine, but I couldn’t easily find a replacement front for it, and the back was completely exposed so there was no easy way to fit the ducting to the back of the fan.
I ended up buying a 250mm “high pressure” fan, but unfortunately this didn’t work out well for me. Despite being described as high pressure, I can confirm that these consumer-available bathroom fans are absolutely useless and not worth buying if you want anything more than a faint breeze.
I had it in place for 1 week, during which time we not only quickly noticed it struggling to remove the steam from the room, but that it was also slowing filling up with water that was condensing in the fan as it struggled the clear the room.
First iteration. Note the side exiting fan which required twisted ducting – not ideal. You can also see the hole in the sarking is larger than needed – that’s because there was a pre-existing hole that I took advantage of which was longer than I needed.
Unfortunately given how much of a complete failure this fan had been, I had to remove it and find an alternative.
The solution was a 150mm pro-series Simix inline fan capable of delivering 166l/s (597m3/hr) air throughput. For comparison, the previous attempted fan was maybe 69l/s (250m3/hr) at best and even that seems dubious given how poorly it performed.
Nimbus is a big fan of this model.
I couldn’t find these at any consumer hardware store and ended up taking advantage of a friend with an electrical trade account who was able to place an order with the supplier for me.
The one key difference with this solution vs my previous attempt is that the fan is inline, which means the bathroom needed a vent installed, with ducting from the intake vent to the fan, then ducting to the outlet vent. This does have some noise advantages since you can place the fan motor further away from the intake.
Second iteration. Mounting it on framing timber is a little overkill but I had framing timber and not plywood handy. Because of the solid roof, I found it easier to mount to the underside of the roof, rather than building a platform on the attic floor.
This solution worked much better – the fan is able to extract a considerable amount of air and whilst a bit noisy due to high RPM and small diameter, it does a good job of clearing the bathroom throughout the shower and not letting it build up too much moisture.
When researching this project, it was suggested that you shouldn’t be clever and mix duct sizes (eg a 200mm fan into a 150mm vent), so I kept the same spec throughout – if I had gone for a larger roof vent and duct in the beginning, I might have gone with something larger to get more throughput but also larger fans tend to be quieter (as a general rule).
The other big positive of this approach is that I was able to solve the skylight condensing problem by putting the new intake vent directly into the side of the skylight wall to rapidly extract out the air.
This is working extremely well – whilst we have the existing damage from past moisture build up which will require remedial work (complete repaint, maybe some new plastering), since putting in this new vent we’ve had very little moisture build up since the fan keeps the air moving in the skylight preventing condensation. And since heat rises, all the steam from the shower naturally gravitates towards the vent anyway.
This photo illustrates the issue with the placement of the old vs new fan – the old one did nothing for all the stream that wafted up into the skylight space, vs the new one keeping that space clear.
I found it really hard to find an intake vent that wasn’t horribly ugly and plasticky, so ended up paying a bit of a premium for an aluminium model. It ended up being a right pain to install so maybe I should have gone for a cheap nasty plastic 150mm that would have been simple to fit, but I think it was worth it and looks good (once I fix all the paintwork anyway). I even managed to get it dead centre which given I was cutting it out from inside the attic due to inability to get ladder high enough in the bathroom was a pretty good outcome.
Anyway despite the challenges, I’m pretty happy with this setup now. It’s working reliably, I’ve checked the ducting a few times to make sure there’s been no moisture build up or water ingress from outside (both good!) so I’m expecting this solution to last for a long time.
I still need to fix the plasterboard and paint job in the bathroom. It’s kind of stuck pending access to a more flexible ladder/indoor scaffold type system just due to the height of the bathroom roof and very limited placement points for ladders. Still it annoys me daily so it’ll get fixed sooner or later when I get really sick of it looking so bad.
Rough cost for the project was around $500-600 NZD in parts – the most expensive bits being the fan motor, and then the roof vent kit – a wall vent solution would be a fair bit cheaper.
If I was doing this project again from scratch, I’d probably have done a few things differently.