Great server crash of 2012

Leave a reply

In a twist of irony, shortly after boarding my flight in Sydney for my trip back to Wellington to escape the heat of the AU summer, my home NZ server crashed due to the massive 30 degree heatwave experienced in Wellington on Christmas day. :-/

I have two NZ servers, my public facing colocation host, and my “home” server which now lives at my parent’s house following my move. The colocation box is nice and comfy in it’s aircon controlled climate, but the home server fluctuates quite significantly thanks to the Wellington climate and it’s geolocation of being in a house rather than a more temperature consistent apartment/office.

After bringing the host back online, Munin showed some pretty scary looking graphs:

localhost flew too close to the sun and plummeted to it's doom

localhost flew too close to the sun and plummeted to his doom

I’ve had problems with the stability of this system in the past. Whilst I mostly resolved this with the upgrades in cooling, there are still the odd occasions of crashing, which appears to be linked with summer months.

The above graphs are interesting since they show a huge climb in disk temperatures, although I still suspect it’s the CPU that lead to the actual system crash occurring – the CPU temperature graphs show a climb up towards 60 degrees, which is the level where I’ve seen many system crashes in the past.

What’s particularly annoying is that all these crashes cause the RAID 6 to trigger a rebuild – I’m unsure as to why exactly this is, I suspect that maybe the CPU hangs in the middle of a disk operation that has written to some disks, but not all.

Having the RAID rebuild after reboot is particularly nasty since it places even more load and effort onto an already overheated system and subjects the array to increased failure risk due to the loss of redundancy. I’d personally consider this a kernel bug, if a disk operation failed, the array should still have a known good state and be able to recover from that – fail only the blocks that are borked.

Other than buying less iffy hardware and finding a cooler spot in the house, there’s not a lot else I can do for this box…. I’m pondering using CPU frequency scaling to help reduce the temperature, by dropping the clock speed of the CPU if it gets too hot, but that has it’s own set of risks and issues associated with it.

In past experiments with temperature scaling on this host, it hasn’t worked too well with the high virtualised workload causing it to swap frequently between high and low performance, leading to an increase in latency and general sluggishness on the host. There’s also a risk that clocking down the CPU may just result in the same work taking longer on the CPU potentially still generating a lot of heat.

I could attack the workload somewhat, the VMs on the host are named based on their role, eg (prod-, devel-, dr-) so there’s the option to make use of KVM to suspend all but key production VMs when the temperature gets too high. Further VM type tagging would help target this a bit more, for example my minecraft VM is a production host, but it’s less important than my file server VM and could be suspended on that basis.

Fundamentally the host staying online outweighs the importance of any of the workloads, on the simple basis that if the host is still online, it can restart services when needed. If the host is down, then all services are broken until human intervention can be provided.

Google Convoy

2 Replies

Spotted this when doing street view around Pyrmont Sydney – there’s at least two Google cars since one has captured this picture, looks like there might even be a third….

The watchers are watching the watchers.

Unsure why there would be a convoy of cars, the few occasions when I’ve seen Google imaging cars in the past, they’ve always been standalone. Having said that, Pyrmont contains Google’s Australian headquarters, so maybe there’s a few special projects on the go.

linux.conf.au 2013 plans

Leave a reply

It’s nearing that important time of year that the NZ-AU open source flock congregate that important and time honoured tradition of linux.conf.au. I’ve said plenty about this conference in the past, going to make an effort to write a lot more this year about the conference.

There’s a bit of concern this year that there might not be a team ready to take up the mantle for 2014, unfortunately linux.conf.au is a victim of it’s own success – as each year has grown bigger and better, it’s at the stage where a lot of volunteers consider it too daunting to take it on themselves. Hopefully a team has managed to put together a credible bid for 2014, it would be sad to lose this amazing conference.

As I’m now living in Sydney, I can actually get to this year’s conference via a business class coach service which is way cheaper than flying, and really just as fast once taking the hassles of getting to the airport, going through security and flying into account. Avoiding the security theatre is a good enough reason for me really – I travel a lot, but I actually really hate all the messing about.

If you’re attending the conference and departing from Sydney (or flying into Sydney from NZ to then transfer to Canberra), I’d also suggest this bus service – feel free to join me on my booked bus if you want a chat buddy:

Depart Sydney, Sunday 27th Jan at 11:45 on bus GX273.
Depart Canberra, Saturday 2nd Feb at 14:00 on bus GX284.

The bus has WiFi and power and extra leg room, so should be pretty good if you want to laptop the whole way in style – for about $35 each way.

Museum of Contemporary Art Australia

Leave a reply

On one of our trips down to Circular Quay, we recently discovered the Museum of Contemporary Art Australia.

It’s a free museum with a range of interesting and unique exhibits and worth visiting if you have a couple of hours time to spare – since it’s right on Circular Quay it’s really easy to get to, plus it’s also right by The Rocks which is worth a visit.

Circular Quay area

This is what happens when you erect scaffolding in a flight path…

Mail order plane anyone?

OCD alcoholism: drunken night of beer bottle smashing followed by having to re-assemble them all again.

This is a visual representation of my inbox.

This is a visual representation of emails pouring into my inbox. :-/

At what stage are artists just really taking the piss?

The colour! OMG!

The best work of art in the whole place :-P

Of course not all of Sydney is as excited about artwork….

Is this sign art itself?

The MCA was a really good visit, well worth checking out if you have time in Sydney – plus it’s free, which is a miracle for Sydney really. ;-)

Based on how much I enjoyed MCA, I’m even more keen to make a trip down to Tasmania in the near future to go visit MONA – I sadly missed it on my last trip as it hadn’t quite opened at that stage, but hearing lots of awesome stuff about it.

Sydney’s Powerhouse Museum

Leave a reply

The Powerhouse Museum in Sydney is a science/tech/design museum offering a range of exhibits including space, robotics, history, fashion and other geeky and design related topics.

I went there on a special event day so the usual $12 entrance fee had been halved (yay!) and spent a few hours having a good look around the museum.

No tech museum would be complete without a steam exhibit – the powerhouse actually has some of the engines in a powered state, although there wasn’t a whole lot going on when I was there.

Before data centers, these were the power houses of industry.

Before data centers, these were the power houses behind the world’s industry.

Old destination board from a railway station.

The man walking in front of a steam engine with a red flag to limit it’s speed seems about as hopeless as the RIAA/MPAA wanting to stop digital downloads…. you can’t restrict new technology for long.

There’s also a good exhibit of space technology, including an actual F-1 rocket engine, the most powerful liquid fuelled rocket ever developed and the machine responsible for powering the Saturn V which took humanity to the moon.

(from the left) F-1 rocket engine, a sounding rocket (research), several models of famous space craft and satellites and more.

F-1 Engine! These things are NOT small!

$\m/ squeeeee$

\m/ :-D

Rocket thruster used in command modules.

1/3 scale Soyuz pair coupled together.

Retro computer inside the space station module mockup.

Replica Mars Rover – The Soviets sure made some weird looking hardware.

There are a range of robotics exhibits, including some neat demonstrations of industrial robotic arms that are bit more uncommon to see.

All hail the robotic overlords!

Everyone loves hexapods!

Plus a bunch of other random bits:

Weird looking aircraft

Electrifying Touch

At times the selection of exhibits feels a bit disjointed, things certainly don’t flow quite as well as some of the other science and technology museums I’ve been to and some areas a bit worn and dated – having said that, they are in the process of renovations, so it might be fairer to re-evaluate it in a year or so.

That being said, it’s worth a visit, just for some of the awesomeness they have there – plus how often can you take a picture of you and your partner standing underneath an F1 rocket engine? :-)

Houston, set engines to snuggly!

Bondi Beach

Leave a reply

Seeing as I’m living in Sydney, I felt compelled to head out to visit Bondi Beach – it’s a pretty famous location, so figured it’s worth getting out there to check it out at least once.

All set up for a day at the beach, geek style!

Getting out to Bondi is a bit of a mission, since unfortunately the rail link doesn’t go the whole way – instead you catch the train to Bondi Junction, and then switch to bus in order to get to the beach.

You can buy a (pricey) ticket at any of the city train station machines that includes both the train and bus fare – costing about $8.80 return at an off-peak time.

The bus is the worst part of the trip, in future I’ll probably consider just walking it as it’s only 3.2km from the railway station – whilst the buses *do* have air con, any benefits are lost with the squeeze with numerous tourists squished into it, personally I’d rather be hot outside than hot squeezed into a metal tin with random strangers….

Bondi Beach. It has sand. Waves. Usual beach things really.

People are really the biggest issue with Bondi – it’s a nice beach in it’s own right, but the volume of tourists and shops around it kind of ruins the enjoyment somewhat, particularly when combined with the transport hassles. In summer the crowds can be massive, as much as 30 thousand+ – and it’s not a particularly large beach really.

The smaller beaches in walking distance to the south like Tarramara and Bronte look more appealing if you’re more of an introverted type, or take the ferry over to Manly for some of the nice beaches over there.

If you’re a tourist on holiday, you’re probably better off spending your time in Sydney to explore the city and waterfront and save the beach trips for when visiting Brisbane’s gold coast.

Having said that, if you’re in Sydney and don’t mind lots of people, it’s a nice beach and looks great for swimming and surfing, plus it’s well patrolled by surf life savers.

One of the perks of Bondi Beach is being able to meet pretty ladies. Although they won’t always appreciate you taking pictures of them. ;-)

Aside from the beach, there are a large number of eateries along the beach and in the surrounding streets – (like Funky Pies for some delicious vegetarian/vegan goodness), as well as a walk up over the rocky cliffs.

Walkway from the southern end of Bondi Beach.

Looking back at Bondi

It was worth the visit, but I don’t think Bondi will become one of my regular visited locations, the transport hassles make it a bit difficult to get there – it’s easier for me to get out to Manly on the ferry than it is to get to Bondi on the train/bus.

It would be a different story if living locally or having a private transport option- some of my colleagues at work get there by bike which seems to work quite well for them.

Das Keyboard Ultimate Silent

8 Replies

With the recent move to Sydney, I’ve had to leave my beloved IBM Model M keyboards back in New Zealand – sadly they’re a bit heavy and large to effectively pack into my luggage without sacrificing some much needed clothes.

Even if I was to bring them over here to Australia with me, the Model M’s are too loud for me to use in a shared office environment – my Model M was previously banned from my last employers office after they could hear it through two walls and down a phone at the other end….

Instead I’ve brought a Das Keyboard Ultimate Silent. I’ve been a fan of the Das Keyboard idea for a while – just like the IBM Model M they’re traditional clicky mechanical keyboards, but have modern features such as USB, lighter bodies and (love them or hate them) Windows keys that are useful for both Windows and MacOS users.

Das Keyboards come in both labelled (Professional) and unlabelled (Ultimate) revisions, and the option of either standard loud clicky keys or the “silent” model – considering I’m working in a shared office space, I elected to go for the silent edition.

Mmmmm sleek black sexiness.

I’ve been using the Ultimate Silent for about two months now, general impressions are:

It’s an excellent keyboard that’s well worth the $150 AUD price tag. I’ve had heaps of comments from co-workers on how great it feels to type on, command line power geeks can’t be wrong. ;-)
The keys still have the tactile feedback of a mechanical clicky keyboard. Whilst the responsive spring-back is a little more subdued than on the Model Ms, it still delivers a very nice feel.
The blank keys are AWESOME. People who try to use my computer are always really put off at first, however if you’re a touch typist it won’t take long to get used to it.
It’s not exactly silent -“quieter” is a more accurate term, I certainly have the loudest keyboard in the office, but it’s nowhere near as loud as an actual Model M is. The sound is more a subdued tap as the keys hit the bottom of the keyboard when typing, rather than the audible click of a traditional clicky keyboard.
My colleagues are a pretty good bunch of people since they haven’t murdered me for loudly typing and stretching the “silent” label to the limits. ;-)
I have occasional issues with finding a particular symbol key (things like ^ or &), but I can touch type almost any of the 104 keys on it without an issue.

Personally I’ll keep using the IBM Model Ms as my personal keyboard, they’re great quality keyboards and I love the fact I can keep using something designed in 1980s (mine were manufactured in 1994) on my computer for possible the rest of my life, but I’d be very content with using a Das Keyboard personally as well as professionally if I didn’t already have the Model M.

It always amazes me how often geeks will spend huge money on their computers and then neglect the keyboard or buy something that features lots of flashy lights and special keys, but ignore the most important requirement of good typing ergonomics and performance.

I haven’t tried the clicky version of the Das Keyboard myself so I can’t really compare it – I expect you’d find that the clicky version has a even nicer feedback (like the Model M) but the silent is the better investment if you work near anyone else.

I brought mine from AusPCMarket who have local stock and it arrived in a couple of working days without an issue, otherwise buy direct from Das Keyboards.

Light Rail of Sydney

Leave a reply

Ending up living in Sydney rather than Melbourne has brought one unfortunate side effect – a lack of awesome trams spread around the city. Thankfully it turns out one doesn’t have to go all the way to Melbourne just to get a light rail fix.

Sydney has two forms of light rails – the monorail which is to be decommissioned in 2013 and the more recent tram link that runs from Central station, through Pyrmont and out into Glebe.

mmmmmm light rail here, light rail there, light rail everywhere!

I haven’t been on the Sydney trams yet, based on the location of my apartment, central station and my office in Pyrmont, the tram tends to take me away from the direction I want to actually travel in – although that being said it may be useful if I ever want to go quickly from work to Central station and then transit onto the commuter rail network.

I’m hoping that Sydney considers adopting more light rail – whilst the commuter rail network is very effective at linking main centers, there’s a lot of Sydney that’s only linked by buses, which aren’t particularly fast and seem to be at their limits in regards to capacity.

Extending the tram lines out to places like Surry Hills and Bondi would be a huge plus, the linking of commuter rail and light rail has been very successful for Melbourne and there’s no reason why it couldn’t be replicated here as well.

Sydney Harbour Bridge Walk

Leave a reply

One of my biggest annoyances about living in Auckland is that the only way to cross the Auckland Harbour Bridge was via car or bus – so whilst my apartment would have been within feasible walking distance to the city, it was utter impossible to walk it.

Sydney has at least got this right and their famous harbour bridge is crossed by car, train, cycle and foot, which offers an excellent chance to see the bridge up close and get some amazing views over the city and harbour.

Views don’t get much more iconic than this. Amazingly hot dude and some building thingy in the background.

It’s an easy walk to get to if you’re a tourist – you just need to take the Northern Rail line Central, Wynward or Town Hall train stations and depart at the Milsons Point railway station on the North Shore, which is right where the bridge crossing starts for both pedestrians and cyclists.

Both the train and driving across the bridge offer views, but it’s nowhere near as excellent as checking it out on foot – and of course driving across will incur you some lovely tolls.

It’s hard to get a sense of scale – for reference there are two rail lines running through the right side tower, each capable of fitting a double decker train though it – or in the case of the left side, a road lane and a pedestrian path.

If I find my current 1 bedroom CBD apartment rent painful, I’d hate to think how much these places would be costing…

Sydney Opera House!

There’s one metric fuckload of steel in this.

It’s also really high!

Circular Quay, the international passenger terminal for cruise ships and the rocks.

I love these apartments at the city end of the bridge for being a neat varying stack of towers and roof gardens, but at the same time hate it’s blocky concreteness.

Interspersed concrete and greenary awaits at the end of the crossing – descend the steps to end up in “The Rocks”, the heritage part of Sydney

The crossing is well worth doing, it’s a real shame that Auckland hasn’t added a walk and cycleway of their own – not only is it a great recreational addition, but it also makes the lifestyle choice of walking or cycling to work from the North Shore feasible.

Australian National Maritime Museum

Leave a reply

As part of my daily walk to work in Pyrmont, I pass the impressive Australian National Maritime Museum – considering they have both a Destroyer and a submarine parked in the harbor outside, I could hardly resist a visit for long. :-)

The museum has the usual range of maritime artifacts, exhibits about life at see, immigration, ship guns and steam engines, as well as a few neat items such as a maritime helicopter and small yachts and motor boats suspended inside.

Weird helicopter – kind of looks like some weird bug when viewed from underneath.

Of course for me, the real attraction was what was parked outside…

Above or below, either way we have your warfare needs sorted. :-)

Nowhere near as large as the cruise ships that now dock at Sydney, but still impressive when looking at them up close.

The best part is that yes, you can board both the destroyer and the submarine and explore them for yourselves! Unfortunately the day that I was there the submarine was closed, but I’m sure I’ll be back at some point and able to go inside it to explore more.

I had a good wander around the HMAS Vampire, a 1956 darling class destroyer built in Australia, and the last generation of warships before weapon systems moved from traditional cannons/big guns to missile based systems.

If you filmed a porno here, the title would be “one shade of gray”.

Military budgets don’t seem to extend towards having normal height ceilings.

Bit obsolete in the age of the drones and smart bombs, but nothing is quite as impressive as some big cannons.

I love the sci-fi look to these AA cannons.

Pyrmont Bridge with the wooden boat festival in the foreground.

Fuck your NBN and your UFB, fibre to the warship is where it’s at!

Server rooms have come a long way since. :-)

More computer users need to keep this in mind.

Early generation Nagios outage notification device?

Boom, boom, boom, boom, boom.

I can’t resist some good thick chains…. ;-)

Sir, we have a firing solution for the Apple store from the front turret!

I love warning signs like these “this shit be bad bro, try not to hang around here”

The rounds fired by these machine guns must be massive… probably tear a small boat apart.

Standing on the deck of the weapon of the 20th century looking out at the new weapons of the 21st century…. mega corporations.

If your target is too quick to move, there’s also the smaller patrol boats you can use to chase them down. They can’t have been too comfortable to spend time on, even sitting at the dock on a calm day the boat had a noticeable rocking to it.

The little offspring of the destroyer?

Whilst I couldn’t get inside on this trip, I got some great pictures of the HMAS Oberon diseal-electric submarine parked up alongside the destroyer. Whilst it’s not hard to find warships at maritime museums around the world, near-operational submarines are a little less common.