Tag Archives: geek

Anything IT related (which is most things I say) :-)

linux.conf.au: day 4

Another successful day of Linux geeking has passed, this week is going surprisingly quickly…

Some of the days highlights:

  • James Bottomley spoke on the current state of Linux UEFI support and demonstrated the tools and processes to install and manage keys and hashes for the installed software. Would have been interesting to have Matthew Garrett at LCA this year to present his somewhat different solution in comparison.
  • Avi Miller from Oracle did an interesting presentation on a new Linux feature called “Transcendent Memory“, which is a solution to the memory ballooning problems for virtualised environments. Essentially it works by giving the kernel the option to request more memory from another host, which could be the VM host, or even another host entirely connected via 10GigE or Infiniband, and having the kernel request and release memory when required. To make it even more exciting, memory doesn’t have to be just RAM, SSDs are also usable, meaning you could add a couple memory hosts to your Xen (and soon KVM) environments and stack them with RAM and SSD to then be provided to all your other guests as a memory ballooning space. It’s a very cool concept and one I intended to review further in future.
  • To wrap up the day, Michael Schwern presented on the 2038 bug – the problem where 32-bit computers are unable to keep time any further and reset to 1901, due to the limits of a 32-bit time buffer (see wikipedia). Time is something that always appears very simple, yet is extremely complex to do right once you consider timezones and other weirdness like leap years/seconds.
The end of time is here! Always trust announcements by a guy wearing a cardboard and robes.

The end of time is here! Always trust announcements by a guy wearing a cardboard and robes.

The conference presentations finished up with a surprise talk from Simon Hackett and Robert Llewellyn from Red Dwarf,  which was somewhat entertaining, but not highly relevant for me – personally I’d rather have heard more from Simon Hackett on the history and future expectations for the ISP industry in Australia than having them debate their electric cars.

Thursday was the evening of the Penguin Dinner, the (usually) formal dinner held at each LCA, this year rather than the usual sit down 3-course dinner, the conference decided to do a BBQ-style event up at the Observatory on Mount Stromlo.

The Penguin Dinner is always a little pricey at $80, but for a night out, good food, drinks and spending time with friends, it’s usually a fun and enjoyable event. Sadly this year had a few issues that kind of spoilt it, at least for me personally, with some major failings on the food and transport which lead to me spending only 2 hours up the mountain and feeling quite hungry.

At the same time, LCA is a volunteer organised conference and I must thank them for-making the effort, even if it was quite a failure this year – I don’t necessarily know all the behind the scenes factors, although the conflicting/poor communications really didn’t put me in the best mood that night.

Next year there is a professional events coordinator being hired to help with the event, so hopefully this adds value in their experience handling logistics and catering to avoid a repeat of the issue.

On the plus side, for the limited time I spent up the mountain, I got some neat photographs (I *really* need to borrow Lisa’s DSLR rather than using my cellphone for this stuff) and spent some good time discussing life with friends lying on the grass looking at the stars after the sun went down.

Part of the old burnt-out observatory

Part of the old burnt-out observatory

Sun setting along the ridge.

Sun setting along the ridge.

What is it with geeks and blue lights? ;-)

What is it with geeks and blue LEDs? ;-)

The other perk from the penguin dinner was the AWESOME shirts they gave everyone in the conference as a surprise. Lisa took this photo when I got back to Sydney since she loves it [1] so much.

Paaaartay!

Paaaartay!

[1] She hates it.

linux.conf.au: day 3

Having reached mid-week, my morning wakeup is getting increasingly difficult from late nights, thankfully there were large amounts of deep fried potato and coffee readily available.

Breakfast of champions - just add cheese and it would be a meal.

Breakfast of champions – just add cheese and it would be a meal.

Coffee Coffee Coffee Coffee Coffee Coffee Coffee Coffee Coffee

Coffee Coffee Coffee Coffee Coffee Coffee Coffee Coffee Coffee

The day had some interesting talks, most of the value I got was out of the web development space:

  • Andy Fitzsimon did an interesting presentation on design and how to approach designing applications or websites and the terminologies that developers use.
  • Sarah Sharp presented on “vampire mice”  – essentially a lot of USB devices don’t correctly obey the USB power suspend options, the result is that by enabling USB suspend for all your devices and disconnecting those that don’t obey, considerable power can be saved – one audience member found he could save 4W by sleeping all his USB devices. I also discovered that newer versions of Powertop now provide the ability to select particular USB devices for power-save mode.
  • There was a really good talk by Joel Stanley, probably one of the most interesting talks that day, on how they designed and built some hardware for doing digital radio transmissions using a radio circuit connected into an Android phone and the challenges encountered of doing hardware integration with Android.
  • We had an update on IPv6 adoption by Geoff Huston – sadly as expected, we’re dangerously low on IPv4 space, yet IPv6 adoption isn’t taking place particularly quickly either, with Internode still being the only major AU ISP with dual stacked addressing for consumers. On a side note, really awesome to see a former keynote presenter come back as a regular presenter and make a talk, having community engagement really adds to my respect for them.
  • My friend Adam Harvey did another awesome web development talk, this time presenting on some of the new CSS3 techniques including animation and transitions with some demonstrations on how these can work.
Open source radio reciever with Android phone coupled.

Open source radio receiver with Android phone coupled.

users: delighted, presenter: smug :-P

users: delighted, presenter: smug :-P

Spot the possum!

Spot the possum!

With all the talks this week, I’m feeling particularly motivated to do some more development this week, starting with writing some new proper landing pages for some of my projects.

Playing with new HTML5/CSS3 effects having been inspired to upskill my web development skills.

Playing with new HTML5/CSS3 effects having been inspired to upskill my web development skills.

linux.conf.au: day 2

The second day of linux.conf.au has been and gone, was another day of interesting miniconf talks and many geeky discussions with old and new friends.

Jethro: Booted

Jethro: Booted, with the power of coffee!

The keynote was a really good talk by Radia Perlman about how engineers approach developing network protocols and an interesting talk of the history of STP and the designed replacement, TRILL. Great to see a really technical female keynote speaker at LCA this year, particularly one as passionate about her topic as Radia.

The conference WiFi is still pretty unhappy this year, I’ve been suffering pretty bad latency and packet loss (30-50%) most of the past few days – if I’ve been able to find an AP – seems they’re only located around the lecture rooms. Yesterday afternoon it seems to have started improving however, so it may be that the networking team have beaten the university APs into submission.

No internet makes sad Jethro sad. :'(

No internet makes sad Jethro sad. :'(

Of course, some of the projectors decided not to play nicely, which seems pretty much business as usual when it comes to projectors and functioning…. it appears that the projector in question would complain about the higher refresh rates provided by DVI and HDMI connected devices, but functioned correctly with VGA.

Someone did an interesting talk a couple of LCA’s ago on the issue, apparently many projectors lie about what their true capabilities are and request resolutions and refresh rates from the computer that are higher than what they can actually support, which really messes with any modern operating system’s auto-detection.

Lending my VGA enabled Thinkpad to @lgnome whist a @chrisjrn observes.

Lending my VGA enabled Thinkpad to @lgnome whist a @chrisjrn observes.

A startled @colmiga approaches!

A startled @colmiga approaches!

Geeks listening intently

Geeks listening intently to concurrent programming.

@lgnome pushing some crazy new drugs to all the kiddies

@lgnome pushing some crazy new drugs to all the kiddies

A few of my friends were delivering talks today, so I spent my time between the Browser miniconf and Open Programming miniconf, picked up some interesting new technologies and techniques to look at:

  • Adam Harvey’s PHP talks were great as usual, always good to get an update on the latest developments in the PHP world.
  • Francois Marier from Mozilla NZ presented on Content Security Policy, a technique I wasn’t aware of until now. Essentially it allows you to set a header defining which sites should be trusted as sources of CSS, Javascript and image content, allowing a well developed site to be locked down to prevent many forms of XSS (cross site scripting).
  • Francios also spoke briefly about HTTP Strict Transport Security, a header which can be used by SSL websites to fix the long standing problem of users being intercepted by a bad proxy and served up a hacked HTTP-only version of the website. Essentially this header tells your browser that your site should only ever be accessed by HTTPS – anything that then directs your browser to HTTP will result in a security block, protecting the user, since your browser has been told that the site should only ever be SSL from it’s previous interaction. It’s not perfect, but it’s a great step forwards, as long as the first connection is made on a trusted non-intercepted link, it makes man-in-the-middle attacks impossible.
  • Daniel Nadasi from Google presented on AngularJS, a modern Javascript framework suitable for building complex applications with features designed to reduce the complexity of developing the required Javascript.

After that, dinner at one of the (many!) Asian restaurants in the area, followed by some delicious beer at the Wig and Pen.

Either I've already had too many beers, or there's a giant stone parcel in my way.

Either I’ve already had too many beers, or there’s a giant stone parcel in my way.

Onwards to delicious geekiness!

Onwards to delicious geekiness!

Delicious hand pulled pale ale.

Delicious hand pulled pale ale.

The beetroot beer is an interesting idea. But some ideas should just not be attempted. :-/

The beetroot beer is an interesting idea. But some ideas should just not be attempted. :-/

Native Australian night life!

Native Australian night life! This little fellow was very up close and friendly.

Linux.conf.au native wildlife. ;-)

Linux.conf.au native wildlife. ;-)

Another great day, looking forwards to Wednesday and the rest of the week. :-)

linux.conf.au: day 1

First proper day of linux.conf.au today, starting with breakfast and the quest of several hundred geeks to find and consume coffee.

Some of us went a bit overboard to get their exact daily coffee fix....

Some of us went a bit overboard to get their exact daily coffee fix….

After acquiring coffee, we started the day with a keynote by the well known Bdale Garbee, talking about a number of (somewhat controversial) thoughts and reflections on Linux and the open source ecosystem in regards to the uptake by commercial companies.

Keynote venue.

Keynote venue.

Bdale raised some really good points, particularly how GNU/Linux isn’t a sellable idea to OEM vendors on cost – many vendors pay nothing for Microsoft licensing, or even make a profit due to the amount of preloaded crapware they ship with the computers. Vendors are unlikely to ship GNU/Linux unless there is sufficient consumer demand or feature set that makes it so good

My take on the talk was that Bdale was advocating that we aren’t going to win the desktop with a mass popularity – instead of trying to build a desktop for the average joe, we should build desktops that meet our own needs as power uses

It’s an interesting approach – some of the more recent endeavours with desktop developers has lead to environments that newer users like, but power users hate (eg GNOME 3), as a power user, I share this view, I’d rather we develop a really good power user OS, rather than an OS designed for the simplest user. Having said that, the nice thing about open source is that developers can target different audiences and share each other’s work.

Bdale goes on to state that the year of the Linux desktop isn’t relevant, it’s something we’re probably never going to win – but we have won the year of Linux on the mobile, which is going to replace conventional workstations more and more for the average use and become the dominant device used.

It’s something I personally believe as well, I already have some friends who *only* own a phone or tablet, instead of a desktop or tablet, and use it for all their communications. In this space, Android/Linux is selling extremely well.

And although it’s not a conventional GNU/Linux space we know and love and it still has it’s share of problems, a future where Android/Linux is the dominate device OS is much more promising than the current Windows/MacOS duopoly.

The rest of the day had a mix of miniconf talks – there wasn’t anything particularly special for me, but there were some good highlights during the day:

  • Sherri Cabral did a great talk on what it means to be a senior sysadmin, stating that a proper senior sysadmin knows how to solve problems by experience ( not guess work), works to continuously automate themselves out of a job with better tools and works to impart knowledge onto others.
  • Andrew Bartlett did a brief update on Samba 4 (the Linux CIFS/SMB file system implementation) – it’s production ready now and includes proper active directory support. The trade off, is that in order to implement AD, you can’t use an external LDAP directory or Kerberos server when using Samba 4 in an AD server mode.
  • Nick Clifford did an entertaining presentation on the experiences and suffering from working with SNMP, turns out that both vendor and open source SNMP implementations are generally quite poor quality.
  • Several interesting debates over the issues with our current monitoring systems (Nagios, Icinga, Munin, etc) and how we can fix them and scale better – no clear “this is the solution” responses, but some good food for thought.

Overall it was a good first day, followed up by some casual drinks and chats with friends – thankfully we even managed to find an open liquor store in Canberra on a public holiday.

Poor @lgnome expresses his pain at yet another closed liquor store before we located an open location.

Poor @lgnome expresses his pain at yet another closed liquor store.

 

 

linux.conf.au: day 0

It’s time for the most important week of the year – linux.conf.au – which is being held in Canberra this year. I’m actually going to try and blog each day this year, unlike last year which still has all my photos in the “too be be blogged folder”. :-)

Ended up taking the bus down from Sydney to Canberra – at only around $60 and a 3 hour trip, it made more sense to take the bus down, rather than go through the hassle of getting to and from the airports and all the security hassles of flying.

Ended up having several other linux.conf.au friends on the bus, which makes for an interesting trip – and having a bus with WiFi and power was certainly handy.

I am geek, hear me roar!

I am geek, hear me roar!

Horrifying wail of the Aucklander!

Horrifying wail of the Aucklander!

The road trip down to Canberra wasn’t particularly scenic, most of the route is just dry Australian bush and motorways, generally it seems between city road trips in AU tend not to be wildly scenic unlike most of the ones I take in NZ.

Canberra itself is interesting, my initial thoughts on entering the city was that it’s kind of a cross between Rotorua and post-quake Christchurch – most of the city is low rise- 5-10 story buildings and low density sprawl, and extremely quiet with both the university and parliament on leave. In fact many have already commented it would be a great place to film a zombie movie simply due to it’s eerily deserted nature.

Considering it’s  a designed city, I do wonder why they choose such a sprawled design, IMHO it would have been way better to have a very small high density tower CBD which would be easily walk-able and massive park lands around them. Canberra also made the mistake of not putting in light rail, instead relying on buses and cars as primary transport.

Neat fountain in town

Neat fountain in town

The Aussies can never make fun of us Kiwis and sheep again... at least we don't have THIS in our capital city O_o

The Aussies can never make fun of us Kiwis and sheep again… at least we don’t have THIS in our capital city O_o

Impressively large transmission tower for such a small city.

Impressively large transmission tower for such a small city.

Once nice side of Canberra, is that with the sprawl, there tends to be a lot of greenery (or what passes for greenery in the aussie heat!) around the town and campus, including a bit of wildlife – so far I’ve seen rabbits, cockatoos, and lizards, which makes a nice change from Sydney’s wildlife viewing of giant rats running over concrete pavements.

Sqwark!

Sqwark!

The evening was spent tracking down the best pub options near by, and we were fortunate enough to discover the Wig and Pen, a local British-style brewery/pub, with about 10 of their own beers on hand pulled taps. I’m told that when the conference was here in Canberra in 2005, the attendees drank the pub dry – twice. Hopefully they have more beer on stock this year.

First beer casualty from the conference - laptop being stood vertically to drain, whilst charging a cellphone.

First beer casualty from the conference – laptop being stood vertically to drain, whilst charging a cellphone.

Normally every year the conference provides a swag bag, typically the bag is pretty good and there’s usually a few good bits in there, as well as spammy items like brochures, branded cheap gadgets (USB speakers, reading lights, etc).

This year they’ve cut down hugely on the swag volume, my bag simply had some bathroom supplies (yes, that means there’s no excuse for the geeks to wash this week), a water bottle, some sunblock and the conference t-shirt. I’m a huge fan of this reduction in waste and hope that other conferences continue on with this theme.

Arrrrrr there be some swag me mateys!

Arrrrrr there be some swag me mateys!

The conference accommodation isn’t the best this year – it’s clean and functional, but I’m really not a huge fan of the older shared dorm styles with communal bathroom facilities, particularly the showers with their coffin-style claustrophobic feel.

The plus side of course, is that the accommodation is always cheap and your evenings are filled with awesome conversations and chats with other geeks.

Looking forwards for the actuals talks, going to be lots of interesting cloud and mobile talks this year, as well as the usual kernel, programming and sysadmin streams. :-)

Ebook Debate

Lisa and I recently decided to purchase a Kindle – having first moved to Auckland and then Sydney, we were unable to take many books with us, something we both regretted – as the plan isn’t to settle in Sydney long term, it’s not worth shipping them over to only then have to ship them elsewhere again.

In the last 5+ years almost all the reading I’ve done has been on my laptop, but this has some unfortunate side effects:

  • It’s never as easy to lie on the beach or at the park with a laptop as it is for with a book, as a result, we do far less reading or lying outside than we would like.
  • Years of working in IT has turned me into a screen skim reader – I was always a fast reader, but in order to keep up with analysing logs and complex information, I’m become a very impatient skim reader. If I try to read any large textual content such as books on my computer, I’m liable to skip through content.
  • Computers make me context-shift – I’ll be reading a book, and then jump into email, then IM and lose all depth and concentration on what I was reading. Sometimes I just want a book to be a book.
  • LCD screens are very difficult to read outside in bright sunlight – this is particularly an issue, since we want to try and get outside and out of our apartment more to escape the crippling heat of Sydney.

We pondered getting a small tablet, such as the iPad Mini or Google Nexus 7, but these share a lot of the same problems as using our laptops, such as the LCD screen being poor for reading and the multi tasking nature of the device leading to context shifting and loss of focus.

In the end, we decided to buy one of the current generation of basic Kindle models, and if we like it, upgrade to having two of the newer generation Paperwhite Kindles once they launch in the AU market.

We had a bit of a mission getting one – the Paperwhite is currently only available in the US market and Amazon won’t ship to AU – but when we tried to order the regular model via Amazon, it kept refusing to ship to AU for some unstated reason:

Hi Amazon, I have a bug report for you...

Hi Amazon, I have a bug report for you…

I ended up giving up and shelling out an extra $20 to buy from Dick Smith’s who had the model we wanted in stock. It’s been a long time since I shopped retail, it’s certainly an interesting experience….

With the expensive prices, display models that were two generations out of date, display Kindles that hadn’t even been setup leaving them unusable for demonstrations and pushy sales pitches trying to offload warranties, cases and screen protectors, it’s no wonder that online shopping is decimating the AU retail market.

Aside from the purchasing hassles, the Kindle is shaping up to be a great device – at least based on what I’ve experienced so far, which is only the basics, as Lisa keeps stealing it away from me….

The e-ink display is excellent in daylight, although it suffers a bit in our apartment, which has somewhat dull lighting levels – the newer Paperwhite model with the front-lit display would help resolve this issue for us, so I expect we will look to upgrade once it’s released.

In bright sunlight or even just outdoors in general, the display is clear and easy to read, something that is a huge difference compared to my conventional LCD laptop and I expect we’ll make more park and beach trips for reading & relaxing now that we have it.

Kindle in the park. [lovingly stolen from Lisa's instagram feed since I forgot to take a picture myself]

Kindle in the park. [lovingly stolen from Lisa’s instagram feed since I forgot to take a picture myself]

I had some reservations about the Kindle, the DRM around their book store and the amount of control that they have over it is of some concern (see FSF for details), but we’ve decided to use it primarily as a side loaded device, where we download and store all our ebooks on our laptops (using a tool like Calibre) and then side load them onto the Kindle, which is easy to do via USB transfer or emailing them to the Kindle itself.

When buying books, I’ll stick to getting them in a downloadable DRM-free format, so that I can copy them to any device in future – this also solves the backup issue for the Kindle, since anything on my laptop or servers is backed up reliably.

Of course I have to try resisting hacking the Kindle and bricking it in some way, or I’ll have an angry fiancée on the warpath, so for now I’ll keep trying to treat it as a book only.  ;-)

Apache Redirect & Varnish

Apache’s mod_rewrite is an awesome module providing all sorts of impressive capabilities, however it’s sometimes overkill when all you want is to redirect an entire site or path to a different one. In these situations, the mod_alias’s Rewrite function is often the ideal solution, with it’s simple syntax.

Redirect permanent /  http://newhost.example.com/
Redirect permanent /test.html http://newhost.example.com/test.php

Unfortunately I’ve found a situation where certain syntax can cause failures in some environments – consider the following:

Redirect permanent /  http://newhost.example.com

This example will work happily when a user is accessing Apache directly, however it will fail horribly when accessing via a Varnish cache, eg:

$ wget oldhost.example.com
 --2012-11-27 11:26:54--  http://oldhost.example.com/
 Resolving oldhost.example.com (oldhost.example.com)... 172.16.1.1
 Connecting to oldhost.example.com (oldhost.example.com)|172.16.1.1|:80... connected.
 HTTP request sent, awaiting response... 301 Moved Permanently
 Location: unspecified
 ERROR: Redirection (301) without location.

This was the source of a lot of headaches since the site still *worked* correctly when connecting directly to it, but kept failing whenever accessed via the varnish caches.

The cause is simply a missing trailing / in the redirect – configuration should actually look like:

Redirect permanent /  http://newhost.example.com/

It’s a nasty trap for the unwary sysadmin, since it works perfectly when accessing the server directly – I’m not entirely sure whether it’s an issue with Apache allowing this syntax, or whether it’s an issue with Varnish not understanding how to proceed – maybe a bit of both.

RFC 2616 isn’t clear, it states that if the abs_path is empty, it’s the same as abs_path being equal to /, and that when requesting a URL with an undefined abs_path, it should be treated as /… but it doesn’t clarify whether an application serving a URL should or shouldn’t append the trailing /.

I guess ideally Apache should be patched to serve up URLs with a trailing / and Varnish should accept redirects with or without it to protect both applications from blame.

Android ICS & the MTP pain

With Android 3.0, Google switched the way of accessing files on Android devices from USB mass storage device behaviour, to Media Transfer Protocol, a standard initially created by Microsoft, but later developed into part of the USB standard for media devices.

For most Android users, new ICS phones are where they will first encounter this new approach to sharing file content. Phones previously upgraded from Android 2.x may still provide the conventional mass storage device behaviour, but any new ICS devices force the option of either MTP or the camera-focused PTP.

MTP is one of those technologies designed with the best of intentions – the fundamental design around it is that it abstracts the underlying filesystem, enabling the phone/device to store the files in whatever way suits it best and then exposing them to the computer.

Unlike the traditional mass storage mount, the files can be available to both the phone and the computer at the same time, whereas previously Android applications would lose access to the “sdcard” storage space when mounted on the computer.

It’s a great idea, except that it’s nowhere near as compatible with computers as a generic mass storage device is. Whilst Windows has native support for MTP, thanks to MS being the original creator of the standard, MacOS has no native support and requires a download, and Linux systems are all kinds of fail when it comes to MTP.

Considering I use a Linux laptop, this situation is very unhelpful – reading online suggests various options (the arch wiki was quite helpful):

  • Use of GNOME with GVFS or KDE’s KIO to connect the storage using the GUI’s device handling functionality. Unfortunately with my tests performed using Nautilus (GNOME’s file manager) on Debian Testing (recent app versions), I wasn’t able to connect to the device, instead experiencing a lot of timeout errors.
  • Linux has a FUSE driver that is able to read MTP, with corresponding usertools called mtpfs – however in my tests, this driver has some serious usability, stability and trust issues, thanks to it repeatedly failing and randomly unmounting, as well as hiding many files that are known to exist.
  • An alternative driver called “go-mtpfs” (written in Go) which provides user space connectivity to MTP Android devices at reasonable speeds and a reasonable UI for mounting the drives.
  • Failing these, a network transfer over WiFi could be an option, such as Samba or SSH. Bluetooth is also a possibility, but the performance is quite poor for numerous file transfers.

I’ve ended up using go-mtpfs which appears to work quite nicely, at least with my ICS Google Nexus phone, however it’s annoying that it’s not in the Debian repositories yet.

Whilst I understand the design decisions and issues solved by Google moving to using MTP rather than USB Mass Storage, the sudden deprecation of a feature that previously existed is a real pain for a user like myself who is wondering why their new phone won’t connect to their computer…

Personally I think Google should have included a legacy USB Mass Storage option for at least another release (ie till Android 5) to have provided time for desktop OS developers to improve the handling of this protocol.

It would also have been useful if they had included a “Install Android Driver” option to expose a small USB mass storage drive (eg 20MB) that includes drivers/apps for common platforms such as MacOS and Linux, so that users could connect and load on drivers without needing

For now, go-mtpfs seems to work OK, meanwhile one of the GVFS/GNOME developers is re-working the MTP handling to properly support Android devices, so hopefully we won’t have the headaches around this protocol for too much longer…

Great server crash of 2012

In a twist of irony, shortly after boarding my flight in Sydney for my trip back to Wellington to escape the heat of the AU summer, my home NZ server crashed due to the massive 30 degree heatwave experienced in Wellington on Christmas day. :-/

I have two NZ servers, my public facing colocation host, and my “home” server which now lives at my parent’s house following my move. The colocation box is nice and comfy in it’s aircon controlled climate, but the home server fluctuates quite significantly thanks to the Wellington climate and it’s geolocation of being in a house rather than a more temperature consistent apartment/office.

After bringing the host back online, Munin showed some pretty scary looking graphs:

localhost flew too close to the sun and plummeted to it's doom

localhost flew too close to the sun and plummeted to his doom

I’ve had problems with the stability of this system in the past. Whilst I mostly resolved this with the upgrades in cooling, there are still the odd occasions of crashing, which appears to be linked with summer months.

The above graphs are interesting since they show a huge climb in disk temperatures, although I still suspect it’s the CPU that lead to the actual system crash occurring – the CPU temperature graphs show a climb up towards 60 degrees, which is the level where I’ve seen many system crashes in the past.

What’s particularly annoying is that all these crashes cause the RAID 6 to trigger a rebuild – I’m unsure as to why exactly this is, I suspect that maybe the CPU hangs in the middle of a disk operation that has written to some disks, but not all.

Having the RAID rebuild after reboot is particularly nasty since it places even more load and effort onto an already overheated system and subjects the array to increased failure risk due to the loss of redundancy. I’d personally consider this a kernel bug, if a disk operation failed, the array should still have a known good state and be able to recover from that – fail only the blocks that are borked.

Other than buying less iffy hardware and finding a cooler spot in the house, there’s not a lot else I can do for this box…. I’m pondering using CPU frequency scaling to help reduce the temperature, by dropping the clock speed of the CPU if it gets too hot, but that has it’s own set of risks and issues associated with it.

In past experiments with temperature scaling on this host, it hasn’t worked too well with the high virtualised workload causing it to swap frequently between high and low performance, leading to an increase in latency and general sluggishness on the host. There’s also a risk that clocking down the CPU may just result in the same work taking longer on the CPU potentially still generating a lot of heat.

I could attack the workload somewhat, the VMs on the host are named based on their role, eg (prod-, devel-, dr-) so there’s the option to make use of KVM to suspend all but key production VMs when the temperature gets too high. Further VM type tagging would help target this a bit more, for example my minecraft VM is a production host, but it’s less important than my file server VM and could be suspended on that basis.

Fundamentally the host  staying online outweighs the importance of any of the workloads, on the simple basis that if the host is still online, it can restart services when needed. If the host is down, then all services are broken until human intervention can be provided.

Google Convoy

Spotted this when doing street view around Pyrmont Sydney – there’s at least two Google cars since one has captured this picture, looks like there might even be a third….

The watchers are watching the watched.

The watchers are watching the watchers.

Unsure why there would be a convoy of cars, the few occasions when I’ve seen Google imaging cars in the past, they’ve always been standalone. Having said that, Pyrmont contains Google’s Australian headquarters, so maybe there’s a few special projects on the go.