Since building my new KVM server in January, I’ve been experiencing random occasional system crashes – sometimes months apart, other times a couple in a week.

I’ve been trying to trace the cause, but this fault is hard to diagnose – there’s never been anything output to display, nor anything in system or even BIOS logs.

• Unlikely to be a kernel panic, no output to console, nor syslog, plus running CentOS 6 kernels which are usually pretty damn stable being rebuilt RHEL kernels…
• Unlikely to be some weird disk fault, I replaced all the disks recently but still experiencing the same issue – and when the fault occurs, all the RAID arrays get upset, even independent arrays on different controllers.
• Possibly a motherboard firmware issue, however upgraded to latest BIOS version and unable to find any similar problems online.
• Possibly CPU/Memory/Motherboard hardware faults.

However with the recent addition of a properly configured Munin server to my network, I’ve started graphing all the temperature sensors from my server – what I found, is that the AMD Phenom II 810 CPU is running hot – very hot infact, at around 60-70c, and the crashes were occuring once the CPU peaked at 70c.

I had initially discounted thermal problems, since the case has great cooling and I’ve never historically had calling problems with the stock cooler on AMD CPUs, especially since the CPU is not being overclocked.

However unlike many other systems I’ve built, this particular host is always heavily loaded – I’m running about 20-30 KVM virtual machines on it and there’s always a whole bunch of active processes, plus disk encryption CPU overhead.

And looking at the stock cooler, it’s not surprising that it’s been overheating – it’s basically a block of plain aluminium  – there aren’t even any heat pipes, unlike the stock cooler that ships with the black edition model.

My stock AMD heatsink - dusty, boring, poor thermal conductivity and no sexy heatpipes.

So I’ve replaced the heatsink with a nice new Zalman CNPS9700LED copper cooler – it’s a big beast, 790g and certainly wouldn’t fit in a lot of cases – but once installed, you can feel how the large fan blows air out over all the copper fins – there’s a really good airflow with the design to ensure heat gets radiated off quickly.

Pretty new cooler, all installed - without a case window, this pretty LED lighting is lost on me though.

Here’s the pretty graphs showing the difference that this cooler made to my server – please excuse some of the gaps, Munin has been having a bit of fun with virtualised workloads and timeouts….

It’s well worth the ~ $100 NZD based on the thermal difference it made – my CPU has gone from 60-70c down to 30-40c and so far, the server is running solid without fault.

From the graph, the CPU (green) is running a good 20+ degrees cooler than it previously did, but in addition, the motherboard chipset (blue) is also running cooler – most likely caused by the CPU cooler fan pulling in air right over the heatsink on the motherboard, assisting it with cooling.

(I’m running the cooler at low speed with the stock Zalman thermal paste too – if you turned the fan speed up higher or used fancier thermal pastes, lower temperatures or more thermal conductivity might be possible.)

In terms of the hardware supplied by Zalman, it’s a pretty good package – the cooler comes with some decent thermal paste, application brush, cooler, and connectors for various model CPUs.

My only complaint was that the design with the AM3 socket, means that the cooler outtake doesn’t line up with the rear case fan – however this is a lesser problem, since the hot air radiates out all over the cooler and is quickly removed by the fan anyway.

In terms of whether it’s fixed my issue, it remains to be seen – the crashes were not always consistent and I won’t call it as “fixed” until I get 4 months solid run time without occurrences, but I’m optimistic.

Lately I’ve been feeling pretty annoyed at some of the things going on in my life, so trying to make some improvements, looking at what I dislike about my lifestyle currently.

• Credit Cards – A necessarily evil – I do a lot of travel and they are vital for that – but at the same time, terrible habit of putting all my purchases onto them and not really realizing how much is being spent until I get the bill… Moving to using EFTPOS more to better track regular spending.
• Services I don’t need – this is an easy one to get into, been looking at what I pay for – maybe I don’t need 5 SIP trunks and/or 20 domains – it’s easy to say that something is only $15 a month or so and doesn’t really matter,  but over a year it adds up….
• Paper Mail – physical bills and letters are extremely annoying, working to move everyone who sends me paper materials to electronic forms and culling as many mailing lists as possible.
• Email – I get way too much email and the fact is, about 50% of it I don’t care about… I’m being pretty brutal lately and unsubscribing from lots of different lists and groups.
• Computers – As part of my goal of reducing the complexity of my computers, by the end of the month, I will have decommissioned one of my two collocation servers and my flat server infrastructure is down to a single tower and router/switch, as well as having completed a number of VM reductions and improvements.
• Stuff – Been going pretty well at my goal of reducing the amount of stuff I have, down to a final few servers to sell off, and a few boxes of bits to sort through/out.
• Car – Having recently obtained my full license and not needing one much for work now, tempted to consider selling off my car – after all it’s expensive to run and TBH, I prefer to walk to most places. Thinking something I might visit later this year, at least at my current flat parking isn’t that expensive.
• I fucking hate TVs – We have a 32″ TV in the lounge – whilst it’s not connected to public broadcasting, it’s far too easy to just blob infront of it and waste time away watching stuff. I’d much rather sell the TV to discourage myself from wasting so much time and instead spend it on the computer doing more geeky things. And I have more than enough LCDs to watch things on anyway. :-)
• Business – There’s a number of things I do that cost a lot, make little money and add to stressful life – which makes little sense, so I’m culling some things and focusing on those I enjoy and which do help me in my savings goals.
• Games – I need to play a few more games, do more fun hacking, in general, taking some de-stressing time away from computers, but in a way that’s more interactive than just passive entertainment.
• Open Source – There’s a lot of content I’m working on, I need to force myself to make the time to get it out into the public.

So a number of areas to work on improving – want to obtain a better life style, stop wasting so much money and generally feel like I’m doing something a bit more productive with my life.

I think part of it is that very scared of falling into a “work, come home, watch stuff, sleep, repeat” habit and starting to feel like it’s more of a possibility.

I want to feel motivated, complete geeky projects and do amazing things – one of the most depressive things I’ve seen has been people I know who do nothing but watch TV all evening after work, never doing anything exciting, challanging or geeky.

(of course, each to their own, but to me, there’s nothing worse than that life style, I need more challenge, to make something valuable and worthwhile).

Hopefully by posting this, I’ll have a rough list of aims that you can all accuse me of failing if I start to stray to motivate me further. ;-)

As we’ve previously established, I use Twitter. A lot. Too much, some might say. ;-)

I’ve been using twitter since 2009, during that time I’ve made a huge number of friends all over the world and keep in touch with many of them on an almost daily basis.

I’ve even met someone special on Twitter, stayed with friends in AU met purely on twitter and gotten myself into plenty of trouble and debates :-)

But Twitter has it’s downsides – time, dramas,  network lock-in and ever expanding social scene which is difficult to address.

• Time: Twitter consumes massive amounts of time and I consider it highly addictive for an infoholic like myself – I need to keep refreshing, getting the next message, reloading.
• Chains: Whenever I consider leaving twitter, there is always the realization that a vast number of the awesome people that I know could not have been met via any other means and that I wouldn’t keep in touch with them otherwise – my social circle has been massively expanded by Twitter, I can go to most major cities in AU and NZ now and have friends there I can meet with, which is pretty amazing for someone who spends most of his time on his computer writing code.
• Proprietary Network: Most people don’t care so much, but I really dislike how Twitter is a single proprietary entity that is controlling all this communication –  Ideally I’d like to be using an open platform, for example StatusNet/Identica – but that then brings one back to the issue of being locked into Twitter due to the user base all being there….
• Business: Your twitter profile is your personal conversational space, but it’s searchable for all employers, investors and journalists to access in future – there are already tones of cases of information being used against people in ways they didn’t expect which has come back to bite them. Of course one could make their profile private…. but that then discourages new people from following and making new friends.

And then there’s the dramas….. Twitter is basicly a giant high school clic at times and this can be pretty stressful at times.

• Follows, Unfollows: People follow and unfollow all the time – since I’ve started using twitter, the selection of people I follow has changed heaps – maybe the person’s interests changed, maybe I kept getting into arguments with them, maybe they keep reminding me of that hawt evening long ago or maybe they just turn out to be douchebags. However unfollows tend to get taken personally and maybe Twitter users in general have this problem where we place too much value on a follow-based relationship.
• Twittercide: Sometimes people get tired of Twitter – maybe for one of the reasons above – and end up quitting, often just deleting their account without saying goodbye – this actually really upsets me, would you just suddenly stop talking with any IRL friends without saying anything?
• Past Hurts: In close communities, particular combinations like small cities like Wellington + Twitter, it’s easy to keep bumping into past regrets, exs or people you generally dislike. Worse even when you see people praising people who you know are complete jerks.

So as you can probably tell, there’s a lot about Twitter than I’m unhappy about – so I’m looking at making some improvements to the way I follow people and use it:

• Enforce Limits: 150 friends, 100 other max. Science suggest that we can only maintain social relationships with 100-230 people at a time, and I’m not going to argue with science. That’s for conservatives to do.
• Value Communication. Sure it’s amusing to follow certain people, but I don’t really need to read 100 messages a day about their partner dramas or the type of cushions their cat sits on. The key question I’ve been asking myself is, does this tweet really add value to my life? I learn some excellent things at times from industry peers, valued friends and such, if I’m going to limit the amounts, try and follow those who provide quality content – after all, my time is short and valuable, make sure I’m using it productively.
• Realise that I don’t need to follow everyone. For a long time I’ve only really followed people who start engaging with me, but maybe I don’t need to follow people even if they do – if they want to have good conversations with me, that’s great – I’ll happily engage, but maybe they post too much crap at other times to be worth a follow.
• Determine Friends from Fans: Fans will follow then get tired and unfollow you the next day without saying a word, friends are people you regularly engage with and wouldn’t want to miss – not everyone who follows is really a friend – and maybe I should try and cull who I follow back to be those who I actually care about.
• I can block retweets. Some otherwise great people retweet some complete crap. But it’s possible to block just their retweets and I’m going to use this with more frequency to improve the messaging quality.
• Less Time: I can get *amazing* amounts of geekery done when I’m not distracted by twitter every 5minutes – I’m noticing too many evenings where I do little more than twitter, and I’d rather spend a week of evenings doing geekery and then catching up with friends at a specific time.

As much as it pains me to say it, I’m pretty tired with Twitter. But I can’t quit, there’s too much value in the relationships there – so I’m trying to find a middle ground, between not having it and being totally addicted.

Maybe long term I’ll move off twitter, perhaps more use of my blog and IM will eliminate some of the uses I have for it, but whilst those allow me to maintain current relationships, I’m not sure if it really enables me to grow and find new ones.

Other ideas include more use of email lists and chatrooms around specific topics to hang out with like minded indiviuals. Or maybe write some bots to automate social interaction for me and send me summaried updates :-)

I’ve been driving around in my mighty Toyota Starlet 1997 for about 18 months+ now and have finally gotten tired of only having a radio as my only source of audio.

I can get away with using radio when in the CBD with good alternative stations like Active who don’t have too many ads, but when doing roadtrips often there are large sections with no coverage or only poor quality commerical stations.

So I decided to buy a new stereo and settled on a Sony CDXGT500U stereo – primarily due to it meeting my two requirements in the cheapest formfactor – both an AUX 3.5mm input jack AND a USB socket for taking MP3s (sadly no Ogg or Flac tho).

Being a sucker for DIY I decided to have a go at installing it myself – I didn’t need anything too flash like new speakers or cable runs, just wanted the inputs really. Fortunately the installation of the stereo was pretty easy, but I ran into the good old problem of proprietary connectors/standards used by the different vendors.

1. There’s no single standard for the mounting of devices in the car dash – in the case of this stereo, the mounting brackets supplied aren’t required and instead it bolts directly into the Japanese-style mounts.
2. Sony doesn’t use a standard for their stereos.
3. Neither does Toyota use a standard for their cars.

To make it work (without going to the pain of soldering/custom wire wrapping) I had to buy *two* different adapters – once for Sony->ISO and another for Toyota->ISO which cost a good $15 each from retailers.

We all love lots of daisy chained adapters!

On the plus side, I now have a new stereo installed, dragging my car out of the 80s and into 2011. It’s also the most expensive thing in the car now, although being a Starlet, it’s hardly a theft magnet.

This Starlet be totally pimped yo!

By now most people have heard of the Sony Playstation Network getting hacked and around 75 million accounts worth of information being obtained.

Ignoring the whole fact that someone owned Sony so badly and that they’re not even sure if credit card details got exploited, I want to examine the information that is being stored with Sony.

There are three key bits of information obtained from the breach:

1. Login credentials of PSN users.
2. User identify information, consisting of phone number, email address and age.
3. Possibly credit card information.

The last mention is the most important – obviously any credit card breach is bad (also PCI-DSS compliance, WTF Sony?), but Sony isn’t sure if the card DB has been exposed or not at this stage and is making a general just-in-case recommendation.

Login credentials may be an issue depending how smart you are – if you’re one of those people who uses the same login on every site, this is a clear example of why you shouldn’t, and you can now enjoy changing the login details on every single site you use… (how many more provider compromises does it take till you learn this is bad??)

So assuming you didn’t use credit cards and used unique credentials, this limits the exposure to user identity information – this is causing huge outcry in the media, with some great quotes from different countries police stating how this is going to lead to widespread identity theft.

Which raises the following points:

• Why are bank and other key systems requiring identification so poorly setup that all that you need is name, age and address to obtain?
• All these details are already available online for anyone with a bit of sense, it’s hard to keep all this stuff private in the days of social networking.
• What are the penalties for companies not conducting the proper validation and security checks on people signing up to things like loans?

Sure it’s bad that the information got compromised, but let’s consider that most of the identity information is already public.

Birthdates are easy to get with the widespread popularity of social networking, same for addresses which can be found from domain records, social networking, websites and more, along with contact details.

If this information is enough to then take out a loan or a bank account, then I think those providers have some pretty heavy explaining to do – far too many have sloppy validation checks which don’t reflect the realities of the 21st century.

Just last week, I had to “validate” my home address to obtain a driver’s license. All that’s required to prove my identity is some photo ID and a service bill with an address on it.

Faking a bill is hardly complex, most laser printers will make something that’s good enough to pass any regular inspection, it’s a step that is only going to catch out the most clueless of exploiters.

Wake up companies, seriously….

I know that some providers to take precautions, even when this may lead to some customer inconvenience/annoyance.

• National Bank (NZ) would refuse to tell me anything about my account, unless I rang them from a number that matched their records for my account.
• Visiting banks in person often requires photo ID, which can be faked, but takes a bit more effort.
• My approach in business has always been to ensure a customer was emailing/calling from a known account, otherwise we would call back to confirm requests on their recorded number.

Although some of these approaches are becoming less trust worthy…

• Email accounts are commonly broken into – because of this, if we get unusual requests or password reset requests, we often call back the client to confirm.
• With the adoption of VoIP technologies, it’s becoming easier to assume someone’s phone number and send/recieve phone calls on their behalf.

Sadly there isn’t really a truly valid fix, there’s no identification that can be issued that can truly validate people’s identity and secret words or passwords are usually weakened by the fact that humans suck and choose terrible words or reuse them often.

I think the best fix is simply making sure service providers validate information such as ensuring customers have their last invoice & account number before making changes and that financial institutions or credit agencies follow strict security procedures such as photo identification.

This late post is part of my 30 days of geek challenge.

I figured it would be a bit too naracistic to review my own software and a bit boring to review some of my ever day applications, so instead I’m going to do a post about a rather geeky application – KVM virtualisation.

About Virtualisation

For those unfamiliar with virtualisation (hi Lisa <3), it’s a technology that allows one physical computer to run multiple virtual computers – with computers getting more and more powerful compared to relatively stable workloads, virtualisation allows us to make much better use of system resources.

I’ve been using virtualisation on Linux since RHEL 5 first shipped with Xen support – this allowed me to transform a single server into multiple speedy machines and I haven’t looked back since – being able to condense 84U of rackmount servers down into a big black tower in my bedroom is a pretty awesome ability. :-)

Background – Xen, KVM

I’ve been using Xen in production for a couple years now, whilst it’s been pretty good, there have also been a large number of quite serious bugs at times – combined with the lack of upstream kernel support, it’s given Xen a bit of a bad taste.

Recently I built a new KVM server at home running RHEL 6 to replace my data center, which was costing me too much in power and space. I chose to dump Xen and switch to KVM, which is included in the upstream Linux kernel and is a much smaller simpler code base, since KVM relies on the hardware virtualisation capabilities of the CPU rather than software emulation or paravirtualisation.

In short, KVM is pretty speedy since it’s not emulating much, instead giving the CPU the hardwork. You can then combine paravirtualisation for things like network and storage to boost performance even further.

My Platform

I ended up building my KVM server on RHEL 6 Beta 2 (before it was released) and am currently running around 25 virtual machines on it with stable experiences.

Neither the server or guests have needed restarts after running for a couple months without interruption and on a whole, KVM seems a lot more stable and bug free than Xen on RHEL 5 ever was for me. **

(** I say Xen on RHEL 5, since I believe that Xen has advanced a lot since XenSource was snapshotted for RHEL 5, so it may be unfair to compare RHEL 5 Xen against KVM, a more accurate test would be current Xen releases against KVM).

VM Supend to Disk

VM suspend to disk is somewhat impressive, I had to take the host down to install a secondary NIC (curse you lack of PCI hotswap!) and KVM suspended all the virtual machines to disk and resumed them on reboot.

This saves you from needing to reboot all your virtual systems, although there are some limitations:

• If your I/O system isn’t great, it may actually take longer to write the RAM of each VM to disk than it would take to simply reboot the VMS. Make sure you’re using the fastest disks possible for this.
• If you have a lot of RAM (eg 16GB like me) and forget to make your filesystem on the host OS big enough to cope…..
• You can’t apply kernel updates to all your VMs in one go by simply rebooting the host OS, you need to restart each VM that requires the update.

In my tests it performed nicely, out of 25 running VMs, only one experienced an issue, which was a crashed NTP process, quickly identified by Nagios and restarted manually.

I/O Performance

I/O performance is always interesting with virtualised systems. Some products, typically desktop end user focused virtualisation solutions, will just store the virtual servers as files on the local filesystem.

This isn’t quite so ideal for a server where performance and low overhead is key – by storing a file system ontop of another filesystem, you are adding much more overhead to the block layer which will translate into decreased performance, not so much around raw read/write, but around seek performance (in my tests anyway).

Secondly, if you are running a fully emulated guest, KVM has to emulate virtual IDE disks, which really impacts performance, since doing I/O consumes much more CPU. If your guest OS supports it, paravirtualised drivers will make a huge improvement to performance.

I’m running KVM guests inside Linux logical volumes, ontop of an encrypted block device underneath (which does impact performance a lot) however I did manage to obtain some interesting statistics showing the performance of paravirtualisation vs IDE emulation.

View KVM IDE Emulation vs Paravirtualisation Results

They show noticeable improvement in the paravirtualised disk, especially around seek times… of interest, at the time of the tests, the other server workloads were idle, so the CPU was mostly free for I/O.

I suspect if I were to run the tests again on a CPU occupied server, paravirtualisation’s advantages would become even more apparent, since IDE emulation will be very susceptible to CPU load.

The above tests were run on a host server running RHEL 6 kernel 2.6.32-71.14.1.el6.x86_64 ontop of an encrypted RAID 6 LVM volume, with 16GB RAM, Phenon II Quad Core and SATA disks.

In both tests, the guest was a KVM virtual machine running CentOS 5.5 with kernel 2.6.18-194.32.1.el5.x86_64 and 256MB RAM – so not much memory for disk caching – to a 30GB ext3 partition that was cleanly formatted between tests.

Bonnie++ 1.03e was used with CLI options of -n 512 and -s 1024.

Note that I don’t have perfect guest to host I/O comparison test results, but similar tests run against a RAID 5 array on the same server suggests that may be around a 10% performance impact with KVM paravirtualisation which is pretty hard to notice.

Problems

I’ve had some issues with stability which I believe I traced to one of the earlier beta kernels with RHEL 6, since upgrading to 2.6.32-71.14.1.el6.x86_64 the server has been solid, even with large virtual network transfers.

In the past when I/O was struggling (mostly before I had upgraded to paravirtualised disk) I experienced some strange networking issues, as per the post here and identified KVM limitations around the I/O resource allocation space.

Other than the above, I haven’t experienced many other issues with the host and future testing and configuration is ongoing  – I should be blogging a lot of Xen to KVM migration notes in the near future and will be testing CentOS 6 more throughly once released, maybe some other distributions as well.