Tag Archives: geek

Anything IT related (which is most things I say) :-)

LCA2011: Day 01

Monday (yesterday) was the first day of the conference, so much great content and stuff on I’m only just catching up with blogging about it now.

I was one of the afternoon presenters, with my Indefero Source Code and Project Management presentation. The videos are still getting processed, but my slides are now available as a PDF downloaded.

There’s some pictures taken of me doing my talk by @chrisjrn which make me look like a bossy dictator. Which is probably accurate. ;-)

Me looking excited at the conference introduction, whilst @LGnome works hard on his slides like a good boy. (pic: @chrisjrn)

I look a bit old, but I think that's the sunburn/dry skin :-/ (pic: @chrisjrn)

I am Jethro, hear me roar! (pic: @chrisjrn)

If you don't use this software, I will come to your workplace and hurt you. (pic: @chrisjrn)

Chris is taking  a large number of pictures during the conference, you can see them all on his flicker feed here.

Also, here’s some blurry camera phone pictures by me from the conference:

Some of the LCA crowd waiting for the conference to start.

South Bank busway, with train line in the background.

Geeks hit the town - lookout Brisbane!

OMG! It's taller than 2 stories!

It's a bridge!

Brisbane looks awesome at night. Especially when standing in the middle of the river.

Maybe I don't have enough chargers already..... there's also a laptop and another cellphone not yet pictured O_o

Indefero – Source Code & Project Management

This blog post is for the benefit of those who attended my Indefero presentation at linux.conf.au 2011 and want a copy of the slides/links.

Download indefero_source_code_management presentation [PDF].

Download indefero_source_code_management presentation [ODT]

The talk was also recorded but I’m not sure when the video will be up online – I shall post the link to it once it appears. :-)

You can now watch my presentation online, or download the Ogg/Vorbis at Blip. :-D

LCA2011: Day 00

Technically today isn’t an LCA day, but rather a “travel to LCA” day – figured since I won’t be tweeting as much, I should at least try to blog post every day on the events. If I’m really good, I may even upload my conference notes O_o

My trip started off with my first international flight via the new international terminal at Wellington – have to say, I really do like the architecture, although I think the lack of viewing spaces to look out onto the runway is a little sad. :-(

New Wellington international terminal

Sadly my flight wasn’t the best, it seems that during the process to move my flights from Thursday to Sunday 23rd, AirNZ had managed to completely forget that I was vegetarian.

I think it’s a pretty silly flaw in their system – seeing as I book all my flights with my AirNZ Airpoints ID, my basic preferences (such as diet) should really be set there.

The hostesses where pretty good and managed to find me some cheese and crackers, but they don’t carry additional extra vegetarian meals so couldn’t help me too much.

Other than that, the flight was uneventful and even the kids on the plane were reasonably well behaved.

Me crusing along on the AirTrain with a poorly adjusted camera. Note the Linux t-shirt to attract other fellow geeks at the airport.

After getting to Brisbane, I took the AirTrain from the Airport to South Bank Station for just $15 return using a special conference discount.

I’m actually extremly impressed with Brisbane’s public transport system! It’s how Wellington’s *should* be.

AirTrain:

  • The wait between trains is 25-30mins, I only had to wait a couple before it came.
  • It’s very smooth, fast and modern.
  • At $15 conference rate, it’s pretty affordable too, certainly beats taking a taxi.

Regulars Trains/Buses

  • Very, Very Cool.
  • Excellent use of RFID technologies – the “go cards” in use are plentiful and can be brought at vending machines around the place. Once purchased, you swipe on and off trains AND buses.
  • Brisbane has dedicated roads for buses, often running under ground which it runs natural gas powered buses along – almost like a bus-powered subway system.
  • Modern, fast, accurate timing and bus order displays.

And other stuff from today:

  • Caught up with a whole bunch of awesome friends again.
  • Had a bit of a scare that my 3G SIM was flawed, but after correcting the APN settings, the 3G on my laptop is now working with dodo.com.au :-D

Over all, a good start. :-)

OMG OMG LCA

Well I’m still getting organised in a somewhat mad panic (not enough hours in the day!) for my linux.conf.au trip and associated AU-based holiday.

Key information is that I will be landing in Brisbane at 16:00 AU time on Sunday 23rd, I figure with customs and airtain travel time, I should be getting to Urbanest around 17:00-18:00, from where I will then be looking for awesome geeks to catch up with for beer and or delicious food.

I will be keeping my NZ mobile number, but will have an AU 3G SIM card with me for my laptop, so I’ll be online during the conference and whilst around Brisbane.

I should also be reachable via my NZ office number –  my Android mobile runs SIP, which runs via an OpenVPN tunnel, back to the Amberdms Asterisk server – which is a pretty cool way to get very cheap international calling when in a wifi hotspot :-D

New KVM Server :-D

As per my recent post about how I have more computers than a small country,  I’ve taken the step to build a new server to run at home for all my development, backup and VM storage.

I’ve managed to condense down my server racks of stuff into a big huge tower case, taking 10 hard drives and up to a EATX server motherboard – whilst it may be pretty big, it’s nowhere near as large as a small data centre is. ;-)

The shiny black monolith of awesomeness

Hardware specifications are:

  • ASUSTek M4A78T-E Motherboard
  • AMD Phenon II X4 810 CPU (4 cores, single socket)
  • 12GB DDR3 RAM (planning to boost to max of 16GB)
  • 4x 1TB 7200RPM SATA drives for archival/file storage. (RAID 6)
  • 6x 160GB 7200RPM SATA drives for virtual machine space. (RAID 5)
  • 2x 4x SATA port controller cards for PCIe-4x
  • Lian Li PC-A71F chassis + additional 4x 3.5″ hotswap chassis.
  • Vantec Ion2+ 600W modular PSU
  • NexStar SATA docking bay + 2x 2TB 5600RPM SATA drives for external offsite backup purposes.

Software Specifications are:

  • RHEL 6 Beta x86_64 (yes, you heard correctly – running beta + jethro hax to get newer version of KVM, with plans to upgrade to CentOS 6 once released)
  • Full disk encryption across all drives to prevent data theft should physical access be compromised.
  • KVM virtualisation -All my previous systems have been Xen, but with the newer hardware I had an opportunity to upgrade to KVM – which is great, I’m finding it far less buggy than Xen has ever been.

Cool Stuff:

  • I’ve been running a number of performance tests using bonnie++ which I will post later this week – or maybe next week due to time pressures – comparing the different RAID levels and disk encryption.
  • Aside from the silly side-mounted hard drives (more on this later) I’m loving the Lian Li case, they very rarely disappoint. The sleek black finish and the smooth minimalistic door on it really helps make it look sexy and awe-inspiring.
  • Yes, the RAM/CPU is a little lacking, plan is to upgrade the MB, CPU and RAM to a bigger (maybe server dual socket) board later this year or early 2012.
  • The whole system even with the disks and fans spinning along at a reasonable load is quiet enough for me to sleep with easily. Although, having said that, I’ll sleep through almost anything. ;-)
  • Use of the multiple 160GB drives is in order to boost the I/O performance of VM disk operations by spreading load across a large number of spindles.

Not-so-cool Stuff:

  • The 16GB memory limit is going to be a pain, I may have to replace the MB sooner than desired.
  • I used up all my PCIe 16x slots in order to fit both PCIe 4x SATA controllers, so I’ve lost the ability to stick more video cards or other I/O controllers – need more PCIe 4x slots in my next motherboard.
  • The dust filters on this version of the case appear to be more awkward to remove whilst running, unlike some of their past models.
  • The side mounting hard drive case makes it difficult to close the case sides without hitting power cables or SATA cables…. using 90′ angled connectors helped for SATA data cables, but the SATA power cables are still being annoying.

Server nudity for all you geek perves out there!

I’ll have some more blog posts over the next week or so (even with pre LCA chaos) to detail some of the things I’ve learnt about Xen to KVM migration and other useful bits relating to virtualisation on RHEL 6.

Happy Birthday to me!

<?php

for ($i=0; $i <= 22; $i++)
{
    print "Jethro is oo";
    
    for ($j=0; $j < $i; $j++)
    {
        print "ooo";
    }
    
   print "old\n";
}

if ($i == 88)
{
    die("horrible horrible death");
}
else
{
    print "Live long, and prosper\n";
}

?>

The OOM killer is a nasty nasty bully

As part of my two weeks of annual leave, I’ve been making good of the spare time to work on upgrading a lot of my servers, adjusting configurations and performing a large shuffle of virtual machines between some of the hosts I have in different data centers.

As part of this work, I’ve been upgrading what was previously a DR-only host to run as full production after some nice memory and disk upgrades.

Unfortunately I ran into the beloved “Memory squeeze in netback driver” bug as per Xensource bug 762.

This delightful bug leads to a situation where although the server has about 8GB of available memory, Xen runs out of memory for networking to the VMs after a certain number of guests are started.

It’s a known fault with something to do with the Xen dom memory ballooning – one workaround is to force the domain to a certain memory size – easy enough to do, one change in the bootloader and another in the xen configuration files.

However I had to be clever. I thought to myself “Why not just tell the Xen dom to just set the memory now using xm mem-set command and save a reboot?”. Sadly my brilliant idea didn’t extend to checking how much memory the host was actually using….

Since it had been running for a while, a few processes had decided to take advantage of the additional memory and didn’t take kindly to having to fit into the new size, promptly consuming the allocated 256MB plus the swap space on the host.

If you’ve never exhausted a Linux box of memory, what happens next is never fun – Essentially the kernel invokes the Out Of Memory killer, which goes and kills of processes that it thinks are most deserving of being terminated to free resources.

Whilst this sounds like a smart feature, the OOM killer isn’t actually that smart and can do some undesirable activities – in this case, it went and terminated almost all the processes on the server, including both cron and SSH in an attempt to free memory.

I had setup a script to automatically restart the server should another remote server be unable to establish an SSH connection after 10mins whilst working on the changes just-in-case I did something silly and killed networking, however with cron terminated, this script isn’t getting executed.

So I now have a box that can do nothing other than ping, located in a data centre requiring a technician to power cycle it – the nightmare of any sysadmin. :-(

These situations are pretty rare these days thanks to most workloads being inside virtual machines or on servers with lights out management, but they still happen from time to time sadly. :-(

This bug is also one of the reasons why I’m really enjoying KVM on RHEL 6 over Xen on RHEL 5, so far it appears far more stable, less buggy and generally less “hacky” in nature.

Interestingly, only seen this bug on x86_64 xen hosts… many of the bugs I find with Xen seem to be architecture specific bugs and often don’t happen on i386 or vice-versa.

Sadly most of my production boxes still have another 12-24 months of life before I can justify upgrading them all to shiny new KVM hosts with LOM capabilities, I look forwards to when I can.

Meanwhile, I think some research into the OOM killer is needed, to find out how I can best configure it not to kill key processes.

The OOM killer isn’t entirely stupid, it does a number of metrics to try and make the best of a bad situation as per the documentation, but at the end of the day it’s just a really nasty tool for a problem you never ever want.

Too many computers….

I’m currently looking over my network and computers and realising that maybe, just maybe, I have a few too many….

For example, here’s my production network:

That doesn’t include a lot of the test kit that’s racked up, that’s all production or development hosts that need to stay running.

The main issue is that I’ve ended up with a bit too much VM bloat, which isn’t a problem in terms of hardware (since one just sticks another 16GB of RAM into a box to fix that) but rather management and maintenance, the time it takes to check things like backups, verify correct operation, keep system secured, etc.

The other major issue, is that I’ve ended up with too much development kit in a small office/data centre, mostly running staging and development virtual machines but also with several specialist physical hosts (eg blade server, some sun kit, switching, etc)

Don’t get me wrong, it’s a brilliant lab to work with and very useful for spinning up projects to develop with, but the sad reality is that the cost problem doesn’t justify keeping so much physical kit.

Doing the math, I’ve found that if I pay commercial rent, power and internet services, this data centre would cost me about $10,000+ a year. For that price, I could buy a *very* nice server every couple years to run everything as a virtual machine instance.

For $10k, I could even buy a really high end machine every 3 years and rack it into a data center, however NZ’s insane data cap limitations mean that I still need to retain a local server for accessing the file server and backup servers.

So as much as I love the lab, I think I will make the decision to pull the plug and reduce/virtualise a lot of the kit. So if you’re after some decent servers, switches and other infrastructure at a good price, let me know, there’ll prob be some good auctions going up in the next few weeks or so.

Kiwicon 2010

I’m spending all weekend at Kiwicon 2010, a IT security/hacking/cracking conference being held in Wellington – this is the 4th Kiwicon, I’ve been to each once since the beginning and it just keeps getting better and better every year.

It’s an interesting mix of networking security, *nix security, windows exploits and other cool geeky or life hacks, well worth attending. :-)

I’ve been taking notes from certain talks and will be posting them up there as well as emailing anyone who wants a copy.

Will add a hyperlink to all notes uploaded on this page. :-)

30 days of geek running late

For those of you following 30 days of geek, apologies, but I’ve an extremely busy week so running a little bit behind. Will try and catch up soon as soon as one large project is out of the way. :-(