I just arrived back in Wellington today after a week away at linux.conf.au 2009 in Hobart, Tasmania. More on the conference in future postings, but I can summarise it as “awesome” meanwhile.

This particular posting is a tale of how my file server was cursed to lose it’s uptime:

Whilst I was away there was a large power outage in Wellington, across the entire city including the CBD and Suburbs for about 30mins.

My trusty home file/xen server kept on running without a problem on UPS, but it seems that it could not escape the fate of being taken offline due to power issues…

I arrived home this afternoon and noticed that the server was making more noise than normal, due to the intake dust filter being a bit clogged.

I took off the front panel, cleaned out the dust filter and began to put it back on, when I knocked the fan filter metal across the fan speed control circuit.

A spark later, I realised I had shorted the exposed fan control circuit at the front of the case which promptly caused the PSU to cut power to the server.

The server survives the Wellington power outage, then dies and loses my uptime when I simply clean out the dust… just typical…. :-(

(Fortunately after re-plugging the mains cable the server came back online successfully. Phew!)

LINUX.CONF.AU
First up in 2009 is the most important event of the entire year – linux.conf.au – which will be held in Hobart, Tasmania. If you will be attending, you can see my flight details on the wiki.

I will be staying at the uni accommodation as usual. Whilst a hotel would be a bit nicer, the uni accommodation always ensures you end up hanging out with heaps of other awesome geeks.

AMBERDMS
It’s now been 3 months since I left Prophecy to work on Amberdms fulltime. During this time I’ve been doing a very large amount of programming and I’m scheduled to release my first version of the Amberdms Billing System under an open-source license by the end of Jan – hopefully before I go away to linux.conf.au.

This billing system offers a wide range of features, including:

• Double-entry accounting
• PDF invoice generation
• Various financial reports and tools
• Timesheets & billing for hours worked
• Data-usage billing – ideal for ISPs or other service providers
• Migration utilities from SQL-Ledger.

The code is all written in PHP and I’m preparing developer documentation to assist 3rd party integration and encourage community contributions.

THIS BLOG
I haven’t forgotten to add an RSS feed, just haven’t gotten around to it yet. Will try and make some time to add it in Feburary.

HIGH-HA SERVER SETUP
At the moment I have a server being shipped up to Auckland NZ and my old production server in Napier is going to be upgraded to Centos 5. This will provide me with two identical servers to be configured in a geographically distributed cluster to provide high-availability for my software-as-a-service products.

This will be going live in January before I release the billing system – will probably write up some articles about problems and challenges I deal with and the design used.

I’ve recently built an Asterisk pabx to run all my phone systems for Amberdms. In testing I found everything to be working correctly, with the exception of my Linksys SPA942 desk phone.

The phone worked fine, except for when it received calls, the caller would not be able to hear anything, and the call would disconnect after 15 seconds everytime.

Further testing found that:

• Calls from the Linksys to any phone (internal or external) worked fine.
• Calls to & from software-based phones on the same network segment as the Linksys worked fine.

Whilst one-directional audio is typical of NAT issues, this wasn’t the case since the network was fully routed between the phone and the PABX.

After performing a packet dump on the local router, I found that the dumps showed RTP packets arriving at the phone from the PABX, but the phone was not returning any, meaning that the phone was not sending anything back!

After a long period of debugging, I enabled “sip set debug” on the PABX and found the following message in the output:

set_destination: Parsing <1000> for address/port to send to
[Aug 30 21:17:31] WARNING[14315]: chan_sip.c:5845 set_destination: Can't find address for host '1000'

I had configured the phone with the display string of “Jethro Carr <1000>”, and it seems that the “<1000>” was being interpreted by the PABX and used for the address of the phone – which explains the weird “can’t find adddress” messages I was receiving.

After changing the display string to just “Jethro Carr” the problem was resolved!

Another busy month again! (but then again, are any of them not busy these days?)

WORK

I recently spent a whole weekend migrating a customer from a Windows 2003 Active Directory + Exchange server to a new Linux-based solution using Samba with roaming profiles and Scalix for email/calendering.

Considering the old system was out of RAM and ran out of disk, it was about time for an upgrade. Since the migration, desktop load times have dropped from up to an hour to a couple of minutes and the server is no longer running out of disk.

Amberdms has no major news to report – work is going slowly on the AOConf web-based administrative interface, simply due to the large amount of other tasks going on.

FLAT

In the weekend I got a server rack given to me by a colleague – I’ve now been able to put all my computer equipment into one place, which is going to make it a million times easier when I next move. (It took about 5 hours to get all the cabling set up for the rack).

It’s all been neatly wired up, with ethernet on one side, power on the other, and with all the power strips labeled (eg: “UPS Power” or “Surge Only”) to make it easy to maintain.

I also got myself 305m of Cat5e ethernet cabling, RJ45 connectors and a crimping tool – this is unbelievably useful – no more problems with cables being just 1m too short.

I’ve uploaded a couple of pictures – Back Side, Front Side and my cheap substitute for a rack shelf.

From the rack, I’ve run cables from the gaming machine to my desk, where I have my Mac Mini and display, as can be seen in this picture.

It’s still missing a lot of the gear that I haven’t yet moved to my flat, such as my file server and second UPS, so it’s currently looking a bit empty.

I think it’s really starting to look like a geek flat now… :-)

ARTICLES

I’ve updated my “Introduction to Linux Clustering” document with various spelling/grammatical fixes that people kept reminding me about, as well as using it as an opportunity to explain things better.

I came across an interesting bug with MS Outlook 2003 experienced with a client today.

PROBLEM
Outlook fails to open or save email attachments with the error “Can’t create file”, and suggests possible permissions problem. However, creating a file from any normal means will work without a problem.

SOLUTION
If more than 99 files exist in the outlook temporary directory which have the same attachment name, they will be named FILENAME(#).EXT. However, once the number # reaches 99, outlook will be unable to create any more files and the error occurs.

Location of dir can be found in:
HKEY_CURRENT_USER\Software\Microsoft\Office\${VER}\Outlook\Security

Found in:
C:\Documents and Settings\${USERNAME}\Local Settings\Temporary Internet Files\OLK2BC

RESOURCES
More indepth details on this problem:

• groovypost.com

I haven’t located any Microsoft KB articles about this issue, but I don’t care enough about their software to spend time trying to find it. :-)

I’ve released a new article – Introduction to Linux Clustering. This one has been cooking for a few months now, so I finally finished it off seeing that I’ve got 4 days off thanks to Easter.

Easter is great – one of the few good things the church has given atheists – time off AND chocolate. ;-)

Anyway, you can download the article at: introduction_linux_clustering_1.1.

Big thanks to my friend Mark Rais for reviewing it for me and helping me make improvements. If there’s any mistakes left in it, I’ll blame him. ;-)

In 2007, the NZ government changed daylight savings, by extending it a couple of weeks. Whilst the Linux vendors pushed out updates for tzdata which contains the data needed for your system clock to function, PHP has it’s own timezone database which is causing problems for some systems, where the clock in PHP is being calculated using the old daylight savings time.

Run the following to check if PHP on your server has been updated to the correct timezone:

$ date
$ php -r "echo strftime('%Y-%m-%d %H:%M:%S');";

If the times match, no problem. If they DON’T it means that all dates are being calculated using the old daylight savings time, so from the 16th March until the 6th April, all time shown in PHP applications will be 1 hour behind.

It’s a pity that not all the Linux vendors realised this and released fixes, but I had never even considered that PHP would use a separate timezone database, so it’s quite possible neither did they.

I have tested on the following systems:

• CentOS 4 – PHP 4.3.9 (no problem)
• CentOS 5 – PHP 5.1.6 (wrong time)

It looks like PHP 4 doesn’t suffer from the problem, possibly it uses the system timezone data instead, but I wasn’t able to find a lot of information about this.

Fortunately, you can fix the timezone data without rebuilding PHP. The timezone data can be installed as a separate module (timezonedb), which is used instead of the compiled in timezone data.

I just ordered some parts, to start building a wearable computer! I’ll be using the gumstix LCD kit for the base of it.

The base unit will have touch screen, ethernet, microSD and 3 serial interfaces. From there, I can then hookup anything else I want (wifi, bluetooth, GSM modem, GPS) using the host USB port on the gumstix. :-)

I will have to research options for battery power…. need a Li-PO battery supply with 5V output… since most batteries seem to be 3.7V, I will need a DC->DC transformer to step up the voltage to 5V. Also want to be able to trickle-charge it via USB.

To run USB devices from the gumstix, I will need 5V, and up to 500mA for each USB device, so need something with a good amount of amperage if I want to run a number of devices.

It’ll be a good project to give me a chance to improve my (currently bad) knowledge of C programming as I will need to write custom applications for it. :-)

I’ll post more information once the kit arrives and I start playing with it… Will probably take a number of months to build, since I am really busy these days, but will be worth it.

Uses for the wearable PC:

• Access to email 24/7
• Link into GPS maps
• Typical PDA functions (note taking, calendar, etc)
• Mobile VoIP
• Music/video player
• Because it’s a cool project and has geek cred. ;-)

For the last week I have been in Melbourne undergoing training for the RH436 course and exam.

I am pleased to announce that I have passed the exam, and have now added the RH436 endorsement to my existing RHCE qualification!

For those who are unfamiliar with Redhat’s courses, the RH436 course is one of the 5 main components of the Redhat Certified Architect qualification, which is another level above RHCE.

RH436 is a 4-day course which covers clustering and storage management. This includes the following technologies:

• RAID/LVM
• device-mapper
• udev
• Redhat Cluster Suite
• GFS (a clustered filesystem)

The main reason I was interested in clustering was for automated failovers, and it is clear that Redhat have put together a powerful distribution for clustering.

An example application is for running a mission-critical website. With a cluster, you can setup multiple nodes/servers, and in the event of a failure, the website service will be automatically moved to another node, with the aim of providing minimal disruption.

Of course, there are far more advanced uses and capabilities such as load sharing, shared storage, etc.

Over the next couple of months I will be experimenting with my new clustering knowledge and combining it with technologies like Xen vitalization, in the aim of developing designs for rock-solid, secure, and fault-tolerant cluster implementations, which I will release on my website.

It is certainly clear to me that Xen and clustering will change the way we run servers.

I intend to do some presentations to Wellylug about clustering and Xen over the next few months, which I will also turn into articles and make available on my website.

 < jethro> awh crap....
 < jethro> the LVM howto didn't mention that is wasn't safe to
           move the /  filesystem whilst it was in use
 * jethro deadlocked the server
 < jethro> I sure hope it'll work after reboot
 < friend1> (simulate a tension-filled pause here)
 < friend2> o_O
 < friend1> err
 < friend1> heh
 < friend1> where'd you move it to?
 < jethro> from one physical LVM volume to another with pvmove
 < jethro> I think the LVM is toast now though... :-(
 < jethro> yup, it's toast. #$$@!#%

What does that mean? It means that I found out the hard way that it IS NOT safe to move a live LVM physical volume from one disk to another with the pvmove command IF the physical volume contains the / filesystem.

Apparantly the problem is that the binaries are in use when it tries to move itself and causes the system to dead lock. :-(