For a long time I’ve been asked “when are you getting an RSS feed” and “when is your blog getting updated?”.

I’ve been pretty busy and haven’t had the time to upgrade my custom built system dating back to 2006, so I’ve finally decided to make the jump to WordPress, since it’ll make the maintenance and upgrading of it so much easier in future.

I’m also aiming to start blogging a bit more frequently, beginning with the 30-days blogging challenge that started up on Twitter – more about this soon – as well as being able to blog from my smart phone whilst on the go. :-)

At this stage, some of the old content is still missing, as I am slowly importing and reformatting/fixing a lot of materials – if you want to access the old site for anything, you can still reach it at: https://www.jethrocarr.com/old/

Sadly after almost 5 years of loyal service, my beloved Libretto U100 recently started showing signs of it’s age by crashing randomly and struggling to keep up with my daily tasks.

To replace it, I wanted something both lightweight and portable, but also powerful to keep up with all the large documents and projects I’m working on.

I ended up selecting a Lenovo X201i laptop, Lenovo’s ultraportable model.

THE LAPTOP

With a 12″ widescreen LCD and weight of about 1.5kg, it’s certainly larger than my Libretto, but a great compromise between usability and portability. Having a fullsize keyboard is another major advantage and it certainly makes the laptop more productive on the go.

I also purchase the extended 9-cell battery, which gets me about 7 hours life, enough to get me through a day of customer meetings.

Over all it’s a fantastic machine, and with a Intel Core i5 and 4GB of RAM packed into the system it’s a fast system able to tackle everything I throw at it.

In the weekend I upgraded my Libretto U100 to Fedora 12 (from Fedora 9 previously). I was extremely surprised (and happy) to find that everything worked correctly first time with the exception of the docking station (which I shall blog about later). Considering the rarity and uniqueness of this particular machine, it’s an excellent result.

HUAWEI E220 IS A PITA

However I discovered that my Huawei E220 3G Modem (branded as “Vodem” here in NZ) was now failing to work – when trying to connect, NetworkManager would start, the connection would run for a few seconds and then suddenly disconnect. I would also receive a popup saying that sr0 was unable to be mounted.

The 3G modem would then fail to appear in Network Manager and the kernel log showed lots of weird USB errors.

The Huawei E220 is an interesting device, it has both a 3G modem and also a USB “SCSI CDROM” drive which contains drivers for when plugged into a Windows computer. However this dual-device operation has historically caused no end of different problems across various Linux releases.

In Fedora 12, it seems that the “cdrom” (usbstorage) and 3G Modem (usbserial) drivers fight each other – first the usbserial driver works as expected, connects to the network and Network Manager runs OK. However a second later the “cdrom” tries to get mounted and glitches, breaking both drivers and dropping the connection.

SOLUTION

You can’t work around it by trying some of the past workarounds with older Linux releases such as removing the usbstorage module or apply custom vendor & product options to the usbserial module, either workaround will break the newer version of NetworkManager/ModemManager.

Fortunately the fix is relatively simple – we just need to tell the system to ignore the “cdrom” – which we can do by using Udev. Simply create the file /etc/udev/rules.d/20-custom-huawei.rules with the contents of:

# work around dodgy Huawei modem
SUBSYSTEMS=="scsi" ATTRS{vendor}=="HUAWEI", OPTIONS+="ignore_device"

Then re-plug the Huawei and the system will detect both the 3G Modem and the “cdrom”, however the ignore_device option will cause udev to avoid trying to mount the CDROM and therefore permits the 3G modem to work uninterrupted. :-)

As part of the Amberdms Billing System I needed to add the ability to export the entire MySQL database when logged in as an administrator from the application UI.

This feature was desired to prevent any shoddy hosting companies from preventing users from downloading their data from the application – without it, a hosting provider could refuse to provide the database creating effective vendor lock-in for users, even though the software is open source.

There were a couple different approaches I could use:

• Implement code that reads all the database structure and data rows and writes SQL from that. (this is the phpmyadmin approach)
• Use mysqldump from the CLI

I chose the latter, since it’s much easier to write and maintain than a SQL generator like phpmyadmin uses, however I came across a few challenges:

• I needed to supply a username & password to mysqldump – however, doing this via the CLI would expose the password to anyone with shell access to the server (they could run ps aux to see the password used).
• The databases could be anywhere from 1MB to several hundred, whatever solution was chosen could not require the whole file to be stored in memory.

My approach was to write some code that creates a temporary configuration and export file, then saves the authentication details into the temp file and calls mysqldump and instructs it to use the config file for options.

To provide the file for download, the PHP script then sets the HTTP headers and uses readfile to basically output all the file contents straight to the brower, avoiding any memory issues for the PHP script.

Below is my code, note that there are some support functions used to generate secure, unique temp files as well as perform easier MySQL queries, but it is easy to adapt to whatever framework you are using.

It is important to note that the process that generates your temporary files should make sure the files are readable ONLY by the webserver process, otherwise other users could read the config file and discover the passwords.

/*
	Create temp files for download
*/
$file_config	= file_generate_tmpfile();
$file_export	= file_generate_tmpfile();


/*
	Write authentication information into temp config file 
	this allows us to prevent the exposure of the DB password on the CLI
*/

$fh = fopen($file_config, "w");
fwrite($fh, "[mysqldump]\n");
fwrite($fh, "host=". $config["db_host"] ."\n");
fwrite($fh, "user=". $config["db_user"] ."\n");
fwrite($fh, "password=". $config["db_pass"] ."\n");
fclose($fh);


/*
	Export Database
*/

$dbname = sql_get_singlevalue("SELECT DATABASE() as value");

system("/usr/bin/mysqldump --defaults-file=$file_config $dbname > $file_export");


/*
	Set HTTP headers
*/

$filename = "database_export_". mktime() .".sql";
	
// required for IE, otherwise Content-disposition is ignored
if (ini_get('zlib.output_compression'))
	ini_set('zlib.output_compression', 'Off');

header("Pragma: public"); // required
header("Expires: 0");
header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
header("Cache-Control: private",false); // required for certain browsers 
header("Content-Type: application/force-download");
	
header("Content-Disposition: attachment; filename=\"".basename($filename)."\";" );
header("Content-Transfer-Encoding: binary");

// tell the browser how big the file is (in bytes)
header("Content-Length: ". filesize($file_export) ."");


/*
	Print out the file contents for browser download
*/
readfile($file_export);


/*
	Cleanup
*/
unlink($file_config);
unlink($file_export);

Feel free to use any of this code royalty-free in your own applications. :-)

Sometimes when trying to create new Xen domains, my Xen server will refuse to create the domain, complaining with “Error creating domain: Out of memory. xc_dom_boot_mem_init: can’t allocate low memory for domain”.

This problem is a currently known bug, there are various patches and work arounds being discussed on the Red Hat bug tracker. If you don’t want to rebuild Xen with a patch to try and resolve the issue I found that manually reducing the amount of memory that the host (Domain-0) was consuming would resolve the issue.

You can do this by running (as root):

$ xm mem-set Domain-0 1024M

Set to whatever memory value you want, if your host does nothing other than running Xen for all the VMs, you can set it quite low, I recommend 256MB.

This error only seems to happen on x86_64 servers, according to the bug report the reason was “For each vcpu of xen/ia64, it requires 16M contiguous memory for vhpt, but balloon driver didn’t consider this case when it balloons memory.”

For details, refer to bug 466021 at Red Hat.

A new 0-day attack on the Linux kernel has just been released by Brad Spengler called the “Chedder Bay Exploit” which exploits a flaw in the Linux 2.6.30+ kernel.

This exploit is interesting, in that the code doesn’t look particularly broken, but when compiled the compiler optimisations causes the compiled code to have a security hole.

For more technical details on this exploit and further news, check the LWN.net article or use the CVE reference CVE-2009-1897.

From my quick review of the exploit, it appears the attack uses Pulseaudio to bypass Selinux security if it is enabled and then performs an attack against the /dev/net/tun device, allowing a standard user to gain root access.

Not having pulseaudio or the tun kernel module loaded should prevent this exploit from working, although I have not yet had sufficient time to test this since I received the alert announcement around 3am NZ time.

The exploit affects the 2.6.30+ kernel releases and also some of the test kernel 2.6.18 kernel releases by Redhat.

However, all production kernel releases for RHEL/CentOS do not appear to be vulnerable since the change that introduced the security exploit had not been backported yet.

In my tests on CentOS 5.3 with kernel 2.6.18-128.1.16.el5xen on i386/xen, I was unable to trigger the exploit.

UPDATE 19th July 2009

I have a correction to make based on feedback from Brad Spengler, the exploit uses Pulseaudio if there is no SELinux present, or if it is in the disabled state.

However, if SELinux is enabled, the exploit uses a vulnerability in SELinux to gain privileges without the need of Pulseaudio – this is a situation where attempting to use SELinux to make yourself more secure actually leads to your system being less secure.

Many thanks to Brad for the explanation.

My fileserver recently starting locking up and crashing – with the original CPU and MB being around 5 years old now, it was due for a replacement.

There had been previous problems with that motherboard which have developed over the years – the serial ports had died many years ago, and in order to keep the system stable and to prevent reboots, I had to run the RAM overclocked by +0.1V – if run at normal voltage, it would randomly hang.

About a year ago I upgraded it with 4x 500GB IDE disks and another 2GB of RAM to boost it to 3GB to act as my home file and Xen server.

I decided to keep the existing disks and case, but to replace the motherboard, CPU and RAM with new equipment. To better suit my growing needs I also wanted this box to be a full blown Xen server running backup and development VMs for my company.

The power supply in the server had also recently failed, and I was using a cheap generic replacement supply, so replaced this with a new 500W Vantec.

Upgrades ended up being:

• AMD Phenom II X4 810 – That’s 4 cores at 2.6Ghz with a total cache of 6MB!
• 6GB DDR 3 1333 FSB RAM (Kingston)
• ASUS M4A7BT-E Motherboard
• 500W Vantec PSU
• (second hand) Silicon Image IDE PCI controller card

The upgrades went smoothly thanks to the ease of the Lian Li case – these cases are a bit more pricy, but fantastic to work with, perfectly formed aluminium, sliding out motherboard try and removable drive cages, with everything using a single size of thumbscrew.

REBUILDING INITRD

The most complex part was having to fix the CentOS kernel – whilst the IDE data array was fine, the OS is stored on 2x RAID 1 SATA drives. Due to the fact I had completely changed SATA controllers, Linux was unable to boot since the modules were missing from initrd.

To fix this, you need to re-create the initrd file that the kernel uses using the redhat mkinitrd tool. Personally, I wish Redhat would simply stick *all* supported disk drivers into the initrd file to save me from this hassle, but there might be some technical reason for not doing so…

The steps to take are:
1. Boot off a live CD and chroot to your install

2. Edit /etc/modules.conf to include your new driver (in my case ahci)

3. Delete (or move) the existing initrd in /boot

4. Mount /proc and /sys.

5. Run /sbin/start_udev to populate /dev

6. Execute mkinitrd -v /boot/initrd-MYKERNEL.img MYKERNEL

7. Reboot

We need to mount the special filesystems and create the device structure since mkinitrd uses these for detecting what drivers to include in the initrd file.

CENTOS 5.3 WITH ASUS M4A7BT-E

Once booted everything worked except for the ethernet controller – this particular motherboard requires the atl1e driver, which is now in the mainstream kernel.

For users of older kernels, you can download the source for the module here and build it for your kernel.
Since this is a server, I ran no tests with the onboard video or sound.

PERFORMANCE

I’m currently re-organising a lot of the data and services on this system, moving all services off the host server into virtual machines and setting up LVM on the array, once this is done planning to run some performance tests to see what I can get this powerful new box to do. :-)

Also working on getting another IDE controller – currently I have a two-controller card, which means I have two IDE drives on each channel, which degrades performance since the drives have to take turns when transferring data.

Once I get another controller, I will be giving each IDE disk a dedicated channel and will do some before and after performance tests.

PICTURES! :-D

Removal of old motherboard - note how the entire m/b chassis slides out of the main case for easy access!

4x 500GB IDE disks consisting of my main file storage array.

Very fuzzy image, sorry - the ASUS motherboard includes a seporate header you can use to connect all the case panel connectors to and then plug the single connector onto the motherboard! Great idea! :-)

New motherboard (ASUS M4A7BT-E), CPU (Phenom II X4 810 - 2.6Ghz, 4core, 64bit), 6GB of RAM and PCI IDE controller for existing disks.

Different angle of the assembled motherboard.

Hmmmmm.... my room is going to take a little while to clean up after I've finished assembling this system...

Blue IDE cables, blue power cables, blue heatsinks - seem to have bit of a theme going here...