In the weekend I upgraded my Libretto U100 to Fedora 12 (from Fedora 9 previously). I was extremely surprised (and happy) to find that everything worked correctly first time with the exception of the docking station (which I shall blog about later). Considering the rarity and uniqueness of this particular machine, it’s an excellent result.

HUAWEI E220 IS A PITA

However I discovered that my Huawei E220 3G Modem (branded as “Vodem” here in NZ) was now failing to work – when trying to connect, NetworkManager would start, the connection would run for a few seconds and then suddenly disconnect. I would also receive a popup saying that sr0 was unable to be mounted.

The 3G modem would then fail to appear in Network Manager and the kernel log showed lots of weird USB errors.

The Huawei E220 is an interesting device, it has both a 3G modem and also a USB “SCSI CDROM” drive which contains drivers for when plugged into a Windows computer. However this dual-device operation has historically caused no end of different problems across various Linux releases.

In Fedora 12, it seems that the “cdrom” (usbstorage) and 3G Modem (usbserial) drivers fight each other – first the usbserial driver works as expected, connects to the network and Network Manager runs OK. However a second later the “cdrom” tries to get mounted and glitches, breaking both drivers and dropping the connection.

SOLUTION

You can’t work around it by trying some of the past workarounds with older Linux releases such as removing the usbstorage module or apply custom vendor & product options to the usbserial module, either workaround will break the newer version of NetworkManager/ModemManager.

Fortunately the fix is relatively simple – we just need to tell the system to ignore the “cdrom” – which we can do by using Udev. Simply create the file /etc/udev/rules.d/20-custom-huawei.rules with the contents of:

# work around dodgy Huawei modem
SUBSYSTEMS=="scsi" ATTRS{vendor}=="HUAWEI", OPTIONS+="ignore_device"

Then re-plug the Huawei and the system will detect both the 3G Modem and the “cdrom”, however the ignore_device option will cause udev to avoid trying to mount the CDROM and therefore permits the 3G modem to work uninterrupted. :-)

As part of the Amberdms Billing System I needed to add the ability to export the entire MySQL database when logged in as an administrator from the application UI.

This feature was desired to prevent any shoddy hosting companies from preventing users from downloading their data from the application – without it, a hosting provider could refuse to provide the database creating effective vendor lock-in for users, even though the software is open source.

There were a couple different approaches I could use:

• Implement code that reads all the database structure and data rows and writes SQL from that. (this is the phpmyadmin approach)
• Use mysqldump from the CLI

I chose the latter, since it’s much easier to write and maintain than a SQL generator like phpmyadmin uses, however I came across a few challenges:

• I needed to supply a username & password to mysqldump – however, doing this via the CLI would expose the password to anyone with shell access to the server (they could run ps aux to see the password used).
• The databases could be anywhere from 1MB to several hundred, whatever solution was chosen could not require the whole file to be stored in memory.

My approach was to write some code that creates a temporary configuration and export file, then saves the authentication details into the temp file and calls mysqldump and instructs it to use the config file for options.

To provide the file for download, the PHP script then sets the HTTP headers and uses readfile to basically output all the file contents straight to the brower, avoiding any memory issues for the PHP script.

Below is my code, note that there are some support functions used to generate secure, unique temp files as well as perform easier MySQL queries, but it is easy to adapt to whatever framework you are using.

It is important to note that the process that generates your temporary files should make sure the files are readable ONLY by the webserver process, otherwise other users could read the config file and discover the passwords.

/*
	Create temp files for download
*/
$file_config	= file_generate_tmpfile();
$file_export	= file_generate_tmpfile();


/*
	Write authentication information into temp config file 
	this allows us to prevent the exposure of the DB password on the CLI
*/

$fh = fopen($file_config, "w");
fwrite($fh, "[mysqldump]\n");
fwrite($fh, "host=". $config["db_host"] ."\n");
fwrite($fh, "user=". $config["db_user"] ."\n");
fwrite($fh, "password=". $config["db_pass"] ."\n");
fclose($fh);


/*
	Export Database
*/

$dbname = sql_get_singlevalue("SELECT DATABASE() as value");

system("/usr/bin/mysqldump --defaults-file=$file_config $dbname > $file_export");


/*
	Set HTTP headers
*/

$filename = "database_export_". mktime() .".sql";
	
// required for IE, otherwise Content-disposition is ignored
if (ini_get('zlib.output_compression'))
	ini_set('zlib.output_compression', 'Off');

header("Pragma: public"); // required
header("Expires: 0");
header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
header("Cache-Control: private",false); // required for certain browsers 
header("Content-Type: application/force-download");
	
header("Content-Disposition: attachment; filename=\"".basename($filename)."\";" );
header("Content-Transfer-Encoding: binary");

// tell the browser how big the file is (in bytes)
header("Content-Length: ". filesize($file_export) ."");


/*
	Print out the file contents for browser download
*/
readfile($file_export);


/*
	Cleanup
*/
unlink($file_config);
unlink($file_export);

Feel free to use any of this code royalty-free in your own applications. :-)

Sometimes when trying to create new Xen domains, my Xen server will refuse to create the domain, complaining with “Error creating domain: Out of memory. xc_dom_boot_mem_init: can’t allocate low memory for domain”.

This problem is a currently known bug, there are various patches and work arounds being discussed on the Red Hat bug tracker. If you don’t want to rebuild Xen with a patch to try and resolve the issue I found that manually reducing the amount of memory that the host (Domain-0) was consuming would resolve the issue.

You can do this by running (as root):

$ xm mem-set Domain-0 1024M

Set to whatever memory value you want, if your host does nothing other than running Xen for all the VMs, you can set it quite low, I recommend 256MB.

This error only seems to happen on x86_64 servers, according to the bug report the reason was “For each vcpu of xen/ia64, it requires 16M contiguous memory for vhpt, but balloon driver didn’t consider this case when it balloons memory.”

For details, refer to bug 466021 at Red Hat.

A new 0-day attack on the Linux kernel has just been released by Brad Spengler called the “Chedder Bay Exploit” which exploits a flaw in the Linux 2.6.30+ kernel.

This exploit is interesting, in that the code doesn’t look particularly broken, but when compiled the compiler optimisations causes the compiled code to have a security hole.

For more technical details on this exploit and further news, check the LWN.net article or use the CVE reference CVE-2009-1897.

From my quick review of the exploit, it appears the attack uses Pulseaudio to bypass Selinux security if it is enabled and then performs an attack against the /dev/net/tun device, allowing a standard user to gain root access.

Not having pulseaudio or the tun kernel module loaded should prevent this exploit from working, although I have not yet had sufficient time to test this since I received the alert announcement around 3am NZ time.

The exploit affects the 2.6.30+ kernel releases and also some of the test kernel 2.6.18 kernel releases by Redhat.

However, all production kernel releases for RHEL/CentOS do not appear to be vulnerable since the change that introduced the security exploit had not been backported yet.

In my tests on CentOS 5.3 with kernel 2.6.18-128.1.16.el5xen on i386/xen, I was unable to trigger the exploit.

UPDATE 19th July 2009

I have a correction to make based on feedback from Brad Spengler, the exploit uses Pulseaudio if there is no SELinux present, or if it is in the disabled state.

However, if SELinux is enabled, the exploit uses a vulnerability in SELinux to gain privileges without the need of Pulseaudio – this is a situation where attempting to use SELinux to make yourself more secure actually leads to your system being less secure.

Many thanks to Brad for the explanation.

My fileserver recently starting locking up and crashing – with the original CPU and MB being around 5 years old now, it was due for a replacement.

There had been previous problems with that motherboard which have developed over the years – the serial ports had died many years ago, and in order to keep the system stable and to prevent reboots, I had to run the RAM overclocked by +0.1V – if run at normal voltage, it would randomly hang.

About a year ago I upgraded it with 4x 500GB IDE disks and another 2GB of RAM to boost it to 3GB to act as my home file and Xen server.

I decided to keep the existing disks and case, but to replace the motherboard, CPU and RAM with new equipment. To better suit my growing needs I also wanted this box to be a full blown Xen server running backup and development VMs for my company.

The power supply in the server had also recently failed, and I was using a cheap generic replacement supply, so replaced this with a new 500W Vantec.

Upgrades ended up being:

• AMD Phenom II X4 810 – That’s 4 cores at 2.6Ghz with a total cache of 6MB!
• 6GB DDR 3 1333 FSB RAM (Kingston)
• ASUS M4A7BT-E Motherboard
• 500W Vantec PSU
• (second hand) Silicon Image IDE PCI controller card

The upgrades went smoothly thanks to the ease of the Lian Li case – these cases are a bit more pricy, but fantastic to work with, perfectly formed aluminium, sliding out motherboard try and removable drive cages, with everything using a single size of thumbscrew.

REBUILDING INITRD

The most complex part was having to fix the CentOS kernel – whilst the IDE data array was fine, the OS is stored on 2x RAID 1 SATA drives. Due to the fact I had completely changed SATA controllers, Linux was unable to boot since the modules were missing from initrd.

To fix this, you need to re-create the initrd file that the kernel uses using the redhat mkinitrd tool. Personally, I wish Redhat would simply stick *all* supported disk drivers into the initrd file to save me from this hassle, but there might be some technical reason for not doing so…

The steps to take are:
1. Boot off a live CD and chroot to your install

2. Edit /etc/modules.conf to include your new driver (in my case ahci)

3. Delete (or move) the existing initrd in /boot

4. Mount /proc and /sys.

5. Run /sbin/start_udev to populate /dev

6. Execute mkinitrd -v /boot/initrd-MYKERNEL.img MYKERNEL

7. Reboot

We need to mount the special filesystems and create the device structure since mkinitrd uses these for detecting what drivers to include in the initrd file.

CENTOS 5.3 WITH ASUS M4A7BT-E

Once booted everything worked except for the ethernet controller – this particular motherboard requires the atl1e driver, which is now in the mainstream kernel.

For users of older kernels, you can download the source for the module here and build it for your kernel.
Since this is a server, I ran no tests with the onboard video or sound.

PERFORMANCE

I’m currently re-organising a lot of the data and services on this system, moving all services off the host server into virtual machines and setting up LVM on the array, once this is done planning to run some performance tests to see what I can get this powerful new box to do. :-)

Also working on getting another IDE controller – currently I have a two-controller card, which means I have two IDE drives on each channel, which degrades performance since the drives have to take turns when transferring data.

Once I get another controller, I will be giving each IDE disk a dedicated channel and will do some before and after performance tests.

PICTURES! :-D

Removal of old motherboard - note how the entire m/b chassis slides out of the main case for easy access!

4x 500GB IDE disks consisting of my main file storage array.

Very fuzzy image, sorry - the ASUS motherboard includes a seporate header you can use to connect all the case panel connectors to and then plug the single connector onto the motherboard! Great idea! :-)

New motherboard (ASUS M4A7BT-E), CPU (Phenom II X4 810 - 2.6Ghz, 4core, 64bit), 6GB of RAM and PCI IDE controller for existing disks.

Different angle of the assembled motherboard.

Hmmmmm.... my room is going to take a little while to clean up after I've finished assembling this system...

Blue IDE cables, blue power cables, blue heatsinks - seem to have bit of a theme going here...

Wow, it has been a busy month! I really need to try and update my blog a bit more frequently to stop it from getting stale and requiring a large update to be posted.

AMBERDMS BILLING SYSTEM

After the initial release of the Amberdms Billing System 1.0.0 on 18th Feb 2009, there were several hundred downloads and a number of companies signing up for the trial service.

I recently released version 1.1.0 with a whole bunch of new features, including PayPal support, service usage alerts and better documentation.

I’m currently working on the next release which is focusing on the following:

• ACID compliance in all application areas.
• Taking the current SQL-Ledger migration program from alpha to stable and preparing migration documentation.
• Reoccuring Transactions.
• File import abilities.

I’m also currently investigating the work involved in developing migration applications from various proprietary products such as MYOB, as well as the idea of integrating the Amberdms Billing System with various e-commerce shop applications.

NEW PLANS!

I have recently added new plan options to the hosted version of the Amberdms Billing System, including a basic FREE plan and an entry level 100MB plan for only $10 + GST a month.

That’s a low cost accounting system run by professional engineers with no proprietary lock in! Remember, for every new customer, I can purchase more junk food and spend more time adding even more great features, so PLEASE SPREAD THE WORD! :-)

For details about these new plans, read through the announcement online.

BUT WAIT, THERE’S MORE!

I will also be announcing some exciting new Linux server solution offerings from Amberdms in the next week or so, with some great offerings for small-mid sized companies.

If you want to get information about new products and releases from Amberdms, join the low-traffic amberdms-announce mailing list.

Otherwise, I shall blog them here in due course.

TWITTER

If you are on twitter, follow me and get regular updates on what I’m up to.

Unfortunately, I had just gotten over my email addiction but now Twitter has come along to get me addicted to something new… :-(

INSTANT MESSAGING

I now have an operational jabber/XMPP server. If you want to IM me, you can at message be at jethro@amberdms.com by using any XMPP service such as Google Talk, your own server, or a free account from jabber.org.

After fixing up my laptop yesterday, installing the spare HDD and reloading everything I suddenly find that my spare HDD is faulty. :-(

At first the laptop would just randomly lock up or pause for periods of time, but today it failed badly. With luck, I was able to get the documents for the last 12 hours I had been working on to a USB stick.

I know that it’s the HDD rather than the laptop, since this spare HDD had caused similar problems on the spare laptop, but I think I triggered the real nasty stuff when I loaded about 40 GB of my data onto it giving it a big work out.

I’m back up and running using my spare Libretto U100 laptop, but have had to place an order for a new 1.8″ HDD to put back in my primary unit. (about $580 NZD – ouch!)

I’ll end up with my production laptop working with a new HDD (hopefully another 3 years life!), a working spare laptop to lend to my brother and an additional spare mainboard, RAM & chassis from the scrapped Libretto.

What a hassle! Good thing my backups are frequent and working. :-)

I had a whole pile of “fun” today when my Libretto U100’s HDD suddenly died at 1pm right in the middle of preparations to release my company’s new open-source billing system product on the 16th of February.

Fortunately as I had done a backup just last night, I only lost about 3 hours work, but it was still quite an annoyance as it took about 10 hours to replace the hard drive, pull all the data off the backup server and restore the operating system.

Most of the complexity comes from the very small size of the laptop and the fact that the only way to remove the HDD is to remove almost all parts of the motherboard & expansion cards inside.

Fortunately I had spare components from another Libretto that I had scrapped for parts – without that the problem would have been much more serious, since it only takes 1.8″ IDE HDDs which aren’t that easy to get and can be rather expensive (one Toshiba parts supplier quotes about $500 for a replacement 60GB drive).

The I/O performance on the laptop had been getting pretty bad over the past month and I was starting to suspect drive problems so the failure wasn’t a huge shock, it just came at a really annoying time.

I’ve had 3 years of almost 24×7 use out of the laptop, including carrying it all over the place on almost a daily basis, so I’m pretty pleased with it’s life span overall.

I think my next purchase will be one of the new 64GB IDE 1.8″ SSD drives to get a much faster, reliable laptop – considering that whenever I notice slowness on my Libretto it’s due to disk I/O, a SSD should be a worthy investment to extending it’s usable life.

February has been busy – some of the highlights (and lowpoints) of this month has been the release of my new billing system (good) and repeated problems with my libretto laptop (bad).

AMBERDMS

On the 18th of February, I release my new open-source accounting, service billing and time keeping product, the Amberdms Billing System.

This program has been written from the ground up by me for use by small & medium businesses, with particular focus on supporting the needs of IT businesses, including consultants and ISPs.

Take a look at the Amberdms website for further details about all the exciting features. :-)

In other Amberdms news, the amount of consulting work is increasing and I’ve also started offering budget NZ-based virtual machine services which is also helping to grow Amberdms into the next corporate empire. ;-)

LIBRETTO

My libretto is now back in fully working form. After replacing the hard drive, I had been suffering from instability issues where the machine would randomly hang and crash.

I believe I have tracked these problem to a BIOS issue, as after I reset all the BIOS options back to factory defaults, the issues resolved themselves.

Still, this caused many wasted hours and lots of annoyance.

TWITTER

Everyone keeps pestering me to create a Twitter account, so I have gone and created one. Follow my exploits at www.twitter.com/jethrocarr.

Time will tell if getting Twitter was a good idea… I had just gotten over my email addiction, so I’m not sure how safe I’ll be with Twitter.