Tag Archives: rhel

find-debuginfo.sh invalid predicate

I do a lot of packaging for RHEL/CentOS 5 hosts, often this packaging is backporting of newer software versions, typically I’ll pull Fedora’s latest package and make various adjustments to it for RHEL 5’s older environment – typically things like package name changes, downgrade from systemd to init and correcting any missing build dependencies.

Today I came across this rather unhelpful error message:

+ /usr/lib/rpm/find-debuginfo.sh /usr/src/redhat/BUILD/my-package-1.2.3
find: invalid predicate `'

This error is due to the newer Fedora spec files often not explicitly setting the value of BuildRoot which then leaves the package to install into the default location, which isn’t always defined on RHEL 5 hosts.

The correct fix is to define the build root in the spec file with:

BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)

This will set both %{buildroot} and $RPM_BUILD_ROOT, so no matter whether you’re using either syntax, the files will be installed into the right place.

However, this error is a symptom of a bigger issue – without defining BuildRoot, the package will still compile and complete make install, however instead of the installed files going into /var/tmp/packagename…etc, the files will be installed directly into the actual / filesystem, which is generally ReallyBad(tm)

Now if you were building the package as a non-privileged user, this would have failed at the install phase and you would not have gotten as far as the invalid predicate error.

But if you were naughty and building as the root user, the package would have installed into / without complaint and clobbered any existing files installed on the build host. And the first sign of something being wrong is the invalid predicate error when the find debug script gets provided with no files.

This is the key reason why you are highly recommended to build all packages as a non-privileged user, so that if the build incorrectly tries to install anything into /, the install will be denied and the user will quickly realize things aren’t installing into the right place.

Building as root can be even worse than just “whoops, I overwrote the installed version of mypackage whilst building a new one” or “blagh annoying invalid predicate error” – consider the following specfile line:

rm -rf $RPM_BUILD_ROOT/%{_includedir}

On a properly defined package, this would execute:

rm -rf /var/tmp/packagename/usr/include/

But on a package lacking a BuildRoot definition it becomes:

rm -rf /usr/include/

Yikes! Not exactly what you want – of course, running as a non-root user would save you, since that rm command would be refused and you’d quickly figure out the issue.

I will leave it as an exercise of the reader to determine why I commented about this specific example… ;-)

IMHO, rpmbuild should be patched to just outright refuse to compile packages as the root user so this mistake can’t happen, it seems silly to allow a bad packaging habit to be used when the damages are so severe.

acpid trickiness

3 Replies

Ran into an issue last night with one of my KVM VMs not registering a shutdown command from the host server.

This typically happens because the guest isn’t listening (or is configured to ignore) ACPI power “button” presses, so the guest doesn’t get told that it should shutdown.

In the case of my CentOS (RHEL) 5 VM, the acpid daemon wasn’t installed/running so the ACPI events were being ignored and the VM would just stay running. :-(

To install, start and configure to run at boot:

# yum install -y acpid
# /etc/init.d/acpid start
# chkconfig --level 345 acpid on

If acpid wasn’t originally running, it appears that HAL daemon can grab control of the /proc/acpi/event file and you may end up with the following error upon starting acpid:

Starting acpi daemon: acpid: can't open /proc/acpi/event: Device or resource bus

The reason can quickly be established with a ps aux:

[root@basestar ~]# ps aux | grep acpi
root        17  0.0  0.0      0     0 ?        S<   03:16   0:00 [kacpid]
68        2121  0.0  0.3   2108   812 ?        S    03:18   0:00 hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event
root      3916  0.0  0.2   5136   704 pts/0    S+   03:24   0:00 grep acpi

Turns out HAL grabs the proc file for itself if acpid isn’t running, but if acpid is running, it will talk to acpid to get it’s information. This would self-correct on a reboot, but we can just do:

# /etc/init.d/haldaemon stop
# /etc/init.d/acpid start
# /etc/init.d/haldaemon start

And sorted:

[root@basestar ~]# ps aux | grep acpi
root        17  0.0  0.0      0     0 ?        S<   03:16   0:00 [kacpid]
root      3985  0.0  0.2   1760   544 ?        Ss   03:24   0:00 /usr/sbin/acpid
68        4014  0.0  0.3   2108   808 ?        S    03:24   0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root     16500  0.0  0.2   5136   704 pts/0    S+   13:24   0:00 grep acpi

Mozilla Sync Server RPMs

31 Replies

A few weeks ago I wrote about the awesomeness that is Mozilla’s Firefox Sync, a built-in feature of Firefox versions 4 & later which allows for synchronization of bookmarks, history, tabs and password information between multiple systems. (historically known as Weave)

I’ve been running this for a few weeks now on my servers using fully packaged components and it’s been working well, excluding a few minor hick-ups.

It’s taken a bit longer than I would have liked, but I now have stable RPM packages for RHEL/CentOS 5 and 6 for both i386 and x86_64 available publicly.

I always package all software I use on my servers (and even my desktops most of the time) as it makes re-building, upgrading and supporting systems far easier in the long run. By having everything in packages and repos, I can rebuild a server entirely simply by knowing what list of packages were originally installed and their configuration files.

Packaging one’s software is also great when upgrading distribution, as you can get a list of all non-vendor programs and libraries installed and then use the .src.rpm files to build new packages for the new OS release.

Packaging Headaches

Mozilla Sync Server was much more difficult to package than I would have liked, mostly due to the documentation clarity and the number of dependencies.

The primary source of pain was that I run CentOS 5 for a lot of my production systems,which ships with Python 2.4, whereas to run Mozilla Sync Server, you will need Python 2.6 or later.

This meant that I had to build RPMs for a large number (upwards of 20 IIRC) python packages to provide python26 versions of existing system packages. Whilst EPEL had a few of the core ones (such as python26 itself), many of the modules I needed either weren’t packaged, or were only had EPEL packages for Python 2.4.

The other major headache was due to unclear information and in some cases, incorrect documentation from Mozilla.

Mozilla uses the project source name of server-full in the setup documentation, however this isn’t actually the entire “full” application – rather it provides the WSGI executable and some libraries, however you also need server-core, server-reg and server-storage plus a number of python modules to build a complete solution.

Sadly this isn’t entirely clear to anyone reading the setup instructions, the only setup information relates to checking out server-full and running a build script which will go through and download all the dependencies (in theory, it often broke for me) and build a working system, complete with paster web server.

Whilst this would be a handy resource for anyone doing development, it’s pretty useless for someone wanting to package a proper system for deployment since you need to break all the dependencies into separate packages.

(Note that whilst Mozilla refer to having RPM packages for the software components, these have been written for their own inhouse deployment and are not totally suitable for stock systems, not to mention even when you have SPEC files for some of the Mozilla components, you still lack the SPEC files for dependencies.)

To top it off, some information is just flat out wrong and can only be found out by first subscribing to the developer mailing list – in order to gain a login to browse the list archives – so that you can ind such gems as “LDAP doesn’t work and don’t try as it’s being re-written”.

Toss in finding a few bugs that got fixed right around the time I was working on packaging these apps and you can understand if I’m not filled with love for the developers right this moment.

Of course, this is a particularly common open source problem – the team clearly released in a way that made sense to them, and of course everyone would know the difference between server-core/full/reg/storage, etc right?? ;-) I know I’m sometimes guilty of the same thing.

Having said that, the documentation does appear to be getting better and the community is starting to contribute more good documentation resources. I also found a number of people on the mailing list quite helpful and the Mozilla Sync team were really fast and responsive when I opened a bug report, even when it’s a “stupid jethro didn’t hg pull the latest release before testing” issue.

Getting My Packages

All the new packages can be found in the Amberdms public package repositories, the instructions on setting up the CentOS 5 or CentOS 6 repos can be found here.

RHEL/CentOS 5 Repo Instructions

If you are running RHEL/CentOS 5, you only need to enable amberdms-os, since all the packages will install in parallel to the distribution packages. Nothing in this repo should ever clash with packages released by RedHat, but may clash/be newer than dag or EPEL packages.

RHEL/CentOS 6 Repo Instructions

If you are running RHEL/CentOS6, you will need to enable both amberdms-os and amberdms-updates, as some of the python packages that are required are shipped by RHEL, but are too outdated to be used for Mozilla Sync Server.

Note that amberdms-updates may contain newer versions of other packages, so take care when enabling it, as I will have other unrelated RPMs in there. If you only want my newer python packages for mozilla sync, set includepkgs=python-* for amberdms-updates

Also whilst I have tested these packages for Mozilla Sync Server’s requirements, I can’t be sure of their suitability with existing Python applications on your server, so take care when installing these as there’s always a chance they could break something.

RHEL/CentOS 5 & 6 Installation Instructions

Prerequisites:

Configured Amberdms Repositories as per above instructions.
Working & configured Apache/httpd server. The packaged programs will work with other web servers, but you’ll have to write your own configuration files for them.

Installation Steps:

Install packages with:
```
yum install mozilla-sync-server
```
Adjust Apache configuration to allow access from desired networks (standard apache IP rules).
```
/etc/httpd/conf.d/mozilla-sync-server.conf
```
Adjust Mozilla Sync Server configuration. If you want to run with the standard sqllite DB (good for initial testing), all you must adjust is line 44 to set the fallback_node value to the correct reachable URL for Firefox clients.
```
vi /etc/mozilla-sync-server/mozilla-sync-server.conf
```
Restart Apache – due to the way mozilla-sync-server uses WSGI, if you make a change to the configuration, there might still be a running process using the existing config. Doing a restart of Apache will always fix this.
```
/etc/init.d/httpd restart
```

Test that you can reach the sync server location and see if anything breaks. These tests will fail if something is wrong such as missing modules or inability to access the database.

http://host.example.com/mozilla-sync/
^ should return 404 if working - anything else indicated error

http://host.example.com/mozilla-sync/user/1.0/a/
^ should return 200 with the page output of only 0

There is also a heartbeat page that can be useful when doing automated checks of the service health, although I found it possible to sometimes break the server in ways that would stop sync for Firefox, but still show OK for heartbeat.
```
http://host.example.com/mozilla-sync/__heartbeat__
```
If you experience any issues with the test URLs, check /var/log/httpd/*error_log*. You may experience problems if you’re using https:// with self-signed certificates that aren’t installed in the browser as trusted too, so import your certs properly so they’re trusted.
Mozilla Sync Server is now ready for you to start using with Firefox clients. My recommendation is to use a clean profile you can delete and re-create for testing purposes and only add sync with your actual profile once you’ve confirmed the server is working.

Using MySQL instead of SQLite:

I tend to standardise on using MySQL where possible for all my web service applications since I have better and more robust monitoring and backup tools for MySQL databases.

If you want to setup Mozilla Sync Server to use MySQL, it’s best to get it working with SQLite first and then try with MySQL to ensure you don’t have any issues with the basic setup before doing more complex bits.

Obviously the first step should be to setup MySQL server, if you haven’t done this yet, the following command will set it up and take you through a secure setup process to password protect the root DB accounts:
```
yum install -y mysql-server
/etc/init.d/mysqld start
chkconfig --level 345 mysqld on
/usr/bin/mysql_secure_installation
```

Once the MySQL server is running, you’ll need to create a database and user for Mozilla Sync Server to use – this can be done with:

mysql -u root -p
# or without -p if no MySQLroot password
CREATE DATABASE mozilla_sync;
GRANT ALL PRIVILEGES ON mozilla_sync.* TO mozilla_sync@localhost IDENTIFIED BY  'examplepassword';
flush privileges;
\q

Copy the [storage] and [auth] sections from /etc/mozilla-sync-server/sample-configs/mysql.conf to replace the same sections in /etc/mozilla-sync-server/mozilla-sync-server.conf. The syntax for the sqluri line is:
```
sqluri = mysql://mozilla_sync:examplepassword@localhost:3306/mozilla_sync
```
Restart Apache (very important, failing todo so will not apply config changes):
```
/etc/init.d/httpd restart
```
Complete! Test from a Firefox client and check table structure is created with SHOW TABLES; MySQL query to confirm successful configuration.

Other Databases

I haven’t done any packaging or testing for it, but Mozilla Sync Server also supports memcached as a storage database, there is a sample configuration file supplied with the RPMs I’ve built, but you may need to also built some python26 modules to support it.

Other Platforms?

If you want to package for another platform, the best/most accurate resource on configuring the sync server currently is one by Fabian Wenk about running it on FreeBSD.

I haven’t seen any guides to packaging the application, the TL;DR version is that you’ll essentially need server-full, server-core, server-reg and server-storage, plus all the other python-module dependencies – take a look at the RPM specfiles to get a good idea.

I’ll hopefully do some Debian packages in the near future too, will have to work on improving my deb packaging foo.

Warnings, issues, small print, etc.

These packages are still quite beta, they’ve only been tested by me so far and there’s possibly some things in them that are wrong.

I want to go through and clean up some of the Python module RPMs at some stage as I don’t think the SPEC files I have are as portable as they should be, commits back always welcome. ;-)

If you find these packages useful, please let me know in comments or emails, always good to get an idea how people find this stuff and whether it’s worth the late nighters. ;-)

And if you have any problems, feel free to email me or comment on this page and I’ll help out the best I can – I suspect I’ll have to write a Mozilla Sync Server troubleshooting guide at some stage sooner or later.

apt-get install debian

11 Replies

Early last year I wrote how I was concerned about the progress and future of the CentOS project and was considering other options.

As of today, I’ve now shifted my primary workstation (Lenovo X201i laptop) from the somewhat out of date Fedora 13 over to Debian Stable/Squeeze.

The main drives for this change were:

Fedora 13 was out-of-date and lacking security fixes, hence requirement to upgrade.
The logical replacement, Fedora 16, now ships with GNOME 3 which I found to be buggy and a regression to my work flow and requirements (not going into details here, but essentially issues with dual screens, workspaces and customization of toolbars).
Desire to seriously try Debian as a primary system with the purpose of evaluating it’s suitability as a replacement for my CentOS servers.
Requirement for a distribution that made major release upgrades feasible (Fedora can do version jumps, but not recommended, making it a tricky process to find time to do a laptop upgrade/rebuild).
Distribution standardisation across my server & workstation environments.
I needed a full reinstall in order to downsize from a 320GB HDD to a 120GB SSD.
Reliability – my laptop is my primary business machine, if it doesn’t work, I’m going to be living on instant noodles until it starts working. Or even worse, work will buy me a Macbook to use like everyone else. :-/

I chose Debian particularly, since it would be a fine option for replacing my CentOS servers in the future with long life support & stability, it’s large package selection and the fact that it’s committed to freedom and openness (as is Fedora also); all of which made it more attractive than Ubuntu for me, which feels much more desktop and fast release focused.

So far, I’m loving it – the distribution is solid, well built and developer friendly, and the package selection is pretty decent, not to mention apt being nice and snappy (although the SSD sure helps there ;-)

I’ve had a couple minor issues relating to my Lenovo hardware that I’ve been able to resolve and have gotten into building a few Debian packages in order to backport newer versions of programs like Firefox/Iceweasel.

From what I’ve observed with playing with Debian today is that’s a pretty awesome distribution, but not entirely as perfect as some of it’s users like to make out:

The installer is a bit dated – not due to the text installer (I fucking love text installers! \m/), but rather due to it’s lack of support for WPA/2 wifi access points and also the ability to allow the user to make broken systems without warning (eg no /boot partition when you don’t have enough coffee like me).
Debian is often credited for having all the packages under the sun in it – whilst almost true, I did note that a number of my more obscure package choices didn’t exist, sending me running for my compiler a few times.
It would be nice if stable backports tracked some of the common packages that users like updated on older systems – programs like web browsers, instant messengers and maybe even kernels for uncooperative hardware. A user could avoid this by using Debian testing, but there are valid reasons to use stable + some backports over using testing or unstable.
I think rpm has nicer command line options for directly working with installed or to-be-installed packages than dpkg. Having said that, some of this could be user familiarity/likes, so time will tell as I use it more.

Over all though, these are minor issues – I think it’s a fantastic distribution with so much working out of the box, applications appearing well tested and good online documentation and resources.

I’m currently running trials and comparisons of Debian with my CentOS hosts with a view for replacing my current CentOS 5 instances with Debian Stable instances over a phased migration period, as well as testing features like LDAP authentication and KVM, but it’s looking pretty damn good so far.

At this stage I’ve only used CentOS 6 as a KVM host platform and it seems unlikely I’ll end up deploying any CentOS 6 VMs with all the security update release slowness. With only a couple servers on CentOS 6 altogether, I’m pondering switching them over to Debian sooner rather than later to reduce maintenance efforts.

[FYI, this post isn’t intended as an attack/demands at the CentOS developers, rather it’s recognizing they’re a volunteer team and probably lacking resource – and I thank them for their efforts, but it appears long term it’s not a good option for my requirements.]

It does also raise questions about Red Hat’s RHEL future with engineers like myself – with Red Hat no longer offering a free-as-in-beer-with-no-support option and CentOS being too slow, more geeks like myself might move to Debian. And if we do so, when the next enterprise project comes along, will we be recommending RHEL or Debian?

Red Hat offers the advantage of commercial support, but for a company with their own engineers, Debian may be more appropriate and budget friendly.

DHCP, I/O and other virtualisation fun with KVM

4 Replies

I recently shifted from having two huge server racks down to having a single speedy home server running KVM virtual machines, with the intent of packaging all my servers – experimental, development, staging, etc, into a single reliable system which will reduce power and maintenance costs.

As part of this change, I went from having dedicated DHCP & DNS servers to having everything located onto the KVM host.

The design I’ve used, has the host OS running with minimal services – the host just runs KVM, OpenVPN, DHCP and a DNS caching nameserver – all other services run as guest VMs, with a virtual network for the guests and host to communicate over.

Guests run as DHCP clients – this makes it easy to assign or adjust addressing if needed and get their information from the host OS.

However this does mean you can’t get away with hammering the host too badly – for example, running an I/O and network intensive backup can cause some interesting problems when you also need the host for services, such as DHCP.

Take a look at the following log messages from a mostly idle VM – these were taken whilst another VM on the server was running a bonnie++ process to test performance:

Mar  6 10:18:06 virtguest dhclient: 5 bad udp checksums in 5 packets
Mar  6 10:18:27 virtguest dhclient: DHCPREQUEST on eth0 to 10.8.12.1 port 67
Mar  6 10:18:45 virtguest dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Mar  6 10:19:00 virtguest dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Mar  6 10:19:07 virtguest dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Mar  6 10:19:15 virtguest dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Mar  6 10:19:15 virtguest dhclient: 5 bad udp checksums in 5 packets

That’s some messed up stuff – what you’re seeing is that the guest VM is trying to renew the DHCP address with the host server – but the host is so sluggish with having to run the I/O intensive virtual machine that is actually corrupting or dropping the UDP packets, preventing the guest VM from renewing it’s address.

This of course raises the most important question: What happens if the guest can’t renew it’s IP address?

In this case, the Linux/CentOS 5 guest VM actually completely lost it’s IP address after a long period of DHCPREQUEST attempts, fell off the network entirely and caused my phone to go nuts with Nagios alerts.

Now of course in any sane production environment, nobody would be running a bonnie++ processes on a VM on an active server – however there’s some pretty key points still made here:

The isolation is a lie: Guests are only *somewhat* isolated from one another – one guest can still mess with another and effectively denial-of-service attack the other VMs by utilising all the available resources.
Guests can be jerks: Organisations running KVM (or some other systems) with untrusted guest VMs should carefully consider how they are going to monitor and protect the service from users running crazily resource intensive processes. (after all, there will be someone who wants to bonnie++ test their new VM simply for the lols).
cgroups to the rescue? Linux cgroups does have an I/O controller (blkio-cgroup) although whilst this controls read/write flow, it won’t restrict seeks which can also badly impact spinning rust based servers.
WTF DHCP? The approach of the guests simply dropping their DHCP address after losing contact with the DHCP server is a pretty bad design limitation – if the DHCP server is unreachable, it should keep the original address (of course if the “physical” ethernet connection dropped, that would be a different situation, and it should drop it’s address to match).
Also: I wonder what OSes/distributions have the above behavior?

I’m currenting running a number of bonnie++ tests on my KVM server and will have a blog post in the near future detailing these findings in more detail, I’m also planning to look into cgroups and other resource control and limiting functions and will report back on how these fare when you have guest VMs running heavy processes.

Overall it made my weekend of geekery that bit more exciting. :-D

CentOS, RHEL and future possibilities?

10 Replies

Those who know me will know that I’m a long term CentOS user – this actually started from my love of RHEL, back in my early Linux using days when I was running Red Hat 8.0.

Whilst it made financial sense for Red Hat to switch to making their product only available in binary form for their customers, at the same time I can’t help but feel this has damaged the appeal of Red Hat for geeks like myself – I’m no longer able to setup friends, family or customers without the funds for RHEL with a quality, enterprise-grade free (as in beer + freedom) distribution.

I do wonder if this contributes to reduced market awareness in the small business space and also whether it reduces the likeliness of geeks like myself promoting the software – after all, if I can’t run RHEL myself, I’m likely to look at other distributions and options and end up promoting those.

With the lack of a free Red Hat enterprise-grade distribution, there are only a couple options for wanting a Red Hat-style experience:

Fedora – the community developed distribution that forms the future base of RHEL, a fantastic distribution in it’s own right, but with only 12 months support per release, not suitable for server deployments.
CentOS – the community free re-spin of RHEL with their trademarks removed to make it legally redistributable.

I’ve been using CentOS heavily on my servers and Fedora on my workstations, however there are a number of security delays that are concerning me about CentOS which have been recently highlighted in an LWN article.

Essentially, the core problem is that the latest version of CentOS is still only 5.5, whilst Red Hat have had 5.6 out for some time, with numerous security updates in it that have yet to be released for CentOS…..

Having systems vulnerable to known exploits with no upstream patches is always a pretty serious concern to any system administrator…. this is leading me to re-think my usage of CentOS and to consider whether I should consider other platforms.

I’ve never been a huge fan of Debian in the past, but I’m considering giving it a more detailed look and try – Debian has the advantages of a strong community (like Fedora has) but without the limitation of a short support life – although then again, Debian’s releases and support spans are a little less rigid than Red Hat, which is somewhat annoying.

There’s a few server platforms that come to mind – Ubuntu LTS, Mint Linux, Debian, Open/SuSe or of course, Fedora.

The other option is that I could spin my own distribution – based on the number of custom RPMs I already build, rebuilding Red Hat’s update packages for my own needs wouldn’t be too hard, but I really don’t want to get caught up in distribution maintenance for the next 5 years plus it’s not suitable for customer deployments – so even if I decide that a custom built system is best for me, it still doesn’t solve the “what do I install for others?” question.

Maybe I need Fedora LTS – long term support for specific versions of Fedora – 3 or 5 years would be wonderful and meet the needs of server administrators.

This was tried once before, with the Fedora Legacy project, but it didn’t last long – possibly the goal of supporting *all* the releases was too much to reasonably handle, so an approach of selection even/odd number releases only might make it more feasible – I know that I’d certain be willing to contribute.

Anyway, this is a late night concerned system administrator brain dump about the problem, interested in thoughts and comments from others here about distributions they use/would consider in the server environment.