Tag Archives: x201i

How Jethro Geeks – IRL

12 Replies

A number of friends are always quite interested in how my personal IT infrastructure is put together, so I’m going to try and do one post a week ranging from physical environments, desktop, applications, server environments, monitoring and architecture.

Hopefully this is of interest to some readers – I’ll be upfront and advise that not everything is perfect in this setup, like any large environment there’s always ongoing upgrade projects, considering my environment is larger than some small ISPs it’s not surprising that there’s areas of poor design or legacy components, however I’ll try to be honest about these deficiencies and where I’m working to make improvements.

If you have questions or things you’d like to know my solution for, feel free to comment on any of the posts in this series. :-)

 

Today I’m examining my physical infrastructure, including my workstation and my servers.

After my move to Auckland, it’s changed a lot since last year and is now based around my laptop and gaming desktop primarily.

All the geekery, all the time

This is probably my most effective setup yet, the table was an excellent investment at about $100 off Trademe, with enough space for 2 workstations plus accessories in a really comfortable and accessible form factor.

 

My laptop is a Lenovo Thinkpad X201i, with an Intel Core i5 CPU, 8GB RAM, 120GB SSD and a 9-cell battery for long run time. It was running Fedora, but I recently shifted to Debian so I could upskill on the Debian variations some more, particularly around packaging.

I tend to dock it and use the external LCD mostly when at home, but it’s quite comfortable to use directly and I often do when out and about for work – I just find it’s easier to work on projects with the larger keyboard & screen so it usually lives on the dock when I’m coding.

This machine gets utterly hammered, I run this laptop 24×7, typically have to reboot about once every month or so, usually from issues resulting with a system crash from docking or suspend/resume – something I blame the crappy Lenovo BIOS for.

 

I have an older desktop running Windows XP for gaming, it’s a bit dated now with only a Core 2 Duo and 3GB RAM – kind of due for a replacement, but it still runs the games I want quite acceptably, so there’s been little pressure to replace – plus since I only really use it about once a week, it’s not high on my investment list compared to my laptop and servers.

Naturally, there are the IBM Model M keyboards for both systems, I love these keyboards more than anything (yes Lisa, more than anything <3 ) and I’m really going to be sad when I have to work in an office with other people again whom don’t share my love for loud clicky keyboards.

The desk is a bit messy ATM with several phones and routers lying about for some projects I’ve been working on, I’ll go through stages of extreme OCD tidiness to surrendering to the chaos… fundamentally I just have too much junk to go on it, so trying to downsize the amount of stuff I have. ;-)

 

Of course this is just my workstations – there’s a whole lot going on in the background with my two physical servers where the real stuff happens.

A couple years back, I had a lab with 2x 42U racks which I really miss. These days I’m running everything on two physical machines running Xen and KVM virtualisation for all services – it was just so expensive and difficult having the racks, I’d consider doing it again if I brought a house, but when renting it’s far better to be as mobile as possible.

The primary server is my colocation box which runs in a New Zealand data center owned by my current employer:

Forever Alone :'( [thanks to my colleagues for that]

It’s an IBM xseries 306m, with 3.0Ghz P4 CPU, 8GB of RAM and 2x 1TB enterprise grade SATA drives, running CentOS (RHEL clone). It’s not the fastest machine, but it’s more than speedy enough for running all my public-facing production facing services.

It’s a vendor box as it enabled me to have 3 yrs onsite NBD repair support for it, these days I have a complete hardware spare onsite since it’s too old to be supported by IBM any longer.

To provide security isolation and easier management, services are spread across a number of Xen virtual machines based on type and risk of attack, this machine runs around 8 virtual machines performing different publicly facing services including running my mail servers, web servers, VoIP, IM and more.

 

For anything not public-facing or critical production, there’s my secondary server, which is a “whitebox” custom build running a RHEL/CentOS/JethroHybrid with KVM for virtualisation, running from home.

Whilst I run this server 24×7, it’s not critical for daily life, so I’m able to shut it down for a day or so when moving house or internet providers and not lose my ability to function – having said that, an outage for more than a couple days does get annoying fast….

Mmmmmm my beautiful monolith

This attractive black monolith packs a quad core Phenom II CPU, custom cooler, 2x SATA controllers, 16GB RAM, 12x 1TB hard drives in full tower Lian Li case. (slightly out-of-date spec list)

I’m running RHEL with KVM on this server which allows me to run not just my internal production Linux servers, but also other platforms including Windows for development and testing purposes.

It exists to run a number of internal production services, file shares and all my development environment, including virtual Linux and Windows servers, virtual network appliances and other test systems.

These days it’s getting a bit loaded, I’m using about 1 CPU core for RAID and disk encryption and usually 2 cores for the regular VM operation, leaving about 1 core free for load fluctuations. At some point I’ll have to upgrade, in which case I’ll replace the M/B with a new one to take 32GB RAM and a hex-core processor (or maybe octo-core by then?).

 

To avoid nasty sudden poweroff issues, there’s an APC UPS keeping things running and a cheap LCD and ancient crappy PS/2 keyboard attached as a local console when needed.

It’s a pretty large full tower machine, so I except to be leaving it in NZ when I move overseas for a while as it’s just too hard to ship and try and move around with it – if I end up staying overseas for longer than originally planned, I may need to consider replacing both physical servers with a single colocated rackmount box to drop running costs and to solve the EOL status of the IBM xseries.

 

The little black box on the bookshelf with antennas is my Mikrotik Routerboard 493G, which provides wifi and wired networking for my flat, with a GigE connection into the server which does all the internet firewalling and routing.

Other than the Mikrotik, I don’t have much in the way of production networking equipment – all my other kit is purely development only and not always connected and a lot of the development kit I now run as VMs anyway.

 

Hopefully this is of some interest, I’ll aim to do one post a week about my infrastructure in different areas, so add to your RSS reader for future updates. :-)

Lenovo & tp-fan fun

Leave a reply

I quite like my Lenovo X201i laptop, I’ve been using it for a couple years now and it’s turned out to be the ideal combination of size and usability – the 12″ form factor means I can carry it around easily enough, it has plenty of performance (particularly since I upgraded it to an SSD and 8GB of RAM) and I can see myself using it for the foreseeable future.

Unfortunately it does have a few issues… the crappy “Thinkpad Wireless” default card that comes in it caused me no end of headaches and the BIOS has always been a

Thankfully most of the major BIOS flaws have been resolved in part due to subsequent updates, but also thanks to the efforts of the Linux kernel developers to work around weird bits of the BIOS’s behavior.

Sadly not all issues have been resolved, in particular, the thermal management is still flawed and fails to adequately handle the maximum heat output of the laptop. I recently discovered that when you’re unfortunate enough to run some very CPU intensive single-threaded processes, by keeping 1/4 cores at 100% for an extended period of time the Lenovo laptop will overheat and issue an emergency thermal shutdown to the OS.

During this time the fan increases in speed, but still has quite a low noise level and airflow volume, which is very hot to the touch, it appears the issue is due to the Lenovo BIOS not ramping the fan speed up high enough to meet the heat being produced.

Thanks to the excellent Thinkwiki site, there’s detailed information on how you can force specific fan speeds using the thinkpad_acpi kernel module, as well as details on various scripts and fan control solutions people have written.

What’s interesting is that when running the fan on level 7 (the maximum speed), the fan still doesn’t spin particularly fast or loudly, no more than when the overheating occurs. But reading the wiki shows that there is a “disengaged” mode, where the fan will run at the true maximum system speed.

It appears to me that the BIOS has the 100% speed setting for the fan set at too low a threshold, the smart fix would be to correct the BIOS so that 100% is actually the true maximum speed of the fan and to scale up slowly to keep the CPU at a reasonable temperature.

In order to fix it for myself, I obtained the tp-fan program, which runs a python daemon to monitor and adjust the fan speeds in the system based on the configured options. Sadly it’s not able to scale between “100%” and “disengaged” speeds, meaning I have the choice of quiet running or loud running but no middle ground.

Thanks to tpfan’s UI, I was able to tweak the speed positions until I obtained the right balance, the fans will now run at up to 100% for all normal tasks, often sitting just under 50 degrees at 60% fan speed.

When running a highly CPU intensive task, the fan will jump up to the max speed and run at that until the temperature drops sufficiently.  In practice it’s worked pretty well, I don’t get too much jumping up and down of the fan speed and my system hasn’t had any thermal shutdowns since I started using it.

Whilst it’s clearly a fault with the Lenovo BIOS not handling the fans properly, it raised a few other questions for me:

  • Why does the OS lack logic to move CPU intensive tasks between cores? Shuffling high intensive loads between idle cores would reduce the heat and require less active cooling by the system fans – even on a working system that won’t overheat, this would be a good way to reduce power consumption.
  • Why doesn’t the OS have a feature to throttle the CPU clock speed down as the CPU temperature rises? It would be better than having the all or nothing approach that it currently enforces, better to have a slower computer than a fried computer.

Clearly I need some more free time to start writing kernel patches for my laptop, although I fear what new dangerous geeky paths this might lead me into. :-/

apt-get install debian

11 Replies

Early last year I wrote how I was concerned about the progress and future of the CentOS project and was considering other options.

As of today, I’ve now shifted my primary workstation (Lenovo X201i laptop) from the somewhat out of date Fedora 13 over to Debian Stable/Squeeze.

The main drives for this change were:

  • Fedora 13 was out-of-date and lacking security fixes, hence requirement to upgrade.
  • The logical replacement, Fedora 16, now ships with GNOME 3 which I found to be buggy and a regression to my work flow and requirements (not going into details here, but essentially issues with dual screens, workspaces and customization of toolbars).
  • Desire to seriously try Debian as a primary system with the purpose of evaluating it’s suitability as a replacement for my CentOS servers.
  • Requirement for a distribution that made major release upgrades feasible (Fedora can do version jumps, but not recommended, making it a tricky process to find time to do a laptop upgrade/rebuild).
  • Distribution standardisation across my server & workstation environments.
  • I needed a full reinstall in order to downsize from a 320GB HDD to a 120GB SSD.
  • Reliability – my laptop is my primary business machine, if it doesn’t work, I’m going to be living on instant noodles until it starts working. Or even worse, work will buy me a Macbook to use like everyone else. :-/

I chose Debian particularly, since it would be a fine option for replacing my CentOS servers in the future with long life support & stability, it’s large package selection and the fact that it’s committed to freedom and openness (as is Fedora also); all of which made it more attractive than Ubuntu for me, which feels much more desktop and fast release focused.

So far, I’m loving it – the distribution is solid, well built and developer friendly, and the package selection is pretty decent, not to mention apt being nice and snappy (although the SSD sure helps there ;-)

I’ve had a couple minor issues relating to my Lenovo hardware that I’ve been able to resolve and have gotten into building a few Debian packages in order to backport newer versions of programs like Firefox/Iceweasel.

From what I’ve observed with playing with Debian today is that’s a pretty awesome distribution, but not entirely as perfect as some of it’s users like to make out:

  • The installer is a bit dated – not due to the text installer (I fucking love text installers! \m/), but rather due to it’s lack of support for WPA/2 wifi access points and also the ability to allow the user to make broken systems without warning (eg no /boot partition when you don’t have enough coffee like me).
  • Debian is often credited for having all the packages under the sun in it – whilst almost true, I did note that a number of my more obscure package choices didn’t exist, sending me  running for my compiler a few times.
  • It would be nice if stable backports tracked some of the common packages that users like updated on older systems – programs like web browsers, instant messengers and maybe even kernels for uncooperative hardware. A user could avoid this by using Debian testing, but there are valid reasons to use stable + some backports over using testing or unstable.
  • I think rpm has nicer command line options for directly working with installed or to-be-installed packages than dpkg. Having said that, some of this could be user familiarity/likes, so time will tell as I use it more.

Over all though, these are minor issues – I think it’s a fantastic distribution with so much working out of the box, applications appearing well tested and good online documentation and resources.

I’m currently running trials and comparisons of Debian with my CentOS hosts with a view for replacing my current CentOS 5 instances with Debian Stable instances over a phased migration period, as well as testing features like LDAP authentication and KVM, but it’s looking pretty damn good so far.

At this stage I’ve only used CentOS 6 as a KVM host platform and it seems unlikely I’ll end up deploying any CentOS 6 VMs with all the security update release slowness. With only a couple servers on CentOS 6 altogether, I’m pondering switching them over to Debian sooner rather than later to reduce maintenance efforts.

[FYI, this post isn’t intended as an attack/demands at the CentOS developers, rather it’s recognizing they’re a volunteer team and probably lacking resource – and I thank them for their efforts, but it appears long term it’s not a good option for my requirements.]

It does also raise questions about Red Hat’s RHEL future with engineers like myself – with Red Hat no longer offering a free-as-in-beer-with-no-support option and CentOS being too slow, more geeks like myself might move to Debian. And if we do so, when the next enterprise project comes along, will we be recommending RHEL or Debian?

Red Hat offers the advantage of commercial support, but for a company with their own engineers, Debian may be more appropriate and budget friendly.