This is what happens when you’re a computer geek for 10+ yrs…..
Lost in the endless boxes of cables
I might have too many cables….
3 Replies
Tag Archives: geek
Android alarm UI WTF
I like Android, but there are a few times the UX (User Experience) is a bit messed up compared with the way the user thinks. For example, take the newly introduced alarm clock time selection interface added in ICS:
So the alarm time selection gives me the ability to drag the time up/down, simple enough, most users understand dragging on touch screen devices. However if one decides to tap the up/down buttons instead…
I guess the developer decided that the up button should increase the time and the down should decrease which would make sense if it wasn’t for the fact that the user can see the preceding and following numbers which changes their perspective from the arrows being for numerical incrementing to the arrows being for sliding/rotating the displayed numbers on screen.
It’s even more annoying since it worked logically on pre-ICS devices only to be changed and broken in this confusing manner. :-(
Home is where my computer is
Jack asked me yesterday about the ISA video card in the background, by my TV.
Home sweet home! Click the image for a larger version if you want to read the PCB.
I found it when at the Broadbeach Markets, in Gold Coast, Brisbane, Australia one year at a random stall, I don’t normally collect much in the way of trinkets, but this is one of the few that I have. :-)
gdisk, oh glorious gdisk
My file server virtual machine passed the 2TB limit a couple months ago, which forced me to get around to upgrading it to RHEL 6 and moving from MSDOS to GPT based partitions, as the MSDOS partitioning table doesn’t support more than 2TB partitions.
I recently had to boost it up by another 1 TB to counter growing disk usage and got stuck trying to resize the physical volume – the trusty old fdisk command doesn’t support GPT partitions, with most documentation resources directing you to use parted instead.
The problem with parted, is that the developers have tried to be clever and made parted filesystem aware, so it will perform filesystem operations as well as block partition operations. Secondly, parted writes changes whilst you’re making them rather than letting you discard or write the final results of your changes to the partition table.
This breaks really badly for my LVM physical volume partitions – as you can see below, parted has a resize command, but when used against an LVM volume it is unable to recognize it as a known type and fails with the very helpful “Error: Could not detect file system“.
Naturally this didn’t put parted into my good books for the evening – doing a search of the documentation didn’t really clarify whether doing the old fdisk way or deleting and re-creating partitions at the same start and end positions was safe or not, but the documentation suggested that this is a destructive process. Seeing as I really didn’t feel like have to pull 2TB of data off backup, I chose caution and decided not to test that poorly documented behavior.
The other suggested option is to just add an additional partition and add it to LVM – whilst there’s no technical reason against this method, it really offended my OCD and the desire to keep my server’s partitioning table simple and logical – I don’t want lots of weirdly sized partitions littering the server from every time I’ve had to upsize the virtual machine!
Whilst cursing parted, I wondered whether there was a tool just like fdisk, but for GPT partition tables. Linux geeks do like to poke fun at fdisk for having a somewhat obscure user interface and basic feature set, but once you learn it, it’s a powerful tool with excellent documentation and it’s simplicity leads it to being able to perform a number of very tricky tasks, as long as the admin knows what they’re doing.
Doing some research lead me to gdisk, which as the name suggests, is a GPT capable clone of fdisk, providing a very similar user interface and functionality.
Whilst it’s not part of RHEL’s core package set, it is available in the EPEL repositories, hopefully these are acceptable to your environment:
Once installed, it was a pretty simple process of loading gdisk and deleting the partition before expanding to the new size:
Most important is to verify that the start sector hasn’t changed between deleting the old partition and adding the new one – as long as these are the same and the partition is the same size or larger than the old one, everything will be OK.
Save and apply the changes:
On my RHEL 6 KVM virtio VM, I wasn’t able to get the OS to recognize the new partition size, even after running partprobe, so I had to reboot the VM.
Once rebooted, it was a simple case of issuing pvresize and pvdisplay to confirm the new physical volume size – from there, I can then expand LVM logical volumes as desired.
Note that pvresize is a bit annoying in that it won’t show any unallocated space – what is means by free PE, is free physical extents, disk that the LVM physical volume already occupies but which isn’t allocated to logical volumes yet. Until you run pvresize, you won’t see any change to the size of the volume.
So far gdisk looks pretty good, I suspect it will become a standard on my own Linux servers, but not being in the base RHEL repositories will limit usage a bit on commercial and client systems, which often have very locked down and limited package sets.
The fact that I need a partition table at all with my virtual machines is a bit of a pain, it would be much nicer if I could just turn the whole /dev/vda drive into a LVM physical volume and then boot the VM from an LVM partition inside the volume.
As things currently stand, it’s necessary to have a non-LVM /boot partition, so I have to create one small conventional partition for boot and a second partition consuming all remaining disk for actual data.
nagios check_disk trap
Let’s play spot the difference:
[root@localhost ~]# /usr/lib64/nagios/plugins/check_disk -w 20 -c 10 -p /home
DISK OK – free space: /home 111715 MB (4% inode=99%);| /home=2498209MB;2609905;2609915;0;2609925
[root@localhost ~]# /usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /home
DISK CRITICAL – free space: /home 111715 MB (4% inode=99%);| /home=2498209MB;2087940;2348932;0;2609925
Make sure you that you define your units of disk or add % to your Nagios checks otherwise you might suddenly find yourself running to add more disk….
virt-viewer remote access tricks
Sometimes I need to connect directly to the console of my virtual machines, typically this is usually when working with development or experimental VMs where SSH/RDP/VNC isn’t working for whatever reason, or when I’m installing a new OS entirely.
To view virtual machines using libvirt (by both KVM or Xen), you use the virt-viewer command, this launches a window and establishes a VNC or SPICE connection into the virtual machine.
Historically I’ve just run this by SSHing into the virtual machine host and then using X11 forwarding to display the virtual machine window on my laptop. However this performs really badly on slow connections, particularly 3G where it’s almost unusable due to the design of X11 forwarding not being particularly efficient.
However virt-viewer has the capability to run locally and connect to a remote server, either directly to the libvirt daemon, or via an SSH tunnel. To do the latter, the following command will work for KVM (qemu) based hypervisors:
virt-viewer --connect qemu+ssh://user@host.example.com/system vmnamehere
With the above, you’ll have to enter your SSH password twice – first to establish the connection to the hypervisor and secondly to establish a tunnel to the VM’s VNC/SPICE session – you’ll probably quickly decide to get some SSH keys/certs setup to prevent annoyance. ;-)
This performs way faster than X11 forwarding, plus the UI of virt-manager stays much more responsive, including grabbing/ungrabbing of the local keyboard/mouse, even if the connection or server is lagging badly.
If you’re using Xen with libvirt, the following should work (I haven’t tested this, but based on the man page and some common sense):
virt-viewer --connect xen+ssh://user@host.example.com/ vmnamehere
If you wanted to open up the right ports on your server’s firewall and are sending all traffic via a secure connection (eg VPN), you can drop the +ssh and use –direct to connect directly to the hypervisor and VM without port forwarding via SSH.
How Jethro Geeks – IRL
A number of friends are always quite interested in how my personal IT infrastructure is put together, so I’m going to try and do one post a week ranging from physical environments, desktop, applications, server environments, monitoring and architecture.
Hopefully this is of interest to some readers – I’ll be upfront and advise that not everything is perfect in this setup, like any large environment there’s always ongoing upgrade projects, considering my environment is larger than some small ISPs it’s not surprising that there’s areas of poor design or legacy components, however I’ll try to be honest about these deficiencies and where I’m working to make improvements.
If you have questions or things you’d like to know my solution for, feel free to comment on any of the posts in this series. :-)
Today I’m examining my physical infrastructure, including my workstation and my servers.
After my move to Auckland, it’s changed a lot since last year and is now based around my laptop and gaming desktop primarily.
All the geekery, all the time
This is probably my most effective setup yet, the table was an excellent investment at about $100 off Trademe, with enough space for 2 workstations plus accessories in a really comfortable and accessible form factor.
My laptop is a Lenovo Thinkpad X201i, with an Intel Core i5 CPU, 8GB RAM, 120GB SSD and a 9-cell battery for long run time. It was running Fedora, but I recently shifted to Debian so I could upskill on the Debian variations some more, particularly around packaging.
I tend to dock it and use the external LCD mostly when at home, but it’s quite comfortable to use directly and I often do when out and about for work – I just find it’s easier to work on projects with the larger keyboard & screen so it usually lives on the dock when I’m coding.
This machine gets utterly hammered, I run this laptop 24×7, typically have to reboot about once every month or so, usually from issues resulting with a system crash from docking or suspend/resume – something I blame the crappy Lenovo BIOS for.
I have an older desktop running Windows XP for gaming, it’s a bit dated now with only a Core 2 Duo and 3GB RAM – kind of due for a replacement, but it still runs the games I want quite acceptably, so there’s been little pressure to replace – plus since I only really use it about once a week, it’s not high on my investment list compared to my laptop and servers.
Naturally, there are the IBM Model M keyboards for both systems, I love these keyboards more than anything (yes Lisa, more than anything <3 ) and I’m really going to be sad when I have to work in an office with other people again whom don’t share my love for loud clicky keyboards.
The desk is a bit messy ATM with several phones and routers lying about for some projects I’ve been working on, I’ll go through stages of extreme OCD tidiness to surrendering to the chaos… fundamentally I just have too much junk to go on it, so trying to downsize the amount of stuff I have. ;-)
Of course this is just my workstations – there’s a whole lot going on in the background with my two physical servers where the real stuff happens.
A couple years back, I had a lab with 2x 42U racks which I really miss. These days I’m running everything on two physical machines running Xen and KVM virtualisation for all services – it was just so expensive and difficult having the racks, I’d consider doing it again if I brought a house, but when renting it’s far better to be as mobile as possible.
The primary server is my colocation box which runs in a New Zealand data center owned by my current employer:
Forever Alone :'( [thanks to my colleagues for that]
It’s a vendor box as it enabled me to have 3 yrs onsite NBD repair support for it, these days I have a complete hardware spare onsite since it’s too old to be supported by IBM any longer.
To provide security isolation and easier management, services are spread across a number of Xen virtual machines based on type and risk of attack, this machine runs around 8 virtual machines performing different publicly facing services including running my mail servers, web servers, VoIP, IM and more.
For anything not public-facing or critical production, there’s my secondary server, which is a “whitebox” custom build running a RHEL/CentOS/JethroHybrid with KVM for virtualisation, running from home.
Whilst I run this server 24×7, it’s not critical for daily life, so I’m able to shut it down for a day or so when moving house or internet providers and not lose my ability to function – having said that, an outage for more than a couple days does get annoying fast….
Mmmmmm my beautiful monolith
This attractive black monolith packs a quad core Phenom II CPU, custom cooler, 2x SATA controllers, 16GB RAM, 12x 1TB hard drives in full tower Lian Li case. (slightly out-of-date spec list)
I’m running RHEL with KVM on this server which allows me to run not just my internal production Linux servers, but also other platforms including Windows for development and testing purposes.
It exists to run a number of internal production services, file shares and all my development environment, including virtual Linux and Windows servers, virtual network appliances and other test systems.
These days it’s getting a bit loaded, I’m using about 1 CPU core for RAID and disk encryption and usually 2 cores for the regular VM operation, leaving about 1 core free for load fluctuations. At some point I’ll have to upgrade, in which case I’ll replace the M/B with a new one to take 32GB RAM and a hex-core processor (or maybe octo-core by then?).
To avoid nasty sudden poweroff issues, there’s an APC UPS keeping things running and a cheap LCD and ancient crappy PS/2 keyboard attached as a local console when needed.
It’s a pretty large full tower machine, so I except to be leaving it in NZ when I move overseas for a while as it’s just too hard to ship and try and move around with it – if I end up staying overseas for longer than originally planned, I may need to consider replacing both physical servers with a single colocated rackmount box to drop running costs and to solve the EOL status of the IBM xseries.
The little black box on the bookshelf with antennas is my Mikrotik Routerboard 493G, which provides wifi and wired networking for my flat, with a GigE connection into the server which does all the internet firewalling and routing.
Other than the Mikrotik, I don’t have much in the way of production networking equipment – all my other kit is purely development only and not always connected and a lot of the development kit I now run as VMs anyway.
Hopefully this is of some interest, I’ll aim to do one post a week about my infrastructure in different areas, so add to your RSS reader for future updates. :-)
MS Volume License Service Center
Occasionally I have to touch Microsoft software, thankfully most of our customers have their licenses entered into the Volume Licensing Service Center these days which makes finding the install media a lot easier than rummaging through CD wallets in the office.
The volume license center isn’t perfect by a long shot but for the most part it’s a pretty effective way of getting keys and software downloads for purchased software [1], with the glaring exception of a major defect with the download functionality:
So close, yet so far
The download interface helpfully gives you some advice to use a download manager – because lets face it, browser downloaders suck universally – Firefox, Chrome and IE all have poor quality download functionality.
However by download manager, Microsoft actually mean “some Microsoft application you should download and run to download the file”. I’ve *never* had a good experience with vendor download managers, not to mention the fact I’m wanting to download this file to a GUI-less Linux KVM host, so this option is right out.
The next logical option is to download with the browser and just grab the download URL – however as shown above, clicking the download button won’t provide a real URL it instead runs a bit of javascript which then directs the browser to the actual download URL.
It’s not uncommon behavior, but it’s damn annoying – browsers know how to download a file, you don’t need javascript to make it happen and it breaks the ability to copy and paste the link directly into a download manager.
When annoying companies use javascript to obfuscate the download URL, the next trick is to start the download with the browser, then go to the download window and copy the real URL out from there.
However, this still fails:
Thou shalt not pass!
Looks like Microsoft is doing some clever checks, possibly with cookies, user agent and IP, or some combination of all of above and refuses the download manager you’ve chosen. >:-(
With a bit of digging around it would be possible to make a solution to work around this, but it’s a major PITA that they do all this pointless obfuscation and I don’t know if I do enough downloads from there to justify the effort to find and make a proper solution to work around Microsoft’s failings.
What really annoys me, is that I’ve already BROUGHT the product and you need a license key in order to EVEN USE IT after you’ve downloaded – it’s not like someone else is going to figure out the randomly generated download path for my session, download the ISO and somehow get a free copy of Windows Server….
I can go to The Pirate Bay and download Microsoft ISOs in a matter of minutes, there’s no point trying to restrict the download ability of your paying customers, the ISOs are already shared – usually pre-hacked to remove activation.
The result is needing to download the ISO over a DSL line to my workstation and upload it back over that same DSL line (oh god the slowness) to get it to the customer’s server, something which is extremely annoying and wasteful for my data cap.
I wish Microsoft would just make their ISO archive available for download off FTP already. :-(
[1] As a side note, I really, really wish I could just buy MS software online via this center and be done without having to deal with NZ’s resellers who don’t add any value, just overhead to purchasing this stuff.
IRD online services registration
I recently signed up with IRD’s (New Zealand’s Tax Department) online Kiwisaver service, so I could view the status of my payments and balance of New Zealand’s voluntary superannuation scheme.
The user sign up form is pretty depressing (and no, not just because it’s about signing up to tax rather than cool stuff):
The 70s called, they want your security consultants back.
My first concern is passwords being limited to a maximum of 10 characters, it’s way too short for many good passwords (or even better, passphrases), any system should take at least 255 chars without complain.
Secondly, the “forgotten password phrase” is the most stupid thing I’ve ever seen, it’s basically a second password field – if you forget your password, you can contact them and give them this second password…. except that if you’re stupid enough to forget the first password, how the hell are you going to remember a secondary normally never-used password?
I’d also love to know how secure the secondary password phrase requirements are, because since it gives you access into the account, the security is no stronger than whatever you put in here – and how likely are users to choose something good and secure as their “backup phrase”?
This is some pretty simple security concepts and I’m a bit dismayed that IRD managed to get these so wrong – at least it shouldn’t be hard to correct….
Exchange, I will have my revenge!
It’s been a busy few weeks – straight after my visit to Christchurch I got stuck into the main migration phase of a new desktop and server deployment for one of our desktop customers.
It wasn’t a small bit of work, going from 20 independent 7-year old Windows XP desktops to new shiny Windows 7 desktops and moving from Scalix/Linux to Exchange/Win2008R2. It’s not the normal sort of project for me, usually I’ll be dealing with network systems and *nix servers, rather than Microsoft shops, but I had some free time and knew the customer site well so I ended up getting the project.
The deployment was mostly straightforwards, and I intended to blog about this in the near future, I honestly found some of the MS tech such as Active Directory quite nice and it’s interesting comparing the setup compared to what’s possible with the Linux environment.
However I still have no love for Microsoft Exchange, which has to be one of the most infuriating emails systems I’ve had to use. We ended up going with Exchange for this customer due to it working the easiest with their MS-centric environment and providing benefits such as ActiveSync for mobiles in future.
However with myself coming from a Linux background, having grown up with solid and easy to debug and monitor platforms like Sendmail, Postfix and Dovecot, Exchange is an exercise in obscure configuration and infuriating functionality.
To illustrate my point, I’m going to take you on a review of a fault we had with this new setup several days after switching over to the Exchange server…..
* * *
On one particular day, after several days of no problems, the Exchange server suddenly decided it didn’t want to email the upstream smarthost mail server.
The upstream server in question has both IPv4 and IPv6 addresses, something that you tend to want in the 21st century and it’s pretty rare that we have problems with it.
With Exchange 2010 and Windows Server 2008, both components have IPv6 enabled out-of-the-box – we don’t have IPv6 at this particular customer, since the ISP haven’t extended IPv6 beyond the core & colo networks, so we can’t allocate ranges to our customers using them at this stage.
For some unknown reason, the Windows server decided that it would make sense to try connecting to the smart host via IPv6 AAAA record, despite there being no actual upstream IPv6 connection. To make matters worse, it then decided the next most logical thing was to just fail, rather than falling back to the IPv4 A record.
The Windows experts assigned to look at this issue, decided the best solution was to “disable IPv6 in Exchange”, something I assumed meant “tell Exchange not to use IPv6 for smarthosts”.
With the issue resolved, no faults occurring and emails flowing, the issue was checked off as sorted. :-)
Later that night, the server was rebooted to make some changes to the underlying KVM platform – however after rebooting, the Windows server didn’t come back up. Instead it was stuck for almost two hours at “Applying computer settings….” at boot – even once the login screen started, it would still take another 30mins before I could login.
This is the digital equivalent of watching paint dry.
After eventually logging in, the server revealed the cause of the slow startup as being the fault of the “microsoft.exchange.search.exsearch.exe” process running non-stop at 100% CPU.
After killing off that process to get some resemblance of a responsive system, it became apparent that a number of key Exchange components were also not running.
I waded through the maze that is event viewer, to find a number of Exchange errors, in particular one talking about being unable to connect to Active Directory LDAP, with an error of DSC_E_NO_SUITABLE_CDC (Error 0x80040a02, event 2114).
Every time I have to use event viewer I miss syslog, tail and grep even more.
Naturally the first response was to review what changes had been made on the server recently. After confirming that no updates had been made in the last couple of days, the only recent change was the IPv6 adjustment made by the Windows engineers earlier in the day.
Reading up on IPv6 support and Windows Server 2008, I came across this gem on microsoft.com:
"From Microsoft's perspective, IPv6 is a mandatory part of the Windows operating system and it is enabled and included in standard Windows service and application testing during the operating system development process. Because Windows was designed specifically with IPv6 present, Microsoft does not perform any testing to determine the effects of disabling IPv6. If IPv6 is disabled on Windows Vista, Windows Server 2008, or later versions, some components will not function."
I then came across this blog post, from someone who had experienced the same error string, but with different cause. In his post, the author had a handy footnote:
"The biggest red herring I found when troubleshooting this one from articles others had posted was related to IPv6. I see quite a few people suggesting IPv6 is required for Exchange 2007 and 2010. This is NOT true. As a matter of fact, if the server hosting Exchange 2007 or 2010 is a DC, then IPv6 must be enabled otherwise simply uncheck the checkbox in TCP/IP properties on all connected interfaces. You don't need to buggar with the registry to "really disable it"....just uncheck the checkbox."
The customer’s Windows 2008 R2 server is responsible for both running Exchange 2010 as well as Active Directory
To resolve the smart host issues, the Windows team had disabled IPv6 altogether on the interface, resulting in a situation where Exchange was unable to establish a connection to AD to get information needed to startup and run.
To resolve, I simply enabled IPv6 for the server and the Exchange processes correctly started themselves within 10 seconds or so as I watched in the Services utility.
This resolved the “Exchange isn’t functioning at all issue”, but still left me with the smarthost IPv6 issue. To work around the issue for now, I just set the smarthost in Exchange to use the IPv4 address, but will need a better fix long term.
With the issue resolved, some post-incident considerations:
- I’m starting to see more cases where a *lack* of IPv6 is actually causing more problems than the presence of it, particularly around mail servers.
- Exchange has some major architectural issues – I would love to know why an internal communication issue caused the search indexer process to go nuts at 100% CPU for hours.I’ve broken Linux boxes in terrible ways before, particularly with LDAP server outages leaving boxes unable to get any user information – they just error out slowly with timeouts, they don’t go and start chewing up 100% CPU. And I can drop them into a lower run level to fix and reboot within minutes instead of hours.
- I did a search and couldn’t find any official Microsoft best practice documentation for server 2008, nor did Windows Server warn the admin that disabling IPv6 would break key services.
- If Microsoft has published anything like this, it’s certainly not easy to find – microsoft.com is a complete searching disaster. And yes, whilst they have a “best practice analyzer tool”, it’s not really want I want as an admin, I want a doc I can review and check plans against.
- I’m seriously tempted to start adding surcharges for providing support for Microsoft platforms. :-/
* * *
Overall, Exchange certainly hasn’t put itself in my good books, issues like the IPv6 requirement are understandable, but the side effect of the search indexer going nuts on CPU makes no sense and it’s pretty concerning that the code isn’t just “oh I can’t connect, I’ll close/sleep till later”.
So sorry Microsoft, but you won’t see me becoming a Windows Server fanboy at any stage – my Linux Sendmail/Dovecot setup might not have some of Exchange’s flashier features, but it’s damn reliable, extremely easy to debug and logs in a clear and logical fashion. I can trust it to operate in a logical fashion and that’s worth more to me than the features.