Author Archives: Jethro Carr

Point & click Procmail with MailGuidance

Procmail is a rather old, but still very useful Unix/Linux application commonly used for writing mail filter rules on Linux servers. I typically use it for user-level filtering, such as defining mailbox filters for all my emails.

It’s also useful for handling shared email addresses, such as support mailboxes receiving a range of emails. Procmail allows these emails to be re-directed to multiple people, different folders or almost any other action desirable.

To make it easier to manage Procmail rule sets in this scenario, I built a tool called “MailGuidance”. It’s an open source PHP/MySQL application which allows a user to create Procmail filters in a web environment and having it then generate the appropriate configuration in the background on the server.

Define whom in your organisation should be getting emails for each matching filter.

MailGuidance is intended for small organisations or an individual seeking a web-based way of managing their procmail rules, it’s intentionally simple and does limit the power of procmail somewhat in exchange for making an easy to use experience for users.

Easy web based interface where filters can be enabled/disabled per user.
User “holiday mode” where all emails to that user get redirected to another until they return, so that nothing gets forgotten.
Optional email archiving into different folders.
Configurable behavior for archiving and unmatched mail.
Works perfectly with IPv6. :-)

Configurable behaviors.

Going away? Send all that albino monkey porn you’ve subscribed to through to your colleague instead!

The best use case for MailGuidance so far has been for handling server log and error emails, by filtering and then redirecting them to the appropriate people/teams to avoid spamming system administrators with irrelevant messages.

I spent some time this weekend tweaking it a bit more and have now packaged some releases and opened up the repository publicly – you can download stable version 1.0.0 or read more about it on my project page here. RPMs are available for users of RHEL/clones.

Radius Rapid Rotate (R3)

Leave a reply

I’ve been spending a bit of time lately going through my private source code repositories and tidying up things for public release.

A while ago I had a customer who required their FreeRadius traffic accounting logs to be collected from a few servers and saved onto a mounted network drive. It’s a simple enough problem, however there’s a few requirements that make it slightly trickier than it sounds:

Extremely important that log files weren’t lost or corrupted in any circumstance.
The archive location was a mounted network drive, this means no guarantee that the filesystem would always be mounted and writable.
The rotated files need to be named with the server hostname so that files from multiple servers could be collated in a single location without clashing.
Regular frequent execution period, eg every 5 or 10 minutes.

The solution I wrote was “Radius Rapid Rotate” (or R3 since I’m a lazy typist). This utility rotates FreeRadius log files in a manner which meets all the above requirements.

It would have been possible to write this all into an existing application, such as logrotate, however logrotate isn’t intended for such frequent execution and won’t do log rotation onto a network mount in a manner that will handle a dodgy network mount gracefully.

Whilst this application is FreeRadius focused, it would be easy to port to use for other purposes if suitable.

Introducing FlatTraffic

2 Replies

FlatTraffic is an AGPL web interface for analyzing NetFlow records and showing statistics designed to make it clear and easy to determine which hosts of the network are consuming data.

It’s still in beta stage, the application is functional and is documented, but may have bugs and need a few tweaks here and there to bring it up to a stable grade… I’m releasing now so that people can start using and breaking it to get a well tested piece of code to enable a 1.0.0 release.

I’d be lying if I said this was a complete list of my computers….

As you are probably aware, New Zealand (and Australia to a lesser degree) are victims of the much hated internet data cap, an unfortunate response to the economic pressures of providing internet services in our markets.

This is a particular issue when you have situations such as flatmates sharing a connection or a a collection of servers behind an internet link which are hungrily consuming the data cap every second.

To help keep the peace with flatmates I started writing this application when I was back in Wellington to report on traffic usage, using a SQL DB of NetFlow records collected by the gateway. It got put on hold somewhat after moving to Auckland and getting a fat DSL plan from Snap NZ, however it recently got resurrected so that I could track down which host on my home server was chewing through the much smaller data cap at it’s new home at my parents place (sadly my full tower beauty wouldn’t fit into my plane luggage).

FlatTraffic is focused at being a geek home/small server environment tool rather than a general purpose NetFlow analyzer – there are more powerful tools already available for that, my design focus with FlatTraffic is simplicity and doing one job really well.

FlatTraffic assumes you’re using it in a conventional ISP customer situation and allows you to configure the monthly date that your service renews on, so that it will show data usage periods that match your billing period. You can also configure other key options such as 1000 vs 1024 bytes and what automatic DB truncating options should be turned on.

Graphical configuration options, eat your heart out Microsoft developers.

There are currently four reports defined in FlatTraffic:

Traffic consumed by protocol.
Traffic consumed by host (with reverse DNS lookup resolution of host IPs)
Traffic consumed per day.
Traffic consumed by configured network range.

Helpful daily totals, aligned with your ISP’s billing period.

FlatTraffic doesn’t replace a NetFlow collector, you still need to understand the principles of setting up NetFlow traffic accounting and configuring a collector that stores records into a SQL database.

I’ve included some sample scripts for use with flowd (from the flow-tools collection) however I’m going to work on adding support for some better collectors. There’s also work needed for IPv6, since whilst the app UI is IPv6 compatible, the NetFlow reporting is strictly IPv4 only currently.

(Unfortunately I also have issues in that the iptables module I’m using to generate NetFlow records don’t seem to have an ip6tables version, so I’m a bit stuck for generating IPv6 records currently without adding a device between my server and the WAN connection :-( ).

In my own environment I hand out static DHCP leases to all my systems along with having configured reverse DNS so when doing a host report I can clearly see which host is responsible for what usage – if you have dynamically addressed hosts doing lots of traffic, things won’t be too helpful until you fix the leases for at least the high users.

To keep performance reasonable when working with huge NetFlow databases, FlatTraffic queries summary data for the selected date period and then caches into MySQL MEMORY tables to make subsequent reports quick and non resource intensive.

Please sir, can I have some more flow records?

I’m currently using it with NetFlow DBs with several months worth of data without issue, but it needs further and wider testing to determine how scalable it really is. I’ve worked to avoid putting much memory hungry logic in PHP, instead FlatTraffic tries to do as much as possible inside MySQL itself and uses some easily indexable queries.

To get started with FlatTraffic, visit the project page and install from either RPM, Source Tarball or direct from SVN – and send me feedback, good or bad. If you’re using another type of NetFlow collector other than flowd and would like support take a look at this page. Also note that there’s no reason why FlatTraffic couldn’t end up using other sources of data, it’s not architecturally limited to just NetFlow if you can get similar traffic details in some other form that would do fine.

If you end up using this application, please let me know how you find, always good to know what is/isn’t useful for people.

Munin 2.0.x on EL 5/6 with IPv6

2 Replies

I’ve been looking forwards to Munin 2 for a while – whilst Munin has historically been a great monitoring resource, it’s always been a little bit too fragile for my liking and the 2.x series sounds like it will correct a number of limitations.

Munin 2.0.6 packages recently became available in the EPEL repository, making it easy to add Munin to your RHEL/CentOS/OracleEL 5/6 servers.

Unfortunately the upgrade managed to break value collection for all my hosts, thanks to the fact that I run a dual-stack IPv4/IPv6 network. :-(

Essentially there were two problems encountered:

Firstly, the Munin 2.x master attempts to talk to the nodes via IPv6 by default, as it typical of applications when running in a dual stack environment. However when it isn’t able to establish an IPv6 connection, instead of falling back to IPv4, Munin just fails to connect.
Secondly, the Munin nodes weren’t listing on IPv6 as they should have been – which is the cause of the first problem.

The first problem is an application bug, or possibly a bug in one of the underlying libraries that Munin-node is using. I haven’t gone to the effort of tracing and debugging it at this stage, but if I get some time it would be good to fix properly.

The second is a packaging issue – there are two dependency issues on EL 5 & 6 that need to be resolved before munin-node will support IPv6 properly.

perl-IO-Socket-INET6 must be installed – whilst it may not be a package dependency (at time of writing anyway) it is a functional dependency for IPv6 to work.
perl-Net-Server as provided by EPEL is too old to support listening on IPv6 and needs to be upgraded to version 2.x.

Once the above two issues are corrected, make sure that the munin configuration is correctly configured:

host *
allow ^127\.0\.0\.1$
allow ^192\.168\.1$
allow ^fdd5:\S*$

I configure my Munin nodes to listen to all interfaces (host *) and to allow access from localhost, my IPv4 LAN and my IPv6 LAN. Note that the allow lines are just regex rather than CIDR notation.

If you prefer to allow all connections and control access by some other means (such as ip6tables firewall rules), you can use just the following as your only allow line:

allow ^\S*$

Once done, you can verify that munin-node is listening on an IPv6 interface. :-)

ipv4host$ netstat -na | grep 4949
tcp 0 0 0.0.0.0:4949 0.0.0.0:* LISTEN
ipv6host$  netstat -na | grep 4949
tcp 0 0 :::4949 :::* LISTEN

I’ve created packages that solve these issues for EL 5 & EL 6 which are now available in my repos – essentially an upgraded perl-Net-Server package and an adjusted EPEL Munin package that includes the perl-IO-Socket-Net package as a dependency.

Commuting in Sydney

Leave a reply

I’ve now been in Sydney for 5 weeks, settling into a new job, a new lifestyle and an entirely new city. Still very much in the tourist phase, there’s heaps we still need to see and do and only just starting to get settled really.

Sydney opera house!

The first two weeks here were spent staying with some of Lisa’s relatives out in Hornsby Heights – nice suburban area, but it takes a bus and a train in order to get into the CBD, which is a 1.5 hour per-way trip – 3 hours a day, or even more depressingly 15 hours a week just to get to and from work.

Because of this commute we haven’t really done much in the first two weeks whilst here, most of my time was either traveling or looking for a place to live with Lisa.

Sydney residents seem to complain about the train service, but it’s actually one of the best I’ve used, really the only thing that lets it down is the lack of a smart card system like Melbourne’s Myki

Instead it uses magnetic stripe tickets which are purchased via ticket machines at every station. These tickets can be for single trips, returns, weekly, monthly or even yearly – I had enough trouble keeping the paper ticket in one piece for a week, so unsure how well monthly or yearly tickets are going to last. :-/

Buttons! All the buttons!

The trains do vary in quality, some of the ones being run are a bit beat up and graffitied, but they always seem to be on time and pretty reliable.

It’s the first city I’ve been in which runs double decker trains – Sydney tends to run them as two pairs of locomotives, each with 4 carriages – effectively 8 carriages, or 16 if you count the fact that they’re double decker and probably fit about twice of the average carriage.

Inside one of the newer trains – note the upper and lower levels!

The suburban train network has the best views, most city training tends to be underground into subway stations, which do tend to be quite hot and cramped – thankfully Sydney has seemed to learn to build big, the newer stations in some suburbs are massive spacious underground caverns.

Unfortunately this large station entrance is an exception to the rule…

Of course you don’t necessarily have to take the underground rail….

MONORAIL!!! I get to walk past it every day on the way to work, so cool.

There’s an active cycling scene in Sydney, particularly around where I live and work, although I’m not sure how anyone survives cycling in hot Sydney days which are pretty horrific survival experiences at times. :-/

Whilst there are cycle lanes, they can be a bit scary as a pedestrian as a lot of cyclists seem to consider themselves immune to the cyclist traffic lights and will sometimes ride right at you against the red light whilst pedestrians are crossing…. There’s also a few wonderful design failures such as shared pedestrian/cyclist zones that are no larger than 1 bike each way at a time leading to people riding a bit too close for comfort.

Thankfully we have now found a place and we’ve finally settled in somewhat – now living in an apartment on Clarence St, right in the middle of the CBD which makes my job in Pyrmont only a short 20minute walk, meaning I can actually spend time enjoying my evenings.

Tree lined home street!

There’s a few pretty awesome perks to my commute, which takes me over Darling Harbour via Pyrmont Bridge and offers some pretty neat views.

Pyrmont bridge in the evening – note the monorail track which goes over it.

Harbor view – just off to the left is the rear end of a warship and a submarine at the maritime museum – but I’ll post more about these later on….

Generally things are going pretty well here, quite a culture shock compared to NZ, but we are getting out and about learning new places and things to see.

Wellington in New Years

Leave a reply

For those of you back in NZ, I will be flying to Wellington for New Years and will be in the city from 25th Dec until 4th of January. My lovely fiancee will be there for a subset of that with me, as she is spending some time with family visiting Sydney before flying into NZ.

If you’re around it will be great to catch up over coffee/beer/other for a few days whilst I’m there.

Unfortunately there won’t be an Auckland visit this trip, so if you’re up there, why not come down to Wellington for a few days instead? You know you want to. :-)

Blessed is the DSL

Leave a reply

As all you loyal readers should have noticed by now is that my blog has been a little quiet since I moved to Sydney. I can assure you it’s not from a lack of interest, but rather a lack of time and essential living resources such as DSL conspiring to prevent me from posting.

I’ve been in Sydney for just over 4 weeks now and have a few blog posts to write, so will complete these over the next few days. :-)

We managed to find a flat and moved into it 2 weeks ago, right in the middle of the CBD with only a 25min walk to my work office – sadly DSL took a little longer to get sorted (thanks Telstra @#$%^&*U) but I now have a nice shiny DSL connection with the good people at Internode.

It’s been an interesting experiment using 3G as my primary internet connection these last few days – as first it’s OK, but it becomes noticeable at how bad performing it is after only a few days. Slow bandwidth making downloads of small files a noticeable factor, latency making SSH connections laggy and non-responsive.

Even simple stuff like listening to music is hard – I have all my music on my server in NZ and simply play it directly off the samba share across my VPN and use cachefilesd to cache recently accessed files on my laptop – this works great on fixed line connections, but fails horribly on 3G which is generally fast enough most of the time for 256k MP3, but has nowhere near enough of a data cap and the connection drops really mess with playback.

The poor 3G performance is not helped by the fact that AU city 3G networks are generally poor – combination of factors, bad coverage, buildings and huge volumes of users, I actually found that NZ’s mobile network was generally better performing all round.

On the other hand, the new LTE networks (“4G”) here are stunningly impressive – colleague is frequently getting 24mbit (and as much as 32mbit) download on his 4G Samsung S3, which is even faster than my CBD ADSL2+ line. Of course this means you’ll chomp though your 1GB monthly plan in about 4.2 minutes… :-/

It was an interesting experience, but I’m now very happy to have a real connection back. Not having a home server is a bit of an adjustment, I’m down to just a DSL modem plugged into my Mikrotik RB493G…

I may look at putting in a larger file server cache locally here at some point, currently looking at the best option for pulling large content from NZ to AU and holding it in cache for the optimal time, almost need a cache that I can instruct to pre-seed on demand – eg “cache all recent accesses from my NZ file server, but also cache the following 4GB file I’ve just requested so that I can watch it when I get home tonight”.

As I write this, we’ve only had DSL for 5 hours and have already pulled 2GB with just casual browsing…. I think the 200GB cap will be enough for us, but one of the perks of living in AU is that I could get up to a 1.2TB cap if I really wanted. ;-)

Cuckoo Clock NZ

Leave a reply

Having arrived in Sydney, I’m staying with some of Lisa’s relatives who have kindly provided us with a room for a while until we get our own place sorted out.

One of the things they have in their house, is a proper mechanical cuckoo clock, which I find highly amusing every time it pops open and emits chirps. I decided it would be fun to write a twitter cuckoo clock.

It’s pretty simple code-wise, just need to generate a tweet every hour with a cuckoo for each hour on a 12-hour clock and a bit of general sanity checking, such as checking what time the last tweet was posted, so if crond goes nuts it won’t spam the feed.

Behold, the amazingness of the Twitter cuckoo clock.

I decided to make it slightly more interesting, so every time it tweets, there is a 1-in-10 chance of it posting some other message from a list of defined messages, as per the above example.

You can check it out at @cuckooclocknz and you can check out the small bit of Python that powers it on my repos. I was tempted to make some for AU, but I was lazy and just did NZ, since my servers are running in NZ timezone and there’s only one timezone for the whole country unlike AU…

Slowly getting more used to Python coding, I’m not a huge fan yet, there’s some nice things about it, like the enforced indenting structure, but some odd things that throw me after years of PHP and Perl, such as for loops and the stricter type handling that need getting used to.

Twitter Auto Delete

1 Reply

Despite me making a clean break from Twitter earlier this year, I’ve ended up back on it on a casual basis, mostly due to the number of my friends on there who only chat or are only reachable via it. :-(

I decided that this time I’d like to treat Twitter more like an IRC chat room, ie a place to chat casually with friends, but not as a formal permanent record – so I made some tweaks to how I was using it:

Primary interaction with Twitter is via PrplTwtr, a plugin for Pidgin, which makes Twitter act like any other chat room, to avoid the habit of having Twitter open in my browser being an invasive distraction. If friends @reply me or DM me, I get a new IM message notification, but otherwise I can ignore it happily.
I wrote a small script that automatically goes and deletes all my Twitter messages after 24 hours – this is enough time for me to chat comfortably with friends, but makes it hard for outsiders to go and data mine my feed and it’s less of a permanent recordable cached record, or link to my tweets long term.

It’s not a perfect setup, whilst it prevents someone from casually going back and seeing my history and engagements with others, it doesn’t stop someone recording my tweets over an extended period to build up their own data pool about me, and of course I have no way of knowing if when I delete a tweet, if it really disappears from the pool of information that Twitter sells to data miners to use.

But it’s good enough that I can chat with friends and keep up-to-date with their lives without leaving a huge digital footprint for any randoms to trawl through.

There are some auto-deleter services around, but I didn’t trust any of them to not do malicious things with my account (eg spamming their presence), plus I wanted it to delete all my tweets *except* my blog post feed.

I found that there’s a pretty decent Twitter module for Python and decided to use this as an exercise to finally learn some proper Python, something I’ve somewhat avoided for lack of a good learning exercise.

The result is a simple Twitter auto-deleter script that is called by Cron every 4 hours and runs a check and deletes any tweets older than 24 hours – the basics is pretty simple really:

39    # query my user status list
40    mytimeline = api.GetUserTimeline(screen_name=user_name,count=query_quantity,include_rts=True)
41    
42    for status in mytimeline:
43    
44        if re.match("^New Blog Post", status.text):
45            #print "Blog post! No delete wanted"
46            continue
47    
48        if status.created_at_in_seconds < cond_time_before:
49            api.DestroyStatus(status.id)
50       
51            print "Deleting Tweet:"
52            print "- Created At: " + status.created_at
53            print "- Content: " + status.text

Note that with GetUserTimeline, you need to specify include_rts=True as an explicit option, so that it includes anything you’ve retweeted in the timeline returned.

Favorites are special wee critters and require a separate GetFavorites call, I don’t use Favorites, so wanted this delete to remove any favorites created by accidental miss-clicks.

You can check out my source here – if you want to run it on your own server, you’ll need to use your account to setup a dev API key and access tokens etc. And you may want to adjust things like the deletion of favorites or retention of blog posts.

I’ve pondered turning this into a simple web-hosted service for people to use, so if you’re the sort of person who can’t use this script yourself but would like the ability to auto-delete your tweets, let me know and I’ll ponder doing it if there’s interest.

I’m sure Twitter will probably kill off more and more of these API calls in future, but at the moment they’re exposing just enough logic to enable me to do this. :-)

Do note that if you run this on a big account, you will hit the maximum API call limit VERY quickly, hence a configured query quantity limit to restrict how many tweets are loaded per execution – you could get away with several hundred every 60mins if you wanted to delete all your twitter history as fast as possible without actually blowing away the account.

Whirlwind Month

Leave a reply

Since I left Auckland at the end of August, it’s been a pretty hectic month, with flights all over the place and adventures in Wellington, Hastings, Melbourne and Sydney!

After we left Auckland, Lisa and I enjoyed a relaxed week in Wellington just spending time catching up with friends, having coffee, beer and just basically having a week of holiday.

Wellington wanders <3

After a week in Wellington, we drove up to Hawke’s Bay for one last road trip in my trusty Toyota Starlet and spent the weekend with Lisa’s family, before I departed leaving her there, so that I could spend two weeks in Wellington tidying up affairs down there, whilst Lisa worked on tidying up and reducing the amount of stuff she had stored at her parents place.

Spending so long apart really sucks, but we both needed to do so in order to get stuff done, and there’s just not enough space at my parents place for us to stay for much longer than one night.

I ended up managing to sell off my beloved Toyota Starlet and a number of other items including my gaming computer, lots of computer cables, software, old flat stuff and other bits and pieces – it’s amazing how much stuff you collect over time, but I’ve managed to get it down so that I have two suitcases plus one cubboard at my parents place only – essentially only storing anything that will have use once I return to NZ in a couple of years.

After packing two suitcases, I flew out of Wellington on the 15th of September on a one-way ticket to start my AU adventure!

Kitty wants to come too!

I arrived in Melbourne looking for work and managed to spend a few days there doing job interviews, getting some basics like my bank accounts and cellphone sorted out and generally tripping around a new city and trying not to freak out at the fact that I’ve just moved country and in a single step changed EVERYTHING in my daily life.

I ended up booking and staying at the Miami Hotel, a 3.5 star place out in West Melbourne near a friends house, since I wanted a bit of my own space – it’s somewhere I’d happily recommend to anyone traveling on a budget, thanks to it’s around $90 a night

Thankfully I do have a number of friends in Melbourne, plus some Twitter friends I had not yet met in person, who helped take me around the place and to get some new sights.

It’s not a hipster bar, until you sit on crates drinking organic beer and watching a DJ in baggy pants wearing a cap with ears designed to look like a horse re-mix popular tracks.

Melbourne CBD, from up on Rooftop Bar

More Melbourne!

There’s no better introduction to Melbourne than starting my first day by visiting a hipster bar, followed by a rooftop bar looking out over the city and catching up with all my Melbourne Twitter friends. :-)

I also spent some time wandering the Victoria Markets, huge amount of stalls and fresh produce – even a meat hall which my friends had great delight dragging me into. :-/

Fluffy kittens! And a fuzzy kitten adventure tube!

I did a lot of walking whilst in Melbourne, I was staying up in West Melbourne, so ended up walking through the CBD almost daily and getting a feel of the city and learning new places.

Tanqueray tram, just for @pikelet

Trams! Tower blocks! Melbourne!

The Yarra river – I personally wouldn’t go swimming in it any time soon, looks a bit murky. :-/

Rain clouds moving into the city.

Misty walking bridge.

I also ended up training and tramming around the city lots, the transport system in Melbourne is amazing, so rapid and easy to get around, I got across town in about 15mins by jumping on a train, then a tram and using the Android Tram Hunter and Train Tracker applications to check schedules and routes.

Woot, trains!

Melbourne at night, when coming out of Flinder’s Station

It’s a pretty amazing city – it took me a few days to get settled a bit more, but keen to spend a lot more time there.

Sadly my plans to get work there were interrupted, I had only been in Melbourne for a few days before I got the sad news that my grandfather in Wellington has been diagnosed with terminal cancer and has less than 4 weeks to live. :-(

I decided to take the time to fly back to NZ sooner rather than later, so that I could spend some time with him whilst I still can, so booked a flight back to Wellington at short notice – sadly this meant that I had to pass on a couple interesting opportunities that had arisen, but I guess there’s never really a good time for this sort of thing to happen.

After having booked my return flight, I then got a call at short notice from a contract recruiter, who had a position for a 3-month contract in Sydney for a Linux engineer – I managed to re-arrange my travel plans, and instead of flying directly back to Wellington, ended up flying into Sydney, spending a day there to interview and then flying back to Wellington from there.

Doesn’t get much more iconic than the Sydney Harbor Bridge. Unlike Auckland, they were actually smart enough to put rail, walk and cycle crossings on it.

Circular Quay Panorama

Sydney CBD from the botanical gardens down by the opera house.

Sydney CBD viewed from Pyrmont.

I will have to make some time to actually go on and explore these awesome machines – looks like it’s possible to actually go and tour inside the submarine!

Fuck yeah Monorail!

After all this crazy tripping around Melbourne and Sydney, I flew back to Wellington for a few days to spend time with family and then flew to Hawke’s Bay to spend time with Lisa.

Meanwhile the job interview in Sydney returned good results, so this weekend, Lisa and I fly to Sydney to start my new 3-month contract in Sydney from the 1st of October!

Going to be a lot of fun and will offer some new challenges in a whole new city, 3 months will give us a chance to sample Sydney and figure out if we want to stay there for longer term, or we can shift to Melbourne at a later stage.