Tag Archives: code

Any posts relating to software that I have developed, including contributions to other projects, but excluding any packaging work.

Git & GitHub Enabled

7 Replies

GitHub Goodness

I’ve been developing software for a little while now and have build up a few repositories for my applications, with all my open source ones being available publicly. Sometimes people find some of my applications useful and I get thanks, patches or PHP hate mail. :-)

For a long time I’ve been using SVN for storing my code, along with Indefero as my project tracker at projects.jethrocarr.com. It’s a good combination, very similar to Google Code in many respects and generally a lightweight application with a good core feature set.

The only issue has been that with the explosion in popularity of Git and the socalisation of coding with sites such as GitHub, users have gotten tired of the “diff and email patch” approach when submitting contributions and want to take advantage of shiner features such as pull requests which make contributes much easier, as well as more recognisable to others.

Whilst I’m keen to do as much as possible to make it easier to get commits and users, switching to any one particular hosted provider is of concern to me – whilst they may be popular now, will they still be as popular in 10 years time? (Remember SourceForge anyone?).

The solution is that undergoing the pain to migrate existing repositories from SVN to Git (a lot more messing around than you might think) opens up the ability to pull and push to multiple repositories, which means that I now have all my open source projects on GitHub and in addition have my own hosted Indefero server which has a full copy of all my code.
This allows me to engage with users on GitHub, whilst still maintaining control of my own issue tracker and full copies of my repositories and data. It also avoids users from setting up their own GitHub repositories of my projects and having them confused as official ones – with my own in place, it’s a starting place for forks to occur from.
I’m going to trial this for a few months – if it all works well, I’m going to take a look at adding in easy support to Indefero to create and push/pull from a GitHub repository automatically as part of creating a new project. And if needed I could add additional Git providers to mirror to as well (eg BitBucket) should other popular hives of activity appear.

Amberdms Billing System 2.0.0

9 Replies

It’s been a long while since my last release of the Amberdms Billing System (ABS), but at last I’ve finished merging in and testing all the new features that were worked on during my time running Amberdms with two other great coders and prepared a new stable release and documentation.

If you’re not familiar with ABS, it’s an open source billing system providing double-entry accounting, invoicing (with PDF generation), customer management, service billing (including telco usage billing) and time sheeting and billing functionality.

It’s been used by a couple smaller ISPs in New Zealand as well as various open source users around the world and is extremely flexible and powerful software (in my biased opinion). :-)

Accounting and billing isn’t a sexy application… but it needs to get done.

The major developments of this release include:

New invoice templates using HTML/CSS with Webkit as a rendering engine to produce stunning PDF invoices.
Numerous improvements and additions to the customer management page.
Credit notes & customer credit balance management.
Easier bulk handling of payments with bulk payment interface and (beta) bank statement import function.
Support for VoIP billing, including charging customers for all calls made based on a configurable call record database as the source.
Service bundling to group multiple services together to form packages.

If you’re running a business, particularly a service or technology orientated company, I invite you to take a look and check out ABS. Even if you’re using an existing accounting system like Xero, ABS is a great fit for the billing requirements and is a great base to use, rather than building your own in house billing platform.

You can read the release announcement details here, or go directly to the open source project page and download the installation guides and source code.

SMStoXMPP

3 Replies

Having moved to AU means that I now have two cell phones – one with my AU SIM card and another with my NZ SIM card which I keep around in order to receive the odd message from friends/contacts back home and far too many calls from telemarketers.

I didn’t want to have to carry around a second mobile and the cost of having a phone on roaming in AU makes it undesirably expensive to keep in touch with anyone via SMS messaging, so went looking for a solution that would let me get my SMS messages from my phone to my laptop and phone in a more accessible form.

I considered purchasing an SMS gateway device, but they tend to be quite expensive and I’d still have to get some system in place for getting messages from the device to me in an accessible form.

Instead I realised that I could use one of the many older Android cellphones that I have lying around as a gateway device with the right software. The ability to run software makes them completely flexible and with WiFi and 3G data options, it would be entirely possible to leave one in NZ and take advantage of the cheaper connectivity costs to send SMS back to people from within the country.

I was able to use an off-the-shelf application “SMS Gateway” to turn the phone into an SMS gateway, with the option of sending/receiving SMS messages via HTTP or SMTP/POP3.

However emails aren’t the best way to send and reply to SMS messages, particularly if your mail client decides to dump in a whole bunch of MIME data. I decided on a more refined approach and ended up writing a program called “SMStoXMPP“.

Like the name suggestions, SMStoXMPP is lightweight PHP-based SMS to XMPP (Jabber) bi-directional gateway application which receives messages from an SMS gateway device/application and relays them to the target user via XMPP instant messages. The user can then reply via XMPP and have the message delivered via the gateway to the original user.

For me this solves a major issue and means I can leave my NZ cell phone at my flat or even potentially back in NZ and get SMS on my laptop or phone via XMPP no matter where I am or what SIM card I’m on.

To make conversations even easier, SMStoXMP does lookups of the phone numbers against any CardDAV address book (such as Google Contacts) and displays your chosen name for the contact. It even provides search functions to make it even easier to find someone to chat to.

Chatting with various contacts via SMStoXMPP with Pidgin as a client.

I’ve released version 1.0.0 today, along with documentation for installing, configuring gateways and documentation on how to write your own gateways if you wish to add support for other applications.

Generally it’s pretty stable and works well – there are a few enhancements I want to make to the code and a few bits that are a bit messy, but the major requirements of not leaking memory and being reliably able to send and receive messages have been met. :-)

Whilst I’ve only written support for the one Android phone base gateway, I’m working on getting a USB GSM modem to work which would also be a good solution for anyone with a home server.

It would also be trivial to write in support for one of the many online HTTP SMS gateways that exist if you wanted a way to send messages to people and didn’t care about using your existing phone number.

Download SMStoXMPP source code & documentation here
Or add my Yum repositories and install onto CentOS 5/6.

Holy Relic of the Server Farm

Leave a reply

At work we’ve been using New Relic, a popular software-as-a-service monitoring platform to monitor a number of our servers and applications.

Whilst I’m always hesitant of relying on external providers and prefer an open source solution where possible, the advantages provided by New Relic have been hard to ignore, good enough to drag me away from the old trusty realm of Munin.

Like many conventional monitoring tools (eg Munin), New Relic provides good coverage and monitoring of servers, including useful reports on I/O, networking and processes.

Bro, I’m relaxed as bro. (server monitoring with New Relic)

However where New Relic really provides value is with it’s monitoring of applications, thanks to a number of agents for various platforms including PHP, Ruby, Python, Java and .NET.

These agents hook into your applications and profile their performance in detail, showing details such as breakdown of latency by layer (DB, language, external, etc), slow DB queries and other detailed traces.

For example, I found that my blog was taking around 1,000ms of processing time in PHP when serving up page content. The VM itself had little load, but WordPress is just not a particularly well oiled application.

Before and after installing W3 Total Cache on my blog. Next up is to add Varnish and drop server times even further.

Toss out your crufty DBA, we have a new best friend! (just kidding DBAs, I still love ya)

New Relic will even slip an addition into the client-side content which measures the browser-side performance and experience for users visiting your website or application, allowing you to determine cause of slow page loads.

Generally my issue is too much large content + slow links

There’s plenty more offered, I haven’t even looked at all the options and features yet myself – best approach is to sign up for a free account and trial it for a while to see if it suits.

New Relic recently added a Mobile Application agent for iOS and Android developers, so it’s also attractive if you’re writing mobile applications and want to check how they’re performing on real user devices in the wild.

Installation of the server agent is simply a case of dropping a daemon onto the host (with numerous distribution packages available). The application agents vary depending on language, but are either a case of loading the agent with the application, or bundling a module into your application.

It scales well performance wise, we’ve installed the agent on some of AU’s largest websites with very little performance impact in most cases and the New Relic interface remains fast and responsive.

Only warning I’d make is that the agent uses HTTP by default, rather than HTTPS – whilst the security impact is somewhat limited as the data sent isn’t too confidential, I would really prefer the application use HTTPS-only. (There does appear to be an “enterprise security” mode which forces HTTPS agents only and adds other security options, so do some research if it’s a concern).

Pricing is expensive, particularly for the professional account package with the most profiling. Having said that, for a web company where performance is vital, New Relic can quickly pay for itself with reduced developer time spend on issues and fast alerting to performance related issues. Both operations and developers have found it valuable at work, and I’ve personally found this a much more useful tool than our Splunk account.

If you’re only interested in server monitoring you will probably find better value in a traditional Munin setup, unless you value the increased simplicity of configuration and maintenance.

Note that New Relic is also not a replacement for alert-monitoring such as Nagios – whilst New Relic can generate alerts for performance issues and other thresholds, my advice is to rely on Nagios for service and resource overload/failure and rely on New Relic monitoring for alerting to abnormal or negative performance trends.

I also found that I still find Awstats very useful – whilst New Relic has some nice browser stats and geography stats, Awstats is more useful for the “how much traffic and data has my website/application done this month” type questions.

It’s not for everyone’s requirements and budget, but I do highly recommend having an initial trial of it, whether you’re running a couple of servers or a massive enterprise.

NamedManager 1.5.1

Leave a reply

I’ve pushed a new release of NamedManager version 1.5.1, this release is a minor bug fix release providing:

Bug fix for handling of TXT records, where extra slashes would be entered into the record due to an input validator bug.
The Bind configuration writer now runs the Bind-supplied validators for configuration and DNS zone files and refuses to reload Bind without them passing

The first change is naturally important if you’re using TXT records as it does fix a serious issue with the handling of TXT records (no security problems, but corrupted zonefiles would result at times).

Even if you’re not using TXT records, the second change is worth upgrading to as it makes the Bind configuration generator much more robust and prevents any potential future bugs from ever feeding Bind a bad zonefile.

Pre-1.5.1, we relied on Bind’s reload process to validate the files, however this suffers an issue where the error might not be reported back to the user and they would only discover the issue next time Bind restarts. This changes prevents a new zonefile from being loaded into place until the validator passes it, so the worst case is your DNS just refuses to accept changes, whilst logging loudly in the web interface back to you. :-)

If you upgrade, take advantage of this feature, by adding the following to /etc/namedmanager/config-bind.php or wherever you have installed your Bind component configuration file to:

$config["bind"]["verify_zone"]    = "/usr/sbin/named-checkzone";
$config["bind"]["verify_config"]  = "/usr/sbin/named-checkconf";

NamedManager 1.5.1 can be found at the project page or in my packaged repositories.

Presenting NamedManager

11 Replies

A while ago I had a project to build a DNS management application for a client, which has since been refined and improved further, and finally released as “NamedManager” now that I’ve had time to re-do the documentation for a public audience.

NamedManager is an AGPL web-based DNS management system designed to make the adding, adjusting and removal of zones/records easy and reliable by a simple yet effective interface.

Rather than attempting to develop a new name server, NamedManager supports the tried and tested Bind name server and can integrate nicely into existing complex Bind configurations including servers with multiple views without clobbering custom configurations.

Configuring zone records with NamedManager.

It’s written in PHP 5 and uses a MySQL database for storing the DNS record information, which is then converted into Bind compatible configuration files and copied to the name servers, an act which ensures that any loss of the NamedManager application or database will not result in a loss of DNS services.

It’s a stable application, having been in some large production environments for over a year, although there’s certainly more work wanted on the application, such as the addition of IPv6 PTR records and improved UI around SRV and SPF record entry.

NamedManager includes an interface for tracking the sync status of the latest changes across all your name servers, as well as understanding the differences between internal only and publicly accessible name servers and generating the appropriate NS records for domains automatically.

An included daemon can (optional) watch the Bind name server logs and send them back to the web interface, so that you can watch all your name servers via an AJAX log interface to make it easier to watch for issues or debug queries.

Server status report – see if your hosts have synced DNS changes and are reporting logs.

Both forwards domains for IPv4 and IPv6 are supported, and IPv4 reverse domains are also supported (IPv6 reverse to come in future release), along with the ability to import Bind zone files (works for most, unless yours is too ugly/complex) into the application.

View of all the domains active on this DNS cluster with NamedManager.

For developers, NamedManager features a SOAP API which can be used to manage DNS records – this has been used to hook into other provisioning tools (eg: cloud instance management tools) to reduce manual effort for keeping records clean and tidy,

The code structure of NamedManager would make it possible to add support for additional name servers as desired, I would be keen to see support for PowerDNS and Amazon Route 53 as options in future – as always, patches welcome. ;-)

If you’re interested in checking it out, view the NamedManager project page here and follow the instructions to install from RPM, source tarball or SVN.

Point & click Procmail with MailGuidance

Leave a reply

Procmail is a rather old, but still very useful Unix/Linux application commonly used for writing mail filter rules on Linux servers. I typically use it for user-level filtering, such as defining mailbox filters for all my emails.

It’s also useful for handling shared email addresses, such as support mailboxes receiving a range of emails. Procmail allows these emails to be re-directed to multiple people, different folders or almost any other action desirable.

To make it easier to manage Procmail rule sets in this scenario, I built a tool called “MailGuidance”. It’s an open source PHP/MySQL application which allows a user to create Procmail filters in a web environment and having it then generate the appropriate configuration in the background on the server.

Define whom in your organisation should be getting emails for each matching filter.

MailGuidance is intended for small organisations or an individual seeking a web-based way of managing their procmail rules, it’s intentionally simple and does limit the power of procmail somewhat in exchange for making an easy to use experience for users.

Easy web based interface where filters can be enabled/disabled per user.
User “holiday mode” where all emails to that user get redirected to another until they return, so that nothing gets forgotten.
Optional email archiving into different folders.
Configurable behavior for archiving and unmatched mail.
Works perfectly with IPv6. :-)

Configurable behaviors.

Going away? Send all that albino monkey porn you’ve subscribed to through to your colleague instead!

The best use case for MailGuidance so far has been for handling server log and error emails, by filtering and then redirecting them to the appropriate people/teams to avoid spamming system administrators with irrelevant messages.

I spent some time this weekend tweaking it a bit more and have now packaged some releases and opened up the repository publicly – you can download stable version 1.0.0 or read more about it on my project page here. RPMs are available for users of RHEL/clones.

Radius Rapid Rotate (R3)

Leave a reply

I’ve been spending a bit of time lately going through my private source code repositories and tidying up things for public release.

A while ago I had a customer who required their FreeRadius traffic accounting logs to be collected from a few servers and saved onto a mounted network drive. It’s a simple enough problem, however there’s a few requirements that make it slightly trickier than it sounds:

Extremely important that log files weren’t lost or corrupted in any circumstance.
The archive location was a mounted network drive, this means no guarantee that the filesystem would always be mounted and writable.
The rotated files need to be named with the server hostname so that files from multiple servers could be collated in a single location without clashing.
Regular frequent execution period, eg every 5 or 10 minutes.

The solution I wrote was “Radius Rapid Rotate” (or R3 since I’m a lazy typist). This utility rotates FreeRadius log files in a manner which meets all the above requirements.

It would have been possible to write this all into an existing application, such as logrotate, however logrotate isn’t intended for such frequent execution and won’t do log rotation onto a network mount in a manner that will handle a dodgy network mount gracefully.

Whilst this application is FreeRadius focused, it would be easy to port to use for other purposes if suitable.

Introducing FlatTraffic

2 Replies

FlatTraffic is an AGPL web interface for analyzing NetFlow records and showing statistics designed to make it clear and easy to determine which hosts of the network are consuming data.

It’s still in beta stage, the application is functional and is documented, but may have bugs and need a few tweaks here and there to bring it up to a stable grade… I’m releasing now so that people can start using and breaking it to get a well tested piece of code to enable a 1.0.0 release.

I’d be lying if I said this was a complete list of my computers….

As you are probably aware, New Zealand (and Australia to a lesser degree) are victims of the much hated internet data cap, an unfortunate response to the economic pressures of providing internet services in our markets.

This is a particular issue when you have situations such as flatmates sharing a connection or a a collection of servers behind an internet link which are hungrily consuming the data cap every second.

To help keep the peace with flatmates I started writing this application when I was back in Wellington to report on traffic usage, using a SQL DB of NetFlow records collected by the gateway. It got put on hold somewhat after moving to Auckland and getting a fat DSL plan from Snap NZ, however it recently got resurrected so that I could track down which host on my home server was chewing through the much smaller data cap at it’s new home at my parents place (sadly my full tower beauty wouldn’t fit into my plane luggage).

FlatTraffic is focused at being a geek home/small server environment tool rather than a general purpose NetFlow analyzer – there are more powerful tools already available for that, my design focus with FlatTraffic is simplicity and doing one job really well.

FlatTraffic assumes you’re using it in a conventional ISP customer situation and allows you to configure the monthly date that your service renews on, so that it will show data usage periods that match your billing period. You can also configure other key options such as 1000 vs 1024 bytes and what automatic DB truncating options should be turned on.

Graphical configuration options, eat your heart out Microsoft developers.

There are currently four reports defined in FlatTraffic:

Traffic consumed by protocol.
Traffic consumed by host (with reverse DNS lookup resolution of host IPs)
Traffic consumed per day.
Traffic consumed by configured network range.

Helpful daily totals, aligned with your ISP’s billing period.

FlatTraffic doesn’t replace a NetFlow collector, you still need to understand the principles of setting up NetFlow traffic accounting and configuring a collector that stores records into a SQL database.

I’ve included some sample scripts for use with flowd (from the flow-tools collection) however I’m going to work on adding support for some better collectors. There’s also work needed for IPv6, since whilst the app UI is IPv6 compatible, the NetFlow reporting is strictly IPv4 only currently.

(Unfortunately I also have issues in that the iptables module I’m using to generate NetFlow records don’t seem to have an ip6tables version, so I’m a bit stuck for generating IPv6 records currently without adding a device between my server and the WAN connection :-( ).

In my own environment I hand out static DHCP leases to all my systems along with having configured reverse DNS so when doing a host report I can clearly see which host is responsible for what usage – if you have dynamically addressed hosts doing lots of traffic, things won’t be too helpful until you fix the leases for at least the high users.

To keep performance reasonable when working with huge NetFlow databases, FlatTraffic queries summary data for the selected date period and then caches into MySQL MEMORY tables to make subsequent reports quick and non resource intensive.

Please sir, can I have some more flow records?

I’m currently using it with NetFlow DBs with several months worth of data without issue, but it needs further and wider testing to determine how scalable it really is. I’ve worked to avoid putting much memory hungry logic in PHP, instead FlatTraffic tries to do as much as possible inside MySQL itself and uses some easily indexable queries.

To get started with FlatTraffic, visit the project page and install from either RPM, Source Tarball or direct from SVN – and send me feedback, good or bad. If you’re using another type of NetFlow collector other than flowd and would like support take a look at this page. Also note that there’s no reason why FlatTraffic couldn’t end up using other sources of data, it’s not architecturally limited to just NetFlow if you can get similar traffic details in some other form that would do fine.

If you end up using this application, please let me know how you find, always good to know what is/isn’t useful for people.

Cuckoo Clock NZ

Leave a reply

Having arrived in Sydney, I’m staying with some of Lisa’s relatives who have kindly provided us with a room for a while until we get our own place sorted out.

One of the things they have in their house, is a proper mechanical cuckoo clock, which I find highly amusing every time it pops open and emits chirps. I decided it would be fun to write a twitter cuckoo clock.

It’s pretty simple code-wise, just need to generate a tweet every hour with a cuckoo for each hour on a 12-hour clock and a bit of general sanity checking, such as checking what time the last tweet was posted, so if crond goes nuts it won’t spam the feed.

Behold, the amazingness of the Twitter cuckoo clock.

I decided to make it slightly more interesting, so every time it tweets, there is a 1-in-10 chance of it posting some other message from a list of defined messages, as per the above example.

You can check it out at @cuckooclocknz and you can check out the small bit of Python that powers it on my repos. I was tempted to make some for AU, but I was lazy and just did NZ, since my servers are running in NZ timezone and there’s only one timezone for the whole country unlike AU…

Slowly getting more used to Python coding, I’m not a huge fan yet, there’s some nice things about it, like the enforced indenting structure, but some odd things that throw me after years of PHP and Perl, such as for loops and the stricter type handling that need getting used to.