Tag Archives: xmpp

Don’t abandon XMPP, your loyal communications friend

Whilst email has always enjoyed a very decentralised approach where users can be expected to be on all manner of different systems and providers, Instant Messaging has not generally enjoyed the same level of success and freedom.

Historically many of my friends used proprietary networks such as MSN Messenger, Yahoo Messenger and Skype. These networks were never particularly good IM networks, rather what made those networks popular at the time was the massive size of their user bases forcing more and more people to join in order to chat with their friends.

This quickly lead to a situation where users would have several different chat clients installed, each with their own unique user interfaces and functionalities in order to communicate with one another.

Being an open standards and open source fan, this has never sat comfortably with me – thankfully in the last 5-10yrs, a new open standard called XMPP (also known as Jabber) has risen up and had wide spread adoption.

500px-XMPP_logo.svg

XMPP brought the same federated decentralised nature that we are used to in email to instant messaging, making it possible for users on different networks to communicate, including users running their own private servers.

Just like with email, discovery of servers is done entirely via DNS and there is no one centralised company, organisation or person with control over the system -each user’s server is able to run independently and talk directly to the destination user’s server.

With XMPP the need to run multiple different chat programs or connect to multiple providers was also eliminated.  For the first time I was able to chat using my own XMPP server (ejabberd) to friends using their own servers, as well as friends who just wanted something “easy” using hosted services like Google Talk which support(ed) XMPP, all from a single client.

Since Google added XMPP into Google Talk, it’s made the XMPP user base even larger thanks to the strong popularity of Gmail creating so many Google Talk users at the same time. With so many of my friends using it, is has been so easy to add them to my contacts and interact with them on their preferred platform, without violating my freedom and losing control over my server ecosystem.

Sadly this is going to change. Having gained enough critical mass, Google is now deciding to violate their “Don’t be evil” company moral and is moving to lock users into their own proprietary ecosystem, by replacing their well established Google Talk product with a new “Hangouts” product which drops XMPP support.

There’s a very good blog write up here on what Google has done and how it’s going to negatively impact users and how Google’s technical reasons are poor excuses, which I would encourage everyone to read.

The scariest issue is the fact that a user upgrading to Hangouts get silently disconnected from being able to communicate with their non-Google XMPP using friends. If you use Google Talk currently and upgrade to Hangouts, you WILL lose the ability to chat with XMPP users, who will just appear as offline and unreachable.

It’s sad that Google has taken this step and I hope long term that they decide as a company that turning away from free protocols was a mistake and make a step back in the right direction.

Meanwhile, there are a few key bits to be aware of:

  1. My recommendation currently is do not upgrade to Hangouts under any circumstance – you may be surprised to find who drops off your chat list, particularly if you have a geeky set of friends on their own domains and servers.
  2. Whilst you can hang onto Google Talk for now, I suspect long term Google will force everyone onto Hangouts. I recommend considering new options long term for when that occurs.
  3. It’s really easy to get started with setting up an XMPP server. Take a look at the powerful ejabberd or something more lightweight like Prosody. Or you could use a free hosted service such as jabber.org for a free XMPP account hosted by a third party.
  4. You can use a range of IM clients for XMPP accounts, consider looking at Pidgin (GNU/Linux & Windows), Adium (MacOS) and Xabber (Android/Linux).
  5. If you don’t already, it’s a very good idea to have your email and IM behind your own domain like “jethrocarr.com”. You can point it at a provider like Google, or your own server and it gives you full control over your online identity for as long as you wish to have it.

I won’t be going down the path of using Hangouts, so if you upgrade, sorry but I won’t chat to you. Please get an XMPP account from one of the many providers online, or set up your own server, something that is generally a worth while exercise and learning experience.

If someone brings out a reliable gateway for XMPP to Hangouts, I may install it, but there’s no guarantee that this will be possible – people have been hoping for a gateway for Skype for years without much luck, so it’s not a safe assumption to have.

Be wary of some providers (Facebook and Outlook.com) which claim XMPP support, but really only support XMPP to chat to *their* users and lack XMPP federation with external servers and networks, which defeats the whole point of a decentralised open network.

If you have an XMPP account and wish to chat with me, add me using my contact details here. Note that I tend to only accept XMPP connections from people I know – if you’re unknown to me but want to get in touch, email is best at first.

Fixing Blogging

I’m finding an increasing number of friends and people using services like Tumblr or Google Plus as blogging services, or at least as a place to make posts that are more detailed an indepth than typical micro-blogging (aka Twitter/Facebook).

The problem with both these services, is that they deny interaction from external users who aren’t registered with their service.

With traditional blogging platforms such as WordPress, Blogger, or other custom developed blogs, any visitor to the blog could read it and post comment – the interfaces would vary, the ease of posting would vary and the method of validation of posting would vary, but you could 99% of the time still be able to post comments and engage with the author.

This has not been the case with social networks to date – platforms like Twitter or Facebook require a user to be logged in, in order to communicate with others – however this tends to work OK, since they’re mostly used for person-to-person messages and broadcasting, rather than detailed posts you will sent to users outside of those networks (after all 140 char tweets aren’t exactly where you’ll debate things of key meaning).

The real issue starts with half-blog, half-microblog services such as Tumblr and Google Plus, which users have started to use for anything from cat pictures to detailed Linux kernel posts, turning these tools into de-facto blogging platforms, but without the freedom for outsiders to post comments and engage in conversation.

 

Tumblr is one of the worst networks, as it’s very much designed as a glorified replacement for chain email forwards – you post some text or some pictures and all your friends “reblog” your page if they like it and users all pat themselves on their back at how witty and original they all are.

But to make a comment, one must reblog the post, add a comment and have it end up in the pages long list of reblog and like statements at the bottom of the post. And if the original poster wants to comment on that, you’d have to re-blog their blog. :-/

Yo dawg, we heard you like to reblog your reblogging.

The issue is that more people are starting to use it for more than just funny cat pictures and treat it as a replacement to blogging, which makes for a terrible time engaging with anyone. I have friends who use the service to post updates about their lives, but I can’t engage back – makes me feel like some kind of outcast stalker peering through the windows at them.

And even if I was on Tumblr, I’d actually want to be able to comment on things without reblogging them – nobody else cares if Jane had a baby, but I’d like to say “Congrats Jane, you look a lot less fat now the fork()ed process is out” to let my friend know I care.

Considering most Tumblr users are going to use Facebook or Twitter as well, they might as well use the image and short statement posting features of those networks and instead use an actual blog for actual content. Really the fault is due to PEBKAC – users using a bad service in the wrong way.

 

Google Plus is a bit better than Tumblr, in the respect that it actually has expected functionality like posts you can comment on, however it lacks the ability for outsiders to post comments and engage with the author – Google has been pretty persistent with trying to get people to sign up for an account, so it’s to be expected somewhat.

I’ve seen a lot of uptake with Google Plus by developers and geeks, seemingly because they don’t want the commitment of actually using a blog for detailed posts, but want somewhere to post lengthy bits of test.

Linus Torvalds is one particular user whom I might want to follow on Google Plus, but there’s not even RSS if you wanted to get updates on new posts! (To get RSS, you’d have to use external thirdparty services).

Tumblr at least has RSS so I can still use it in my reader like everything else, even if I can’t reply to the author….

Follow Linus! Teenage fanboy Jethro squeee!

And of course with no ability for posting comments by outsiders, I can’t post Linus comments requesting his hand in marriage after merging a kernel bug fix for my laptop. :-(

 

So with all these issues, why are users adopting these services? After all, there are thousands of free blogging services, several well known and very good ones, all better technical options.

I think it’s a combination of issues:

  • Users got overwhelmed by RSS – we followed everything we loved, then got scared by the 10,000 unread posts in our readers – and resolved by simply not opening the reader in fear of the queue waiting for us. The social media style approaches used by Google+, Tumblr and of course Twitter and Facebook focus less on following every single post by users, but rather what’s happening here and now – users don’t feel bad if they miss reading 1,000 posts overnight, they just go on to the next.
  • Users love copying. The MPAA & RIAA love this fact about humans, we love to copy and share stuff with others. Blogging culture tends to frown on this, but Tumblr’s reblogging style of use makes it more acceptable and maintains a credit trail.
  • Less commitment – if I started posting pictures of funny cats or one paragraph posts on this blog, it wouldn’t be doing it justice or up to the level of quality readers expect. However on social network based services, this is OK, there’s no expectation of a certain level of presentation and effort into a post. A funny cat picture followed by the post about you raging about by GNU Hurd will always be better than BSD is acceptable – on a blog, you’d drop the funny cat and be expected to write a well detailed post explaining your reasoning. Another label would be that it’s “more casual”, than conventional blogs.
  • Easier interactions with your readers (at least with Google+) – there’s no standards with blogging for handling notifications to users about changes to your blog or replies to comments. Even WordPress, one of the most popular platforms, doesn’t provided native email notifications to comments.
  • I noticed a major improvement in the level of interaction between myself and my readers after adding Subscribe to Comments Reloaded plugin to this site, using email notifications to users about replies to my blog post. And considering how slack many people are with checking their email, I do wonder how much better it would be if I added support for notification to new posts and comment replies via Twitter or Facebook.
  • Conventional blogs tend to take a bit more effort to post comments, some go overboard with captcha input fields that take 10 attempts or painful comment validation. I’ve tried to keep mine simple with basic fields and dealing with spam using Akismet rather than captcha (which has worked very well for me).

In my opinion the biggest issue is the communication, notification and interaction issue as noted above. I don’t believe we can fix the cultural side of users such as the crap they post or the inability to actually make the effort to read their RSS but we can go someway towards improving the technology to reduce/eliminate some of the pain points, to encourage use of the services.

There have been some attempts to address these issues already:

  • Linkback techniques such as Pingback address the issue of finding out who’s linking to your blog (although I turned this off as I found it really spammy and I get that information out of awstats anyway).
  • RSS handles getting updates of new posts on a polling basis and smarter RSS readers offer better filtering/grouping/etc.
  • Email notifications for blog comments and updates.

But it’s not good enough yet – what I’d actually like to see would be:

  • Improvement of linkback techniques to spam pages less, potentially with the addition of some AI logic to determine whether the linkback was just “check out this cool post!” or some actual useful content that readers of your post would like to read (such as a rebuttal).
  • Smarter RSS readers that act more like social network feeds, to give users who want more of a “live stream” feel what they want.
  • Live commenting technology – not all users have push email, so email notifications kind of suck for many users. A better solution would be to use the existing XMPP standard to send notifications to the user’s XMPP server (anyone using Gmail already has an XMPP service with them and numerous geeks run their own – like me ;-), so the user gets a chat message pop up. If the message format was standardized, it would be possible to have the IM client recognize it was a blog comment reply and to hand off to the installed RSS reader to handle for better UX – or fall back to posting text with a link to the reply for support with any XMPP standard client.
  • (I did see that there is an outdated plugin for XMPP on WordPress  as well as some commercial live-commenting packages that hook into social networks, but I really want a proper open source solution that does everything in one plugin, so there’s a more seemless UX – rather than having 20 checkboxes for which method the user would like notifications via.)
  • Whilst mentioning XMPP, we could even consider replacing RSS with XMPP based push notifications – blog servers sending out a push message when they get an update, rather than readers polling services. Advantage is near-instant update of new posts and potentially less server load of not having thousands of wasted polls when there hasn’t been any update to fetch.
  • Comment reply via notification support. If you send someone an XMPP IM, email, tweet, virtual sheep or whatever to alert to a comment or blog post, they should be able to reply via that native medium and have the blog server interpret, validate and integrate that reply into the page.

My hope is that with these upgrades, blogging platforms will extend themselves to be better placed for holding up against social networking sites, making it easier to have detailed conversations and long running threads with readers and authors.

Moving to a new generation communication platform build around the existing blogging platforms would be as much of an improvement for real time social responsiveness as shifting from email to Twitter and hopefully, the uptake in real time communications will bring more users back to decentralised, open and varied platforms.

I’m tempted to give this a go by building a WordPress plugin to provide unified notifications using XMPP / Email / Social Media, but it’ll depend on time (lol who has that??) and I haven’t done much with WordPress’s codebase before. If you know of something existing, I would certainly be interested to read about it and I’ll be taking a look at options to build upon.

100% pure freedom phone?

As per my earlier rant about Android’s openness, I’m not particularly happy with all the binary components on my phone, nor am I particularly happy with the Android Market’s control and lack of clarity around licensing.

There’s multiple issues with propietary software and why I’ve always been an advocate for not just open source, but more importantly, software freedom. In particular, I try and structure all my computing environments so that I can:

  • Always customize the applications I use if needed – this could be bug fixes, feature changes, etc.
  • Having advertisement and tracking/spyware free software. I’d rather pay good money for software than have it advertisement supported or selling my information to others.
  • Have no dependence on gatekeepers running centralized services – I prefer to run distributed federated services, such as SMTP, (Email) and XMPP (IM) for communications, rather than relying on proprietary networks (eg imessage, skype).
  • Full control and responsibility for the security and privacy of my own data, rather than outsourcing to cloud providers.

It seems it would be possible to replace most of the proprietary components that Google supplies with open source components, but in a quick search I didn’t find any Android distributions that have this bundled up into an easy packaged solution.

One of the more popular distributions, Cynaogenmod has some nice features and is open source, but isn’t specifically designed to be *only* open source, whereas I want a distribution that focuses on making it easy to find, install and manage open source software only.

So I’m making plans to do a custom build of Android for my phone which will feature only free as in freedom software components, with the exception of the hardware driver binary blobs.

  • Replace Android Market with the all-open source F-Droid application – this market is 100% open source and both the client and server are open source, so you can even start your own market. One particularly good feature, is the ability to install older versions, I’ve been bitten in the past with updates introducing bugs in the past with no rollback.
  • Email is well handled with open standards IMAP and IMAP IDLE – I’ve been using K9 Mail for some time (open source build of Android’s email client with additional tweaks) and it works beautifully. With the IMAP IDLE functionality, my phone gets notified about the new mail message within a few seconds of the mailserver completing the processing of the message through to my inbox.
  • Replace contact sync with an LDAP contact directory and sync tool to go against that. LDAP is supported by most address book applications and is something I want to use for all my contacts to make it easier to move between applications.
  • Obtain an XMPP client to replace google talk with support for any XMPP/Jabber server desired, whether Google or another server. Considering I use my own XMPP server already, this is something that’s been on my list for a while.
  • Use aCal with an open source CalDAV server (such as DAViCal) for sharing calenders between devices.
  • Replace google maps with open street maps.
  • This would also offer the advantage of not needing to use Google’s cloud services for storing my address book information, something I’ve never particularly liked the idea of, but was somewhat forced upon in the early days of Android 1.5.

As part of this change, I would also end up dumping Android Market and going with only open source applications for Android – the downside will of course be less application selection, but the up side would be less crapware, no adware applications and full control to install any version and manage applications better.

And the end result would be a truly free, open source Android OS on my phone, which I have full control over, with all data stored on either my phone or servers under my control.

I’ll be fitting in the work as I get time slowly replacing components till I have a reliable fully open stack on my phone and blogging my progress. :-)