I recently had the pleasure to speak at the 2018 Mobile Refresh conference held here in Wellington and did a talk introducing how we run some parts of the Sailthru Mobile platform, along with recommendations and advice to anyone else also building backends to support their mobile applications.
It’s more entry-level than some of my other infrastructure talks as it’s focused on people that are primarily mobile developers with maybe a limited set of infrastructure awareness.
The previous owners of our house had left us with a reasonably comprehensive alarm system wired throughout the house, however like many alarm systems currently in homes, it required an analogue phone line to be able to call back to any kind of monitoring service.
To upgrade the alarm to an IP module via the monitoring company would be at least $500 in parts and seemed to consist of hooking the phone line to essentially a VoIP ATA adaptor which can phone home to their service.
As a home owner I want it internet connected so I can do self-monitoring, give me the ability to control remotely and to integrate it with IP-based camera systems. Most of the conventional alarm companies seem to offer none of things, or only very expensive sub-standard solutions.
To make things worse, their monitoring services are also pretty poor. Most of the companies I spoke to would receive an alarm, then call me to tell me about it/check with me and only then send someone out to investigate. The existing alarm company the previous owner was using didn’t even offer a callout security service!
Spark (NZ incumbent telco) recently brought out a consumer product called Morepork (as seen on stuff!) which looks attractive for your average non-techie consumer, but I’m not particularly keen to tie myself to Spark’s platform and it is very expensive, especially when considering I have to discard an existing functional system and start from scratch. There’s also some design weaknesses like the cameras being mains dependent, which I don’t consider acceptable given how easy it is to cut power to a house.
So I decided that I’d like to get my existing alarm IP connected, but importantly, I wanted to retain complete control over the process of generating an alert and delivering it to my phone so that it’s as fast as possible and also, as reliable as possible.
Not only did I want to avoid the human factor, but I’m also wary of the proprietary technologies used by most of the alarm companies off-the-shelf solutions. I have some strong doubts about the security of a number of offers, not to mention life span (oh sorry that alarm is EOL, no new mobile app for you) and the level of customisation/integration offered (oh you want to link your alarm with your camera motion detection? Sorry, we don’t support that).
I did some research on my alarm system and found it’s one of the DSC PowerSeries range which is a large Canadian company operating globally. The good thing about them being a large global player is that there’s a heap of reference material about their products online.
With a quick search I was able to find user guides, installer guides, programming guides and more. They also include a full wiring diagram inside the alarm control centre which is exceptionally useful, since it essentially explains how you can connect any kind of sensors yourself which can save a whole heap of money compared to paying for an alarm company to do the installation.
I wish all my electronic devices came with documentation this detailed.
The other great thing about this alarm is that since DSC is so massive, there’s an ecosystem of third party vendors offering components for it. Searching for third party IP modules, I ran into this article where the author purchased an IP module from a company known as Envisalink and used it’s third party API to write custom code to get alarm events and issue commands.
A third party API sounded perfect, so I purchased the EnvisaLink EVL-4 for $239 NZD delivered and did the installation myself. In theory the installation is easy, just a case of powering down the alarm (not touching any 240V hard wired mains in the process) and connecting it via the 4 wire keypad bus.
In my case it ended up being a bit more complex since the previous owner had helpfully never given me any of the master/installer alarm codes, so I ended up doing a factory reset of the unit and re-programming it from scratch (which means all the sensors, etc) which takes about a day to figure out and do the first time. The plus side is that this gave me complete control over the unit and I was able to do things like deprogram the old alarm company’s phone number to stop repeat failed callout attempts.
Once connected, the EnvisaLink unit was remarkably hassle free to setup – it grabbed a DHCP lease, connected to the internet and phoned home to the vendor’s free monitoring service.
EnvisaLink unit installed at the top above the alarm control circuit. A++ for LED ricing guys!
The EnvisaLink hardware is a great little unit and the third party programmer’s interface is reasonably well documented and works without too much grief. Unfortunately the rest of the experience of the company selling it isn’t particularly good. Specifically:
I haven’t dug into the communications between the unit and it’s vendor, I sure hope it’s SSL/TLS secured and doesn’t have the ability to remotely exploit it and upgrade it, but I’m not going to chance it. Even if they’ve properly encrypted and secured comms between the unit and their servers, the security is limited to the best practices of the company and software which already look disturbingly weak.
Thankfully my requirements for the module is purely it’s third party API so I can integrate with my own systems, so I can ignore all these issues and put it on it’s own little isolated VLAN where it can’t cause any trouble and talk to anything but my server.
So having sorted out the hardware and gotten the alarm onto the network, I now needed some software that would at least meet the basic alerting requirements I have.
There’s an existing comprehensive Java/Android-based product (plainly labeled as “DSC Security Server”) which looks very configurable, but I specifically wanted something open source to make sure the alarm integration remained maintainable long term and to use Google Push Notifications for instant alerting on both Android (which supports long running background processes) and iOS (which does not – hence you must use push notifications via APNS).
I ended up taking advantage of some existing public code for handling the various commands and error responses from the Envisalink/DSC alarm combination but reworked it a bit so I now have a module system that consists of “alarm integrators” exchanging information/events with the alarm system and “alarm consumers” which decide what to do with the events generated. These all communicate via a simple beanstalk queue.
This design gives ultimate simplicity – each program is not much more than a small script and there’s a standard documented format for anyone whom wants to add support for other alarm integrators or alarm consumers in future. I wanted it kept simple, making it the sort of thing you could dump onto a Raspberry Pi and have anyone with basic scripting skills be able to debug and adjust it.
I’ve assembled these programs into an open source package I’m calling “HowAlarming”“, hopefully it might be useful for anyone in future with the same alarm system or wanting a foundation for building their own software for other alarms (or even their own alarms).
The simplest solution to get alerts from the system would be by sending SMS using one of the many different web-based SMS services, but I wanted something I can extend to support receiving images from the surveillance system in future and maybe also sending commands back.
Hence I’ve written a companion Android app which receives messages from HowAlarming via push notifications and maintains an event log and the current state of the alarm.
UX doens’t get much better than this.
It’s pretty basic, but it offers the MVP that I require. Took about a day to hack together not having done any Android or Java before, thankfully Android Studio makes the process pretty easy with lots of hand holding and easy integration with the simulators and native devices.
TBD if I can hack something together in a day not having done any native app development before that’s better than many of the offerings from the alarm companies currently around, they need to be asking themselves some hard questions. At the very least, they should get someone to write some apps that can pull their customer’s alarm state from their current phone-home infrastructure – there’s probably good money to be made giving existing customers on non-IP era alarms upgrades given the number of installations out there.
So far my solution is working well for me. It’s not without it’s potential problems, for example alarm communications are now at the mercy of a power/internet outage whereas previously as long as the phone line was intact, it could call out. However this is easily fixed with my UPS and 3G failover modem – the 3G actually makes it better than previously.
The other potential issue is that I don’t know what insurance would classify this nature of self-monitoring as. I have mine declared as “un-monitored” to avoid any complications, but if your insurance conditions require monitoring I’m unsure if a home-grown solution would meet those requirements (even if it is better than 90% of the alarm companies). So do your research and check your contracts & terms.
Coming back to New Zealand from Australia, I was faced with needing to pick a telco to use. I’ve used all three New Zealand networks in the past few years (all pre-4G/LTE) and don’t have any particular reason/loyalty to use any specific network.
I decided to stay on the 2degrees network that I had parked my number on before going to Sydney, so I figured I’d put together a brief review of how I’ve found them and what I think about it so far.
Generally there were three main incentives for me to stay on 2degrees:
And of course their pricing is sharp – coming into the New Zealand market as the underdog, 2degrees started going after the lower end prepay market, before moving up to offer a more sophisticated data network.
For $29, I’m now getting 1GB of data, 300 minutes AU/NZ and unlimited SMS AU/NZ. But also received an additional once-off bonus of 2GB data for moving to a no-commitment plan and another 200MB per month as a bonus for my data shared device; it’s insanely good value really.
Of course good pricing and features aren’t any good if the quality of the service is poor or the data-rate substandard. 2degrees still lack 4G/LTE in Wellington (has just been introduced in Auckland) which is going to set them back a bit, however they do still deliver quite a decent result.
Performance of my 1 year old Samsung Galaxy Note 2 (LTE/4G model operating on 3G-only network) was good with a 22.16 Mb/s download and 2.56 MB/s upload from my CBD apartment. It’s actually faster than the apartment WiFi ISP provider currently. (Unsure why the ping below is so bad, it’s certainly not that bad when testing… possibly some issue with the app on my device).
It does pay to have a good device – older devices aren’t necessarily capable of the same speeds, the performance with my 4 year old Lenovo X201i with Qualcomm Gobi 2000 built-in 3G hardware is quite passable, but it’s not quite the speed demon of my cellphone at only 6.16 Mb/s down and 0.36 Mb/s up. Still faster than many ADSL connections however, I was only getting about 4 Mb/s down in my Sydney CBD apartment recently!
Whilst I haven’t got any metrics to show, the performance outside of the cities in regional and rural areas is still reasonable. 2degrees roams onto Vodafone for parts of their coverage outside the main areas, which means that you need to make sure your phone/device is configured to allow national data roaming (or you’ll get *no* data coverage), and it also means you’re suspectable to Vodafone’s network performance, which is one of the worst I’ve used (yes AU readers, even worse than Vodafone AU).
Generally the performance is perfectly fine for my requirements – I don’t download heaps of data, but I do use a lot of applications that are latency and packet loss sensitive. I look forwards to seeing what it’s like once 2degrees get their LTE network in Wellington and I can get the full capability out of my phone hardware.
2degrees is also trailing a service of offering free WiFi access – I’m in the trial and have been testing, generally the WiFi is very speedy and I was getting speeds of around 21 Mb/s down and 9 Mb/s up whilst walking around, but it’s let down by the poor transition that Android (and presumably other vendors) make when moving between WiFi and 3G networks. Because the WiFi signal hangs on longer than it can actually sustain traffic, it leads to small service dropouts in data when moving between the two networks – this isn’t 2degrees’ fault, rather a limitation of WiFi and the way Android handles it, but it reflects badly on telco hybrid WiFi/GSM network approaches.
It’s not all been perfect – I’ve had some issues with 2degrees, mostly when using them as a prepay provider. The way data is handled for prepay differs to on-plan, and it’s possible to consume all your available data, then eat through your credit without any warning, something that cost me a bit more than I would like a couple of times when on prepay.
This is fixed with on-plan, which gives you tight spend control (define how much you want to cap your bill at) and also has a mode that allows you to block non-plan based data spend, to avoid some unexpected usage generating you an expensive bill. I’d recommend going with one of their plans rather than their prepay because of this functionality alone, not to mention that the plans tend to offer a bit better value.
On the plus side, their twitter support was fantastic and sorted me out with extra data credit in compensation. Their in-store support has also been great, when I went to buy an extra SIM ($5) to data share to my laptop, the guy at the counter told me about a promotion, gave me a free SIM and chucked 200MB/month on it, all that I wasn’t expecting.
It’s a nice change, generally telco customer service is some of the worst examples around, so it’s nice to have a positive interaction, although 2degrees do need to make an effort to stop having certain spend protections limited to their plan customers and not prepay – A good customer service interaction is nice, but not having to talk to them in the first place is even better.
So how do I find 2degrees compared to the other networks? I’ve found NZ networks generally a mixed bag in the past – Telecom XT has been the best performing one, but I’ve always found their pricing a bit high and Vodafone is just all round poor in both customer service and data performance. With the current introduction of 4G/LTE by all the networks, it’s a whole another generation of technology and what’s been a good or bad network in the past, may no longer apply, but we need to wait another year or so for the coverage and uptake to increase to see how it performs under load.
For now the low cost and free data sharing of up to 5 devices will keep me on 2degrees for quite some time. If someone else was paying, maybe I’d consider Telecom XT for the bit better performance, but the value of 2degrees is too good to ignore.
Like anything, your particular use case and requirements may vary – shop around and see what makes sense for your requirements.
Traditional computer operating systems have been around for a while, long enough that concerns around physical security have been well addressed. We understand the value and power that the information on our computers can provide to an attacker, so we have locked them down with features such as disk encryption, passphrase protected lock screens and techniques to prevent unwanted DMA attacks via high speed buses.
Yet despite the massive development of mobile devices technology in the past several years, a number of these features didn’t manage to make their way into the mobile operating systems as defaults. Whilst we take the time to setup disk encryption on our laptops and maybe desktops, we tend not to bother securing our mobile devices, possibly due to the perception of them being less risky to have exposed, or that they are less attractive targets.
Even a relatively paranoid IT geek like myself with an encrypted laptop, secure passphrases, and VPNs, still had a mobile phone that was protected with nothing more than it’s physical proximity to myself. Anyone gaining physical access to my phone could unlock it, whether it be by guessing a trivial unlock pattern, or by attaching it to another computer and reading the unencrypted filesystem.
And as these mobile devices have increased in functionality, so has the risk of an attacker getting hold of the device. When a mobile phone did nothing but phone calls and txts, having someone gain access would be more of a annoyance when they rack up a bill or prank call your contacts, than a serious risk.
But rather than leave it there, we started adding other productivity features – email, so we could keep in touch on the go. Instant messaging. Fully featured web browers that sync account details, bookmarks and history with your desktop. Banking applications. Access to shared storage solutions like Dropbox. Suddenly a mobile device is a much more attractive target.
And even if we decide that the mobile apps are too limited in scope, there’s the risk of an attacker using the information such as credentials stored on the device to gain full access to the desktop version of these services. Having an email application that limits the phone to the inbox can reduce risk by protecting your archives, but not if the attacker can obtain your full username/passphrase from the device and then use it to gain full access with some alternative software.
Remember that obtaining credentials from a device isn’t hard – the credentials have to be kept in some decrypted format somewhere on disk, so even if they’re hashed/obfuscated in some form, they’ll still have the key that enables them to be exposed somewhere on disk.
A quick grep through the /data/ volume on my phone revealed numerous applications that had my passphrases in plain text, extremely easy pickings for an attacker.
Mmmm plain text passwords. :-)
I was getting increasingly concerned with this hole in my security, so recently having replaced my Galaxy Nexus with a Samsung Galaxy Note II, I decided to set it up in a more secure fashion.
Android added disk encryption in Android 3, but it’s suffered two main issues that limits it’s usefulness:
The first point I can’t do much about without digging into the low guts of Android, however the second is fixable. My personal acceptable trade-off is a weaker lock screen using a pattern, but being able to have a secure disk encryption passphrase. This ensures that if powered off, an attacker can’t exploit my data and the passphrase is long and secure, but if the phone is running, I take a compromise of security for convenience and ease of use.
There’s still the risk of an attacker installing malware on the non-encrypted OS portion of the mobile device, however if I lose physical access of my phone in an untrusted environment (eg border security confiscation) I can reload the OS from backup.
To setup disk encryption on Android 4 without losing pattern unlock, instead of adjusting via the settings interface, you need to enable it via the shell -easiest way is via the ADB shell in root mode.
Firstly you need to enable developer mode in Settings -> About Phone by tapping the build number multiple times, until it tells you that the developer mode has been unlocked. Then inside Settings -> Developer options, change the “Root Access” option to “Apps and ADB”.
Enable ADB root for all the fun stuff!
Secondly, you need a workstation running the latest version of ADB (ships with the Android ADK under platform-tools) and to connect your phone via USB. Once done, you can enable disk encryption with the following commands (where PASSWORD is the desired encryption passphrase).
user@laptop # adb root user@laptop # adb shell root@phone:/ # root@phone:/ # /system/bin/vdc cryptfs enablecrypto inplace PASSWORD
Your Android device will then restart and encrypt itself. This process takes time, factor up to an hour for it to complete it’s work.
Android phone undergoing encryption; and subsequent boot with encryption enabled.
Once rebooted, your existing pattern based unlock continues to work fine and all your private data and credentials are now secured.
The Linux kernel has had a long history in the mobile space, with the successes and benefits of the OS in the embedded world transferring across to the smart phone and tablet market once devices evolved to a level requiring (and supporting) powerful multitasking operating systems.
But whilst there had been other Linux-based mobiles before, it wasn’t until Android was first released to the world by Google that Linux began to obtain true mass-maket consumer acceptance. With over 1 billion devices activated by late 2013, Android is certainly the single most successful mobile Linux distribution ever and possibly even the single largest mobile OS on basis of number of devices sold.
Whilst Open Source and Free Software [By Free Software I mean software that is Libre, ie Free as in Freedom, rather than Free as in Beer] had historically succeeded strongly in the server space, it always suffered limited mass market appeal in the desktop. With the sudden emergence and success of Android, proponents of both Open Source and Free Software camps could enjoy a moment of victory and success. Sure we may not have won the desktop wars, and sure it wasn’t GNU/Linux in the traditional sense, but damnit, we had a Linux kernel in every other consumer device, something worth celebrating!
Whilst Android still features the Linux kernel, it differs from a conventional GNU/Linux system, as it doesn’t feature the GNU user space and applications. When building Android, Google took the Open Source Linux kernel but threw out most of the existing user space, instead building a new Apache-licensed user space designed for consumers and interaction via touch interfaces.
For Google themselves, Android was a way to prevent vendors like Microsoft or Apple getting a new monopoly in the mobile world where they could then squeeze Google out and strangle their business in the new emerging market – a world where Microsoft or Apple could dictate what browser or search engine that a user could use would not be in Google’s best financial interests and it was vital to take steps to prevent that from being possible.
The proposition to device vendors is that Android was an answer to reducing their R&D costs to compete with incumbent market players, making their devices more attractive and allowing some collaboration with their peers via means of a common application platform which would attract developers and enable a strong ecosystem, that in turn would make Android phones more attractive for consumers.
For Google and device vendors, this was a win-win relationship and it quickly began to pay off.
Yet even as soon as we started consuming the delicious Android desert (with maybe a slightly dubious Google advertising crust we could leave on the side), we found the taste souring with every mouthful. For whilst Google and device vendors brought into the idea of Android the operating system, they never brought into the idea of the Free Software movement which had lead to the software and community that had made this success possible in the first place.
To begin with, unlike the GNU/Linux distributions pre-dating Android which generally fostered collaboration and joint effort around a shared philosophy of working together to make a better system, Android was developed in a closed-room model, with Google and select partners developing new features in private before throwing out completed releases to coincide with new devices. It’s an approach that’s perfectly compliant with Open Source licensing, but not necessarily conducive to building a strong community.
Even the open source nature of the OS was quickly tainted, with device vendors taking Android and instead of evolving the source code as part of a community effort, they added in their own proprietary front ends and variations, shipped devices with locked boot loaders preventing OS customisation and shoved binary drivers and firmware into their device kernels.
This wasn’t the activity of just a few bad vendors either. Even Google’s own popular “Google Nexus” series targeted at developers of both applications and operating system requires proprietary blobs to get hardware such as cellular radios, WiFi, Cameras and GPUs to function. [Depending whom you ask, this is a violation of the Linux kernel’s GPLv2 license, but there is disagreement amongst kernel developers and the fears that a ban on kernel proprietary drivers will just lead to vendors moving the proprietary blobs to user space, a legally valid but still ethically dubious approach.]
Google’s main maintainer for AOSP recently departed Google over frustrations getting Qualcomm to release drivers for the 2013 revision of the popular Nexus 7 tablet, which illustrates the hurdles that developers face when getting even just the binaries from vendors.
Despite all these road blocks thrown up, a strong developer community has still managed to form around hacking on the Android source code, with particular credit to Cyanogenmod a well polished and very popular enhanced distribution of Android, Replicant which seeks to build a purely free Android OS replacing binary blobs along the way, and FDroid a popular alternative to the “Google Play” application store offering only Free Software licensed applications for download.
It’s still not perfect and there’s a lot of work left to do – projects like Cyanogenmod and Replicant still tend to need many proprietary modules if you want to make full use of the features of your device. The community is working on fixing these short comings, but it’s always much more frustrating having to play catch up to vendors, rather than working collaboratively with them.
But whilst this community effort can resolve the issue of proprietary drivers and applications and lead us to a proper Free Software Android, there is a much more tricky issue coming up which could cause far greater headaches.
In order to resolve the issue of Android version fragmentation amongst vendors causing challenges for application developers, Google has been introducing new APIs inside a package called “Google Play Services”, which is a proprietary library distributed only via the Google Play application store.
Any application that is reliant on this new library (not to mention over existing proprietary components such as Google Cloud Messaging used for push notifications) will be unable to run on pure Free Software devices that are stripped of non-free components. And whilst at the immediate moment the features offered by this API are mostly around using specific Google cloud-based APIs and features which are non-free by their very nature, there’s nothing preventing more and more features being included in this API in future, reducing the scope of applications that will run on a Free Software Android.
If Google Play Services proves to be a successful way for Google to enforce consistency and conformity on their platform to tackle the fragmentation issues they face, it’s not inconceivable that they’ll push more and more library functions into proprietary layers distributed via the Play Store like this.
But if Google chooses to change Android in this way, I feel that it will be inappropriate to continuing calling Android an Open Source or Free Software operating system. Instead it will be better described as a proprietary operating system with an open core – in similar fashion to that of Apple’s MacOS.
Such an evolution could lead to two distinct forks of Android being created:
Some readers will ponder why having some proprietary components is such a concern – who really wants to hack around with drivers or application compatibility APIs? Generally they’re not the most exciting part of computers [subjectively speaking of course] and on some level I can understand this mindset.
But proprietary software chunks are more than just being an annoyance to developers who want to tinker. Proprietary software makes your device opaque, obscuring what the software is doing, how it works and how it can be (ab)used.
The Google Play application has the capability to install content on your phone, a feature often used by users to install applications to their device from their browser. But does the source code of the the Google Play application ensure that it can never happen without your awareness? There’s already due cause to distrust the close association between companies like Google and the NSA, without the ability to see inside the software’s source code, you can’t be sure of it’s capabilities.
Building applications around proprietary APIs like Google Play Services removes the freedom of a user to decided to replace calls to proprietary systems to free ones. It may be preferable to use a Free Software mapping API rather than Google’s privacy lacking Maps offering for example, but without the source code, it’s not possible to make this change.
Even something as innocent as a driver or firmware of hardware such as the GSM modem could be turned into a weapon by a powerful adversary, by taking advantage of backdoors in the firmware to deliver malware to spy on an individual – whether for the “right” reasons or not, depends on your moral views and whom is doing the spying at the time.
Admittedly a pessimistic view, but I’ve laid out my personal justifications for taking this approach before and believe we need to look at how this technology could potentially (hopefully never) be used against individuals for immoral reasons.
I think Android illustrates the differences between Open Source and Free Software extremely well. Whilst Android is licensed under an Open Source license, it doesn’t have the same philosophy of Free Software.
It’s source code is open because it provided Google with a commercial advantage, not because Google believe that user freedom is important. Google and their partners have no qualms about making future applications and/or features proprietary, even at the detriment to developers and users by restricting their freedoms to understand and modify the software in their device.
Richard M. Stallman (RMS), the founder of the Free Software movement, wrote about the differences in Free Software vs Open Source and tells how whilst these two different ideologies have overlapping goals, at times they also differ. In some ways the terminology Open Source can be dangerous, as it lets us lose sight of the real reasons why software needs to be Free for Freedom’s sake above all.
Interestingly, despite how strongly I feel about Free Software, I’ve found it somewhat personally easy to ignore concerns of proprietary software on mobiles for a prolonged period of time. In many ways, I see my mobile as just a tool and not a serious “real” computer like my GNU/Linux laptop where I conduct most of my digital activities. It’s possibly a result of my historical experiences with the devices, starting off using mobiles when they were just phones only and having had them slowly gain more capabilities around me, but always being seen as “phones” rather than “pocket computers”.
I’m certainly a digital native, a child of the internet generation separated from my parent’s generation by being the first to really grow up with widely available internet connectivity and computers. But to me, computers are still laptops and servers, despite having a good understanding of the mobile space and using mobile devices every day to possibly excessive amounts.
Yet for the current and next generation growing up, mobile phones and tablets are *the* computer that will define their learning experiences and interaction with the world – they may very well end up never owning a conventional computer, for the old guard of Windows, Linux and PC are gone, replaced with iOS, Android and handhelds.
It’s clear that mobiles operating systems are the platform of the future, it’s time we consider them equals with our conventional operating systems and impose the same strict demands for privacy and freedom that we have grown to expect onto the mobile space. I know that personally I don’t trust my Android mobile even one tenth as much as I trust my GNU/Linux laptop and this is unacceptable when my phone already has access to my files, my emails, my inner most private communications with others and who knows what else.
So the question is, how do we get from the Kinda-Propietary/Android we have now, to the Free/Android that we need?
I know there are some who will take a purist approach of running only pure Free Software Android and ignoring any applications or features that don’t run on it as-is. Unfortunately taking this approach will inevitably lead to long term discrepancies between the mass market Android OS and the Free Software purists pulling the OS feature set in different directions.
A true purist risks becoming a second class citizen – we are already at the stage where not being able to run popular applications can seriously restrict your ability to take part in our world – consider the difficulties of not being able to load applications needed to use public transport, do banking (online or NFC banking) or to communicate with friends due to all these applications requiring a freedom impacting proprietary layer.
It will be difficult to encourage users and application developers to use a Free Software Android build if they discover their existing collection of applications that rely on various proprietary APIs and library features no longer work, so we need to be somewhat pragmatic and make it easier for them to take up Free Software and still run proprietary applications on top of a free base, until such time as free alternatives arise for their applications.
I think the solution is a collection of three different (but all vital) efforts:
Whether we like it or not, Google’s version of Android will be the platform than the majority of developers target long term. It doesn’t suit all developers, but it has suited most Free (as in beer and/or Freedom) and paid application developers for Android well enough for a long period already that I don’t see it being easy to de-rail that momentum.
If we can re-implement Google’s proprietary layers to a level sufficient for maintaining compatibility with the majority of these applications, it opens up some interesting possibilities. A Free/Android mobile with a Free/PlayServices API layer developed using the documented API calls published by Google is entirely possible and would allow users to run a Free/Android mobile and still maintain support for the majority of public applications being released for the Android platform, even if they use more and more proprietary API features.
Such a compatibility layer will enable users to run applications on their own terms – a user might decide to only run Free as in Freedom software, or they could decide that running proprietary software is OK sometimes -and that’s an acceptable choice, but the user is the one that should be making it, not Google or their device vendor.
Potentially we could take this idea a step further and re-implement features like contact and setting synchronisation against a Free Software server that technically capable users can choose to setup on their own servers, giving them the benefits of cloud-type technologies without loss of freedoms and privacy that takes place if using the Google proprietary features.
I’m not alone in these concerns – neither RMS or the Free Software Foundation (FSF) have been idle on this issue – RMS has an excellent write up on the freedom of Android here, and on a more mainstream level, the FSF is running campaigns promoting freeing Android phones and encouraging efforts to keep the platform Free as in Freedom.
I’m currently taking steps to move my Android Mobile off various proprietary dependencies to Free Software alternatives – it’s going to be slow and gradual and it will take time to determine replacements for various applications and libraries.
I haven’t done much in the way of Android application development, but I’m not afraid to pick up some Java if that’s what it takes to fill in a few gaps to get there – and if it means reverse engineering some features like Google Play Services, I’ll go down that path if need be.
Because Free Software computing is vital for privacy, vital for security and vital for a free society itself. And if the cost is a few weekends hacking at code, it’s a price well worth paying.
Having moved to AU means that I now have two cell phones – one with my AU SIM card and another with my NZ SIM card which I keep around in order to receive the odd message from friends/contacts back home and far too many calls from telemarketers.
I didn’t want to have to carry around a second mobile and the cost of having a phone on roaming in AU makes it undesirably expensive to keep in touch with anyone via SMS messaging, so went looking for a solution that would let me get my SMS messages from my phone to my laptop and phone in a more accessible form.
I considered purchasing an SMS gateway device, but they tend to be quite expensive and I’d still have to get some system in place for getting messages from the device to me in an accessible form.
Instead I realised that I could use one of the many older Android cellphones that I have lying around as a gateway device with the right software. The ability to run software makes them completely flexible and with WiFi and 3G data options, it would be entirely possible to leave one in NZ and take advantage of the cheaper connectivity costs to send SMS back to people from within the country.
I was able to use an off-the-shelf application “SMS Gateway” to turn the phone into an SMS gateway, with the option of sending/receiving SMS messages via HTTP or SMTP/POP3.
However emails aren’t the best way to send and reply to SMS messages, particularly if your mail client decides to dump in a whole bunch of MIME data. I decided on a more refined approach and ended up writing a program called “SMStoXMPP“.
Like the name suggestions, SMStoXMPP is lightweight PHP-based SMS to XMPP (Jabber) bi-directional gateway application which receives messages from an SMS gateway device/application and relays them to the target user via XMPP instant messages. The user can then reply via XMPP and have the message delivered via the gateway to the original user.
For me this solves a major issue and means I can leave my NZ cell phone at my flat or even potentially back in NZ and get SMS on my laptop or phone via XMPP no matter where I am or what SIM card I’m on.
To make conversations even easier, SMStoXMP does lookups of the phone numbers against any CardDAV address book (such as Google Contacts) and displays your chosen name for the contact. It even provides search functions to make it even easier to find someone to chat to.
Chatting with various contacts via SMStoXMPP with Pidgin as a client.
I’ve released version 1.0.0 today, along with documentation for installing, configuring gateways and documentation on how to write your own gateways if you wish to add support for other applications.
Generally it’s pretty stable and works well – there are a few enhancements I want to make to the code and a few bits that are a bit messy, but the major requirements of not leaking memory and being reliably able to send and receive messages have been met. :-)
Whilst I’ve only written support for the one Android phone base gateway, I’m working on getting a USB GSM modem to work which would also be a good solution for anyone with a home server.
It would also be trivial to write in support for one of the many online HTTP SMS gateways that exist if you wanted a way to send messages to people and didn’t care about using your existing phone number.
I’ve been using Firefox Mobile on Android for a while (thanks to the fact that it means I can use Firefox Sync between my laptop and mobile to share data). Overall it’s pretty good and the last few releases have fixed up a lot of the past stability issues and UI problems, it’s in a pretty decent state now.
One of the unfortunate problems I’ve had with it until recently is that the application was refusing to import custom certificate authorities. Whilst Android has it’s own CA store, add on browsers (inc Firefox Mobile) can have their own CA stores and the manageability of these can vary a lot.
In the case of Firefox Mobile, the ability to manage certificates was not ported across from the desktop version, meaning that none of my web applications would validate against my custom CA.
However as a passable solution, it’s now possible to import the CA file by downloading a PEM version of the CA certificate in the browser. Just upload a copy of the PEM formatted certificate to a webserver and download the file with the browser to install.
Installing CAs into Firefox Mobile (PEM formatted file).
Now the biggest problem left is sites and applications that have poorly written user agent detection and assume that the only mobile devices that possibly exist are devices that have the iPhone or stock Android user agent. :-( *glares at Atlassian in particular*
Recently I’ve been using a few of my older Android smartphones for various projects where I’ll need to have the phones in a remote location or having the phones in a non-easily accessible location.
A conventional Linux system would be remote managed via SSH, but I need access to the graphical environment of the Android phones to run standard GUI touch applications. Generally I avoid VNC due to the fact it’s an unencrypted, insecure protocol, but in my case I was able to resolve the security concerns by tunnelling all my connections via a SSL VPN, which made VNC acceptable.
A few friends have asked me about VNC solutions for Android – in my case, I’ve been using a program helpfully named “VNC server” (click here for app store) on a mix of devices including older Android 2.2 devices running Cyanogenmod.
Generally it works well, but there are a few limitations with the application to be aware of:
Once the application is installed and configured, it’s easily accessed using any regular VNC application from your computer. The main commands are:
Note that there is one confusing quirk – when the phone screen locks, VNC displays the last thing that was on the screen. This can be confusing when you login and the phone doesn’t respond to any actions.
Something isn’t quite right here….
The fix is to press the home key which makes the lock screen display and then to swipe in the usual manner by clicking and dragging the mouse. Sometimes the lock screen can take a while to render on VNC, so I tend to end up pressing home and then dragging before it even renders the lock screen.
You will also want to disable screen rotation, otherwise if you leave the phone in a rotated state inadvertently, it makes for a very annoying user experience.
Control all the phones!
So far I’ve found it pretty useful, but because of the way the Android OS handles memory, I would hesitate before relying on it 100% – there’s always the risk that my phone may have another process wanting to consume memory and it then suspending the VNC process to allocate more. I suspect there are some tweaks/hacks I could apply to the platform to make it more robust and there may be some stuff already in the app store that will help.
Having said all this, it’s worth nothing that cheap cost and high feature set of Android phones makes them an idea hacking platform and I’m using them for a few projects already such as cheap GSM SMS gateways, as well as considering using the older phones as wireless IP cameras.
Sadly Android is giving me further reason for dissatisfaction with the discovery that Google Play refuses to correctly detect the current country I’m in and is blocking content from them.
Being citizens of New Zealand and living in Australia, both Lisa and I have a mixture of applications for services in both countries – a good example being banking apps, where we tend to want to be able to easily manage our financial affairs in both countries.
The problem is, that all my phones are stuck as being in New Zealand, whilst Lisa’s phone has decided to be stuck to AU. This hasn’t impacted me much yet, since the Commbank application isn’t region locked and being treated by Google as being “inside” NZ I can install the ANZ Go Money application from within AU.
However because Lisa’s phone is the other way around and stuck in AU, she’s unable to install both the ANZ Go Money and Vodafone NZ My Account application, thanks to the developers of both applications region locking them to NZ only.
What makes the issue even more bizarre, is that both of Lisa and my Google Accounts were originally created in New Zealand yet are behaving differently.
Thou shalt not pass!
There’s a few particularly silly mistakes here. Firstly ANZ shouldn’t be assuming their customers are always in NZ, us Kiwis do like to get around the world and tend to want to continue managing our financial affairs from other countries. The app is clearly one that should be set to a global region.
I suspect they made this mistake to try and avoid confusing AU and NZ ANZ customers with different apps, but surely they could just make it a bit clearer in the header of the app name…
Secondly Google… I’m going to ignore the fact that they even offer region locking for applications (urgghhg DRM evil) and going to focus on the fact that the masters of information can’t even detect what fucking country a user is in properly.
I mean it must be so hard to do, when there are at least 4 ways of doing it automatically.
Sure, none of these are guaranteed to be perfect, but using at least 3 of the above and selecting the majority answer would lead to a pretty accurate system. As it is, I have several phones currently powered up around my apartment in AU, none of which are validating as being in AU at all:
Hey guys, I’m pretty sure my OPTUS CONNECTED PHONE is currently located in AU.
Instead there is an option buried somewhere in Google’s maze of applications and settings interfaces which must be setting the basis for deciding the country our phones are in.
But which one?
To make things even more annoying is that I can’t even tell if making a change helps or not. How do I tell if the phone is *reading* the new settings? I’ve done basic stuff like re-sync Google, but also have gone and cleared the Google Play service cache and saved settings on the phone, to no avail.
It’s a bit depressing when even Apple, the poster child for lock down and DRM makes region changing as simple as signing into iTunes with a new country account and installing an application that way.
Restricting and limiting content by geographic regions makes no sense in a modern global world and it’s just sad to see Google continuing to push this outdated model instead of making a stand and adopting 21st century approaches.
If anyone knows how to fix Google to see the correct country it would be appreciated… if you know how to get Google to fix this bullshit and let us install the same applications globally, even better.
Having just flown from Sydney AU to Christchurch NZ, my Galaxy Nexus suddenly decided to finally offer me the Android 4.2.2 upgrade.
Since I got the phone in 2012, it’s been running Android 4.1 – I had expected to receive Android 4.2 in November 2012 when it was released by Google since the Galaxy Nexus is one of Google’s special developers phones which are loved and blessed with official updates and source code.
However the phone has steadily refused to update and whilst I was tempted to build it from source again, seeing as 4.2 lacks any particular features I wanted (see release changes), there was little incentive to do so. However after 4.2.2 was magically revealed to me following changing countries, I decided was nagged to death to update and ended up doing so… sadly I wish I hadn’t….
Google have messed with the camera application yet again completely changing the UI – the menu now appears where ever you touch the screen, which does make it easier to select options quickly in some respects, but they’ve removed the feature I use the most – the ability to jump to the gallery and view the picture you just took, so it’s not really an improvement.
Secondly the Android clock and alarm clock interface has been changed yet again – in some respects it’s an improvement as they’ve added some new features like stop watch, but at the same time it really does feel like they change the UI every release (and not always in good ways) and it would be nice to get some consistency, especially between minor OS revisions.
However these issues pale in comparison to the crimes that Google has committed to the lock screen…. Lock screens are fundamentally simple, after all, they only have one job – to lock my phone (somewhat) securely and prevent any random from using my device. As such, they tend to be pretty consistent and don’t change much between releases.
Sadly Google has decided that the best requirement for their engineering time is to add more features to the lock screen, turning it into some horrible borg screen with widgets, fancy clocks, camera and all sorts of other crap.
Go home lockscreen, you’re drunk. So, so, drunk.
Crime 1 – Widgets
The lock screen now features widgets, which allow one to stick programs outside of the lockscreen for easy access (defeating much of the point of having a lock screen to begin with) and offering very limited real benefit.
Generally widgets serve very limited value, I use about 3 widgets in total – options for tuning on/off hardware features, NZ weather and AU weather. Anything else is generally better done within an actual application.
Widgets really do seem to be the feature that every cool desktop “must have” and at the same time, have to be one of the least useful features that any system can have.
Crime 2 – Horribly deforming the pattern unlock screen
With the addition of the widgets, the UI has been shuffled around and resized. Previously I could unlock by starting my swipe pattern from the edge of the device’s physical screen and drawing my pattern – very easy to do and quick to pick up with muscle memory.
However doing this same unlock action following the Android 4.2 upgrade, will lead to me accidentally selecting the edge of the unlock “widget” and instead of unlocking, I end up selecting a popup widget box (as per my screenshot) and then have to mess around and watch what I’m doing.
This has to the single most annoying feature I’ve seen in a long time purely because it impacts me every single time I pickup the phone and as a creature of habit, it’s highly frustrating.
And to top this off, Android now vibrates and makes a tone for each unlock point selected. I have yet to figure out what turns this highly irritating option off, I suspect it’s tied into the keyboard vibration/tone settings which I do want…
Crime 3 – Bold Clocks
We’ve had digital clocks for over 57 years, during which time I don’t believe anyone has ever woken up and said “wow, I sure wish the hours were bolder than the minutes”.
Yet somehow this was a good idea and my nicely balanced 4-digit 24-hour clock is unbalanced with the jarring harsh realisation that the clock is going to keep looking like a <b> tag experience gone wrong.
I’m not a graphical designer, but this change is really messing with my OCD and driving me nuts… I’d be interested to see what graphic designers and UX designers think of it.
So in general, I’m annoyed. Fucked off actually. It’s annoying enough that if I was working at Google, I’d be banging on the project manager’s door asking for an explanation of this release.
Generally I like Android – it’s more open than the competing iOS and Windows Mobile platforms (although it has it’s faults) and the fact it’s Linux based is pretty awesome… but with release I really have to ask… what the fuck is Google doing currently?
Google has some of the smartest minds on the planet working for them, and the best they can come up with for a new OS release is fucking lock screen widgets? How about something useful like:
The changes made in the 4.2 series are shockingly bad, I’m at the stage where I’m tempted to hack the code and revert the lockscreen back to the 4.1 version just to get my workflow back… really it comes down to whether or not the pain this system causes me ends up outweighing the costs/hassle of patching and maintaining a branch of the source.