Tag Archives: android

Android ICS & the MTP pain

4 Replies

With Android 3.0, Google switched the way of accessing files on Android devices from USB mass storage device behaviour, to Media Transfer Protocol, a standard initially created by Microsoft, but later developed into part of the USB standard for media devices.

For most Android users, new ICS phones are where they will first encounter this new approach to sharing file content. Phones previously upgraded from Android 2.x may still provide the conventional mass storage device behaviour, but any new ICS devices force the option of either MTP or the camera-focused PTP.

MTP is one of those technologies designed with the best of intentions – the fundamental design around it is that it abstracts the underlying filesystem, enabling the phone/device to store the files in whatever way suits it best and then exposing them to the computer.

Unlike the traditional mass storage mount, the files can be available to both the phone and the computer at the same time, whereas previously Android applications would lose access to the “sdcard” storage space when mounted on the computer.

It’s a great idea, except that it’s nowhere near as compatible with computers as a generic mass storage device is. Whilst Windows has native support for MTP, thanks to MS being the original creator of the standard, MacOS has no native support and requires a download, and Linux systems are all kinds of fail when it comes to MTP.

Considering I use a Linux laptop, this situation is very unhelpful – reading online suggests various options (the arch wiki was quite helpful):

  • Use of GNOME with GVFS or KDE’s KIO to connect the storage using the GUI’s device handling functionality. Unfortunately with my tests performed using Nautilus (GNOME’s file manager) on Debian Testing (recent app versions), I wasn’t able to connect to the device, instead experiencing a lot of timeout errors.
  • Linux has a FUSE driver that is able to read MTP, with corresponding usertools called mtpfs – however in my tests, this driver has some serious usability, stability and trust issues, thanks to it repeatedly failing and randomly unmounting, as well as hiding many files that are known to exist.
  • An alternative driver called “go-mtpfs” (written in Go) which provides user space connectivity to MTP Android devices at reasonable speeds and a reasonable UI for mounting the drives.
  • Failing these, a network transfer over WiFi could be an option, such as Samba or SSH. Bluetooth is also a possibility, but the performance is quite poor for numerous file transfers.

I’ve ended up using go-mtpfs which appears to work quite nicely, at least with my ICS Google Nexus phone, however it’s annoying that it’s not in the Debian repositories yet.

Whilst I understand the design decisions and issues solved by Google moving to using MTP rather than USB Mass Storage, the sudden deprecation of a feature that previously existed is a real pain for a user like myself who is wondering why their new phone won’t connect to their computer…

Personally I think Google should have included a legacy USB Mass Storage option for at least another release (ie till Android 5) to have provided time for desktop OS developers to improve the handling of this protocol.

It would also have been useful if they had included a “Install Android Driver” option to expose a small USB mass storage drive (eg 20MB) that includes drivers/apps for common platforms such as MacOS and Linux, so that users could connect and load on drivers without needing

For now, go-mtpfs seems to work OK, meanwhile one of the GVFS/GNOME developers is re-working the MTP handling to properly support Android devices, so hopefully we won’t have the headaches around this protocol for too much longer…

Android OpenVPN & Jelly Bean

2 Replies

Last night my Galaxy Nexus finally got the Jelly Bean update pushed to it via Over-The-Air – I’m not sure why it’s taken until now to get it, but ICS has been working fine so I never bothered to build Android from source again.

It was slightly disturbing that the update came down over 3G data, whilst I have a fair bit of cap, a lot of NZders are on pretty low cellphone datacaps and the update is around 160MB.

The upgrade was pretty seamless, however it broke my Openvpn for Android setup, preventing me from connecting to any of my servers or email. According to the application, there is a known issue that when the OS updates, you need to re-establish the trust relationship with the Android keystore, which you can do by editing the VPN and re-selecting the certificate and selecting “allow”.

Unfortunately, that didn’t work for me, it would keep repeating the error and refusing to run.  There wasn’t much useful in adb logcat either:

I/ActivityManager(  303): Displayed de.blinkt.openvpn/.MainActivity: +213ms
I/ActivityManager(  303): START {act=android.intent.action.MAIN cmp=de.blinkt.openvpn/.LaunchVPN (has extras) u=0} from pid 4071
I/ActivityManager(  303): START {flg=0x20000 cmp=de.blinkt.openvpn/.LogWindow u=0} from pid 4071
I/keystore(  130): uid: 1000 action: t -> 1 state: 1 -> 1 retry: 4
I/keystore(  130): uid: 1000 action: x -> 1 state: 1 -> 1 retry: 4
V/OpenSSL-keystore( 4071): keystore_bind_fn
V/OpenSSL-keystore( 4071): keystore_engine_setup
V/OpenSSL-keystore( 4071): keystore_loadkey(0x5c30c3d0, "1000_USRPKEY_mobile-jethro", 0x0, 0x0)
I/keystore(  130): uid: 10067 action: b -> 7 state: 1 -> 1 retry: 4
W/keystore_client( 4071): Error from keystore: 7
V/OpenSSL-keystore( 4071): Cannot get public key for 1000_USRPKEY_mobile-jethro

I had a read and came across this bug report in Android, suggesting that the names of some certificates could be a problem.

My certificate was mobile-jethro.p12, so I named it to mobile.p12 and imported it again – which resolved the problem! Bit of a nasty character handling bug it seems….

RS232 Shifter Board

2 Replies

As part of my project to make an Arduino based watchdog for my home server, I went and picked up a proper soldering iron today and assembled the RS232 Shifter Circuit I had purchased from Sparkfun/Mindkits.

I ended up getting a Goot soldering iron – it wasn’t too cheap, but the reviews I was reading online suggested it was the next best thing to a proper soldering station and a quality Japanese tool. It’s also 46W, so much better for the lead-free solder I’m using, compared to the crappy 15W DSE brand generic iron I was using previously.

Tool of the gods.

It made a massive difference to the soldering efforts, the lead-free solder melted without issue and flowed smoothly – plus the fine point conical tip made getting the solder into key areas a lot easier.

The tips are changeable and I’ve been advised by an electronics fixing friend to get a set of tip types, as certain ones suit different tasks better than others.

Whilst the new soldering iron is a huge improvement, my soldering skill is still a bit poor, I had a few text-book joins where the solder flowed perfectly and bonded with a smooth shiny surface, but a lot of my joins tend to be slightly one sided, where the solder hasn’t flowed right around the components lead and PCB hole which makes then a bit weak.

I’m also tending to leave too much solder on the component lead, which leaves larger joins than really needed, so need to work on how much solder I apply.

Parents, avert your children’s eyes from this soldering monstrosity!

On the flip side, the top looks great ;-)

After constructing the circuit, I gave it a test by connecting the TX and RX of the shifter circuit to the TX and RX on the Arduino, which basically uses the Arduino as an UART to RS232(over USB) adapter.

Then I connected the new DB9 port to my laptop and typed messages into the Arduino serial monitor, to have them passed through my new circuit and into my laptop’s DB9 port.

Amazingly, things worked first time and I’m able to successfully receive data via the shifter board – in my quick test TX from the shifter board wasn’t working, but it may have been a serial speed/config issue that I just need to tweak in the OS, rather than the board itself.

Arduino IDE to Minicom via an Arduino, my new shifter circuit and my laptop’s USB to RS232 adapter.

The next step is connecting this shifter circuit to the digital I/O ports on the Arduino and using the software serial library to add a second serial port to the device. I can then do the same to the HTC Magic serial break out and I’ll basically have an Arduno with 3 serial ports in total –  the onboard UART, the DB9 PC via shifter and the HTC magic.

Once that’s done, it’s just a case of writing the software needed to turn the collection of hacks into a hardware watchdog and serial remote access system. :-)

And once that’s all working, I’ll build a final solution using one of the Arduino prototyping shields from Freetronics, which will include all the shifter circuitry onboard and just provide a 2×5 IEC pin header to plug a ribbon cable directly into the server’s motherboard serial port header.

Keeping Android Wifi Awake

6 Replies

I run a number of backgrounded applications on my Android phone, such as Nagios (server monitoring) CSipSimple (VoIP/SIP), OpenVPN (SSL-based VPN) and IMAP idle (push email).

Whilst this does impact battery life somewhat, I’ve got things reasonably well tuned so that the frequency of polling and keepalives is just long enough to prevent firewall timeouts, but long enough to avoid excessive waking of 3G & wifi hardware.

(For example, the default OpenVPN keepalive of 10 seconds is far more aggressive than what is actually needed, in reality, I was able to drop my phone back to one keepalive every 5 minutes – short enough to keep sessions active, but long enough that the transmitting hardware can sleep regularly whilst it isn’t needed).

However one problem I wasn’t able to fix, was the amount that the wifi disconnected – this would really screw up things, since services such as IMAP idle and SIP were trying to run over the VPN, but this would be broken by the VPN being dropped when wifi turned itself off.

I found the fix thanks to a friend who came around and told me about the hidden “Advanced” menu in the wifi network selection page:

When on the wifi network selection screen, you need to press the menu key (not sure what the option is for newer Android phones that don’t have the menu key any more?) and then a single “Advanced” menu item will appear.

Selecting this item will give you a couple extra options, including the important “Keep Wi-Fi on during sleep” option that stops the phone from dropping the wifi connection whenever you turn off the screen.

This resolved my issues with backgrounded services and I found it also made the phone generally perform better when doing any data-related services, since wifi didn’t have to renegotiate with the AP as frequently.

It’s not totally perfect, Android seems to sometimes have an argument with the AP and then drop the connection and waste a minute trying to reconnect, but it’s a lot better than it was. :-)

DAViCal, awkward name, great features

5 Replies

A reoccurring theme of this blog is that I love to be able to use open standards and open source for storing and accessing my information – biggest example is of course IMAP for email, but I also use tools such as Mozilla Sync Server for self-reliant synchronization and backup of client device information, without using external cloud providers.

I’ve been a user of Evolution for almost a decade now – sometimes criticized as the “Outlook of Linux”, Evolution provides mail, calendering, contacts and todo lists to the GNOME desktop, with a pretty large but sometimes slightly buggy feature set. For me personally, it’s always done a great job and it’s my key business productivity tool.

I moved all my mail onto an IMAP server years ago, which makes it easy to shift clients if I ever need to – in my case, pretty much just needing to access mail from both my laptop and smart phone.

However this hasn’t been the case for other key data such as calendering and contacts. A few years ago, the open source calendering solutions available weren’t that well developed, and many clients suffered limitations such as read-only functionality.

Thankfully this has been changing – most clients (*glares angrily at Microsoft Outlook*) now support CalDAV and CardDAV quite reliably, which gives us an open standard that works across different programs, platforms and device types.

  • CalDAV is an open standard for the exchange of calendering and task/todo/memo information between a client and a server.
  • CardDAV is an open standard for the exchange of contact/address book information between a client and a server.

These two standards have a number of implementations, both open source and proprietary, of note is Apple Calender Server, which is Apple’s open source implementation; and DAViCal, an open source LAMP based server solution that is becoming quite popular.

I’ve used both solutions – my employer runs an Apple Calender Server after getting fed up at not having free/busy between engineers. Whilst we ended up running a MacOS server, the Linux ports have improved and there are resources for setting it up on a Linux or even BSD host.

Apple Calender Server works reasonably well with Evolution, I never have any issue booking events, however Evolution appears unable to accept or deny meeting requests, forcing me to go to the calender server web-interface which is actually pretty horrific.

I decided I wouldn’t use it for my own personal calendering, even if I went to the effort for porting it onto my Linux servers, it wouldn’t really be the solution I ideally want as it lacks a lot of features and isn’t as easy to configure as other Linux services.

Instead I had a look at DAViCal. It’s a feature-packed calendering and contacts application developed primarily by Morphoss in Wellington NZ, started by Andrew McMillian of ex-Catalyst IT fame.

Despite having an annoyingly tricky name to type (you try typing it for the 100th time at 3am without typoing on the capitalization!!), the software itself appears reliable and worked across a number of devices when I ran tests.

It’s not perfect, I have some issues with the user interface design, which very functional and effective, it’s not that intuitive to a new user, exposing far too many options to them at the beginning, ideally have a simple/advanced option so a user who just wants to add user calenders and do basic stuff can do so, then dig into more detailed ACLs, tokens, shared calenders, etc as needed.

Naturally it’s open source, so I should stop complaining and hack up some code to demonstrate what I think might be better. Maybe if people would stop stealing my car I’d have time to get something done. :-/

Main Screen Turn On! (Maybe some more 1-2-3 clear setup flows here would be nice, the wall of text is kind of offputting for visual people like myself).

Options! All the options!

The web-based interface is only for administration, there isn’t a web-based calender app provided with DAViCal, instead choose any CalDAV client you wish to use with it, whether it’s web based or client-side.

I haven’t given DAViCal’s feature set a full work out yet, at this stage I’ve just setup my personal calendar, contacts and todo list on both Evolution and my Android ICS phone but haven’t touched meeting requests, shared calenders and free/busy information.

Partly my testing is a bit limited since I’m only running Evolution 2.30.3 (Debian Stable) which is a little outdated and it looks like there’s some functionality missing/broken that might not be an issue any more.

On the mobile side, I’m using “aCal”, an open source Android application written by the DAViCal developer, providing a CalDAV calender, todo list and read-only contact/address book synchronization.

This now means I can add, edit and delete calender and task entries on either my Android phone or Linux laptop via evolution and have it propagate to the other device – although unfortunately this is based on polling, rather than push (looks like push is possible theoretically via an extension to the standard, works with iCal).

Tasks & calendar entries in a bright sunny UI

I can also get read-only copies of all my contact information from Evolution synced through to my phone, but sadly there isn’t support for editing contacts on the Android phone just yet.

I did also consider using LDAP for my address book entries, but CardDAV looks like a better designed solution, it’s very rare that I don’t see “LDAP” and “headache” mentioned in the same sentence, and this comes from someone maintaining and supporting LDAP enterprise environments…

Essentially the main problem with LDAP, is that there isn’t an exact standard for address entries, so what works for one client, might not work 100% for another, along with limited selection of decent applications for actually managing LDAP address databases.

Also some clients treat LDAP assuming it’s going to be a million+ record store and expose different UI compared to that of smaller address books which harm user experience (*glares at Evolution*).

aCal & Android ICS address book integration - note the uneditable edit screen on the right, read-only for now :-(

The other main issue with aCal is that it doesn’t sync with the native OS calender program, but instead provides it’s own. Digging through the documentation and mailing list, this appears to be due to the native application lacking support for some of the functionality needed for a proper CalDAV implementation, so a sync solution would leave certain features missing, although I’d still like the option.

Of course these are limitations of aCal, not DAViCal or the standards themselves – there are some other CalDav & CardDAV sync programs available in the Android market under non-open licenses, which you have the option of trying.

The nice thing about using standards is that you can have multiple vendors competing to make the best product/tool for their customer’s needs, not simply using lock-in to maintain/force a customer base. :-)

Overall DAViCal seems really nice and in my testing has been quite reliable – I’m now moving on to more rigorous testing and am in the process of migrating my calender and contacts information into it, once I start using it daily in the real world the true testing begins.

Keen to take a look at what options I have around exposing some information publicly, eg sharing schedule free/busy with friends on different servers.

Leosticks are a gateway drug

2 Replies

At linux.conf.au earlier this year, the guys behind Freetronics gave every attendee a free Leostick Arduino compatible board.

As I predicted at the time, this quickly became the gateway drug – having been given an awesome 8-bit processor that can run off the USB port and can provide any possibility of input/output with both digital and analogue hardware, it was inevitable that I would want to actually acquire some hardware to connect to it!

Beware kids, this is what crack looks like.

My background into actual electronics hasn’t been great, my parents kindly got me a Dick Smith starter kit when I was much younger (remember back in the day when DSE actually sold components! Now I feel old :-/) but I never quite managed to grasp all the concepts and a few attempts since then haven’t been that successful.

Part of the issue for me is I learn by doing and having good resources to refer to, back then it wasn’t so easy, however with internet connectivity and thousands of companies selling components to consumers offering tutorials and circuit design information, it’s never been easier.

Interestingly I found it hard to get a real good “you’re a complete novice with no clue about any of this” guide, but the Arduino learning resources are very good at detailing how their digital circuits work and with a bit of wikipediaing, got me on the right track so far.

Also not having the right tools and components for the job is an issue, so I made a decision to get a proper range of components, tools, hookup wire and some Arduino units to make a few fun projects to learn how to make this stuff work.

I settled on 3 main projects:

  1. Temperature monitoring inside my home server – this is a whitebox machine so doesn’t have too many sensors in good locations, I’d like to be able to monitor some of the major disk bays, fans, motherboard, etc.
  2. Out-of-band serial management and watchdog restart of my home server. This is more complex & ambitious, but all the components are there – with a RS232 to TTY conversion circuit I can read the server’s serial port from the Arduino and use the Arduno and a transistor to control the reset header on the motherboard to power-restart if my slightly flaky CPU crashes again.
  3. Android controlled projects. This is a great one, since I have an abundance of older model Android phones available and would like a project that allows me to improve my C coding (Arduino) and to learn Java/Dalvik (Android). This ticks both boxes. ATM considering adding an Android phone to the Arduino server monitoring solution, or maybe hooking it into my car and using the Android phone as the display.

These cover a few main areas – to learn how to talk with one wire sensor devices, to earn how to use transistors to act as switches, to learn different forms of serial communication and to learn some new programming languages.

Having next to no current electronic parts (soldering iron, breadboard and my general PC tools were about it) I went down the path of ordering a full set of different bits to make sure I had a good selection of tools and parts to make most circuits I want.

Ended up sourcing most of my electronic components (resister packs, prototyping boards, hookup wire, general capacitors & ICs) from Mindkits in NZ, who also import a lot of Sparkfun stuff giving them a pretty awesome range.

Whilst the Arduinos I ordered supply 5V and 3.3V, I grabbed a separate USB-powered supply kit for projects needing their own feed – much easier running off USB (of which I have an abundance of ports around) than adding yet-another-wallwart transformer. I haven’t tackled it yet, but I’m sure my soldering skills will be horrific and naturally worth blogging about in future to scare any competent electronics geek.

I also grabbed two Dallas 1-wire temperature sensors, which whilst expensive compared to the analog options are so damn simple to work with and can be daisy chained. Freetronics sell a breakout board model all pre-assembled, but they’re pricey and they’re so simple you can just wire the sensors straight back to your Arduino circuit anyway.

Next I decided to order some regular size Arduinos from Freetronics – if I start wanting to make my own shields (expansion boards for the Arduinos), I’d need a regular sized unit rather than the ultrasmall Leostick.

Ended up getting the classic Arduino Eleven/Uno and one of the Arduino USB Droids which provide a USB Host port so they can be used with Android phones to write software than can interface with hardware.

After a bit of time, all my bits have arrived from AU and the US and now I’m already to go – planning to blog my progress as I get on with my electronics discovery – hopefully before long I’ll have some neat circuit designs up on here. :-)

Once I actually have a clue what I’m doing, I’ll probably go and prepare a useful resource on learning from scratch, to cover all the gaps that I found hard to fill, since learning this stuff opens up so many exciting projects once you get past the initial barrier.

Arduino Uno/Eleven making an LED blink. HIGH TECH STUFF ;-)

Push a button to make the LED blink! Sure you can do this with just a battery, switch and LED, but using a whole CPU to read the button state and switch on the LED is much geekier! ;-)

1-wire temperature sensors. Notably with a few more than one wire. ;-)

I’ll keep posting my adventures as I get further into the development of different designs, I expect this is going to become a fun new hobby that ties into my other two main interests – computers and things with blinky lights. :-)

Mozilla Firefox Sync Awesomeness

1 Reply

When I recently rebuilt my laptop, I switched back from Google’s Chromium browser to instead use Mozilla Firefox. Whilst I really liked Chromium, there were a few specific reasons which push me more towards Firefox:

  1. I trust the Mozilla Foundation to uphold my desire for producing a great browser more than I trust Google. This isn’t because I believe Google to be evil, but rather that Google’s motivations are to sell advertising, whereas Mozilla’s is to produce a great browser.
  2. The performance issues that pushed me away from Firefox 3 have largely been rectified, reducing the benefit of Chromium.
  3. Standardising my systems on Firefox, means I can make use of Firefox Sync

This last one is of particular interest to me – Firefox Sync is a feature which allows all your Firefox instances to synchronise data between themselves, using a freely provided public server or your own server.

I think this is one of the most compelling user feature improvements since they released version 1.5 IMHO – the ability to synchronise my bookmarks, credentials, history and best of all, open tabs lists, allows me to effortlessly have multiple devices for browsing without dealing with the frustrating issue of the bookmark or saved password being on the other machine.

For example, I setup a Firefox instance on my Linux laptop, Windows VM and my phone and could see all the tabs between them:

Tabs from Linux laptop's POV

Tabs from Windows VM's POV

Tabs from Firefox on Android's POV

I’ve found this tab synchronization to be one of the most useful features, it makes it so much easier to find an article I was in the process of reading to finish whilst I have free time when traveling.

If you’re one of those people with multiple devices (desktops, laptops, work machine, personal machine, tablets, phones, etc) it makes life so much simpler. And as long as you have Firefox 4+, it ships as a native feature.

For details about using Firefox Sync and how it works see Mozilla’s details on the service and/or read the getting started with Firefox Sync FAQ.

 

As you probably know if you ever read my blog, I’m pretty keen on using an entirely open source stack – and so Firefox Sync is naturally fully open source, including both the client (Firefox) and the server components.

This lack of an open source server kills Chromium for me, since the Chrome sync relies on your Google Account and their servers. :-(

Using Mozilla’s open source sync server isn’t as easy as I’d like it to be… building a working server from their source code and limited documentation is a bit of a mission, particularly when some documentation is outdated and doesn’t apply to the latest commit, or when troubleshooting documentation barely exists.

However I’ve managed to successfully package RPMs against CentOS 5 for sync server and dependencies and am currently running further tests before I release them. Ideally I’ll make some time to build them against CentOS 6 as well (done them against CentOS 5 first, since that’s my current production host OS of choice – also the hardest since the version of python it ships with is too old).

With support for SQLite, LDAP, MySQL and Memcache it’s quite flexible and designed to scale to the huge user volumes that Mozilla undoubtedly have – I’ve been running tests with SQLite, but will be having a play with MySQL and OpenLDAP over the next few days as it would be nice to align it to my existing LDAP server.

Expect another blog post later this week with details on how to obtain the RPM packages, along with instructions on setting up your own sync server. It only took me about 3 days full effort of packaging weird python dependencies and getting a working set of configuration and spec files for the sync server to make this stuff work, so hopefully someone is actually interested!

100% pure freedom phone?

2 Replies

As per my earlier rant about Android’s openness, I’m not particularly happy with all the binary components on my phone, nor am I particularly happy with the Android Market’s control and lack of clarity around licensing.

There’s multiple issues with propietary software and why I’ve always been an advocate for not just open source, but more importantly, software freedom. In particular, I try and structure all my computing environments so that I can:

  • Always customize the applications I use if needed – this could be bug fixes, feature changes, etc.
  • Having advertisement and tracking/spyware free software. I’d rather pay good money for software than have it advertisement supported or selling my information to others.
  • Have no dependence on gatekeepers running centralized services – I prefer to run distributed federated services, such as SMTP, (Email) and XMPP (IM) for communications, rather than relying on proprietary networks (eg imessage, skype).
  • Full control and responsibility for the security and privacy of my own data, rather than outsourcing to cloud providers.

It seems it would be possible to replace most of the proprietary components that Google supplies with open source components, but in a quick search I didn’t find any Android distributions that have this bundled up into an easy packaged solution.

One of the more popular distributions, Cynaogenmod has some nice features and is open source, but isn’t specifically designed to be *only* open source, whereas I want a distribution that focuses on making it easy to find, install and manage open source software only.

So I’m making plans to do a custom build of Android for my phone which will feature only free as in freedom software components, with the exception of the hardware driver binary blobs.

  • Replace Android Market with the all-open source F-Droid application – this market is 100% open source and both the client and server are open source, so you can even start your own market. One particularly good feature, is the ability to install older versions, I’ve been bitten in the past with updates introducing bugs in the past with no rollback.
  • Email is well handled with open standards IMAP and IMAP IDLE – I’ve been using K9 Mail for some time (open source build of Android’s email client with additional tweaks) and it works beautifully. With the IMAP IDLE functionality, my phone gets notified about the new mail message within a few seconds of the mailserver completing the processing of the message through to my inbox.
  • Replace contact sync with an LDAP contact directory and sync tool to go against that. LDAP is supported by most address book applications and is something I want to use for all my contacts to make it easier to move between applications.
  • Obtain an XMPP client to replace google talk with support for any XMPP/Jabber server desired, whether Google or another server. Considering I use my own XMPP server already, this is something that’s been on my list for a while.
  • Use aCal with an open source CalDAV server (such as DAViCal) for sharing calenders between devices.
  • Replace google maps with open street maps.
  • This would also offer the advantage of not needing to use Google’s cloud services for storing my address book information, something I’ve never particularly liked the idea of, but was somewhat forced upon in the early days of Android 1.5.

As part of this change, I would also end up dumping Android Market and going with only open source applications for Android – the downside will of course be less application selection, but the up side would be less crapware, no adware applications and full control to install any version and manage applications better.

And the end result would be a truly free, open source Android OS on my phone, which I have full control over, with all data stored on either my phone or servers under my control.

I’ll be fitting in the work as I get time slowly replacing components till I have a reliable fully open stack on my phone and blogging my progress. :-)

Android Market Immaturity

Leave a reply

Whilst I’m on the war path of Android, there’s a number of major issues that the Android Market has which have been causing me great annoyance lately. It feels very much like Google rushed out a Market application that meets their major requirements, but haven’t put much thought into a lot of how the market will behave in the real world.

 

1. Application Update Management

My IT background has a large component of working with enterprise and corporate organisations, in particular, telecommunications companies. These companies are often known for their annoyingly slow habits of deploying new software:

  1. Determine new software version to use.
  2. Document installation, deployment procedures.
  3. Complete strict testing of applications.
  4. Deploy application.
  5. Test and ensure functionality. If a fault occurs, rollback using the documented procedures.

On the other end of the spectrum, the Android Market has the following behavior:

  1. Find updates. (automatic updating can be turned on/off per application).
  2. Install them.
  3. Don’t like the application following the update? Software bugs? Tough, deal with it.

Whilst I’m hardly going to advocate making test plans for deploying Android updates, I think Google need to take some lessons from the enterprise environments – software will always surprise you with bugs, so plan for rollback options.

There will be times when you update an android application, only to discover that it’s changed in some undesirable way, or that it’s developed a bug in a key feature that you use every day or maybe just doesn’t suit you as much as the older release.

I’ve experienced this issue in the past, where a twitter client update broke posting images via twitter for about a month, before a subsequent update fixed it. Whilst this was occurring, I had no means to be able to go and downgrade the application to the older version that had worked fine.

Sure, it’s not as scary an issue as 10,000 customers not having internet like the telco world, but for that user who’s suffering a bug that impacts something they use daily, it’s a big fucking deal.

Add versioning and rollback support. Seriously. Please. Linux has had this sorted for years (decades?), you can always downgrade a package on a distribution to an earlier version if so desired.

Whilst it is possible to downgrade an application on Android if you can locate the .apk file elsewhere, if the application is only available via the Android Market, there is no approach other than earlier phone application backups that you might have created.

 

2. Vanishing Applications

I’ve been using Android for some time now, since around Android 1.5, during this time I’ve used a lot of different applications and have experienced the annoying issue of applications that I like and use being removed from the Android market place.

What tends to happen is:

  1. User find a nice application that meets their needs, downloads and installs.
  2. Developer pulls the application from the market – this can be any number of reasons – trademarking, unhappiness at application quality, removing a free app and going commercial only, no longer any desire to maintain, or even due to removal by Google for malware.
  3. User ends up buying a new phone, or re-installs a new Android OS image and wants to install all their favorite applications again.
  4. User is unable to find their application on the market to download again.

Once this happens, the only option is to try and recover the application from an existing phone, find it floating around online or if it’s an open source application, find the public repository (even abandoned apps tend to keep the source around) and download and compile the application.

Otherwise the user is left with trying to find an alternative application (if one exists) that could be better or worse than what they previously had.

This particular problem has bitten me enough that I’m always actively seeking for open source options and choosing them, even if a proprietary application is slightly better – the knowledge that I can always build the app myself if it vanishes is a key point.

Unfortunately it’s not that easy to always tell which apps are open source or proprietary thanks to the Android Market’s unclear licensing information:

 

3. Clear licensing information.

Android Market will not report what license a particular application has when viewing the applications details or even when downloading the application.

This is a problem as there’s no way in the market application to tell whether an application is free as in freedom or free as in beer, which is a big problem for any users like myself wanting to choose software options that are under an open source license.

There have been numerous requests to Google to change this, something that surely must not be  a hard feature to add, but there’s been no visible progress on this issue.

For now I’m taking more efforts to research applications before installing them, and using F-Droid, the open source only repository as a first stop to find applications.

 

4. Freedom & Censorship

The use of the Google Market application offers some handy features such as the ability to remotely install software onto the phone via browsing the market website, a legitimate and useful function for some.

This connection to Google also allows Google to remove applications that are undesirable – the intent of this is to remove known malware and malicious content from devices, once again, a legitmate and valid use.

The downside, is that there is the capability for Google to use this connection to install or remove other software components in future, for either their own motives or that of a court order.

Consider something like a wikileaks application providing leaked data, or an application to bypass censorship which causes embarrassment or problems for the US governement. As a US company, Google could be ordered to remove that application from devices worldwide, a very plausible and concerning scenario – even if a user is confident about the ethics of Google, it wouldn’t stop a court order forcing software to be removed.

If this scenario seems far fetched, remember that Amazon removed a particular book from all their e-readers after a copyright dispute, removing not only the book, but all the user prepared notes for them.

I’m a strong supporter of computing freedom, having vendors like Google becoming gatekeepers and controllers of what we can and cannot run is concerning, particularly as the future of legislative policy appears to be tighter and nastier, particularly with the US.

 

Can it be fixed?

It would be pretty straightforwards for Google to fix issues 1-3:

  • Add version awareness to the market place, so a user could downgrade applications – even if it was limited so a user could only download to a version they previously had, I would be happy.
  • Keep pulled applications in the market place (with exclusion of apps removed for malware/malicious purposes – in that case, it should be removed and labelled as such) at least for users who have downloaded them in the past, so we can continue to use our favorite apps. A warning that this application has been abandoned or some other term would be fine.
  • Provide licensing information for applications, along with search abilities to find applications by license type. A link to the upstream source would be a nice touch too.

The 4th issue is a little more complex as the ability to remotely manage software has valid features and isn’t as simple as just removing.

Ideally I think the best approach would be to adjust the structure of Google’s Android integration, so the hooks into Google having control over the phone can be changed to allow/always prompt/disable approach.

This still allows for all the current functionality, but gives users with concerns about Google’s abilities to control how their phone behaves.

I’m pessimistic about Google actually going and fixing these things – they aren’t major selling points to attracting new users to Android, but I think they need to be addressed for Android to be more reliable and usable long term.

Android: the free-ish mobile platform

5 Replies

I’ve been using Android for a while now, starting with the somewhat underpowered HTC Magic (G2) before moving up to a much snappier Google/Samsung Nexus S which is now my current model.

Whilst I’m a fan of the platform overall, I’m encountering more and more issues every day with the fact that Android is being positioned as the poster child of open source in the mobile space (with other alternatives like Meego and WebOS way behind in terms of market share and consumer awareness), yet Android is only partially open source, still relying on large proprietary chunks.

With the recent release of Android 4 (Ice Cream Sandwich), I decided I would run through the steps of compiling Android from source code – I’m a firm believer of only running things that you have the ability compile yourself, and have gone through the exercise of building Linux From Scratch and custom distributions in the past to gain understanding of how the Linux OS is assembled and functions.

Android’s open source release (AOSP) is available from source.android.com which provides instructions for downloading the (large!) source tree, tools and building them into a functional device.

Doing so was an interesting experience – some of the first issues encountered are:

  1. AOSP is limited to working out-of-the-box with only a select number of devices – any official “Google” phones, like the Nexus S or Galaxy Nexus are supported, along with a couple additional vendor models (such as the Xoom tablet).
  2. If you have a non-google supported phone, you’re on your own – depending on your vendor, it may be a simplish, painful or maybe impossible task to obtain the required binary blobs. And that doesn’t cover whether or not the phones have locked or unlocked bootloaders.
  3. Even the Google AOSP supported phones can’t run a pure open source stack, proprietary downloads are supplied by Google for specific hardware components for each model and for a specific OS release. Should Google decide to stop supporting a device with future Android versions (as has happened with earlier devices, you won’t easily be able to support the hardware).
  4. The source is big, the build is hungry and you’ll want some performance to build it. I allocated around 40GB for the checked out source and build space and used most of it, along with 8GB of RAM and a few cores on my Phenom. On the plus side, the build only took a few hours, not the days-long efforts some online had predicted.
  5. Google’s build instructions are a bit lacking, given a week, even a Google intern would be able to make a massive improvement to it, I ended up finding many useful commands online that weren’t documented on AOSP’s home page – such as how to package a build into an OTA style .zip for deployment.
  6. The Linux kernel isn’t compiled as part of the Android build process. Instead, Android used a supplied pre-build binary kernel and just includes it into the finished OS image. If you need to adjust the kernel, it must be built separately and then placed into the correct location in the Android sources. This process isn’t documented anywhere on the AOSP homepage that I could find.

The base AOSP build provided me with core functionality including:

  • Functional base operating system and all hardware (thanks to binary blobs from Google for my Nexus S).
  • Communications – calls, txts, wifi, bluetooth, internet browsing
  • Contacts/Address Book.
  • Ability to install applications from direct .apk download or transfer using adb from PC.
  • A generally working and usable device overall.

But I didn’t have a number of needed functions that are typical of off-the-shelf Android devices:

  • No support for Google Account synchronization – without this, I was unable to download synced contacts and any other information from the cloud account.
  • No android market, the only way to install applications is from third party markets, direct download of .apk files or self-compiling applications.
  • No google service-based applications –  google maps, gmail, google talk, etc
  • No face unlock ability – I expected this would be part of ICS, but seems it’s part of Google’s application set…. this mix of having open source and proprietary components is one of the biggest problems with Android, you aren’t always sure what is or isn’t open source.

To get these other needed functions that are typical of an Android phone, you need the Google apps package (or at least the market application so others can be downloaded).

The killer part is that this package isn’t freely available. From what I’ve been able to find, it seems Google only provides their application package to vendors who pass their tests for Android compatibility to maintain quality.

Personally I think this is a lot of crap, considering the shocking quality of some Android devices that have come out – at the very least, the Android Market place application should be freely available, so users can at least choose to download applications, even if Google decides a particular vendor doesn’t deserve their Google apps.

In the end I managed to source a package of the Google applications for ICS thanks to the efforts of the Cyanogenmod team, but this is a shocking approach – not only is there an uncertainty about having the latest versions, but having users trawling through the internet to find tarballs on some forum is an easy avenue for attack and getting malware onto phones.

The fact that I’m so reliant on hackery to get key functionality back for my phone if I choose to build from source, rather than using my phone vendor’s build images is giving me solid reasons to consider the feasibility of dumping Google’s components from my phone and finding open source replacements for them all.

Whilst Google deserves credit for making the base OS comparability easy to build for users of Google-approved devices, the fact that they’ve allowed vendors to get away with binary blobbing drivers everywhere and keeping key functionality proprietary (market, etc) is pretty bad.

Chasing down binary blobs in order to get a device to work as expected is much more reminiscent of days spend pirating software, not of a healthy open source project and it makes Android feel much more hacky and crappy than it should be.

And the fact that the open source build will work with so few of the phones on the market out-of-the-box is just appalling for an OS that’s called open source – I should be able to go pickup any Android phone in the market and be able to compile AOSP for it and have all the hardware supported, not just select models.

Part of this is the fault of device manufacturers, but IMHO, Google should have put down some restrictions in the use of the Android trademark, so that a device could only actually be an “Android” device if it was fully open source and featured an unlocked bootloader.

I’d even accept a compromise where binary blobs are needed for specific hardware, as long as the blob wrapper can be compiled against different kernels and is free to redistribute (aka firmware style), so that I could buy a phone running Android 2 and happily go and build Android 4 for it and expect it to work.

Overall, I’m happy that I could build a functional image for my Nexus S without too much pain, but disappointed that so much of the feature set we are used to with Android isn’t actually open.