ChromeMarks - Progress update

Hi all.

Thought it was time for a progress update since its been a while since the last release of the app.

The main piece of functionality I've been working on is decryption/encryption. Originally I wasn't going to do this - I thought this was an additional feature that users would have to explicitly enable in Chrome and they could easily disable it in Chrome as well. At best, this would be a potential pro feature only as well. But things have changed over the last few months.

Firstly, the encryption option ("Encrypt all synced data") has been on the dev channel for a while, but recently was released to the stable channel. This meant there was now a much bigger chance of users getting encrypted bookmarks. Indeed, when setting up sync, you can imagine a user reading "encrypt all synced data" and thinking that isn't a bad idea, why not?

Unfortunately Google knows the version of my app can't handle encryption, so they've been treating this in one of two ways. If you already have bookmarks in ChromeMarks, Google would simply not send you any updates. At. All. If you are syncing ChromeMarks from the beginning (i.e. after a reset/reinstall) then Google took a different tack and decided to throw an obscure "database id" error:

I raised a chrome bug and they confirmed this was due to encrypted bookmarks and the only solution was to disable encryption of the bookmarks, or to upgrade the app fullstop.

Add to that, a growing number of users have contacted me with encrypted bookmarks that they are sure they never enabled. Indeed some users have only ever used the stable release so not even had access to the encryption functionality. For those using dev/canary/nightly releases, I suspect a release a while ago probably had a bug in it that enabled encryption by default - you never even knew it happened. But once enabled, it was done and since users weren't aware, they would complain my app was broken since it couldn't possibly be their bookmarks... Of course all the mails start with the phrase "my bookmarks sync fine between my home Chrome pc and my work pc, so it must be your app and nothing I've done".

A handful of users have also contacted me saying they just can't disable the encryption fullstop. In the Chrome "customise sync" box, the options to disable it (see screenshot above) are just not user editable. I don't know why - it may be something to do with the age of the users account and maybe you can't disable the encryption on the fly? They can disable it by deleting the Chrome sync data from the dashboard and re-enabling sync, but that's a long winded thing to explain to a user especially when they don't think they have done anything wrong.

In the last few weeks, Google have changed they responses they return and now, if you have encrypted bookmarks, you get a nasty server error returned (see below). Google knows to return this as the app sends its version number with the request and Google knows this version can't handle encrypted bookmarks. 

Unfortunately if I add in code to detect encrypted bookmarks then Google will detect that and will then assume the app can handle encryption and will no longer send the errors! Grrr!

So I've started to implement decryption into the app by default. Several challenges along the way, though, and I thought I'd explain some of them here to detail some of the trickiness and the reason for the long pause between releases:

1. Chrome is written in c and Android isn't. To work out how to implement this, I had to understand the intricacies of the Chrome code, find the bits that do the encryption, decryption and work out how to do the same in Android. Android is in Java. Which isn't the same as c! So I basically had to write a line by line match for the existing c code into a different language which worked a completely different way and had different dependencies and expectations!

2. I tried in Java first, but struggled to work out how to do basic decryption. I found some Java code on the web and took some of the basic unit tests in chrome to test against, but it never worked. After several days pulling my hair out, I found some obscure documentation on the web that explained some of the Java decryption classes are at bit level not byte level. After a bit of division by 8, suddenly one of the Chrome test packs passed!

3. I spent another week writing more Java to keep testing it and adding in more functionality to match Chrome, including code to decode the keys that Chrome stores in the sync database that are the key to decrypting your bookmarks. This started working well, so when I was confident I'd got it all good, decided to stick some of it in the app and test drive it.

4. It failed immediately when run on Android. I discovered that the security classes in Android's Java have been crippled at birth. Basically several years ago, the US government decided they didn't want the free Java code to include the best encryption code, also for free. With the normal panic over terrorism, they wanted to restrict many counties of the world to only have access to the weak encryption in java and save all the strong stuff for USA and the countries it trusts. Android has a bit of an old version of Java in it and Android is also available over the world. So it lacks the tough stuff - the stuff Chrome uses and requires by default. Grrrr.

5. This basically meant all the Java stuff I had done tested locally was effectively worthless as it would not run in Android. Of course nowadays, the restrictions in Java have been lifted and there are other libraries you can also plugin to do the job instead. But I can't upgrade the version of Java on your device. And I couldn't use the other libraries as they annoyingly have to have files copied into your Java installation and again that isn't possible on Android.

6. So I went back to Chrome and tried to see if I could compile some of the Chrome code as native code and see if that would work instead. But after another few days playing, it became obvious that I'd need a significant amount of the Chrome code base to get anywhere, and I'm definitely nowhere near a reasonable c coder to attempt that! Plus this was also a few weeks before they started actually checking in the initial code to the Chrome code base that might bring Android compatibility (note to readers, don't get excited...its a long way off before we will a nightly release of Chrome that runs natively on Android). But like the chrome devs, I still can't compile a working version for Android.

7. So instead I tried to see if I could compile some of the open source c libraries instead such as nss or openssl. Although I could find some posts out there from people who could compile it for android, I just couldn't get the damn thing to compile under cygwin. I tried ubuntu but got almost the same problems. In the end I went back to Java and persisted and managed to find a website that listed an alternative implementation of the missing encryption algorithm and found a way to simply invoke it manually (read: hook/crook/square peg/round hole) instead of via the default Java security classes - this avoided the need to install it on the device. Sorted.

8. Then started plugging it into Android. However Android doesn't have the operating system crypt classes that Windows and Macs have by default (or linux that would actually compile and work), so encryption on Android is increeeeeeeeeeedibly slow. I haven't had enough time to see if I can work out by how much...!

9. Then I had to find a way around what is sure to be the biggest gripe. Chrome uses your actual account password to encrypt and decrypt the bookmarks. Without your password, it ain't possible. If the app asks for your password, there will be uproar from people whingeing "why does this app need my password, it will have access to my Google checkout and credit card and emails, etc." Sigh. Of course people wouldn't remember that in Chrome where they ticked the radio button that says "encrypt my bookmarks", it says immediately below it "Use my Google account password." So people have already agreed to this - its the way Chrome works, people, and you agreed to it! Don't have a go at me!

10. But I knew this will bring complaints. So I next thought about the screen I'd need to get the users password. Instead of a simple standard popup, I made a full size screen that deliberately looked like the normal Android credentials popup. Here's a shot of the new password request/passphrase request screen:

In it, I then added a screenshot of the box in Chrome mentioning it would use your password and underlined it to make it clear to the user. Hopefully if anyone wants to complain, they can see this and realise they already agreed to Chrome using their password and its clear there is no other option other than disabling encryption completely or using a passphrase instead.

If you click on the "more info" button, you get an image cut from Chrome to prove why the password is requested by [ subtly! ] underlining some key text from Chrome in red:

11. Also started getting annoyed as to how long it takes for the Android emulator to start and run. It takes a large amount of my testing time waiting for the emulator. So I experimented with the latest x86 Android releases and got them up and running in a virtual machine. Starts in about two seconds and works each time by default! Amazing! Unfortunately that did mean the existing native code in the app wouldn't work for a different architecture. I recompiled it for x86 and after a day spent solving an annoying int/unsigned int compare that worked differently on arm compared to x86, it finally works as the arm version does and I use the x86 as my default test environment. Luckily the recent upgrade for androidpit help here as well as the x86 version doesn't have the market licensing installed which the app needs by default.

So that's where I am at the moment. I've yet to get new bookmarks and new folders sent to Google as encrypted items, so haven't tested that bit out, but I can read encrypted bookmarks and decrypt them once the user has supplied the key. I still think this is a pro feature, so I've added encryption to the app by default, but Lite users will only get the first five characters of the bookmark name decrypted (the URL, icon, and folders will all decrypt fine, just bookmark names will be partially decrypted). Can't wait for the one star complaints about that... :-)

Along the way, I had to build my own test Chrome sync server. Obviously the real thing isn't open sourced. There is a python version as part of the Chrome source, but its built for the unit tests so is quite limited in its functionality. I needed something where I could turn on or off decryption and emulate a changed password. This took me a while to build as well, but it's been useful to bang away my initial tests against it locally and not knacker my Google account along the way.

Word of warning for the next release - I've upgraded to the latest version of the Chrome protocols that are used in the stable and nightly versions. Unfortunately this does mean some database changes. Chrome uses the protocol version to determine how it might handle a bookmark or folder change. For example, one version might only accept numeric values for a certain field, but the next version might accept characters instead. If I attempt to upgrade in place, the user may have bookmarks already in their database that came from another version and the Chrome server will not accept that for the latest version number. So for that reason, this release will clear your ChromeMarks database when you upgrade so you are forced to download all your bookmarks from scratch. Sorry.

So some details on what I've been working on and what's remaining. Hopefully this will be  released in a few weeks!

Comments...? Do people like me to waffle away like this? Is it interesting/useful?