ChromeMarks - Progress update

Hi all.

Thought it was time for a progress update since its been a while since the last release of the app.

The main piece of functionality I've been working on is decryption/encryption. Originally I wasn't going to do this - I thought this was an additional feature that users would have to explicitly enable in Chrome and they could easily disable it in Chrome as well. At best, this would be a potential pro feature only as well. But things have changed over the last few months.

Firstly, the encryption option ("Encrypt all synced data") has been on the dev channel for a while, but recently was released to the stable channel. This meant there was now a much bigger chance of users getting encrypted bookmarks. Indeed, when setting up sync, you can imagine a user reading "encrypt all synced data" and thinking that isn't a bad idea, why not?

Unfortunately Google knows the version of my app can't handle encryption, so they've been treating this in one of two ways. If you already have bookmarks in ChromeMarks, Google would simply not send you any updates. At. All. If you are syncing ChromeMarks from the beginning (i.e. after a reset/reinstall) then Google took a different tack and decided to throw an obscure "database id" error:

I raised a chrome bug and they confirmed this was due to encrypted bookmarks and the only solution was to disable encryption of the bookmarks, or to upgrade the app fullstop.

Add to that, a growing number of users have contacted me with encrypted bookmarks that they are sure they never enabled. Indeed some users have only ever used the stable release so not even had access to the encryption functionality. For those using dev/canary/nightly releases, I suspect a release a while ago probably had a bug in it that enabled encryption by default - you never even knew it happened. But once enabled, it was done and since users weren't aware, they would complain my app was broken since it couldn't possibly be their bookmarks... Of course all the mails start with the phrase "my bookmarks sync fine between my home Chrome pc and my work pc, so it must be your app and nothing I've done".

A handful of users have also contacted me saying they just can't disable the encryption fullstop. In the Chrome "customise sync" box, the options to disable it (see screenshot above) are just not user editable. I don't know why - it may be something to do with the age of the users account and maybe you can't disable the encryption on the fly? They can disable it by deleting the Chrome sync data from the dashboard and re-enabling sync, but that's a long winded thing to explain to a user especially when they don't think they have done anything wrong.

In the last few weeks, Google have changed they responses they return and now, if you have encrypted bookmarks, you get a nasty server error returned (see below). Google knows to return this as the app sends its version number with the request and Google knows this version can't handle encrypted bookmarks. 

Unfortunately if I add in code to detect encrypted bookmarks then Google will detect that and will then assume the app can handle encryption and will no longer send the errors! Grrr!

So I've started to implement decryption into the app by default. Several challenges along the way, though, and I thought I'd explain some of them here to detail some of the trickiness and the reason for the long pause between releases:

1. Chrome is written in c and Android isn't. To work out how to implement this, I had to understand the intricacies of the Chrome code, find the bits that do the encryption, decryption and work out how to do the same in Android. Android is in Java. Which isn't the same as c! So I basically had to write a line by line match for the existing c code into a different language which worked a completely different way and had different dependencies and expectations!

2. I tried in Java first, but struggled to work out how to do basic decryption. I found some Java code on the web and took some of the basic unit tests in chrome to test against, but it never worked. After several days pulling my hair out, I found some obscure documentation on the web that explained some of the Java decryption classes are at bit level not byte level. After a bit of division by 8, suddenly one of the Chrome test packs passed!

3. I spent another week writing more Java to keep testing it and adding in more functionality to match Chrome, including code to decode the keys that Chrome stores in the sync database that are the key to decrypting your bookmarks. This started working well, so when I was confident I'd got it all good, decided to stick some of it in the app and test drive it.

4. It failed immediately when run on Android. I discovered that the security classes in Android's Java have been crippled at birth. Basically several years ago, the US government decided they didn't want the free Java code to include the best encryption code, also for free. With the normal panic over terrorism, they wanted to restrict many counties of the world to only have access to the weak encryption in java and save all the strong stuff for USA and the countries it trusts. Android has a bit of an old version of Java in it and Android is also available over the world. So it lacks the tough stuff - the stuff Chrome uses and requires by default. Grrrr.

5. This basically meant all the Java stuff I had done tested locally was effectively worthless as it would not run in Android. Of course nowadays, the restrictions in Java have been lifted and there are other libraries you can also plugin to do the job instead. But I can't upgrade the version of Java on your device. And I couldn't use the other libraries as they annoyingly have to have files copied into your Java installation and again that isn't possible on Android.

6. So I went back to Chrome and tried to see if I could compile some of the Chrome code as native code and see if that would work instead. But after another few days playing, it became obvious that I'd need a significant amount of the Chrome code base to get anywhere, and I'm definitely nowhere near a reasonable c coder to attempt that! Plus this was also a few weeks before they started actually checking in the initial code to the Chrome code base that might bring Android compatibility (note to readers, don't get excited...its a long way off before we will a nightly release of Chrome that runs natively on Android). But like the chrome devs, I still can't compile a working version for Android.

7. So instead I tried to see if I could compile some of the open source c libraries instead such as nss or openssl. Although I could find some posts out there from people who could compile it for android, I just couldn't get the damn thing to compile under cygwin. I tried ubuntu but got almost the same problems. In the end I went back to Java and persisted and managed to find a website that listed an alternative implementation of the missing encryption algorithm and found a way to simply invoke it manually (read: hook/crook/square peg/round hole) instead of via the default Java security classes - this avoided the need to install it on the device. Sorted.

8. Then started plugging it into Android. However Android doesn't have the operating system crypt classes that Windows and Macs have by default (or linux that would actually compile and work), so encryption on Android is increeeeeeeeeeedibly slow. I haven't had enough time to see if I can work out by how much...!

9. Then I had to find a way around what is sure to be the biggest gripe. Chrome uses your actual account password to encrypt and decrypt the bookmarks. Without your password, it ain't possible. If the app asks for your password, there will be uproar from people whingeing "why does this app need my password, it will have access to my Google checkout and credit card and emails, etc." Sigh. Of course people wouldn't remember that in Chrome where they ticked the radio button that says "encrypt my bookmarks", it says immediately below it "Use my Google account password." So people have already agreed to this - its the way Chrome works, people, and you agreed to it! Don't have a go at me!

10. But I knew this will bring complaints. So I next thought about the screen I'd need to get the users password. Instead of a simple standard popup, I made a full size screen that deliberately looked like the normal Android credentials popup. Here's a shot of the new password request/passphrase request screen:

In it, I then added a screenshot of the box in Chrome mentioning it would use your password and underlined it to make it clear to the user. Hopefully if anyone wants to complain, they can see this and realise they already agreed to Chrome using their password and its clear there is no other option other than disabling encryption completely or using a passphrase instead.

If you click on the "more info" button, you get an image cut from Chrome to prove why the password is requested by [ subtly! ] underlining some key text from Chrome in red:

11. Also started getting annoyed as to how long it takes for the Android emulator to start and run. It takes a large amount of my testing time waiting for the emulator. So I experimented with the latest x86 Android releases and got them up and running in a virtual machine. Starts in about two seconds and works each time by default! Amazing! Unfortunately that did mean the existing native code in the app wouldn't work for a different architecture. I recompiled it for x86 and after a day spent solving an annoying int/unsigned int compare that worked differently on arm compared to x86, it finally works as the arm version does and I use the x86 as my default test environment. Luckily the recent upgrade for androidpit help here as well as the x86 version doesn't have the market licensing installed which the app needs by default.

So that's where I am at the moment. I've yet to get new bookmarks and new folders sent to Google as encrypted items, so haven't tested that bit out, but I can read encrypted bookmarks and decrypt them once the user has supplied the key. I still think this is a pro feature, so I've added encryption to the app by default, but Lite users will only get the first five characters of the bookmark name decrypted (the URL, icon, and folders will all decrypt fine, just bookmark names will be partially decrypted). Can't wait for the one star complaints about that... :-)

Along the way, I had to build my own test Chrome sync server. Obviously the real thing isn't open sourced. There is a python version as part of the Chrome source, but its built for the unit tests so is quite limited in its functionality. I needed something where I could turn on or off decryption and emulate a changed password. This took me a while to build as well, but it's been useful to bang away my initial tests against it locally and not knacker my Google account along the way.

Word of warning for the next release - I've upgraded to the latest version of the Chrome protocols that are used in the stable and nightly versions. Unfortunately this does mean some database changes. Chrome uses the protocol version to determine how it might handle a bookmark or folder change. For example, one version might only accept numeric values for a certain field, but the next version might accept characters instead. If I attempt to upgrade in place, the user may have bookmarks already in their database that came from another version and the Chrome server will not accept that for the latest version number. So for that reason, this release will clear your ChromeMarks database when you upgrade so you are forced to download all your bookmarks from scratch. Sorry.

So some details on what I've been working on and what's remaining. Hopefully this will be  released in a few weeks!

Comments...? Do people like me to waffle away like this? Is it interesting/useful?

ChromeMarks - 2011.08.28 release

Another new upgrade today.

Mostly upgrading the internals, but a few new bits.

1. For the Lite app, I've added an "Upgrade Options" screen to the settings. This will show you what features of the full app are available to you should you choose to upgrade. So for example, can the app specifically integrate to your browser? Screenshot below.
2. Upgraded the crash reporter (ACRA) from v3.2 to v4.1. Not much difference to the user, but the info reported is a bit more helpful to me. Juicy bits that I really do need include the user's language, the rotation of their device at the time of error and the size of the user's screen. These help me to better work out what resources in the app might have caused a problem. It's also got functionality to save your email address keyed in such that if another error happens, you won't need to key it in again.
3. For the Lite app. Upgraded the admob ads. Allegedly they should cause less crashes - but talk on the forum is they fixed some stuff but broke some others.. :-) Also taken the opportunity to add in some ads for users that can't receive the ads previously - such as AdFree users. Previously, admob just hid itself away when it couldn't receive an ad. Now, you'll just get my ads instead. Sorry if you got used to no ads but, hey, you are using a free app supported by ad revenue, so please resist the 1 star comments..
4. Added Turkish translation. Thanks to Saner Apaydin for all his work! Another user who did it in record time - I have to keep asking these translators if they remembered to get some fresh air!
5. The usual handful of small bug fixes.

A picture of the new Upgrade Options screen in the Lite app:

Hope you enjoy the release - still more stuff yet to come..!

ChromeMarks update - 2011.08.07

Another new update for ChromeMarks. If a release could have a sub title, this would be the called the  "performance release".

As usual, the list of changes:

1. Finally taken the plunge and upgraded the the ACL (Android Compatibility Library). The most noticeable thing this brings is an improved way of loading the data for the screen and better management of the memory. For users, they should see folders open quicker and navigate quicker. For Honeycomb users, you (hopefully!) shouldn't get the force close when you return to the app after popping off elsewhere.
2. As another benefit of the above, if a folder takes less than a second to open, then the next folder will open without the "please wait...opening..." popup. The popup itself takes a moment to appear and then disappear, so removing it will make the results appear quicker generally. But I still need some kind of "progress" widget to let the user know the app received their touch event. I've added the ChromeMarks logo in the top left corner (Honeycomb users had it there already) and the logo will animate when you open a folder. Unfortunately Android can't animate the logo smoothly as it's busy furiously opening your folder! Once the folder is opened, Android now has the free cpu to animate it smoothly, but the op has finished so there is no need! Aarrgh! I might try to improve it if I can - but for now, it at least wiggles to let you know it's working.. :-)
3. Along the way, I've tweaked and eeked as much more as I can in various places to improve the core functionality and speed as much as I can. I even tried to get the icons to load after the rest of the screen to delay the non-essential stuff. But I had issues with the icons not recycling properly and appearing on the wrong bookmarks after you scrolled. This might appear in a later release, but for now it's commented out.
4. I've added a new fun stat to the "Info & statistics" screen. You can now see a total of much of Google's cpu you've personally used with each sync operation in ChromeMarks. It's not exactly useful, but anyone for playing top trumps?!
5. As requested by a user, since I added the "delete item" to the gestures, I've added it to the long click context menu as well.
6. More issues with Samsung devices. Dear oh dear, Samsung. Why do you have to make your own little changes to core Android and break something that works with *every* other device in the world...? A handful of reports from users experiencing force closes in specific settings screens only when using certain languages, or high cpu in various screens. I have no way of knowing what they broke, but they clearly broke something as no other device has these issues! I also can't reproduce any of it in the emulators (I've tested in 7 different android emulator versions) and can't reproduce on my personal phones. This further proves they've broken something in core android. I've no way of knowing what, but I've closed my eyes, spun round twice, cursed the Samsung deity and changed a handful of lines of code in a vain attempt it might make something work better or worse.
7. I've also added some extra tracing functionality such that I can explain to certain users how to enable it and send me extremely detailed traces. This might help with the high cpu - especially if I spot something in there that doesn't appear on any other device.
8. The usual bug fixes along the way. Thanks to all users who send in the automated errors reports - please don't click "cancel" on it - press send otherwise I'll never know about your bug and won't fix it. You don't have to enter an email or even any text, just press send! No personally identifiable info is sent either, so all you naughty pirates still using the 04.16 release can feel safe in sending the reports!
9. Some translation updates and corrections as well. More languages in the pipeline - Turkish is coming imminently!

Back to Samsung. The reality of it is, I'm actually spending more of my time investigating and trying to fix things that are broken only on Samsung devices, than I spend time making new features for the benefit of *all* my users. I'm afraid if this continues, I'll most likely have no choice but to disable Samsung apps from using this app completely and give up on them. I don't want to, but Samsung won't release the source, they won't answer emails, they won't answer tweets, they won't fix their remote testing developer site, they won't engage with the developer community even on their own website, they constantly make ill advised changes, what choice do I have..?


Enough with my rant... :-)

Next release..? A growing list of issues to fix with the local bookmarks syncing. Want to get an alternate Market integrated and working properly. Several changes to Chrome functionality that I want to pro-actively fix before another "July 12th" happens. A thorough restructuring of the debug messages which should make a noticeable decrease in file size. More gestures (double tap?). And that's just the list of things near the top of the list..

Still many, many  more new features and feedback on the plan for this app, stay tuned.

H.

ChromeMarks update 2011.07.22

Hi all - time for another update to ChromeMarks. Thankfully the changes made on the 12th July haven't had any issues that required a fix here, so these are all new features or fixes to things that have been broken for a while.

1. New features - Gestures. Only in the paid app. You can now specify that flicking up or down performs some action (i.e. up a folder, goto root) and swiping a bookmark or folder left/right will perform something else (i.e. open in new window, edit, delete). You can configure the "zone" that detects flings so it doesn't disrupt with your normal scrolling (i.e flings only work on the left half, right half, middle half, etc) and also configure the detection speed. I'm not too sure with the detection speed - it may be oversensitive, it's hard to tell as it also depends on the screen size and possibly the cpu speed. Some screenshots below.
2. New translations. Swedish, Dutch and Japanese. Thanks to all the translators who have helped (in no particular order) Kees Huiberts (nl), Gerben Markvoort (nl), Kjell Kernen (sv), Yutaka Takeda (ja), aozora (ja), Kazuo Ugai (ja).
3. It's clear from the automated bug reports there are many problems with the manufacturer builds of 3.0.1. There are many posts on the android developer forums where errors have been reported and the stock answer is "how can that possibly happen?" For whatever reason, some of the core java code is corrupted and invokes something else which will always be wrong. I don't know how it can happen, but it clearly is and many apps are affected by things that should *never* fail, but do! I've tried a few fixes here to deliberately avoid running the bits of code that knowingly fail on 3.0.1.
4. Note, the above fix is different to the other issues with Honeycomb generally regarding restarting the app keeps force closing it. That's due to changes Google forced on us with Honeycomb and neglected to make it backwards compatible with existing apps. Grrr. Hopefully a fix in a later release, but it will need a serious rewrite.
5. Some tweaks to the settings screens to dynamically resize the text for each setting's title. Hopefully you won't see text disappearing under a checkbox or drop down widget. Unfortunately I can't do this everywhere else, so popups and other text fields are still fixed sizes.
6. Changed the "Frequently asked questions" to "Help". The content is no longer hard coded answers. I've given a list of the most popular questions and linked them back to the community website. This allows me to provide more in depth answers, where needed, but to also be able to update them real-time. Also allows users to comment on them and get responses to those comments. Unfortunately this does mean the actual answers are no longer translated - though userecho (the website the community site is hosted with) says it will bring that functionality in at a later date.
7. If your first sync failed, you might have ended up with duplicate data on one of the tables in the database for each successive failed sync until it finally worked. It wasn't actually a big problem, but it might have simply made the database a smidge bigger and slower to access. Only talking a couple of extra records - if you had 1000 bookmarks, this would have had a negligible effect. On upgrade, the app will detect this and remove the extra records.

Note, there may have been some other changes that either Google started making from 12th July, or took effect due to me upgrading to the latest version of chrome sync service:

1. Some Chinese users were constantly getting ssl handshake errors. The analytics confirmed these errors were coming out for previous versions of the app as well, so it wasn't related to the last upgrade. It looks like an internet server somewhere in China that passes the traffic through to Google was mis-configured. After exactly a week, the reports started dropping fast. Thanks to Leo for sending me debug logs and for testing every few days and confirming things were working. Good job to, I wouldn't know where to complain about it!
2. Some users have started receiving "throttled" errors in the app when they change bookmarks or make new ones. Google normally throttles users to 100 changes a day. But previously it had always been a specific client software that was throttled (i.e. your desktop browser was throttled, but you could happily make 100 changes in ChromeMarks). The comments from some users might indicate that Google has changed this to throttling per user account - so irrespective of the client software, *you* can only make 100 changes a day. I've tried to get more info from Google, but don't hold your breath. Remember that auto-fill entries also count to your 100-a-day, so if you fill in a lot of form fields and have auto-fill synced, you'll hit your limit fast. More updates if I find out more info.

Some screenshots of the updated main settings screen and the Gestures screen showing some of the options (more off the bottom of the screen):

A reminder that I'm now on twitter as @chromemarks. Follow me if you want to hear about new features or Google outages and feel free to ask questions or comments.

Twitter action!

Just a quick message to say that I'm on twitter as @chromemarks and http://twitter.com/chromemarks. I'll try and tweet on new features I'm working on, new bugs or issues and anything else that might be useful. Send me a message if you have ideas, comments or questions!

ChromeMarks broken - IO Exception errors

Google have made some changes somewhere which has broken all ChromeMarks functionality.

I'm trying to fix it and hope a new release in the next day or so. Please bear with me - remember this app is coded and maintained by a single person during his spare time!

I don't know what they changed. From what I can see, the data is coming back fine, but at the very end of the sync are a couple of fields to denote the final progress and I suspect they've changed something around that area. It's a bit annoying as one of the fundamental purposes of the protocol buffers Google invented and use for this communication are meant to be always backwards compatible.

I guess Google thought that the only people using chromesync would be users of Chrome and since they should be auto-updating, what's the harm in turning off some old server functionality, eh....?

More updates later.

UPDATE: 22.45pm BST. I think I've fixed it. Will update more here in the next hour while I refresh the Market.

UPDATE. So, I'm still not exactly sure what they broke, so I've had to upgrade the app asap to the latest protocols. The app has gone from release 24 to release 27. Google have made quite a few changes along the way, partly why I've been putting this off for so long.

But with a nice bit of Fiddler (where would this app be without it??) and I *think* I've worked out how the new protocols work. They didn't quite fit immediately in, but with a squeeze, they work.

Unfortunately Google have changed the way they track changes and moved from an increasing "number" to a semi-random byte array (may not be random to them, but clients should not expect them to follow any logical pattern). This necessitated a change to the database that ChromeMarks is held in. If I'd done this as normal change in my own time, I would have written a migration into the app to fix this as soon as you upgrade. But I didn't want the app to be offline for too long and my wallet can't afford all those annoying people that vote one-star, so I've chickened out and made the app reset-to-default when you open it to force you to redownload. Sorry! Not my fault!

I've tested what I can in the time so far, but since the bug happened 24 hours ago, I've needed sleep, travel time and a normal job, so please be gentle if there are a few more bugs I didn't spot! They will get fixed as soon as I can. I've only checked it on Froyo as that's the most prevalent - I'll have a try shortly on Honeycomb as that's the most likely to be tetchy.. :-)

Annoyingly I was going to make a release this weekend with some real goodies for the paid app (could I say it was a "gesture" to "fling" these changes in your direction?), but now I've got to merge two changed versions of the app together. So the next official release (barring bug fixes) will probably be next weekend.

As usual, thanks for your support and to the people that mailed me letting me know there were problems, and to everyone else who actually read the note on the Market.

I think I need some sleep...

ChromeMarks updated 2011.07.02

Another update. As normal, the details in more detail (!)

1. Experimental fix for Samsung owners who experience disappearing local bookmarks created by ChromeMarks. The root cause is due to Samsung making naughty changes to the Stock Browser and neglecting to make those changes work with the hundreds of bookmark apps on the Market. Lots of thanks to Emiel who sent me enough files to guess the changes made by Samsung. There may still be some more changes needed here to polish it a bit, but the bookmarks are now visible, at the least. NOTE this has only been tested on the GT-I9000 - other Samsungs may work, or not! Let me know! When you first upgrade, the app will attempt to fix any bookmarks you may already have there and then. Otherwise, the normal sync should do the work anyway.
2. Added Portuguese translation. Though I'm a bit worried as to wether it's Brazillian Portuguese or normal Portuguese! Time will tell... :-)
3. Fixed the Folder browser on Honeycomb devices. This is used in the settings when choosing the Home folder, or when editing a bookmark/folder and choosing the folder it is saved into. It was definitely a Honeycomb bug as it worked fine in all other versions. But it was hard to track down! But for some reason, I made a sprinkling of tweaks that made it suddenly work - I saved the changes immediately and cross the fingers it doesn't stop again!
4. The usual bug fixes along the way.

No pictures this time as nothing UI has changed. There are a sprinkling of UI improvements I want to make soon, Gingerbread needs a bit of TLC now.