The Tudumo Times



11.10.10

Sync!

Last week I uploaded a version with the single most oft-requested feature...and the one that has been most personally draining to develop. To get it done, I put almost all other features on hold. It’s meant many rewrites, different languages, technologies and architectures, changes, pauses, restarts while the cloud landscape changed and while I found extra nooks and crannies in the code that have needed updating.

But here’s the good news: you can now sync your tasks between computers, via the Tudumo server! Change a task on one computer and Tudumo will automatically upload it after a brief pause, and fetch any changes that the server might have. Or click this button anytime:


It sounds so simple when put like that!  Haha, devil in the details, indeed.

You can also use it as an online backup, or when moving your tasks to a new computer. Sync up on the old computer, sync down on the new one.

What will it cost you?

Nothing. Free. As much as possible, I’d like it to remain free for simple sync usage. I’ve put a lot of effort into writing it on the aforementioned cloud architectures and taking much care with efficiency, so costs are low.  Despite the huge effort and needing many changes under the hood, you won’t pay any upgrade fees either - I consider the app still in version 1, so users who bought even a couple of years ago don’t pay a cent extra. In fact, I’ll say that all you’ll get from me this year are more features, for free.

Where to next?

Well, this thing has to settle first. There’s no way that I won’t get bug reports, and what makes sense to me might be utterly irrational to you. There are also some aspects of sync that I’d like tightened up, like syncing task order.

And (future) features?
  • Web view
  • Use Dropbox (or other file sync solutions) for sync. Tudumo will just load changes from the file system, so you have options.
  • Sending actions to predefined friends
    (“oh look. fix the tap. can’t wait!”)
  • A 3D view that you need special glasses for
Yes. I make joke.

So if you’re just interested in the sync - go play with it.  For those that are interested in the technical back-story:

I initially built the sync backend on Google App Engine, in Python, while it was in a very limited test of 10 000 developers. In the early test phases Google heavily restricted the amount of time an operation could take so that meant I had to be really careful with the time each operation took. So, ok - split up whatever I’m sending into tiny bits. Also, there was a relatively high error rate, the technology needed lots of work-arounds (XML? Ah. Tricky. Hand-write that. Send it as text, or as a zipped file? Well - have to hand-write parts of that too) and it just felt unbaked to me. It worked, but wasn't confidence-inspiring.

Then Microsoft released a developer version of Azure which seemed less restricted and more 'enterprisey', so I rebuilt the server in C# for Azure. Now I could re-use some code between the client and server, reducing weirdness in communication. That seemed fairly good until I deployed the app and started testing it, and then they released the pricing. (Yup, should have waited for that!) With Azure you pay even if nobody looks at it, and you have to manage how many 'instances' you use. If you go over 1GB, the cost doubles (approximately). If you need another server (for performance or reliability) you need to manage it yourself and then pay about three times the original price (2 servers + extra storage). You couldn’t send mail at the time - had to use an external server for that. It’s cloud, Jim, but not as we know it.

At some point Google released the Java version of App Engine, all the while increasing their limits and generally making the service more bulletproof.  I’m an old hand at Java and while it’s definitely more effort to write than Python, I wanted the languages on both ends to be relatively close - C# is similar to Java - so coding constructs can be similar. I tried it on a side project and it went well, so I ported the server code again. This time it felt top-notch. With all systems there are hiccups but with some of the cloud providers you can bet that the hiccups will drop over time, as they get better and better at figuring the problem space out. In terms of costs I don’t pay until you really hammer the server. When it gets busy I start paying for usage, but if written carefully the cost isn’t bad at all. For basic sync - which is what most consumer users will need - that’s perfect. No minimum and no (effective) maximum and you can sync every time a task changes instead of e.g. waiting 20 minutes for enough data to send up. And if the server goes down in the middle of the night some exceptionally smart people are worried about getting it back up. Sure there’ll be mishaps, but I’m leveraging Google’s ability so we all benefit from any improvements they make in future. Foursquare went down a week or so ago, for about 17 hours over two days. They have 30-ish people and funding and lots of experience. What chance a single developer or small team will do better, in a 24/7 world?

A view of my web-scale dashboard. Those (daily) limits can be raised in minutes - just apply cash

For my choices, does that make App Engine better than Azure?  No. As usual it’s a case of “use the right tool for the task”. There are still some exceptionally good reasons to use Azure (or Amazon's EC2 stack), and I'll likely use it for some things. Basically, all the cloud services are locked in an arms race which developers and users benefit from.

My choice was to make a long-term problem (cheap development but pay more over time, and continue worrying about any scaling and management) into a short-term one (pay more upfront in development costs, reduce ongoing expenses and risks), but wow - quite a long short-term! Not the most effective way to get the feature out and I “wasted” quite a lot of time bumbling around, surely irritating many users who actually don’t care about the storyline as much as they want to Just. Sync. Tasks.  But now that’s done, with a solid roadmap and predictable costing, and I've learned a lot that will be strategically important in future.

In truth, partly due to the choice of technologies the server part wasn't that hard, once the basic design was up. The client, Tudumo, needed many, many changes under the hood to handle sync in a fine-grained fashion that would work for multiple sync connectors (for example, Outlook) and in a (hopefully) elegant way. No matter how much you follow good design principles, there’s always a family of surprises waiting around each door!

Conclusion

The lessons I learned (expensively) confirmed what I already knew: Do a simple first version, avoid the bleeding edge, don’t optimise prematurely. On the other hand, any hard problem usually has competing principles. I’d rather have Google administering/securing/scaling a server with your data on it, than me. The distant problem was scaling, but the near-term (and ongoing) one was security. I can focus on the application level issues and leave the operating system, web server and whatnot alone. Outsource that to the experts.

8 comments:

  1. Anonymous20.10.10

    Sweet!

    Richard, thanks for your perseverance. Sounds like your coding project was much like one of my typical handyman projects. It always seems so easy until you actually start to do it.

    I was starting to worry that you were no longer supporting the application. That would have been a big shame as Tudumo is by far the best GTD app for me. I'm sure that's the case for many others. Thanks for fighting your way through all the obstacles.

    ReplyDelete
  2. Thanks, Anonymous. To be honest I brought it on myself, but now it's done. I'll enjoy being able to build on that now!

    ReplyDelete
  3. Anonymous1.11.10

    Good job, I use Things on the Mac, and they have been working on this for almost 2 years!

    ReplyDelete
  4. b.e.c.i.d17.11.10

    Great news, excellent story and brilliantly written! Me too, I thought tudumo was an abandoned application, good to hear it ain't. I'm still looking for a way to make my lists portable. Any plans on that? Mobile client? Sync with Outlook? Web view working in mobile we browsers (which ones?)? Keep up good work Richard! Regards.

    ReplyDelete
  5. Mobile web definitely - that's a very high priority for me. If I can support WebKit-based browsers initially (iPhone, Android) I'll be happy. I'll toy around with Outlook again as I'd like to sync with it, but no specific promises there.

    ReplyDelete
  6. Richard, I have fallen in love with Tudumo in the time I've had my trial version. It is coming time for me to get off the fence and either purchase (which I so want to do) or move on another - the one thing holding me back would be the lack of mobile support for Tudumo. I know you have mentioned mobile browser support, and while that would be a great leap in the right direction, is there ever going to be a possibility of a native iOS app? I would be DELIGHTED if so.

    ReplyDelete
  7. There's definitely a possibility, but I do have to focus on the web part for now. I have my test version looking ok on my iPhone because I really want it there too! One of the guys I work with is doing some iPhone dev now, so I'll keep my eye on how much of a hassle it is to port to. One risk with that is having e.g. 5 different codebases to maintain - not a good idea.

    ReplyDelete
  8. Richard,
    Me too fallen in love with 'Tudumo', Its very simple to use, organize and its very much effective as well.
    Eagerly waiting for Andriod version of it. Could you let us know when can we expect a Andriod version of it?

    Thanks for developing such a useful app.

    ReplyDelete