Hacking Instagram

This week saw the release of Instagram. The app has been a huge hit on the iPhone and so it has been eagerly awaited on Android. Will it be as big a hit on Android? Who knows. However it has certainly caused its share of controversy. From my perspective as a long time Android user, I have a couple of complaints. First, it looks like an iOS app, not an Android app. Whenever I see an app like that, I figure the makers are lazy and don't really care about Android users. The app still might be useful, but it creates an expectation of poor quality for me and definitely curbs my enthusiasm. Still, this is something that can be overcome.

My second complaint is that Instagram just does not play well with others. What I mean is that most Android apps make use of Android's use of typed sharing (i.e. I want to share a picture, or a link, or a video, etc.) to allow you to create or find something with one app, and share it with another. Instagram does not allow this. Now it does let you share something to Twitter, Facebook, or Tumblr. Even better, it lets you share to all three at once. That's a great feature. However, your sharing is limited to what the makers decided to put it into, and that's too bad. What if I want to share to Path or Google+? The only place where Instagram allows you to use another app for sharing is for a picture that you have already taken. Then it doesn't let you share the picture, but only a URL to the picture on the Instagram website.

Now I definitely understand part of why Instagram does things this way, or at least I think I do. I think Instagram wants to be a social network, first, and provides a mobile app simply as a way to create content for their network. So any "sharing" is a link back to the Instagram social network. Allowing me to create a photo with Instagram and then share it to Path with no link back to Instagram would be of no value (perhaps even negative value) to Instagram. However, it might be of value of me. And maybe to you too. So here's how to do it.

First off, go ahead and create a picture with Instagram. Take a pic, apply filters, borders, etc. just like usual:

Now when you get to the Upload screen, just exit Instagram. Of course you can upload it and use their sharing options if that is interesting to you. If you want to share you creation in some other way, open up your Gallery app and look for the Instagram folder. Your creation will be in there, and you can then select it and share it using the app of your choice:

You can also copy the photo from you phone; it's stored in /Pictures/Instagram. The point is that you can take advantage of some of the photo editing tools in Instagram, even if you have no interest in the Instagram social network, and even though Instagram doesn't support this kind of usage. Of course there are already a lot of other photo editing/sharing apps available on Android. Some of my favorites are Lightbox, Camera ZOOM FX, and Diptic. 

Diversions in C Minor

Last year we saw the passing of Dennis Ritchie. I was one of the lucky students who used the K&R book as a text book in college. Actually it's pretty amazing that the text books for CS first years were K&R and The Mythical Man-Month. There was pretty much no excuse for me to F up after being taught from those two books. The untimely passing of Ritchie made me recollect that I never really finished K&R. My professor used many of the chapters as reference material, and of course at some point I figured that I already knew everything so well that I didn't need no stinkin' book. So I decided then that I would re-read the book including all of its problems.

Of course I didn't start re-reading K&R right then as I should have. But I did get around to it recently. The very first problem in chapter two asked the reader to calculate the minimum signed chars, shorts, ints, and longs, and the maximum of those plus their unsigned cousins. These are all very straightforward things to calculate for anyone with an understanding of modular arithmetic. The book then mentions a more challenging problem: calculating minimum and maximum floats. No problemo I thought: Obviously that doesn't work, or this would be a pretty boring blog post. Here's where the awesome comes in. On my MacBook Air, this prints 1.175494e-38 for FLT_MIN but the second line prints 1.401298e-45 for the "calculated" minimum float. That's right it calculates a value that is smaller than FLT_MIN.

Experienced C developers will surely recognize what is going on here but I was flummoxed. It was only upon looking at some of the other constants defined in float.h that I started to figure out what was going on. In particular I noticed that on my MBA, FLT_MIN * FLT_EPS = 1.401298e-45. FLT_EPS is "float epsilon" and is defined as the smallest x for which 1+x != x. This is sort of the standard error on the system, i.e. any calculation could be off by a factor of FLT_EPS. I decided to calculate this in my program and then to also use it to calculate FLT_MAX: The FLT_MAX calculation takes a long time, since it uses such a small increment. I'm sure a bigger increment could be used to speed it up.

The JavaScript assembler

Earlier this week, the third part in a series of articles I wrote was published on developerWorks. The series is on CoffeeScript, a language that I first learned of about a year ago. Last summer I read CoffeeScript by Trevor Burnham over the course of a week while on a cruise ship. Later that summer I was at the Strange Loop conference, and serendipitously wound up at a dinner table with several of the guys who develop Clojure. A lot of the conversation that evening was on ClojureScript.

As a  long time web developer I am not surprised by the growing interest in programming languages that compile into JavaScript. The browser has never been more important, and thus JavaScript has never been more important. Most programmers that I know find that JavaScript leaves a lot to be desired. This is not just a matter of taste, JS is fucking weird. In fact for years, a lot of the more "serious" programmers I knew would turn up their noses at JS. They were too good for that trash, so they kept to the back-end. It's become tougher and tougher to avoid JS, so ... use a different programming language that compiles to JS. Brilliant!

This idea is hardly new. Personally, I can't help but think of Google Web Toolkit. For years GWT has let your write in Java and compile to JS. So why bother with CoffeeScript, ClojureScript, and probably some other awesomes that I'm leaving out? One obvious thing is that Java is an existing language and is statically typed. So it has its own baggage (which includes excellent tooling, but still) and of course many developers do not like statically typed languages. In fact, many argue that web development with a statically typed language is an impedance mismatch of the highest order. Personally I think GWT was also hurt by initially defaulting to producing highly obfuscated "production ready" JS. I always thought that was developer unfriendly. I think that may have changed over the years, but sometimes it's hard to overcome an initial impression. I think one of CoffeeScript's great strengths is that it produces very developer friendly JS by default. I've never built any huge apps in either language, but in my limited experience I found that debugging was easier in CoffeeScript. I could usually figure out what I had done wrong just by looking at the error message.

Any discussion like this would be amiss without mentioning Dart. It gives you dynamic and static aspects, in a way that reminds me of ActionScript. Personally I think optional typing works best when use variable:type style declarations, like ActionScript and Scala, but that's a minor nit. Of course when you consider that "native Dart" is supported in a version of the world's most popular browser, the possibilities start to get really interesting. I attended a Dart meetup recently where the Chrome developer tools allowed breakpoints and the like directly in the Dart script that was running. The SourceMap proposal would allow this kind of debugging of other higher level languages like CoffeeScript and ClojureScript.

The future seems bright for languages targeting a JavaScript runtime. I haven't even mentioned node.js and how that makes these languages viable for both client and server applications. To me CoffeeScript, Dart, and ClojureScript are all promising. Each seems to have a large target audience. If you like dynamic languages like Python and Ruby, CoffeeScript fits right in. If you prefer statically typed languages like Java and C++ and all of the nice tools that come with them, then Dart is attractive. If you like pure functional languages like Haskell and Lisp, then ClojureScript is a fun way to get your lambda calculus kicks in the browser. Of course there is a lot of anti-Lisp bias out there, so maybe there's room for another functional language that targets a JS runtime, something like a F#-script or ScalaScript.

Mobile Application Architecture, Part 3

[Note: So it's taken me a long time to get this third part out. I changed jobs in the middle of writing all of this, and there's a lot to learn at the new gig. Also in case you are wondering, this is heavily based on my experiences working on eBay Mobile for Android and Bump for Android. Both apps use an event driven architecture in places, or at least they did while I was working on them. What follows is not the secret sauce from either application, but some good things that were common to both along with some ideas for improving on the patterns.]

This is the third post in a series on the architecture of mobile applications. In the first post we looked at the classical architecture of mobile apps that evolved from web apps. In the second post we looked at event driven architecture and when it should be used. Now we're going to dive a little deeper into event driven architectures and in particular talk about some of the significant design decisions that must be made and the pitfalls that need to be avoided.

So you say you want an event driven architecture? I'll assume you've done your homework and aren't just doing this to because it seems like the cool thing to do. I'll assume that it really fits your application. One of the first things that you will realize as you get started is that event driven architecture does not exempt you from many important aspects. Security is a great example. You must figure out how you will authenticate with your server and how to secure any sensitive information that you send over the wire. Just because you are no longer in the world of HTTP and HTTPS does not mean that you get to send over people's credit card numbers in plain text. It's not any harder to sniff your bits and chances are that you are just as vulnerable to things like man-in-the-middle attacks. So go read up on things like TCP headers and TLS.

Once the basics are in place, you can think more about the architecture of your application. Your servers are going to be sending events to your application. So what is going to receive them? Do you have each UI (controller, Activity, etc.) receive them? How about something that is more long-lived. The most important thing to remember is that whatever receives events generally has to be able to deal with any kind of event. You can never assume that just because your controller only cares about some small set of events that those will be the only events that it ever receives. Thus you probably need to have an event receiver whose lifecyle extends beyond that of any given screen in your application and can handle any and all events. Now keep in mind that it will often delegate events to other components in your application, so no need to worry about this global receiver being stuffed with knowledge of your app. On Android you could use a Service for this or even a subclass of Application. You could also create some other singleton object that is tied to your application's lifecycle.

Now you might ask does this really need to be a singleton? Not necessarily. However I think that in many (most?) circumstances that a singleton will simplify things for you. What happens to those events that are not needed by the UI? Such events probably mutate some internal state of your application. If that internal state is transient, then it probably makes sense for there to be only one copy of it. If it is persistent... 

Inevitably there will be some events coming in from your server that you do care about at the UI level. These things will probably be "second-hand", i.e. the events will have been received by the longer-lived object we talked about above. Now you need to deal with how to get these events to the UI. The most accepted way to do this is to use an Observer pattern. You define an interface for an object that cares about a certain event (or certain set of related events potentially.) Then the object registers itself with the central receiver. When a new event comes in from the server, the central receiver examines it and decides on who to delegate it to.

I must add a few words of caution here. For each event you can easily wind up with a class for the event's data structure, an interface for the observer, and an implementation of the observer. The last of these things is often handled by having the Activity/Fragment/controller implement the interface, or by using an anonymous inner class to implement it. In any case, you wind up with multiple types being defined for each event. So if you have many types of events that your server will push to your application then you can end up with an explosion of code. Android developers should consider using Handlers and Messages as a leaner, more de-coupled solution. The big downside is that you may windup doing manual type checking and casting. This can be avoided by late binding, i.e. keep the data from the server raw as long as possible and do the parsing after the dispatch to the observer.

Finally I will close this long series of posts with a brief word about server side technologies. This post in particular has focussed on how to write your mobile/client applications with an event driven architecture. There is a flip-side to this and that is how to implement an event driven architecture on your server. Just like in our classical architectures, most servers are dominated by HTTP. That is not only the default interface into them, but it is at the heart of the server's architecture. A connection is a short-lived, usually stateless object. There is generally a one-to-one mapping of HTTP requests to either threads or processes. This is a poor fit for event driven architectures where a connection is extremely long lived. It only dies when the client app dies or explicitly closes it for some reason.

Fortunately there are numerous server side technologies that do not use the request-per-thread/process model. Certainly the most hyped one of these is Node.js. The Internet is chock full of benchmarks showing how Node's non-blocking out-scales Apache, Nginx, etc. However there are numerous other event driven app servers in Python, Java, Ruby, etc. Personally I have a bias towards my ex-colleague Jamie Turner's Diesel framework. The point is that there are many choices on the server and I expect that you will see many more moving forward.

Yo ho, yo ho, a startup's life for me

Friday was my last day at Bump Technologies. It's been a little shorter ride there than I expected, but still a very rewarding one. Before Bump I had been at eBay for four years, so you might think that going to a startup was a big change for me. Actually Bump was the sixth startup that I have worked at. Of the non-startups that I worked at was a very small consulting company that was much more like a startup than not. Still I'll leave the number at six. How have those startups fared?

Two burned out pretty quickly. One of those never got more than angel money, while the other got an $8M series A that it burned through more rapidly than I can understand.

One was bought by Oracle and is now part of their business intelligence suite.

One turned profitable, but was shut down by its board because they no longer thought "hockey stick" growth was possible.

One labored for many years on minimal funding before selling its intellectual property and going out of business.

The other of course is Bump.

[Note: While all of the following is based on my experiences in aggregate, I had no one specific in mind while writing this little rant. In particular my comments on 'modern' startups does not reflect Bump.]

My first two startups were in 2000, almost twelve years ago and during the dot-com bubble. A lot has changed in that time. The capital needed for hardware has shrunk amazingly. Commodity hardware plus the rise of virtualized hosting a la Amazon web services, has led to a world where a startup's big spend is on engineering talent. The value placed on engineering is huge. Engineering is a scarce resource, as the supply of engineers is shrinking while the demand is rising. Further the notion of "10X engineers" is widely accepted, and this essentially lifts any kind of ceiling on the value of an engineer. Now as an engineer this is mostly a Very Good Thing, at least from my perspective. However there are some weird things that this leads to.

The high value of engineering can make for a strange situation in companies with non-technical founders. Just imagine a situation where you have to work hard to get something and continuously spend a lot just to keep that something, and you don't really understand what that something does. That can be hard to swallow and keep down. Now multiply that times five or ten or twenty "somethings." Yeah.

Of course startups have always loved young engineering talent fresh out of school. There has always been huge appeal in such engineers because they are very cheap and have much less of a life outside of work, so they are more likely to happily work the long hours necessary in a startup. With the value of engineering going up so much, the appeal of young engineers is much greater than it was twelve years ago. Go back and read the previous paragraph and consider that such an awkward situation can be avoided with younger engineers. The net result is you have "younger" startups in general where more is asked of young, inexperienced engineers. This can lead to some amazing opportunities for talented young engineers. It can also lead to Engineering Disasters like we've never seen before.

Another interesting change in the world of startups is the value of design. I attribute most of this to the amazing success of Apple and the high value they place on design. Now in Apple's case, I would say they place a higher value on user experience and design follows from that, but more on that subtle difference momentarily. The value of design has obviously lead to a greater role for designers. They are more empowered than ever before to insist on certain aspects of a product. However the value of design has also lead to a lot of "armchair designers". Everyone thinks design is important, and design is the kind of thing that everyone can form an opinion on. Of course we all think that our opinion is very valuable, so we feel strongly about our opinions on design.

Think about how often you hear somebody -- doesn't matter what their job is -- talk about how some website or app or whatever is awful or looks terrible or "broken." The bigger more popular the product is, the more likely it will illicit such responses. I hear developers constantly talking about how something is ugly or beautiful. Even better, I hear developers constantly talk about how most other developers don't care about design and that they are special because they do. It's hilarious.

It's at least as bad if not worse among so called product people in startups. When I was working for my first couple of startups, our product managers were people with some kind of unique experience and thus theoretically some unique perspectives into what our customers would want. Not so much today. The qualifications for being a product person in a startup is pretty minimal as best as I can tell. That doesn't stop the from waxing poetic about the aesthetics of websites and apps.

These product managers and developers may all think that they have a lot in common with Steve Jobs, but as I mentioned earlier, I think Apple's approach is a little different than the approach taken by most startups. I think Apple stresses user experience first and realize that this is not the same thing as product or design. In a startup people have to wear many hats, so a lot of these competing principles get rolled together into one product person who may have some dubious credentials.

The worst sin of startup product people is not their strong opinions on what looks good or bad. It's a tendency to base product decisions on their personal preferences. They think that a product should be exactly what they would find most useful or most cool. Now this is not some new and unique thing. Product managers at startups ten years ago were guilty of this too, as are product manager at big companies today. It's just the circumstances of product managers at modern startups make this situation worse, as does the form-over-function corollary that comes from the emphasis on visual design. It's a perfect storm that leads to products being designed around users with a thousand Facebook friends and 5 tumblr blogs who spend more time at speakeasies than at Target. Of course that might really be your user, but it is statistically unlikely.

I've probably said too much about the current state of startups. This is my blog though, so what about me? I seriously doubt that I'll stop at six startups, though of course I hope my current job lasts for a long time. Like so many other people working in Silicon Valley, I want to work on a product that can change the world. That's what has lead me to startups six times in my career. But startups aren't the only kind of companies that can change the world.

Mobile Application Architecture, Part 2

This is the second post about mobile architecture. In the first post, I talked about how the evolution from web applications has led to architecture's centered around asynchronous access to remote resources using HTTP. I also mentioned how Android applications are able to benefit from cloud-to-device messaging (C2DM) to push new data to apps. The most common use for this is for push notifications, but C2DM is not limited to push notifications. The key to pushing data via C2DM is that it uses a persistent socket connection to the C2DM servers. So whenever any new data is sent to those servers, they can send it over this connection to mobile apps. Now you might think that there is something magical or even worse, proprietary, about using sockets within a mobile application. This is not the case at all. Any application can make use of sockets, not just ones written by Google. Actually that's not true. Any native applications on iOS and Android can use sockets. Socket support was recently added to Windows Phone 7 as well. For web apps there is the Web Socket API, but it is not yet implemented across platforms.

So what are the benefits of using sockets? Well there is the data-push capability that you see in C2DM. In general you can make a much faster app by using sockets. There is a lot of wasted time and overhead in making HTTP connections to a server. This wasted time and overhead gets multiplied by the number of HTTP calls used in your application. There is also overhead in creating a socket connection to a server, but you only pay it one time. The extra speed and server push-iness allows for very interactive apps. Sounds great, but how does this affect the application architecture? Here's an updated diagram.



The big difference here is that instead of requests and corresponding responses, we have events being sent from client to server and back again. Now obviously some of these events may be logically correlated, but our communication mechanism no longer encodes this correlation. The "response" to an event from the app does not come immediately and may not be the next event from the server. Everything is asynchronous. Now in our traditional architecture we also had "asynchronous things" but it was different. Each request/response cycle could be shoved off on their own thread or in their own AsyncTask for example. This is a lot different.

Let's take a simple example. One of the first things that many applications require a user to do is to login, i.e. send a combination of name and password and get some kind of authenticated materials in response (token). In a traditional architecture you would send a login request to the server, wait for the response, and then either get the token or display an error if there was a problem with the username+password combination. In an event driven architecture, your application fires a login event that contains the username+password. There is no response per se. Instead at some point in the future the server will fire an event indicating a successful login that includes a token, or an error that may contain a reason as well. You don't know when this will happen, and you may receive other events from the server before getting a "login response" event.

Logging in is just one example. There are numerous other things that are easy to think of in terms of request/response. The whole idea of REST architectures are built around this request/response cycle and HTTP. Doing a search, getting user information, posting a comment, the list goes on and on. Now of course it is possible to recreate the illusion of a request/response cycle -- just fire your request event and block until you get the response event. But clearly that is Doing It Wrong.

So maybe these event driven architectures are not right for all applications. Yes they are sexier, but don't try to put a square peg in a round hole. This kind of architecture is best suited for apps where you are interacting with somebody else. Let's say you are bidding on an auction. Somebody else bids on the same auction and ideally you will see this happen very quickly. Or maybe you sent a message to a friend on some type of social network, and you can't wait to see their response. Social games are also a great example of the types of apps that would benefit from an event driven architecture. Even if the game doesn't have to be real-time, it would probably still benefit from "lower latency" between moves.
Hopefully you have a better idea now on if your application should use a traditional architecture or an event driven one. Of course there is a lot of room in between. You could easily have an architecture that uses both, picking the right interaction model depending on the use case within your app. Maybe most things use a classical model, but then on certain screens you switch to an event-driven one. Or vice versa. Either way you'll want to read my next post where I will talk about the nuances of implementing an event driven architecture.

Mobile Application Architecture, Part 1

The boom in mobile applications has been fueled by excellent application frameworks from Apple and Google. Cocoa Touch and Android take some very different approaches on many things, but both enable developers to create applications that end users enjoy. Of course "excellent" can be a relative term, and in this case the main comparison is web application development. For both experienced and novice developers, it takes less effort to be more productive developing a native mobile application than a web application (mobile or not.) This despite the fact that you really have to deal with a lot more complexity in a mobile app (memory, threads, etc.) than you do for web applications.

Despite having significant advantages over web development, mobile applications often have some similar qualities and this often leads to similar limitations. Most mobile applications are network based. They use the network to get information and allow you to interact with other users. Before mobile applications, this was the realm of web applications. Mobile apps have shown that there is not necessary to use a browser to host an application that is network based. That's good. However the way that mobile apps interact with servers over the network has tended to resemble the way that web applications do this. Here's a picture that shows this.

This diagram shows a traditional mobile application's architecture. In particular it shows an Android app, hence that little red box on the right saying C2DM, which we will talk more about later. Most of this is applicable to iOS as well though. Most apps get data from servers or send data to servers using HTTP, the same protocol used by web browsers. HTTP is wonderfully simple in most ways. Significantly, it is a short-lived, synchronous communication. You make a request and you wait until you get a response. You then use the response to update the state of your application, show things to the user, etc.

Now since HTTP is synchronous and network access is notoriously slow (especially for mobile apps), you must inevitably banish HTTP communication to a different thread than the one where you respond to user interactions. This simple paradigm becomes the basis for how many mobile applications are built. Android provides some nice utilities for handling this scenario like AsyncTask and Loaders. Folks from the Android team have written numerous blogs and presentations on the best way to setup an application that follows this pattern, precisely because this pattern is so prevalent in mobile applications. It is the natural first step from web application to mobile application as you can often re-use much of the backend systems that you used for web applications.

Before we go any further, take another look at the diagram. I promised to talk more about it. That is the cloud-to-device messaging system that is part of Android (or at least Google's version, it's not available on the Kindle Fire for example.) It provides a way for data to get to the application without the usual HTTP. Your server can send a message to C2DM servers, which will then "push" the message to the device and your app. This is done through a persistent TCP socket between the device and the C2DM servers. Now these messages are typically very small, so often it is up to your application to retrieve additional data -- which could very well go back to using HTTP. Background services on Android can makes this very simple to do.

Notice that in the previous paragraph we never once used the word "notification." C2DM was seen by many as Android's equivalent to Apple's Push Notifications. They can definitely fulfill the same use cases. Once your service receives the message it can then choose to raise a notification on the device for the user to respond to. But C2DM is not limited to notifications, bot by a long shot. Publishing a notification is just one of many things that your application can do with the message that it receives from C2DM. What is most interesting about C2DM is that leads us into another type of application architecture not built around HTTP. In part 2 of this post we will take a deeper look at event-driven architectures.

An OWS Protester and a Communist Walk Into a Bar

I've commented a little on some of my thoughts about the Occupy Wall Street movement. As someone who values personal freedom and liberty, I'm at odds with many of the OWS ideas, natch. Still I can empathize on many of their complaints and desires to make our nation better. However recently someone made the statement to me that to describe OWS as class warfare was laughable. I don't think that was a very intelligent statement. Sure many Republican pundits like to use the term class warfare way too much, but that doesn't mean the term is always invalid.

So is class warfare applicable to OWS? Well let's imagine a hypothetical OWS protester and a hypothetical modern Marxist. Why a Marxist? Because class struggle is a key component to Marxism. What would a Marxist think of statements like "We are the 99%" and "we are getting nothing while the 1% is getting everything"? I think the Marxist would agree with these statements. Once the Marxist realized that proletariat was an outdated term with a lot of negative connotations, I think he would be quick to switch over to saying "the 99%" instead. Who would be more likely to say "capitalism has failed democracy", the OWS protester or the Marxist? Seems like both. Who would encourage participation in general strikes? Who would be more likely to say "Join the global revolution?" Ok maybe that's too easy and meaningless. Who would favor using the force of the government to take from one group (the 1%) to give to the other group (the 99%)? Now don't get too angry about that statement. I'm not saying that this should or should not be done, just that it is a statement that I think both an OWS protester and a Marxist would agree upon.

My point is that on many issues, it is impossible to distinguish between an OWS protester and a Marxist. Does that mean that OWS protesters favor class warfare? No, it does not. However it does suggest that there is plenty of room for a reasonable person to think that OWS does indeed favor class warfare. Of course OWS is notorious for its fractured message, but I've tried to pull all of the above from its main websites and events. It seems to me that everything I've quoted would be things that the majority of OWS protesters would agree with. Those just happen to be the same things that a Marxist would agree with.