To pickle or not to pickle

My latest Core Python inspired post is about using repr() and pickle in Python. Oh and a few bits about JSON as well.

It's Called Script-A-Cu-Lus

Not script-a-licious! I wrote an article on script.acu.lous and it is now online at developerWorks.

Eclipse Day 2008

Congratulations to the Eclipse Foundation for releasing Ganymede today. As part of the festivities, Eclipse and Google collaborated on Eclipse Day yesterday. I was fortunate enough to be asked to speak at the event, and it was a great experience. I spoke about how we use Eclipse at eBay. The response was very enthusiastic with lots of folks amazed at how many tools we had built for our developers and how we had standardized tools across such a large development organization. I totally agree that our tools developers are amazing. For me, I don't know how we could survive without having all of the tools we have built! Here are the slides from my presentation, courtesy of SlideShare:

| View | Upload your own

I wanted to really thank Ian Skerrett from Eclipse for organizing everything. The folks at Google were excellent hosts, so I must also thank Robert Konigsberg from Google.

ACM Ajax Seminar

Today I taught a seminar on Ajax to the Bay Area chapter of the ACM. It was a lot of fun. The audience was great, asking lots and lots of excellent questions. The level of enthusiasm was very rewarding. I promised the folks there that I would provide download links. So here they are:

The Slides (OpenOffice format)
The Slides (One big PDF)
All Demo Code

The demo code also includes a Scriptaculous demo that I did not have time for. It uses Ruby on Rails. The Prototype demo uses PHP, and the other ones all use Java. A couple of the demos uses a database, and for those I used MySQL.

Finally I wanted to also thank Sang Shin and Doug Crockford for allowing me to use their material for the seminar.

Blogging about Books

I mentioned some new books that I am reading. The publisher of those books gave me the opportunity to blog about them while I read them. I thought that would be fun, and would give me extra motivation to read the books thoroughly. I am blogging about them directly on their site InformIT. Here is the RSS feed of my blog over there and here is the first post. It is on the Core Python book that I am reading.

Scala vs. Java Performance

Last week I wrote about the 'wallify' problem that was worked on at the BASE meeting. One thing that was mentioned there was that it seemed like a Scala implementation would not have performance that was as good as in Java. I decided to give this a test. I used the Scala code I wrote last week and compared it to Java code that used essentially the same algorithm.

To test, I created a driver. The driver would vary the numbers in the input list, basically 1..N where N was varied. It would then measure the time it took to wallify the list 100 times. The JVM was the 1.5 JVM for OSX with the -server flag set. I was also giving it 1GB of memory (more on that later.) Here is what it looked like in Java:

And in Scala:

Very similar. There was no noticeable performance loss in Scala. Of course there is an obvious problem here. This algorithm should have been linear with respect to the size of the list, N. This is clearly not linear! I took the logarithms of both the size and the time, and found a correlation of about 0.996. This algorithm is actually quadratic.

I thought that maybe my algorithm just sucked. I tried a couple of other Scala ones. One provided by Jorge Ortiz and one created by the group led by Bill Venners. The both had the exact same performance characteristics as my algorithm, i.e. they were quadratic as well.

The only thing I could think of was garbage collection. I put on gc:verbose and watched a lot of garbage being collected. I upped the heap to 1 GB, but it had no effect. So I am not sure if GC is really the culprit or not.

Update: James Iry had a great insight. Java's BigInteger computation is linear once things get big. To keep them small, you can use all 1's. Here is what the performance looked like then, just for my original Scala code.

Ok, that's not quadratic anymore. I wasn't sure if it was linear (I was paranoid about compiler tricks given the list of 1's that was being multiplied) either, so I actually extended the test with more data points. With a few more data points, I got a nice correlation of 0.993 for a linear fit. This confirms that the algorithm itself is linear for small numbers, but is quadratic for larger numbers where high performance integer math becomes significant.

New Books!

Wife: "you got a real heavy book in today by fedex."
Me : "I need to leave work early"

Yep, new books arrived today. Here is the new reading list:

Java Concurrency in Practice by Brian Goetz. I know, I know, I should have already read this. The S3 bulk uploader really convinced me of this.

Refactoring by Martin Fowler. I know, I know, I should have read this many years ago...

core Python. Good to have while hacking on GAE.

Don't Make Me Think by Steve Krug. Wouldn't it be great if designers understood programmers? I can't help with that, but maybe I can understand design? Probably not, but worth a try.

BASE Meeting

Last night I attended my first BASE (Bay Area Scala Enthusiasts) meeting. Organizer Dick Wall had a nice problem to solve for the attendees. The problem was to take a list of integers, and replace each element in the list with the product of all the other elements in the list. For example, if your input was (2,4,6,8) the output should be (4*6*8, 2*6*8, 2*4*8, 2*4*6) = (192, 96, 64, 48). If you want to solve the problem yourself, stop reading and come back when you are ready to read about the solutions.

There is an "obvious" brute force solution, which would be to implement the algorithm exactly as the problem is stated. This is an O(N^2) solution, not so good. You can do better by simply taking a product of all of the elements of the original list, and then replacing each member of the list with the product divided by the element. So for the example above, the product = 2*4*6*8 = 384, and thus the output should be (384/2, 384/4 , 384/6, 384/8). This is O(N) solution, and that is much better. This is the best you can do in terms of speed and space, as you only keep around one extra integer in memory (the product.) However, it doesn't always work.

The problem is when there is a zero in the list. Then the above algorithm will lead to division by zero errors. Dick pointed this out, and pointed out that there are really three casses. If there is exactly one zero, then every element in the return list will be zero, except for the one at the same index as the lone zero from the input list. If there are two or more zeroes, then every element in the return list will be zero. With that in mind, here was my solution:

def wallify(list:List[Int]):List[BigInt] ={
  val one = BigInt.int2bigInt(1)
  val noZeroes = list.filter(_ != 0)
  if (noZeroes.length == list.length){
    val prod = list.foldLeft(one){(p,e) => p*BigInt.int2bigInt(e)}
    list.map(prod/_)
  } else if (list.length - noZeroes.length == 1){
      list.map((el) => 
        if (el == 0) 
          noZeroes.foldLeft(one){(p,e) => p*BigInt.int2bigInt(e)} 
        else 
            0
      )
  } else {
    list.map((el) => 0)
  }
}

I had to leave early, and didn't finish my code until today. Here is the code that the group, led by Bill Venners:

def wallify(input: List[Int]): List[Long] = {
  def boringCalcProduct(longInput: List[Long]): Long = {
    var product = 1L
    for (ele <- longInput) {
      product *= ele
    }
    product
  }

  def calcProduct(input: List[Long]): Long =
    // input.foldLeft(1L)(_ * _)
    (1L /: input)(_ * _)
  
  val zeroCount = input.filter(_ == 0).size
  val longInput: List[Long] = input.map(ele => ele.toLong)
  zeroCount match {
    case 0 =>
      val product = calcProduct(longInput)
      longInput.map(product / _)
    case 1 =>
      val noZero = longInput.filter(_ != 0)
      val product = calcProduct(noZero)
      longInput.map(ele => if (ele == 0) product else 0)
    case _ =>
      input.map(ele => 0L)
  }
}

iPhone 3G

It was fun to watch the WWDC keynote today on MacRumors. It was even more fun to watch things on summize as Twitter held up admirably during the keynote. I was impressed with the new iPhone. I really like the white version, as it seems like a nod to the original iPod to me (an owner of the original.) It made me think, should I get rid of my Blackberry 8830 and switch to the iPhone?

iPhone 3G

• The screen on the iPhone is awesome. The Blackberry's is nice, but it's not really close.
• Much better for listening to music. Blackberry is functional for this, but hardly elegant.
• Web browsing is much better on the iPhone, especiall now that it has a 3G network. The Blackberry browser is good, and Opera works great, but mobile Safari is very nice.
• The iPhone has a camera. I miss having a camera, even if the iPhone's is crummy (as are most phone cameras.)
• The iPhone has wi-fi. I often use my Blackberry's network access in places that have free wi-fi.

Blackberry 8830

• I have tried the iPhone keyboard, and greatly prefer my Blackberry's keyboard. This one is not even close.
• The GMail app for the Blackberry is excellent. On the iPhone you can use the GMail as IMAP, but that is not as good, not even close.
  
• Of course the Blackberry receives my email and calendar from my work's Exchange servers. This is the main appeal of Blackberry for many. If I had an iPhone, I think I would have to run a client from my desktop computer at work. Co-workers have told me that this is a problem because our desktop computers run the 64-bit version of Windows 2003, and a lot of Apple's software has issues with this.
• Google Maps is also very nice on the Blackberry, even if Verizon screws us over and doesn't let it access the device's GPS. Then again I have used GPS on phones (both mine and my wife's) and it is useful, but definitely not a replacement car GPS. Anyways, Google Maps on the Blackberry seems a lot better than accessing the web application on the iPhone.

Hmm, it doesn't seem like a slam dunk either way. It seems like the calendar/email push advantage may be negated, depending on the OS issues. If Google made iPhone apps, especially for GMail, it would almost clinch it for me. I've used GMail as IMAP provider with Mail.app and did not like it very much. Blackberry GMail is a killer app for me. Given Google's investment in Android, it seems unlikely that they will create a similar app for the iPhone... I think I will stick with my Blackberry.

Sports, sports, and more sports

The Belmont Stakes
I used to be a really big horse racing fan. I was introduced to it while at Caltech, as we were just a few miles from Santa Anita, one of the world's greatest tracks. I had my favorites back then: Lure, the two-time winner of the Breeders' Cup Mile and Bertrando who should have won the 93 Classic if it wasn't for the shocking upset by Arcangues, who was more than 100-1. In 96, I had the trifecta for the Kentucky Derby, a trifecta that paid $10K+ ... but didn't play it because I didn't have the money to cover all the possibilities (needed $120, too much for me to bet.) I was also a big fan of Holy Bull. I never liked Cigar, as Cigar beat Holy Bull in the race that the Bull broke down... Ultimately it was horses breaking down that really made it hard for me to enjoy horse racing. Horses were clearly becoming more brittle, mostly because of inbreeding. Drugs were being used as a band-aid, but it was obviously a bad situation that would only get worse.

Still, I try to watch some of the big races still each year, and of course that includes the Triple Crown races. I had no particular love of Big Brown, but of course it seemed appealing to have a Triple Crown winner. One of the first videos that I ever watched online was a video of Secretariat winning the Belmont and thus the Triple Crown. I get chills just thinking about, it was such an awesome feat. Anyways, it was sad to see Big Brown lose and disturbing to see his jockey pull him up as soon as he realized that Big Brown was not going to win. Something did not sit right about that to me, like he knew the horse was not sound and should not have been racing. Anyways...

NBA Finals
Good guys 2, forces of evil 0. What more is there to say? Amazingly the Lakers are huge favorites to win in Game 3, and I wouldn't be surprised if they were still being favored to win the series. There is always a Los Angeles bias in Las Vegas. After all, Vegas makes the line based on money bet, not based on their opinion or some other kind of 'actual' odds. Casinos play down the middle, take a cut from boths sides and always profit with no risk.

French Open
Nadal vs. Federer renewed... I could not believe how easily Nadal won this time. I am still unconvinced that these guys are really that great, and that their success is not more attributable to poor compettition. Nadal vs. Borg? Nadal is certainly athletically superior, and there is the racquet technology factor. And I still think that many of the big servers of the 90s would give either one of these guys huge problems, though maybe not on clay.

Gears and Flash

Tonight I launched up Firefox 3, and it informed there was an update to Gears ready to install. I went for it, and sure enough Gears was now working with Firefox 3! I didn't find any official announcement about this, but I didn't let that stop me from finishing an idea I had last week: getting Gears and Flash to work together.

This was not too hard. It is just a matter of using ExternalInterface to create a bridge between the two worlds. I wrote a quick POC, using a simple DB table for storing name/value pairs. I think it would be fun to re-implement the flash.data package that is normally only available in AIR apps...

Here is the JavaScript code:

function dbCall(sql, args){

    var db = google.gears.factory.create('beta.database');
    var rs = null;
    try{

        db.open('database-test');
        db.execute('create table if not exists little_table (key varchar(20), value varchar(100))');
        rs = args ? db.execute(sql, args) : db.execute(sql);
        data = [];
        while (rs.isValidRow()){

            row = {};
            for (var i=0;i<rs.fieldCount();i++){

                var name = rs.fieldName(i);
                row[name] = rs.field(i);
            }

            data.push(row);
            rs.next();
        }
        return data;
    } finally {

        rs.close();
    }
}

Here is the ActionScript code:

import flash.external.ExternalInterface;
 

[Bindable]
private var rs:Array = [];
 
private function load():void{

    rs = ExternalInterface.call('dbCall', 'select * from little_table');

}
private function save():void{
    var key:String = keyIn.text;
    var value:String = valueIn.text;
    ExternalInterface.call('dbCall', 'insert into little_table values(?,?)', [key, value]);
    keyIn.text = "";
    valueIn.text = "";
    load();

}
private function clearData():void{
    ExternalInterface.call('dbCall', 'delete from little_table');
    load();

}

And here is the MXML I used to test it out:

<?xml version="1.0" encoding="utf-8"?>

<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" initialize="load()">
    <mx:DataGrid x="202" y="59" id="grid" dataProvider="{rs}">

        <mx:columns>
            <mx:DataGridColumn headerText="Key" dataField="key"/>
            <mx:DataGridColumn headerText="Value" dataField="value"/>

        </mx:columns>
    </mx:DataGrid>
    <mx:Label x="10" y="251" text="Key"/>

    <mx:TextInput x="74" y="249" id="keyIn"/>
    <mx:Label x="242" y="251" text="Value"/>

    <mx:TextInput x="285" y="249" id="valueIn"/>
    <mx:Button x="460" y="249" label="Save" click="save()"/>

    <mx:Button x="243" y="298" label="Clear" click="clearData()"/>

</mx:Application>

Good vs. Evil '08

No, this is not about Barack Obama vs. John McCain. This is about something much more benign: basketball. So maybe the Magic did not make it too far in the playoffs. You can't be too surprised by Celtics vs. Lakers, and probably not too unhappy either. These are the classic teams of the NBA. However it is much more than that this year. It is good vs. evil.

Good is Kevin Garnett. If either of my sons wanted to become basketball players or play sports compettitively at all, I would have no problem with them idolizing Kevin Garnett. The Celtics have not always played beautifully during the playoffs, but you can never be bored watching KG play. He's all out on every play. He'll do anything for his team and teammates. He's unselfish, almost too a fault. Everytime he plays, he demonstrates the characteristics that you hope that children learn from sports.

Evil is Kobe Bryant. Too harsh, you say? Let's not forget this guy is a rapist. Let's not forget when he sliced open Manu Ginobili's face while trying to draw a foul at the end of a game. Let's not forget all the times that he would react to criticism about shot selection by refusing to shoot during games. Let's not forget that he forced Phil Jackson and Shaquille O'Neal out of town. Let's not forget that Phil Jackson said that Kobe was uncoachable. Let's not forget that prior to this season, he was demanding a trade because the Lakers were losing.

I won't go as far as to say that Kobe embodies everything bad about pro-sports -- that would not be true. But he embodies a lot of bad things. He's not just selfish, he's a borderline sociopath. Yes he seems like a great guy this year, now that the Lakers are winning, he's making commercials again (the rape has faded from sponsors' memories), and he won the MVP award.

I remember the Celtics-Lakers series of the 80's. I always rooted for the Lakers back then. Magic Johnson was not only a great player, but completely likeable. Plus everybody where I lived rooted for the Celtics because they had so many white guys.

I used to go to Lakers games all the time when I was in college in Los Angeles.

I would love to root against the Celtics, just because Boston has enjoyed too much success this year. The Red Sox won the last World Series, and the Patriots went undefeated in the regular season.

However, the past must be put aside. The stakes are bigger. Boston's gotta win this one. Go Celtics.

Party Unity

Yesterday the Democratic Party decided to "settle" the issue of what to do about the Florida and Michigan primaries. Their decision was obviously intended to appease Hillary Clinton to some degree, while still assuring that Barack Obama clinches the Party's nomination. I support Obama over Clinton, but I am not pleased with this decision. Here is why.

First, by cutting the number of delegates in half, the Party has essentially said that a Floridian's vote is worth half as much as say a Californian's. Second, the Party decided to give 40% of the votes of Michigan to Obama. This was a practical matter, because Obama was not on the ballot in Michigan. If he would have been, he would have probably done much better than that, but it is still wrong for the Party to decide how people voted, instead of letting people actually vote.

The other thing that I object to is all of this business about party unity. I hate that term, because I hate party unity. Party unity is the key to the two-party system that forces us all into picking the lesser of two evils. Let's take a look at the current Democratic contest. Obama does not do well with union workers, and what is wrong with that? Why does he have to do well with union workers? Oh, because if he doesn't get the support of union workers, minorities, women, etc. then he can't beat the Republicans' coalition of religious groups, nationalists, upper middle class, etc.

Personally I would love to see the Democratic Party break apart. I would also love to see the Republican Party break apart. I think it would be great to see coalitions formed on more of an issue-by-issue basis. Is there any doubt that we would already be out of Iraq if that was the case? I think most Republicans know it is a mistake to be there, but party unity subverts democracy. If we didn't have party unity, we could probably agree on some kind of pollution credit system to protect the environment.

Party unity is the key to the success of divisive politics. You can use such issues only when you are assured it will not cause you to lose votes from your own party -- i.e. only when you have party unity. Do you think Republicans would bring up things like abortion, immigration, and stem cell research if they thought that they might lose some of the upper middle class voters they count on? No way. But when you know that your own constintuency will still vote for you even when you bring up issues they disagree with you on, then you are free to be evil.

Bulk Upload to Amazon SimpleDB

This weekend I was helping a friend with loading some data to Amazon's SimpleDB. The problem was fairly simple. He had a flat file with 170K lines of data. Each line represented a video from YouTube along with some metadata about the video. He wanted to turn that file into a "table" on SimpleDB, where each line (video) from the file would become a "row" in the table.

I decided to use Java for the task. I found a useful Java library for using SimpleDB. Some users of the library didn't like it, as it uses JAXB to turn Amazon's XML based API into a Java based API directly. That didn't bother me so I used it.

I wrote a quick program to do the upload. I knew it would take a while to run, but didn't think too much about it. I had some other things to do, so I set it running. Some three hours later, it was still going. I felt pretty silly. I should have done some math on how long this was going to take. So I scrapped it and adjusted my program.

Amazon has no bulk API, and this is the source of the problem. So you literally have to add one item at a time to SimpleDB. The best I could do was to parallelize the upload, i.e. load multiple items simultaneously, one per thread. Java's concurrency APIs made this very easy. Here is the code that I wrote.

import java.io.BufferedReader;

import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;

import java.util.concurrent.TimeUnit;

import com.amazonaws.sdb.AmazonSimpleDB;

import com.amazonaws.sdb.AmazonSimpleDBClient;
import com.amazonaws.sdb.AmazonSimpleDBException;

import com.amazonaws.sdb.model.CreateDomain;
import com.amazonaws.sdb.model.CreateDomainResponse;

import com.amazonaws.sdb.model.PutAttributes;
import com.amazonaws.sdb.model.ReplaceableAttribute;

import com.amazonaws.sdb.util.AmazonSimpleDBUtil;


public class Parser {

   private static final String DATA_FILE="your file here";
   private static final String ACCESS_KEY_ID = "your key here";
   private static final String SECRET_ACCESS_KEY = "your key here";
   private static final String DOMAIN = "videos";
   private static final int THREAD_COUNT = 40;
  
   public static void main(String[] args) throws Exception{

       List<Video> videos = loadVideos();
       AmazonSimpleDB service =
           new AmazonSimpleDBClient(ACCESS_KEY_ID, SECRET_ACCESS_KEY);
       setupDomain(service);
       addVideos(videos,service);
   }


   private static List<Video> loadVideos() throws IOException {

       InputStream stream =
           Thread.currentThread().getContextClassLoader().getResourceAsStream(DATA_FILE);
       BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
       List<Video> videos = new ArrayList<Video>();
       String line = reader.readLine();
       while (line != null){

           Video video = Video.parseVideo(line);
           videos.add(video);
           line = reader.readLine();
       }

       return videos;
   }
  
   // This creates a table in SimpleDB
   private static void setupDomain(AmazonSimpleDB service) {

       CreateDomain request = new CreateDomain();
       request.setDomainName(DOMAIN);
       try {

           CreateDomainResponse response = service.createDomain(request);
           System.out.println(response);
       } catch (AmazonSimpleDBException e) {

           e.printStackTrace();
       }
   }   
  
   // adds all videos to SimpleDb
   private static void addVideos(List<Video> videos, final AmazonSimpleDB service) throws Exception{

       // create a thread pool
       ThreadPoolExecutor pool =
           new ThreadPoolExecutor(THREAD_COUNT, THREAD_COUNT, 10,
                   TimeUnit.SECONDS,
                   new ArrayBlockingQueue<Runnable>(videos.size()));
       // Create a task for each video, and give the collection to the thread pool

       for (final Video v : videos){
           Runnable r= new Runnable(){

               public void run() {
                   addVideo(v, service);
               }

           };
           pool.execute(r);
       }
   }
  
   // This adds a single item to SimpleDB

   private static void addVideo(Video v, AmazonSimpleDB service){

       PutAttributes request = new PutAttributes();
       request.setDomainName(DOMAIN);
       request.setItemName(v.getVideoId());
       List<ReplaceableAttribute> attrs = videoToAttrs(v);
       request.setAttribute(attrs);
       try {

           service.putAttributes(request);
       } catch (AmazonSimpleDBException e) {

           e.printStackTrace();
       }
   }
  
   // Turns a video into a list of name-value pairs

   private static List<ReplaceableAttribute> videoToAttrs(Video v){
       ReplaceableAttribute author = new ReplaceableAttribute();
       author.setName("author");
       author.setValue(v.getAuthor());
       ReplaceableAttribute date = new ReplaceableAttribute();
       date.setName("date");
       date.setValue(Long.toString(v.getDate().getTime()));
       // for votes we pad so we can sort

       ReplaceableAttribute votes = new ReplaceableAttribute();
       votes.setName("votes");
       votes.setValue(AmazonSimpleDBUtil.encodeZeroPadding(v.getVotes(), 4));
       return Arrays.asList(author, date, votes);
   }

  

}

And for completeness, here is the Video class:

import java.util.Date;


public class Video {

   private final String videoId;
   private final int votes;
   private final Date date;
   private final String author;
   private Video(String videoId, int votes, long date, String author) {

       super();
       this.videoId = videoId;
       this.votes = votes;
       this.date = new Date(date);
       this.author = author;
   }

   public String getVideoId() {
       return videoId;
   }

   public int getVotes() {
       return votes;
   }

   public Date getDate() {
       return date;
   }

   public String getAuthor() {
       return author;
   }


   public static Video parseVideo(String data){
       String[] fields = data.split(" ");
       return new Video(fields[1], Integer.parseInt(fields[0]), 1000*Long.parseLong(fields[2]), fields[3]);
   }

}

Some interesting things... I played around with the number of threads to use. Everything seemed to max out at around 3-4 threads, regardless of whether I ran it on my two core laptop or four core workstation. Something seemed amiss. I opened up the Amazon Java client code. I was pleased to see it used a multi-threaded version of the Apache HttpClient, but it was hard-coding the maximum number of connections per host to ... 3. I switched to compiling against source so I could set the maximum number of connections to be the same as the number of threads I was using.

Now I was able to achieve much better throughput. I kept number of threads and max number of http connections the same. For my two-core laptop, I got optimal throughput for 16 threads and connections. For my four-core workstation, I got optimal throughput for 40 threads and connections. I think I will re-factor the Amazon Java API and offer it to the author as a patch. There is no reason to hard code the number of connections to three, just make it configurable. The underlying HttpClient code is highly optimized to allow for this.

PECS in Action

One funny bit at JavaOne was Josh Bloch introducing the mnemonic PECS. Well actually what was most funny about it was the picture of Arnold Schwarzenegger that accompanied it... Anyways, PECS stands for producer-extends, consumer-super. Instead of repeating what Josh said or plagiarizing Effective Java, I will give an example of how PECS helped me out yesterday.

One there was an API that existed back in olden times, before Java 1.5 It looked like this:

void runInboundCycles(final Module[] modules)

There was also a runOubound, but you get the picture. The class Module is an interface that has many implementations. This API got tweaked courtesy of Java 1.5:

void runInboundCycles(final List<Module> modules);

Before the change you could do this:

runInboundCycles(new Module[] { new MyModule() } );

A logical uprev would be:

runInboundCycles(Arrays.asList(new MyModule()));

Turns out that won't compile! The Arrays.asList call will return a List and the Java compiler says that is not a List. So instead you have to do something annoying like:

List<Module> modules = new ArrayList<Module>(1);
modules.add(new MyModule());
runInboundCycles(modules);

This is particularly annoying if modules is actually a member variable, as now you cannot declare it to be final. Enter PECS.

My API is using the Modules, thus my parameter is a producer to the API. Remember producer-extends, so refactor the API like this:

void runInboundCycles(List<? extends Module> modules);

Now you can pass a List<MyModule> and the compiler won't complain.

There are a couple of things about this that bother me. When my crazy brain looks at the API, it thinks "the API consumes Modules." Maybe that's just me. The other thing that bothers me is writing ? extends Module because Module is an inteface. Now granted it would suck to have to write ? implements Module just because Module is an interface and write ? extends Module just because Module was a class, so I am not advocating the altnernative. It just feels weird to write extends in front of an interface type. Maybe I have been programming in Java too long.

Armchair Architects

Funny post by @al3x about Twitter architecture. Oh wait it wasn't supposed to be funny, oops. The fact is that he should totally expect more people to diss Twitter and pretend that they could easily solve all of the problems. I am not just being cynical about people, they actually have some good reasons to do so:

• Twitter crashes a lot. If your site did not crash so much, then people would not think you are an idiot and that they could easily do a better job. The people may all be wrong, but that does not matter. Why do you think Microsoft has come to have such a bad reputation? People do not care about what MSFT did to Netscape, Sun, or Apple. They care about BSODs. People hate Vista because Microsoft did not make it as backwards compatible with 3rd party drivers that did lots of bad, hacky things. But now Vista crashes, so people complain about MSFT.
• Twitter seems simple. You put a 140 character limit on updates and what do you expect? Part of Twitter's appeal is its simplicity, but that same simplicity creates expectations and makes people think they could do it themselves better. Maintenance is expected for things that seem complex, like cars or Photoshop, but not for (seemingly) simple things like iPods or Twitter. If you think hard about it, Twitter is much more complex than it seems, but who wants to think hard?
• Ruby developers are obnoxious. Oh this is my favorite. Ruby developers are a small but very vocal group. They love rubbing it in your face that Ruby is so much more expressive or object-oriented or whatever than anything else on the planet. The Rails sub-cult is even worse about this. So when the most high-profile Rails site starts failing constantly, you must expect a lot of smug developers to wag their fingers. It is kind of a shame that Twitter is paying for DHH's bad karma ... but then again @blaine did make that infamous claim about how easy it was to scale Rails. Of course he's gone now, but there is still enough bad karma to go around. How many Ruby developers would admit how bad their software is? Think about that.

SVJUG: JPA 2.0

Last night I went to the monthly SVJUG meeting. Patrick Linskey from BEA, err Oracle, spoke about JPA 2.0. First off, Patrick is an excellent speaker. I belive he was acuired by BEA via the SolarMetric acquisition, and now he's been acquired by Oracle. If nothing else, he is a much better speaker than anyone I've ever heard from Oracle. There were two very interesting things I saw in JPA 2.0.

The most interesting was query introspection. To me this move really makes it possible for JPA to get "pushed down the stack" if you will. I think it will allow for more abstracted frameworks to use JPA and hide it from the programmer. Working recently with Grails made me realize why this is important. Grails adds many JPA EntityManager methods to the domain classes. In many cases, you don't even have to call these methods. A typical Grails crud action does something like def myDomainObject = MyDomainObject.get(params.id) and then myDomainObject.properties = params and that's it. There is no explicit call to save() or persist(), etc. One could argue that this is a very good thing, as code like entityManager.persist(myDomainObject) is clearly boilerplate.

Now it would be pretty hard to get the similar functionality in Java, as you cannot add methods to objects at compile-time. You could make your domain objects extend an abstract class, but we are all too in love with POJOs to allow for that. However, a container of some sort could do these things for you. Introspection APIs into the JPA are a key to such a thing working. I don't know if the ones being added in JPA 2.0 are sufficient, or if that is even what the JPA folks have in mind, it's just an interesting possibility I see.

The other interesting addition to JPA 2.0 is adding a Cache interface to the spec. This is a nod to the implementers (like BEA's Kodo) that all use some type of "second level" cache. The API simply allows you to evict things from the cache, which seems reasonable enough. Cache invalidation is one of the hardest things out there, so it is nice to have explicit APIs for doing this. 

JPA is a topic close to my heart. I started working with Hibernate about six years ago. In some ways there are two philosophies out there when it comes to web-scale systems. One school of thought rejects relational databases. Take a look at Google's Big Table, Amazon's SimpleDB, or Ning's Content Store for examples of this. The other school embraces the relational database and says that they can be scaled. If you saw the eBay presentation at JavaOne, then you know what school eBay belongs to. I think that other RDBMS believers include Yahoo and Facebook. Historically ORMs get in the way of that scaling, and JPA is no exception. I'm not quite ready to give up on them yet.

American Idol

Yes another season of American Idol will finally end tonight. I think the show is really showing its age, of course I've thought the same thing for a couple of years now. Last night's "showdown" demonstrated the evil of the show. It is a show powered by the music industry, you know the same guys who like to sue kids for downloading music, etc. Thus there is evil in its core, and it really came out last night.

Before the show, my wife asked me who I thought would win. I told her that I thought David Cook had the edge. He was more original and a better performer. I thought that David Archuleta was probably a better singer, but was so immature and annoying. He is unable to sing anything but slow songs, and I thought that people had started to catch on to this, including the judges. Ah, but I should have realized that evil that lies in the heart of men and known that these were the reasons that David A. was assured of victory.

Amazingly David A. performed three ballads. I thought this was his best tactic, but that surely he would catch a lot of criticism for it. Nope. Instead he was praised for song choice! In other words, he was praised for embracing his limitation. Even worse, when given the choice of any song to perform he recycled a song from earlier in the season.

David A. played things as safe as he possibly could. He took no chances at all. That is fine, but how many times have we heard the judges blast contestants for this? Not this time. Instead he was praised. To make it even worse, his opponent was criticized for not doing the same thing: Simon Cowell told David C. that he screwed up by not recycling a song.

It was all complete hypocrisy that reeked of an agenda. The evil empire is clearly at work here. The actual performances did not matter. The judges completely contradicted everything they've ever said in the past in an effort to promote one contestant over the other. It will surely work as well.

Don't be too upset, though. The evil empire is falling apart. The internet put a stake in their heart a long time ago. Go listen to some NIN or Radiohead on your iPod and laugh at the music industry. One day MBA students will study them as a perfect example of how to ruin a business by being too conservative and afraid of (technological) change. It's the same short sighted, greedy principles that worked against David C. last night that ruined a huge, billion dollar industry.