Showing posts with label benchmarks. Show all posts
Showing posts with label benchmarks. Show all posts

Tuesday, January 06, 2009

JavaFX Performance

Recently I did a comparison of JavaFX performance vs. Scala. I did this mostly for kicks, and because some people thought that Mr. JavaFX was picking on other, non-Java languages that run on the VM. James Iry duly pointed out that JavaFX should be benchmarked against the likes of Ruby, Groovy, or Scala. It is meant to be a client-side technology, so it should go up against client-side technologies. So I re-did the little performance comparison to match JavaFX against JavaScript, ActionScript (Flash), and C# (Silverlight).

A comparison like this really becomes a comparison of Virtual Machines. For JavaScript there are a lot of choices. I decided to go with some that are supposed to be fast: Google Chrome, Firefox 3.1 (beta 2), and Safari 4 (developer preview.) Because I wanted Chrome involved, I had to go Windows. So I ran everything under Windows XP, running under Parallels on my MacBook. Here is the pretty graph:

I was just a little surprised by these results. JavaFX is indeed the fastest, but just barely. I was somewhat shocked to see Chrome's V8 JS engine right behind. In fact the difference is negligible for small iterations (shown above.) At larger iterations, JavaFX maintained 20-40% margin. As you can see from the graph, Flash and Silverlight were kneck-and-kneck as well, but was always about 7-10x slower than Chrome/JavaFX. Safari and Firefox were very underwhelming.

Of course this was just a micro-benchmark. The code just does a series of recursive calls. So what were are really measuring is the ability of the VMs to unwind these recursive calls. It is not suprising that HotSpot handles this easily. Actually, the same code in straight Java is much faster than the JavaFX version. It is surprising to see how well V8 handles this.

Now does the ability to unwind recursion translate into performance that a web application user would notice? Maybe. It certainly points to JavaFX's or V8's ability to make optimizations to application code. It is probably a more meaningful test than some raw number crunching.

Friday, January 02, 2009

JavaFX vs. Scala

Chris Oliver post a nice little performance comparison of JavaFX vs. Groovy and JRuby. He concluded that JavaFX was 25x faster than these other two languages running on the JVM. Of course I had to see how this compared to Scala. First here's the direct translation to Scala I did of the code:

object Tak{
def tak(x:Int, y:Int,z:Int):Int = if (y >= x) z else tak(tak(x-1, y, z),tak(y-1, z, x),tak(z-1, x, y))
def main(args:Array[String]) = {
0 until 1000 foreach ((i) => tak(24,16,8))
}
}

Here are the results for JavaFX on my MacBook:

$ time javafx -server -cp . tak

real 0m12.847s
user 0m11.926s
sys 0m0.338s

So my system is a little slower than Chris Oliver's. That's why I had to run his bench on my MacBook first, to make a fair comparison to the Scala version. Here are those results.

$ time scala Tak

real 0m9.690s
user 0m9.122s
sys 0m0.261s

Scala not only beat out JavaFX on my system, but was also faster than JavaFX running on Chris Oliver's faster system. The guys at Sun should have never let Odersky go!

Thursday, September 04, 2008

JavaScript Faster than Flash

This is the last benchmark for awhile. Well, at least for today. I converted the JS benchmarks to ActionScript and tested them. The result were surprising, as JavaScript in Safari 4 and Firefox 3.x edged out Flash:


A few notes. I could not convert all of the tests, as two of them (the DOM and Ajax tests) were predicated on browser specific code. I could have done 'equivalent' functionality in ActionScript, but it did not appropriate for comparison. Otherwise the code was translated as is ... for the most part. I did add static type information where possible. There were also a few APIs (on Date and Array) that had be tweaked slightly. I tested similar changes to the JavaScript. The only test where there was any effect was the Date test. The JavaScript used Date.parse, which does not exist in ActionScript. The Date constructor does the same thing. If I switched to using the Date constructor in JavaScript, it was just slightly slower.

It certainly seems that much of the performance advantage enjoyed by Flash upon arrival of Flash Player 9 has been erased. Flash had a strong advantage still in more mathematical calculations (dates, integer and floating point arithmetic) as well as string manipulation. It did very poorly with arrays and regular expressions. I would guess that as the JITs for JavaScript get better, the string advantages will disappear. Flash will probably maintain an advantage in more mathematical computations, especially given its vector graphics features. Hopefully advances in JavaScript will spurn Flash VM progress.

Notes
1.) Tested on both Flash 9 and 10 RC2 on both OSX and Windows. Negligible performance differences in any of the permutations.
2.) Also tested with Silverlight, but only on Windows. It was slower than everything except IE7. However, that was because it was terribly slow at regular expressions and error handling. It clearly had the best JIT as it was able to reduce some of the tests to 0 after a couple of executions.

JavaScript Benchmarks, now with Chrome

As promised yesterday, I did the JS benchmarks again on a Windows machine so I could include Google Chrome. I tried to be pretty inclusive, adding in IE7, IE8 beta 2, Firefox 3.0.1 (current release), Firefox 3.1 with and without JIT, Safari 3.1 (current release), Safari 4 beta, Opera 9.5 and Chrome. This was all run on my workstation, a 4-core, 3.2 GHz box with 8 GB of RAM. Any add-ons, extensions were disabled. Here is the pretty picture.


Once again Safari is the kind. Safari 3.1 beats everything except for Safari 4 beta, which crushes even Safari 3.1. Opera was a little slower than Safari. Chrome was generally comparable to the various Firefox browsers, but overall slightly slower. Like Firefox 3.1+JIT, it was very on error handling! Of course IE was the slowest by far, but at least IE8 is faster than IE7. Maybe IE8 is shipping with debug symbols included (as Microsoft has often done in the past) and the release candidates will be much faster than the betas. Or not.

Anyways, Chrome, and its V8 engine, does well, but does not seem to be ahead of Firefox and is certainly behind Safari and Opera. Maybe they can do better on the Mac!

Wednesday, September 03, 2008

More JavaScript Benchmarking

My old boss sent me this link about Google Chrome performance. It's a good read. It includes a link to an interesting JavaScript micro-benchmark. It included some interesting findings on Chrome vs. Firefox 3, Safari 3.1, and the new IE 8 beta 2. I was curious about some other browsers, namely Firefox 3.1 beta with and without JIT, Safari 4 beta, and Opera 9.5. Of course I made a nice picture of my results.


Interesting results. First off, FF 3.1 with JIT did not crash. It crashed so many times on me yesterday, that I was sure it would crash on this. Even though it did not crash, it was barely faster than FF 3.1 no JIT or FF 3.0.1. In fact, it was really only faster at error handling and the same on everything else. Apparently errors are easy to JIT for TraceMonkey!

Next, Safari 4 beta is fast. If you look at the link above, Safari 3.1 was already the fastest thing out there, so I guess this should not be a surprise. It crushed everything and it did it on the kind of tasks that real developers do a lot: array and string manipulation, regular expressions, and DOM manipulation (technically not part of your JS engine, but practically the most important test.) I am not used to seeing Opera lose when it comes to any kind of benchmark. If you throw out the array manipulation, it and Safari are pretty close.

I will have to boot up Parallels and try out Chrome vs. Safari 4 beta vs. FF 3.1 beta on Windows.

Tuesday, May 06, 2008

Dynamic Language Performance

A couple of days ago, I read Charlie's post explaining the performance boost seen in Groovy 1.6. Reading stuff like this always leaves me with a great feeling. Not only do you learn something, but it makes other things make more sense. It brings order to chaos, or something like that. Around the same time I read that, I was working a new article about Grails, so the Groovy angle was particularly interesting. I love benchmarks, so it was time to have some fun.

I wrote a Groovy version of the same Ruby code I had used to benchmark JRuby. This was an extremely straightforward port. I was amazed at just how similar Groovy's syntax is to Ruby. Here is the code:


def expo(n,p){
def r = n % p
def exp = 0
def div = p
while (r == 0){
exp += 1
div *= p
r = n % div
}
return exp
}

def factor(n){
def factors = new java.util.HashMap<Integer,Integer>()
def s = n * 0.5
def p = (2..s).toArray()
p.each{
if (it) {
def r = expo(n,it)
if (r){
factors[it] = r
}
def val = it*2
while (val <= s){
p[val -2] = null
val += it
}
}
}
return factors
}

def numDivisors(n){
def total = 1
factor(n).values().each{
total *= (it+1)
}
return total
}

def n = 2
def num = 1
def max = Integer.parseInt(this.args[0])
def Integer triangle = 0
while (num <= max){
triangle = n*(n+1) * 0.5
num = numDivisors(triangle)
n += 1
}
println(triangle)


Anyways, here is the chart.

There is definitely a performance boost for long running processes where JIT'ing can happen more easily in 1.6. It was not as dramatic as I thought it might be, but it is there. Of course this is just one silly benchmark that is heavy in integer math, so take that for what it's worth. 

I also compared Groovy and JRuby. This was also surprising: 



Pretty close! Groovy seems to start-up a little slower, but pulled ahead slightly on bigger tasks. Perhaps the apprentice has overtaken the master.

Also, just for kicks, I tried out Scala. Here is the code:


import scala.collection.mutable._

object Euler12{
def expo(n:int, p:int):int = {
var r = n % p
var exp = 0
var div = p
while (r == 0){
exp = exp + 1
r = n % div
div = div * p
}
if (exp == 0 ) 0 else (exp-1)
}

def factor(n:int):Map[int,int] = {
var factors = new HashMap[int,int]()
var s:int = (n/2) + 1
val p = (2 until s).toArray
p.foreach( (num) => {
if (num > 1){
val r = expo(n, num)
if (r > 0){
factors.put(num, r)
}
var i = num*2
while ((i-2) < p.length){
p(i - 2) = 0
i = i + num
}
}
})
return factors
}

def numDivisors(n:int):int = {
var total = 1
factor(n).values.foreach((num) => {
total = total * (num+1)
})
factor(n).values.foldLeft(1)((p,m) => {
p * (m+1)
})
}

def main(args:Array[String]) : Unit = {
val t = new java.util.Date()
var n = 1
var num = 1
val max = Integer.parseInt(args(0))
var triangle = 3
while (num <= max){
triangle = n*(n+1)/2
num = numDivisors(triangle)
n = n + 1
}
println(triangle)
}
}


This turned out to not be fair. Scala's performance is exactly on par with Java and thus blows away JRuby and Groovy. 


I guess that is what happens when you have a language written by a guy who once wrote javac... Actually I would guess this is mostly a function of the static typing in Scala. It certainly bodes well for initiatives to bring features of Scala, like (BGGA-style) closures and type inference, to Java. It seems possible to implement all of this with no impact on performance, even on a JVM that has not been made to support such features. 

Tuesday, April 10, 2007

JavaScript Benchmarks

Whenever I play with a new computer, one of the things I like to do is run some JavaScript benchmarks (BenchJS.) It's a useful benchmark, since if I'm going to use the computer a lot, chances are that I'll spend a lot of time browsing the web on it. So JavaScript performance will be noticeable to me.

I have two computers at work. The first is a Dell Optiplex. It has a 3.0 GHz Pentium 4 with a 1.5 GB of RAM. I've never had a computer clocked at 3 GHz before. That was near the pinnacle of the NetBurst architecture.

The second computer is an IBM ThinkPad T43p. It has a 2.1 GHz Pentium-M with 2 GB of RAM. Again this was near the pinnacle of the single core Pentium-M's. It has some other nice features, maybe I'll blog about it so more later.

Finally, I recently upgraded my home computer. It's still an AMD 64X2 3800+. I upgraded the RAM to 2 GB. I also upgraded the video card, but I don't think that would matter much for JavaScript performance. I've bought a new hard drive for it, but haven't installed it yet. Again I doubt that would matter. I'm also already bought Vista for it, and will upgrade it soon. Maybe this week, or maybe next weekend, whenever I have time.

Right now, all three computers in question are running Windows XP. I ran Firefox 2.0.03, IE 7, and Opera 9.10. Here are the results:















SetupBrowserTest 1Test 2Test 3Test 4Test 5Test 6Test 7Total


Pentium4-3.0 GHz/1.5 GB RAMFirefox 21.5941.3440.5940.5470.2662.4681.5948.407


Pentium4-3.0 GHz/1.5 GB RAMIE 71.5786.2030.5470.7030.4221.7960.68811.937


Pentium4-3.0 GHz/1.5 GB RAMOpera 91.5631.1560.4380.3120.0782.1560.6416.344


Pentium-M/2.1 GHz/2.0 GB RAMFirefox 21.6720.8440.390.4060.1561.610.7195.797


Pentium-M/2.1 GHz/2.0 GB RAMIE 71.5622.9690.3440.750.4531.3280.6728.078


Pentium-M/2.1 GHz/2.0 GB RAMOpera 91.5620.4370.2820.2650.0631.3280.4224.359


AMD 64X2 3800+/ 2.0 GB RAMFirefox 21.6090.9060.3440.6090.1711.5941.8757.108


AMD 64X2 3800+/ 2.0 GB RAMIE 71.5633.3280.3750.6570.6871.3910.7188.719


AMD 64X2 3800+/ 2.0 GB RAMOpera 91.5630.5160.2970.2810.0781.4220.4694.626















Interesting results! From the browsers, Opera was always faster than Firefox and Firefox was always faster than IE. No surprises here. I was most surprised to see how well the Pentium-M performed. I thought the AMD would beat it out, but it did not. The biggest difference between the two was on the last test. This test calculates the day of the week for Christmas day for the next 1000 years. This test accounted for almost all of the Pentium-M's advantage on each system. I'd be interested to see the JS code for this, but one would think that it would come down to a lot of integer math. So it would make sense that the Pentium-M would have an advantage here. Actually, you gotta wonder why the P4 didn't do better on this test.