If you look at the Screen Sizes and Densities page from the Android developers site, you will see a chart that looks something like the one to the right. There are several slices to this pie, but two of them are much than the rest. One is "Normal hdpi" at 74.5% (as of August 2011) and the other is "Normal mdpi" at 16.9%. What does that mean? It means that 91.4% of all devices have a "Normal" sized screen, so roughly 3.5 - 4.3". The hdpi ones have a higher resolution of course. So for those devices you will want higher quality images and what not.
Of course for anything like this, it is natural to compare things to the iPhone. On the iPhone all devices have 3.5" sized screen. However you once again have the situation with different resolutions between the iPhone 3G/3GS and iPhone 4. So similar to Android, you will probably want to include higher resolution artwork to take advantage of the iPhone 4's higher resolution screen. However as a developer you don't get much of an advantage with the higher resolution screen since the overall size is the same. It's not like you're going to make your buttons the same number of pixels and thus fit more buttons on the screen, etc.
No wait a minute, there is a difference between 91.4% and 100%. A good chunk of that difference is because of tablets, with their "Xlarge mdpi" screens. You can expect that segment to continue to grow in the future. But again, this is a similar situation to the iOS. An iPad has a larger screen than an iPhone. If you care about this, you either make a separate app for the iPad, or you incorporate the alternate layouts within your existing app and make a "universal" app. This is the same deal on Android.
To really make a fair comparison, you would have to exclude the xlarge screens on Android. Then the percentage of phones that fall in the combined Normal mdpi/hdpi bucket is even higher. It's still not 100% but it is pretty close. My point is that if somebody wants to talk to you about Android fragmentation and they use screen size as an example, some healthy skepticism is appropriate.