LEICA Barnack Berek Blog: LENS TESTS

Do they really tell us what we want to know?

A large number of test reports on cameras and lenses can be found in many camera magazines and on the internet, and it seems that in many cases these are more confusing than helpful. Owners of Leica equipment often wonder why their supposedly superior equipment in some cases does not perform to their high expectations. The reasons are diverse and difficult to explain.

As we all know, a modern photographic lens is an immensely complicated instrument, with a great number of performance characteristics. For years, photographers have tried to find a valid method to determine the overall performance of a lens. Unfortunately, it is almost as difficult to measure true lens performance as it is to design the lens in the first place. The emphasis here is placed on “true” performance.

Many readers of magazines and the internet are misled to consider resolution as a valid measure in determining the quality of a lens. The obvious argument is that a lens must be good if it is capable of high resolution. Unfortunately, resolution is not a very good measure, especially if it is taken alone and at face value. This is the very reason why some test reports also publish contrast figures. The contrast level of a lens is at least as important for overall performance as the resolution. As a matter of fact, Leitz has at times sacrificed overall resolution of lenses in order to gain a higher contrast level. Contrast is a very difficult to explain criterion. In simple terms, it can be referred to as the ability of a lens to reproduce high resolution at a recognizable level.

Virtually all resolution tests are done with high contrast (black and white) targets which make it very easy to differentiate between minute details. The results are expressed by lines per millimeter, l/mm, where each black or white line is counted. Actual picture taking is different, however, because a lens is rarely subjected to the ideal conditions of a test target. Instead the lens is confronted with multicolored scenes or subjects, all of which need to be distinguished. Regardless of photographing in color or black and white, a photographic lens must (in most cases) reproduce a colored subject or scene. A lesser lens might very well reproduce two almost identical shades of the same color as one color or the same shade of gray, and thus not be able to resolve this slight difference at all, while a better lens will be capable of making that distinction. On the other hand, in a common lens test, when confronted with high contrast test targets, the lesser lens will inevitably show a higher resolution than it is capable of under everyday conditions.

This is a perfect example showing the importance of contrast over resolution. When viewed from a normal distance of approximately two feet, the bottom picture will immediately look better. Upon closer inspection one will notice, however, that the actual resolution of the top picture is substantially better. This can be seen especially by the small test targets in the center. The actual resolution of the bottom picture is only 50% of that on the top. Yet the substantially higher contrast of the bottom picture immediately gives the impression of a better image, even higher resolution. As I pointed , a pure black and white target will show relatively high resolution, even with a low contrast lens. Had the center target been separated only by different shades of the same color, rather than a high contrast target, the differences would not have been visible, in spite of the relatively high resolution as displayed by the top image. This is a rather extreme example. Differences from one lens to another rarely are as great as in this example.

The gradual disappearance of detail with low contrast subject matter

For that and a number of other reasons, lens manufacturers use a computer read out, the modulation transfer function (MTF), to better determine lens performance. Rather than giving resolution figures, the MTF is based on a fixed resolution value, giving the contrast level of the lens over the entire image area at the various f/stops.

MTF functions also rely on a test target. Like in the example above, the target shows vertical and horizontal lines. This is done because most lenses can distinguish fine detail better in one direction than the opposite. This is the reason why MTF function always show two curves, one for the sagittal test patterns and another for the tangential or meridional one. Either one shows the lens performance starting at the center of the image and going out to the edge of the image.

This is the MTF function of the Leica 50mm f/2 Apo-Summicron-M at f/2.

The four curve pairs show the level of contrast at 10, 20, 40 and 80 lines per millimeter resolution,

a performance level higher than any other lens of this type regardless of manufacturer.

Most published MTF curves are based on a rather low resolution of 30 lines per millimeter at the most. The same “standard” is used by most magazines for their resolution figures. It is obvious that 30 l/mm is a criterion that can be met by virtually any lens short of the bottom of a pop bottle. So it is no wonder that the majority of the lenses tested do achieve relatively good figures. Especially in view of the fact that a 100% contrast rendering is unattainable. But even with these relatively low criteria, the much more even performance level of the Leitz lenses is still revealed. Even at maximum aperture they usually perform to levels that many other lenses do not achieve unless stopped down, if at all. If the contrast figures were based on a resolution of 60 l/mm or even higher, it might become apparent that there are substantially greater differences among various brands of lenses than we are usually led to believe. I have often maintained that photographic magazines and websites which get the majority of their revenue from advertising, have to conduct themselves in a manner that is not detrimental to their advertisers. Although these test are generally very good and unbiased, they are also conducted and presented via criteria which work to the advantage of the publication's subsidizers.

I think it is (or should be) obvious that optical companies other than Leica are all very much capable of making first rate equipment. But their merchandise is also in most cases heavily mass produced. The bench-made process, as used by Leitz, gives the Leica a good competitive edge. But the superiority of Leica lenses especially shows itself in extreme situations. This includes lens performance at maximum apertures and performance under adverse conditions. Unfortunately, published tests do not always consider this and subsequently Leica lenses often seem to less of an advantage than generally expected.

One might also notice in the written evaluations of Leica lenses that the writers often seem to be a bit strained to avoid the overuse of superlatives. But we have seen comments like “...the lens against which all others have to be measured (400mm Telyt)...one of the best lenses ever tested (50mm Summicron)...the closest to our test standards ever (40mm Focotar).”

So far we have talked much about optical performance, the sole criterion used in most lens tests. The mechanical quality of a lens, however, is just as important. Poor mechanical design and quality will eventually degrade even the best lens to mediocre performance. Automobile magazines routinely run 50,000 or even 100,000 mile performance tests on automobiles and I have often hoped that something like this would be done with cameras and lenses as well. The performance of a lens, when brand new, is one thing, especially if it is delivered for tests by the manufacturer after careful tweaking and adjusting to assure the best performance possible. But what about its capabilities after it has gone through several months of use, after it has been knocked around in gadget bags, in cars and airplanes for an extended period of time? Heavy use and the rigors of professional use take a great toll and only the best will remain on top. This is where mechanical quality translates into optical performance and shortcuts in quality will show lower grade results immediately.

Price is the final criterion I should mention. Nobody in the optical industry today is able to perform any miracles. Although mass production does have a beneficial influence on the price of lenses and cameras, this advantage quickly fades when we deal with lenses of an exotic nature that are sold in a much smaller quantity and cannot be mass produced. Suddenly we find lenses from competing companies to be equally expensive than their Leica counterparts. Extreme wide angle lenses and long telephoto lenses, especially those that have proven to be close in performance to their Leica counterparts, usually sell for not much less than their equivalent of Leica lenses. This is the case with most of the major Japanese lens manufacturers. Yet when dealing with absolutes in performance, none of them have anything to offer that can compete with virtually all of the newly designed Leica lenses featuring apochromatic design and aspheric lens elements. The heart of any camera system is its line of lenses, and this is still one of the main reasons for choosing a Leica.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------