Dxomark – objectivity or bias?

Hello!

At the moment, the Internet has rightfully become the most important and, to some extent, an irreplaceable source of information about everything in the world. Sometimes this information is reliable, sometimes it requires verification and rechecking. Often this only applies to information obtained from dubious sources that do not value their reputation, which is easy to lose. Sometimes this also happens with respected resources and companies trying to monetize their reputation, previously earned by years of dedicated and honest work. Examples of this can be cited a lot, ranging from world-famous print media, once from a respected and objective source of information turned into 'jeans' and a mouthpiece of other people's opinions, and ending with younger Internet resources specializing in a particular industry in which they are accepted be considered the experts who have risen on this in the internet age. And after a number of facts, the user begins to think about whether it is possible to continue to believe those who were once a measure of quality, objectivity and a source of information about something.

Today I would like to try to answer a similar question, which has repeatedly arisen in me regarding the resource DxOMark, which evaluates the quality of cameras in various devices and creates a rating based on the ratings received.

Dxomark - objectivity or bias?

Being an inveterate gadget lover who has used a huge number of smartphones from different vendors, more than once or twice asked himself the question: on what principle does the resource give ratings? And many times I did not agree with the ratings from the rating. The first time such a question came up sharply before me was when I saw the same ratings in the ratings for the Nexus 6P and iPhone 6 cameras.

Dxomark - objectivity or bias?

My first thought was, 'Are you serious? Are these smartphones the same score? When did this iPhone 6 reach the Nexus 6P's camera? What a weird rating? ' From that moment on, I began to closely monitor the rating of this resource. And each time more and more I came to the conclusion that sometimes these points are drawn and have only a distant relation to reality. For example, according to this resource, Samsung Galaxy Note 8 takes off on a par with Apple iPhone 8 Plus. When I saw these figures, Stanislavsky woke up in me with his sacramental phrase …

Dxomark - objectivity or bias?

And after my gaze fell one line higher, on which the OnePlus 6 is located, the very Stanislavsky began to tear his hair out.

Yes, I fully admit that, not being a professional photographer, I do not understand something and, perhaps, some professional nuances elude me, but I have eyes. And my eyes tell me that something is wrong with the rating.

And now about what made me write all this sheet of text – the camera rating in the Samsung Galaxy S8 , which turned out to be even lower than that for the Galaxy S7 / S7 Edge.

Dxomark - objectivity or bias?

According to DxOMark, the Galaxy S8 shoots worse than the Galaxy S7 / S7 Edge. How why? This is especially strange for the owner of both of these smartphones.

Dxomark - objectivity or bias?

Yes, the Galaxy S8 has the same camera module that was in the 'sevens', maybe that's why the guys decided not to bother and just draw a score? However, even with this option, it should be borne in mind that the matrix in the S8 is different. Not taken into account? Or does it actually give worse results?

It is very hard to believe it.

And now let's move on from theoretical reasoning to direct comparison. Wondering about the adequacy of evaluating these two generations of Galaxy smartphones, I took several paired photos in different lighting conditions.

The main ones were the conditions of insufficient illumination and the presence of backlight, since I believe that in this mode the camera should be evaluated. Firstly, it is in such conditions that the camera can reveal its potential and either give a decent result or ruin the photo. Secondly, in conditions of sufficient lighting (sunlight, clear weather), all cameras of expensive smartphones photograph about the same. Thirdly, a large percentage of everyday shots are taken in either artificial or insufficient lighting conditions, and with a weak camera, the shots turn out to be spoiled.

For the purity of the experiment, all photographs were taken hand-held, in fully automatic mode, with autofocus and were not subjected to any processing or correction.

So, we are trying to confirm or deny the scores given by DxOMark, and indirectly answer the question of whether to trust these ratings in the future, or look for a more impartial assessment.

Left S7, right S8.

First pair of photos: Solid color texture with medium amount of light.

At first glance, the photos seem almost the same and do not have any fundamental differences. However, in comparison with what I saw with my own eyes, the color rendition on the S8 turned out to be more accurate, since in real life the walls have a slight pinkish tint. S7 gave them a slight greenish tint.

Dxomark - objectivity or bias?
S7
Dxomark - objectivity or bias?
S8


If you zoom in on the images, it turns out that more small details are preserved in the photo taken from the S8 and the texture is better visible, which is badly damaged by the noise reduction on the S7.

Dxomark - objectivity or bias?
S7
Dxomark - objectivity or bias?
S8


Then we look into the distance and in cloudy weather we photograph a house with a textured surface.

Dxomark - objectivity or bias?
S7
Dxomark - objectivity or bias?
S8


Again, the result is very close at first glance, and you will hardly be able to notice the difference on the smartphone screen. However, the difference is noticeable on the computer screen – the S7 did not manage to convey the color of the sky and clouds so accurately.

Now we zoom in and again see the aggressive noise control of the 'seven', which has eaten up the details for the sake of a smooth picture. Good or bad, I don't know, but I like detailed photos more.

Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


The next couple of photos is tricky for most mobile phone cameras – grass.

Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


Both photos came out well. At first glance, the photo taken on the S7 is distinguished by great realism and calm colors (for an amateur). And if you take a closer look, it turns out that this realism is achieved due to the same aggressive noise reduction, which, as in the previous photos, again ate the details and texture. However, in this pair of photos, the S7 showed more sharpness across the entire field than the S8, in which the edges of the photo crumbled.

Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


And a couple more photos with a lot of small objects of the same color with contrasting light on the left. Which photo do you think came out better?

Dxomark - objectivity or bias?
S7
Dxomark - objectivity or bias?
S8


The next couple of photos is a single-color, poorly lit corner in a room with artificial light.

Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


This time the S8 decided to show off its noise reduction capabilities, while the S7, on the contrary, decided to 'make some noise'. Who, in your opinion, did better?

Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


The next couple of photos demonstrates the ability of cameras to look into the distance.

Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


You can clearly see the high transparency and contrast of the picture of the S8 (right).

Now let's try to test the ability to display differently illuminated areas in one frame.

Dxomark - objectivity or bias?
S7
Dxomark - objectivity or bias?
S8


S7 again decided that the coolest shade was green, and texture and detail were something that could be sacrificed.

Dxomark - objectivity or bias?
S7
Dxomark - objectivity or bias?
S8


A room with complex light.

Dxomark - objectivity or bias?
S7
Dxomark - objectivity or bias?
S8


Again, the photo taken by the S8 is more transparent and cleaner.

And now we turn to the conditions in which most smartphone cameras fail – twilight.

Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


Dxomark - objectivity or bias?
S7
Dxomark - objectivity or bias?
S8


It's very strange how DxOMark could give these two smartphones the same score.

For me personally, it is obvious who won in these two photos.

Another series of photos in conditions of lack of light.

Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


Dxomark - objectivity or bias?
Dxomark - objectivity or bias?


Photos from the S8 have fewer highlights and flares, and also more detail. And this despite the fact that the technical characteristics and resolution of the cameras are the same.

This, according to DxOMark, is also almost the same result with a slight advantage of S7 … Strange, but I have a completely different opinion.

And finally, a photo of a small bright object in dim artificial lighting conditions.

Rate article
About smartphones.
Add a comment