DxOMark scores shouldn't be your definitive camera rating system

Over the last few years, flagship devices have pushed the boundaries of what a mobile camera can offer. Every year sensors have improved and the additional elements like better image stabilization and triple cameras add new shooting possibilities to already robust offerings.

Keeping track of these upgrades, many of which are tough to appreciate without an acute eye, becomes trickier every year. It’s fallen to rating scores, provided primarily by DxOMark, to quantify these improvements.

To DxOMark’s credit, its testing procedure is very robust. The company tests camera exposure, color, texture, noise, artifacts, and zoom, along with a new selfie and night scores. Scores are given to each element tested, along with an overall score for each camera. It’s these overall scores that are shared by the likes of Google, Samsung, and Apple to show that their phone is the latest and greatest option around.

The new Samsung Galaxy Note 10 5G, HUAWEI Mate 30 Pro, and OnePlus 7 Pro all rank at, or near, the top. There appears to be a never-ending cycle of the “best DxOMark phone” with each flagship release. Perhaps that’s to be expected, but the company’s scores aren’t always free from controversy. Either way, the increasing reliance on one company’s score to judge camera quality is a little problematic for the industry, especially once you realize that DxOMark isn’t just in the business of ranking phone cameras.

What does DxOMark do?

DxO Labs, the company which runs the DxOMark testing suite, is primarily a consultancy company. In other words, the company charges fees to advise camera hardware companies on how to improve their photography products. This is based on its own analysis and expertise in the camera industry.

No review site is guaranteed to be free from bias, but DxO’s business revolves around attracting big companies to it to make use of its expertise, which adds a lot of baggage to their reviews. Ranking test results in a way that encourages consumers to buy certain phones over others complicates everything.

The company claims to run an independent test, but is that really possible when it offers for-profit consultancy at the same time? There’s no reason to believe DxOMark is in anyway rigging results. After all, the company’s business model depends on its reputation and its results tend to roughly fit with the broader consensus on camera hardware.

However, manufacturers that tune their cameras against the testing suite are likely to score higher than those who don’t. We have heard that a few smartphone manufacturers don’t think DxO’s consultancy fees are worthwhile. These manufacturers don’t score highly on DxO’s tests, if the company even reviews these phones at all.

The pay-to-win problem

In addition to consulting with other companies, DxO Labs also sells its DxO Analyzer solution for testing and measuring cameras. Obtaining a license to use the suite is expensive, especially when you factor in the costs of installation and training to familiarize companies with its functions. There’s nothing wrong with this in principle, however, one would assume that a company, say a smartphone manufacturer, that refines its camera hardware using the DxO Analyzer will score highly when DxOMark comes to test the final product.

There is nothing inherently wrong with a company paying for a service that will result in better quality cameras in their smartphones. Helping to create superior photography results is in everyone’s interests. However, there’s a reliance in the media on DxoMark scores to judge camera quality, which gives the company a lot of influence over not just industry imaging quality, but also how consumers perceive smartphone products.

Those who pay to work closely with DxOMark will likely score more highly in the company’s tests, which is then quoted by many other review sites. There’s pressure on smartphone OEMs to pay for DxO’s services simply for the press recognition.

Many of the biggest brands in the smartphone and professional camera markets are customers of DxO. HTC, HUAWEI, Samsung, and Hon Hai Technology Group (Foxconn) are all on the list. These companies appear to get their money’s worth, with each new generation achieving a higher score than the last. But perhaps most importantly, can we be sure that these latest products are really offering tangible improvements to us consumers?

This leads us to perhaps the biggest issue of all with the industry’s reliance on DxOMark. If companies are shaping their camera development around these tests, DxOMark is thereby partly shaping the development trajectory of smartphone products. However, as the tests aren’t entirely comprehensive and weigh certain features ahead of others, that might not be in consumer interests. DxO took a long time to catch up to trends in zoom and night shots. Its scores don’t always reflect the prevailing trends in the market.

Closing thoughts

All of the above considered, we should definitely take DxOMark’s scores with a pinch of salt. A company working closely with smartphone manufacturers to improve picture quality is surely a good thing for consumers, and DxO clearly knows what it’s talking about when it comes to camera quality. However, it’s important to recognize the potential for bias from a company that has a need to sell services to camera developers, while also scoring the results from companies that it works closely with against those it doesn’t. More so when the tests aren’t completely comprehensive or evenly weighted for all possible features.

Do the Pixel 3, Galaxy Note 10, and Mate 30 Pro all feature best in class cameras? Absolutely. Is DxOMark’s ranking system reflective of camera quality? Probably, depending on the weighting of the results. If some OEMs working closely with the test provider are benefiting with better scores, that’s not inherently bad if its producing better cameras. But if we want more transparent testing and results for smartphone camera quality, consumers, reviewers, and those in the industry should want to consult a wider range of sources.