• last month
Transcript
00:00All right, so we wanted to answer the question
00:02once and for all,
00:03what is the best smartphone camera out right now?
00:06Maybe you think it's the iPhone,
00:07or maybe it's a Samsung flagship,
00:09or maybe it's one of those pixels.
00:11Now in years past,
00:12we've done a blind voting using social media polls,
00:15which was really fun and yielded some surprising results,
00:17but that would still depend on the matchups
00:20that I planted at the very beginning.
00:22So I specifically put the iPhone and the pixel
00:25on opposite sides of the bracket,
00:27expecting the best ones to hopefully meet in the finals.
00:30But they never did,
00:31and there's always a bit of human input.
00:32It's actually possible I could have accidentally
00:34put the second best phone up against the best phone
00:36in the first round, and it got eliminated.
00:38So this time, we're gonna solve all that once and for all.
00:41Welcome to the blind smartphone camera test,
00:44scientific edition.
00:45So here's the idea.
00:47We took 16 smartphones that came out during this year,
00:50the flagships, your expected heavy hitters,
00:52but then also some mid-rangers as well,
00:54all the way down to the unexpected,
00:56and then we took the same exact photo
00:58with each one of them.
00:59Harder than it sounds,
01:00because you have to hold perfectly still
01:02for several minutes straight,
01:03but we actually did this
01:05for three different types of photos,
01:07a standard daylight photo, a low-light photo,
01:10and a portrait mode snap.
01:12And then we take all these images, compress them,
01:14strip them of their EXIF data,
01:15and assigned each one a letter.
01:17Now, for maximum statistical confidence,
01:20we would have to ask you guys to vote
01:23on every single possible matchup at least once.
01:26So A versus B, then A versus C,
01:29then A versus D, all the way through,
01:31and then B versus C, B versus D,
01:34over and over, 16 choose two for my stat heads.
01:37But that's clearly a bit too much.
01:39I mean, we wanna be scientific,
01:40but having you sit there and vote over 100 times
01:43for each category was a bit of a tall ask.
01:46So instead, we got a little clever with it,
01:48and we built a power ranking system,
01:51actually an Elo rating algorithm,
01:53which you might've heard of.
01:54They use this in chess, they use this in table tennis,
01:57and basically in any video game you've played
01:59where there's a bunch of different matchups
02:00all happening at once
02:01and you need to form a ranking of them.
02:03But this time we'll be using it
02:05to sort of form a power ranking of smartphone cameras.
02:07So basically when you hit enough sanctioned matchups,
02:10you're given a rating based on the ratings of your matchups.
02:14So you, along with everybody else,
02:16get shown comparisons at random,
02:18and we have this little progress bar along the bottom.
02:20You vote for the better photo each time,
02:22and then after a little while,
02:23we have enough information to determine
02:26your individual customized statistically significant winner.
02:30So we built this whole site
02:31with the Elo rating system in there.
02:32We loaded up all the photos,
02:34and then we actually tossed this first into our Discord
02:37to have some of you beta test this
02:39before anyone else knew it existed
02:40so we could experiment with the data formatting
02:42on the out end so we could make sure it was all organized.
02:44So actually shout out to Discord
02:46for sponsoring this portion of today's video.
02:48So with the launch of their new server subscription feature,
02:51we opened up an exclusive premium area
02:53on our Discord server where members can ask the team,
02:56including me, general questions about tech
02:58or any of the content,
03:00and get exclusive insider info
03:01that you won't find anywhere else like this test.
03:04And also, if you're wondering,
03:06Discord does it right with revenue split.
03:08It's 90-10, so 90% going to the creator.
03:12So your subscription directly supports us,
03:14and if you're a creator looking for more information
03:16on this particular feature
03:17and adding it to your own Discord,
03:19definitely check out the link below.
03:21So anyway, I dropped this site in there.
03:23A bunch of you got in there quick and voted,
03:25and basically we got the data back in this text format.
03:28It looks kind of messy at first,
03:30but actually it's super helpful, and we can work with this.
03:33A summary of how many times every single matchup
03:36got voted on and the votes for each side.
03:38And also, since I had a lot of help
03:40with building this site and loading up all the pictures,
03:42even though I know which phones were involved,
03:45I don't know which phones took which pictures,
03:47so I could blind test myself.
03:50And it was every bit as engaging as I was hoping.
03:54I could kind of tell some of the phones
03:56that I think took certain pictures,
03:58but for the most part, I would just take a glance,
04:00and if I was seeing those two pictures side-by-side
04:02on Instagram or Twitter, I'm just picking the better one.
04:06No pixel peeping or thorough inspection.
04:08It's just picture on the internet, vote.
04:11And in the end, my standard picture winner
04:14was G, my low light winner was K,
04:18and my portrait mode winner was also K.
04:21So everything looked like it was working.
04:23The data processing was working as we wanted.
04:25So the only thing left to do was send it live.
04:29And here we go.
04:31So as soon as it went public, you guys jumped right on it.
04:34It peaked at around 35,000 votes per minute,
04:38and then ended up totaling more than 21 million votes
04:42in about three days.
04:44There were essentially zero bad actors,
04:45and now we have a load of data.
04:48Many of you may have gone through this,
04:49which means you've done the voting,
04:51and now you have your letter,
04:52which means it's time to reveal all the smartphones
04:55that were involved in this test.
04:58And here you go.
04:59A through P, that's all 16 phones right there
05:02in their orders, which means the winner
05:04when I blind took the test myself
05:05for my standard photo was the Pixel 6a.
05:09My winner for portrait mode and night mode
05:13was the Pixel 7.
05:14But what were the overall winners according to statistics?
05:17Were the 20 plus million votes according to science?
05:21I don't know if there's any other collection of data
05:23like this anywhere on the internet,
05:24so this is super cool to get into.
05:25So let's dig into it.
05:27So first of all, overall winner.
05:30So with the highest average ELO rating
05:32across all three categories,
05:35in third place is the Asus Zenfone 9,
05:39in second place is the Pixel 7 Pro,
05:43and first place is the Pixel 6a,
05:46the people's choice camera of 2022.
05:49But there is a lot of fascinating information to unpack here.
05:52So first of all, doing well in one category
05:55did not necessarily guarantee doing well
05:57across any of the other categories, right?
05:59So the Pixel 7 Pro came in second, fourth,
06:03and first in the three categories.
06:05The Pixel 6a came in third, second, and second,
06:09but the Oppo Find X5 Pro
06:11actually came in first for standard,
06:13then third for low light, and 15th for portrait mode.
06:17This basically means that if you wanted,
06:19you could argue for the importance of the standard photo
06:23over the low light or the portrait mode photos.
06:25So let's say the standard photo
06:27is like three times as important to most people
06:29as the other two categories, which feels about right.
06:32With that weighted importance on standard photos,
06:35then the Oppo Find X5 Pro becomes the overall winner,
06:39which is interesting.
06:39And in case you're wondering
06:41what the iPhone was in all this,
06:41the iPhone 14 Pro tested in sixth place,
06:4510th place, and fifth place.
06:47But a natural curiosity for me anyway was to ask,
06:50what were the biggest losers?
06:52Which ones got absolutely destroyed the most in this test?
06:56And your undisputed winners of losing the most
07:00is the Sony Xperia 1 Mark IV and the Moto Edge 30 Ultra.
07:06These phones performed terribly in these tests.
07:08So the Sony, despite its world-class sensor
07:10and extremely advanced apps,
07:12went in full auto like the rest of these phones,
07:14did not produce good pictures,
07:16and ended up with the lowest average ELO rating
07:18across all three categories.
07:20And then the Moto Edge 30 Ultra
07:21actually had the most overall losses.
07:24It took a bizarre sequence of photos,
07:25including an extremely dark standard light photo
07:28and an incredibly over-sharpened low light photo,
07:31that it's the only one
07:32to have 2 million total votes against it.
07:35Now, here's another interesting thought.
07:37Votes per dollar.
07:39If we wanted to sort by bang for the buck basically,
07:42which ones, if we account for the launch price, the MSRP,
07:46which one was performing the best relative to its price?
07:50And the two winners for votes per dollar by far
07:53are the Pixel 6a, no surprise, over 4,200 votes per dollar,
07:58and the Realme 10 Pro Plus also sneaks in there,
08:01the only other phone up over 4,000 votes per dollar.
08:04Sneaky good camera phone for 379.
08:08And then the obvious losers again
08:09are the $900 Moto Edge and the $1,600 Sony Xperia 1 IV.
08:15Also, I just had to, I had to look through the text files
08:17and find the biggest dunk,
08:19just the biggest overall matchup discrepancy
08:22between two phones in any category.
08:24And I found it, that would be portrait mode,
08:26Pixel 7 Pro versus Sony Xperia 1 Mark IV,
08:30with the Pixel winning 98% of the votes here.
08:34So to the 1,523 of you that voted for the Sony here,
08:37not sure what you saw that the rest of us didn't.
08:40But overall, just more than seeing
08:41which ones got the most votes
08:43is seeing why certain phones and certain pictures
08:47got the most votes.
08:49And the best way to do that
08:50was actually just putting all of the photos
08:52in order side-by-side in order of how many votes they got.
08:57And that's how we started to see
08:58some really interesting patterns.
08:59So like, for example, last time in the head-to-head,
09:02we kind of established that in a case of A versus B,
09:05if one photo is a little brighter than the other,
09:08it'll probably win.
09:09But how far can that go?
09:10Is there a such thing as too bright in this case?
09:13And it turned out, yes,
09:15this is all the standard mode photos in order.
09:18And yes, the three darkest photos
09:20are the three biggest losers, hands down.
09:22They crushed any shadow detail in my hair
09:25and the extremely comfortable sweater I'm wearing
09:27on shop.mckbhd.com.
09:29But right above that is three
09:32of the absolute brightest photos.
09:34Like these were just overexposed straight up.
09:36Matter of fact, it looks like we kind of got this order
09:38of great photos, then too bright, then too dark.
09:43So it shows me that good exposure wins,
09:45but if you're gonna miss,
09:46then yeah, people would rather have it be too bright
09:49than too dark.
09:49With night mode altogether and in order,
09:51you can kind of see a bit of the same thing.
09:54So the most neutral correct exposure is generally one,
09:57if they were sharp.
09:58This time there's about four good exposures,
10:00Vivo X80 Pro, Pixel 6A, Oppo Find X5 Pro,
10:04and Pixel 7 Pro.
10:06Then there's the two brights,
10:08which people mostly preferred over the two darks.
10:11Also, there were some seriously wild things happening
10:13with some of these photos.
10:14In low light, the Vivo, I mean, it absolutely nailed it.
10:17It's clearly the low light king.
10:20The S22 Ultra, interestingly, came in fifth place,
10:23even when completely missing focus on my stationary face
10:28and hitting the background for some reason.
10:30The iPhone 14 Pro did an enormous amount of HDR
10:33that had me looking kind of green like a zombie,
10:36but it still came in 11th place here.
10:38The Moto Edge Plus, as I mentioned,
10:40kicked out maybe the most over-sharpened photo
10:42the world has ever seen.
10:44And then in dead last is the iPhone SE,
10:48the only phone in this test that does not have night mode.
10:52And the sad thing is like, it does have OIS,
10:54it does have a decent sensor,
10:56and it does have a new processor,
10:58but Apple just feels the need to protect
11:00their more expensive phones with more premium features,
11:03so they didn't give the SE night mode,
11:05and it suffered big time here.
11:07Then in portrait mode, we have the most variety of them all,
11:11because not only are we judging things
11:12that we normally would about a photo,
11:14like sharpness, color, exposure, et cetera,
11:16but we are also now adding cutout quality
11:20and artificially blurred background quality.
11:23And every phone uses a slightly different focal length
11:26for their portrait mode by default.
11:28Some of them do a 2X portrait mode,
11:30some of them 2.5, some of them 3X,
11:33some of them even do 1X.
11:35But combining all of these factors,
11:37the Pixel 7 Pro absolutely dominated portrait mode.
11:41This was the runaway winner here in this category.
11:43Great cutout, great exposure,
11:46great colors and a nice soft background, it nailed it.
11:49But guess what?
11:50The Pixel 6a was in second,
11:52and the Realme 10 Pro Plus snuck in there in third,
11:55right above what I consider the best cutout
11:58of the entire competition,
12:00which came from this Galaxy S22 Ultra.
12:03I think it's pretty close to perfect at edge detection here,
12:07pretty similar to the Zenfone 9's cutout,
12:09but a better overall photo.
12:11So it got fourth place instead of seventh.
12:13I was also curious if there would be a correlation
12:15between how many votes a phone got
12:17and whether or not they used
12:18more of a telephoto focal length,
12:21just because maybe it would feel more natural
12:23having a blurred background if you're at 3X.
12:25But when we lined it all up,
12:27there was almost no trend line at all and no correlation.
12:30So that was interesting.
12:31Really the only other definite thing
12:33I noticed about portrait mode photos
12:35is that the three worst photos
12:38that got the fewest votes by far
12:41were also by far the brightest photos.
12:43So see, it turns out you can overdo the brightness.
12:47And that, I think that is basically it.
12:51So summary time, just from all the data that we have,
12:54things that we learned.
12:55Number one is that being really good
12:57in one of these categories
12:58did not necessarily guarantee being good at the others.
13:01That's clearly reflected in the data.
13:03Two is that Pixel 6a and the Realme 10 Pro Plus
13:09turned out to be the best bang for the buck
13:10as far as a smartphone camera
13:12from the phones that came out in 2022.
13:15And the Pixel 6a being the overall winner
13:17actually matches with it winning our bracket style
13:21social media polling blind test from last year,
13:24which the Pixel 5a won.
13:26And last but not least, yeah,
13:27brighter photos do typically win,
13:30but you can in fact be too bright.
13:32So I think this mission was a success.
13:35We learned a lot.
13:36I will say congrats to the Pixel 6a
13:38on the People's Choice Award, so to speak.
13:40And you know what?
13:41We're gonna leave up the voting website
13:44till let's say through the end of this year,
13:47till the end of January, how about?
13:48So January 31st is the last day
13:50you'll be able to go to vote.mkbhd.com
13:53and blind test yourself.
13:55And I think we'll replace the letters now
13:56with the actual names of the phones
13:58now that it's been revealed,
13:59but we don't wanna have to pay
14:01for all this terabytes of data processing forever.
14:03So that's your deadline.
14:04So thanks to y'all on the Discord
14:05for helping us beta test this thing.
14:07Thanks to Zach for helping us build this site
14:09and implement the Elo style power ranking system.
14:12And thanks to you for watching.
14:15Catch you guys in the next one.
14:17Peace.