About the Picture Study
Chapters 1 to 4 are everything you need before you start. Chapter 5 links to a deeper write-up, The study in depth, for anyone curious how it works and where it may be weak.
1. Why we are doing this
A news app shows each story with a small thumbnail by default. A few stories have a picture good enough to be shown big, across the width of the screen. Choosing which ones is hard to do with simple rules like picture size, because a large file can still be a logo or an advert, and a modest file can be a wonderful photograph.
We want to find out whether a small program can make that choice well enough to be trusted. To judge a program fairly we first need human judgement to compare it with. And because this is partly a matter of taste, we need to know how much two different people agree. That is where you come in.
The question each time
Does this picture deserve prominent visual space for this story? It does not have to match the headline literally.
2. What you do
You will see 172 pictures, one at a time, each with its headline, in an order that is random and different from anyone else's. For each one you tap one of four choices. Your answer saves immediately and the next picture appears. It takes roughly 20 to 40 minutes, and you can stop at any time and come back with the same link; it resumes where you stopped. You can go Back and change an answer.
Each picture is shown three ways: whole, as a large 16:9 photo, and as a small thumbnail, so you can see how it would actually look.
There are no right answers. Go with your first honest reaction. Please do your answers alone, without asking anyone.
3. The four choices
- Large
- The picture deserves prominent space. Either it meaningfully shows what the story is about (the event, person, place or situation), or it is related to the story and adds real visual value: striking, atmospheric, emotionally engaging or dramatic. These are examples of visual value, not rules.
- Small
- Relevant or useful, but not strong enough to dominate the layout: routine file photos, ordinary portraits, generic but somewhat relevant places or objects.
- None
- Little visual or editorial value: generic stock imagery, buy/sell or finance graphics, corporate logos, podcast or show title cards, repeated promotional graphics, unrelated marketing.
- Unsure
- You cannot decide. This is a perfectly good answer and is counted as one.
If a picture does not load, choose Unsure; the page records that it failed to load.
4. How your answers are used
Your answers are compared with the owner's answers on the same pictures, which were given independently and which you cannot see. Nobody overrules anybody: we measure how closely two people agree, and where they differ. Afterwards, for the pictures where you two disagree and a handful where you agree, you may be asked one short follow-up: why did you choose that?
5. Behind the study
Want the full reasoning, the maths, the alternatives we considered and where we may be wrong? Read The study in depth. It contains no one's labels. If you spot a flaw, please tell us after you finish Pass 1.
What we are trying to achieve
A decision on whether a small image-and-text AI model, running on an ordinary computer, can pick the pictures worth a large frame well enough to use in the app. The target is practical: of ten pictures shown large, most should be good choices, rare ones may be poor, and a logo or stock picture should almost never appear large.
Why two people
If two thoughtful people disagree a lot, then no program can be judged by a single person's taste. Measuring their agreement tells us how much weight any score deserves. Your labels are kept as raw evidence; they are never merged into one answer before the comparison.
How the test is run
- Development set: about 190 pictures used to try ideas. These 172 are drawn from it.
- Try the model: first with no training at all (zero-shot), then with a small trained layer on top of the frozen model.
- Fresh final exam: about 200 new pictures taken at random from the real feed, labelled once, and scored once. The verdict rules (pass, fail, or not enough evidence) are written down before the exam is opened, so the result cannot be tuned afterwards.
Nothing here changes the app readers use today. If the model does not clear the bar, the app keeps small pictures.
6. Privacy
The page stores your link's name, the picture id, your choice, whether the picture failed to load, and the time. No account, no cookies, no tracking, and nothing about you personally. The link is private: please do not share it.