The study in depth

For raters who want the full picture and the reasoning, and who can tell us where it is weak. Every example here is invented; none comes from the 172 pictures, and nothing here reveals anyone's labels. Back to the short version.

1. The goal

A news feed app shows every story with a small thumbnail. Some pictures deserve a big, full-width frame. The app currently decides with size, shape and repeat-count rules. We tested those on a labelled sample: of 96 pictures they gave a large frame, 70 (73%) were ones the owner would not show large. So roughly 3 in 10 large frames were right.

The owner's target: of ten pictures shown large, six or seven should be right, a couple can be less ideal, and a logo or stock picture should be rare. The research question is whether a small pretrained image-and-text model, running on an ordinary PC and later on a phone, can reach that. Nothing the app's readers see depends on the answer yet.

2. Why we widened the definition

Our first rule was strict: large only if the picture conveys the headline's story. We changed it to: does this picture deserve prominent space for this story? A picture can earn that by showing the story or by being related and visually valuable.

Why

• Much news has no picture of "the event". Budgets, rates, court rulings and forecasts have nothing to photograph. A strict rule would label almost everything Small, and the model would learn "never large", which is useless.
• A feed is also a reading experience. Editors routinely run a strong related picture at size. The app's own design notes describe the goal as a photo's visual merit, which the strict wording undersold.
• It still keeps the bad cases out. Logos, stock and promos stay out under both definitions.

Invented examples

3. The three numbers that decide it

Version 1 is a two-way call: FEATURE (Large) against not FEATURE (Small or None). Let "shown large" be what the model picks.

Toy example: the model shows 30 pictures large; 20 are labelled Large, 2 None. Of 40 labelled Large in total, 20 are caught. Precision = 20/30 = 67%, None-among-large = 2/30 = 7%, coverage = 20/40 = 50%. Precision and coverage pull against each other: showing fewer pictures large raises precision and lowers coverage. Needing both is the point.

4. How a model would decide

We use TinyCLIP (MIT licence; about a 40M-parameter image part and a 19M text part). Like CLIP, it maps an image and a piece of text into the same vector space, so closeness can be measured by cosine similarity: cos(a, b) = a·b / (|a||b|).

Step A: zero-shot, no training

Compare the image with text such as the headline and a few fixed prompts ("a striking news photograph", "a company logo", "a generic stock photo"). Score = similarity to the positive prompts minus similarity to the negative ones. One threshold on that score gives Large or not.

Step B: frozen model plus a small classifier

Keep TinyCLIP unchanged. Take a handful of similarities as inputs x and fit logistic regression: P(Large) = 1 / (1 + e^−(w·x + b)). With only about 190 labels, w is penalised (λ|w|²) so it cannot memorise. The threshold is chosen with cross-validation: split the labels into k parts, train on k−1, test on the held-out part, rotate, and pool the results. Pixel width, height and repeat count may be tried as small extra inputs.

If this works, we try a smaller TinyCLIP (cheaper) and SigLIP 2 Base (as an accuracy ceiling, not for shipping).

What these models are

CLIP-style models have two networks: an image encoder (a vision transformer) and a text encoder. Each turns its input into a vector of numbers. They are trained on huge numbers of image–caption pairs with one goal: a picture and its true caption should end up close together, and mismatched pairs far apart. After that, "how well does this text describe this picture?" is just a similarity score. They do not write anything.

How they differ from an LLM

• Output. An LLM generates text one token at a time, each chosen by predicting what comes next. A CLIP-style model outputs only vectors and similarity scores: one fast pass, no generation.
• Size and cost. LLMs run to billions of parameters and usually need a server. These models are tens to a few hundred million, so they can run on the phone, offline, in milliseconds. That is the main reason we use them: the app is on-device only.
• What they can do. An LLM can reason ("this photo shows a different event") and explain itself. A CLIP model has no reasoning and no explanation; it only knows which text looks closest. Judging "deserves prominence for this story" therefore relies on a few similarity scores and the small classifier we train on your labels.
• Consequence. A big LLM might judge better, but it fails our on-device rule. We are testing whether the small, cheap kind is good enough.

5. Why two raters, and what we compute

If two sensible people disagree widely, no model score means much. Agreement is reported as raw percentage and as Cohen's kappa, which discounts agreement expected by luck: κ = (p_o − p_e) / (1 − p_e).

Toy example, 100 pictures: both say Large on 30, both say not-Large on 50, they split on 20 (10 each way). Observed agreement p_o = 0.80. Each person says Large 40% of the time, so chance agreement p_e = 0.4·0.4 + 0.6·0.6 = 0.52. κ = (0.80 − 0.52) / 0.48 ≈ 0.58. Kappa is descriptive here, not a pass mark.

Your Pass 1 answers (Large, Small, None, Unsure) are kept raw. Later we ask "why" only for disagreements plus 10–15 random agreements; asking earlier could prime the judgement we are measuring. An optional experiment gives pictures both call Large a target of 1, both not-Large 0, and a split 0.5. It is only a development ablation, not a claim about true probability.

6. The final exam and the verdict

The 192 existing pictures are development material. After the rubric is settled, about 200 fresh pictures are drawn at random from the real feed (only filtered for size, duplicates and one-publisher dominance), labelled by the owner, and locked away. The model, prompts, threshold and verdict rule are written down and committed first; then the exam is scored once. Model uncertainty is never used to pick exam pictures. A deliberately hard "stress" batch is allowed for development only.

Because the exam is small, we use 95% Wilson intervals: for k successes of n, centre (p̂ + z²/2n) / (1 + z²/n) with half-width z·√(p̂(1−p̂)/n + z²/4n²) / (1 + z²/n), z = 1.96. Example: 21 of 35 shown-large correct is 60%, but the interval is about 44% to 74%.

7. Other approaches

Tried or ruled out in the project records

• Size, shape and repeat-count rules. Tried; failed both acceptance lines (73% wrongly large; 2 of 26 good pictures wrongly shrunk). Route closed.
• MobileCLIP. Its model licence is research-only; we accept MIT, Apache 2.0 and BSD weights only.
• A cloud model judging every picture. The app is on-device only, for privacy and cost. Gemini is used only on the PC, only to analyse mistakes among 60 development pictures, never to label or train, and never on final-exam pictures.
• Distillation, quantisation, Android runtime. Deferred until the offline result justifies the engineering.

Possible, not pursued (our reasoning, not a project decision)

• Fine-tuning the whole model. About 190 labels is too few; it would likely overfit. Hence frozen features plus a tiny classifier.
• Aesthetic, saliency or face-detection scores. The app's rules say a picture must not be demoted merely for lacking people, having a plain background or being an illustration, so a generic beauty score fits poorly. The question is prominence for this story, not beauty.
• Many more raters. Better evidence, but slow and costly for a single-owner project.

8. Where we may be wrong: please tell us

• The 172 pictures come from a sample, not the feed's natural mix, and kappa depends on the mix.
• The owner designed the rubric and is also a rater, so the rubric may lean toward one taste.
• A CLIP-style model reads a headline only up to about 77 tokens, and similarity to a headline may reward literal matches, the thing we widened away from.
• Many model variants will be tried on about 190 pictures, so a good development result may be luck.
• Around 35 predicted-large exam pictures gives wide intervals (chapter 6). INCONCLUSIVE is a likely outcome.
• Images load from publishers' own links, so some may disappear mid-study.

If you see a flaw, a better baseline or a cheaper test, please say so, after you finish your Pass 1 labels so it does not influence them.

9. What happens next

1. You do Pass 1 on 172 pictures; the owner re-reviews their own labels in parallel. Neither sees the other's answers.
2. The owner's rubric is settled, then the fresh ~200-picture exam is drawn and locked.
3. Agreement is computed; you and the owner give short "why" answers on disagreements.
4. Zero-shot, then logistic-regression runs on the development set; then the plan is frozen.
5. The exam is scored once. Outcome: MODEL_ROUTE_SUPPORTED, MODEL_ROUTE_FAILED or INCONCLUSIVE_NEEDS_MORE_LABELS. Only a supported result leads to on-device engineering.