What we measured, including what failed.

Score a model on photos it has never seen.

When we audited our own earlier models, every one had been scored on the same photos it was trained on. The scores looked excellent and told us nothing: a model can memorise its training photos.

Sight now holds back part of every dataset before training starts, and scores the model only on that part. The first model scored this way came in at 0.914. That is lower than the old numbers, and it is the first one we would stand behind.

A language model’s confidence is not a probability.

We asked a language model to read status lights from device photos and to say how sure it was. It reported complete certainty on 155 of 166 readings. One of those was a crop 40 pixels wide, far too small to see a light in, which it called green.

So Sight never uses a model’s own confidence to decide a verdict. Each check has its own thresholds, and “unclear” is always an allowed answer.

Below a certain size, do not read at all.

On devices at least 150 pixels wide, every one of 92 readings matched the human label. Below that, readings became guesses that sounded sure.

Each check therefore carries a minimum size. A crop smaller than that is never read; the crew is asked for a closer photo instead. A missing answer costs a retake. A confident wrong answer costs a failed job.

The phone in the crew’s hand reads better than the cloud.

We ran five text readers over 72 real field photos (60 install photos and 12 meter screens) and checked every answer by eye. A large hosted language model read 41 of the 44 legible serial numbers, at 2.7 seconds each. But it made 5 errors: it misread 2 clear labels, and on 3 photos where the label could not be read at all it returned a confident serial anyway.

Apple’s on-device text recognition, followed by a simple pattern for what a serial looks like, read 38, got 1 wrong, took 0.2 seconds, needed no signal, and said nothing on every unreadable label. RapidOCR’s phone models matched it at 0.6 seconds. Tesseract in a web view read 12; Florence-2 read 1.

So we are moving serial reading onto the device, and any serial that only the language model saw is one for a person to confirm. One thing did not work: on LCD meter screens the on-device readers scored 3 to 5 of 12, against 9 for the hosted model. Those need a small digit reader of their own.

Small models that give a real probability.

A check needs a number it can trust: pass above this, fail below that. Large language models do not give one. A new kind of small “decision model” does: you send it text and a question, and it answers with calibrated probabilities in tens of milliseconds.

We tested two ways of pulling a serial out of recognised text. GLiNER, a small open entity model, picked the wrong text 28 times; a plain pattern did better. Laya, an open decision model, scores near chance until it is fine-tuned on your own examples. So the answer is not one clever model. It is a tiny reader, a rule, and a decision model only where the question is truly fuzzy.

One engine for the server, the browser and the phone.

Technicians work in basements and risers with no signal. If a phone is going to say “retake that one” on the spot, the rules have to run on the phone, and they have to give exactly the answer the server would.

Sight’s check engine was written with no database, no network and no clock, so it can move. The next step is a Rust version compiled to WebAssembly and to native code, tested against the server with one shared suite of cases so the two can never disagree.

Use a language model to read, never to count.

Language models are unreliable at counting things and at saying where they are in a photo. Detection models are good at exactly that, and cannot read.

Sight splits the work. A detection model trained on the customer’s photos finds each part and counts it. Only then is a language model shown a tight crop of one part and asked one question with a fixed set of answers.

When the model and the labels disagree, check the labels first.

We tested a simple rule on one customer’s labels: every status light should sit inside a device. 653 of 656 did. We opened the disagreements one by one, and four of the five photos held labelling mistakes, not model mistakes.

A model can be no better than its labels. We now run rules like this over a dataset before training, so the mistakes are fixed before they are learned.

A check is a small program, not a prompt.

Customers do not buy a model. They buy an answer to “was this job done right?”, and that answer is a rule: two green lights inside this device, a serial that matches the work order, a seal on every opening.

Sight builds each check from a handful of fixed blocks, keeps every version, and freezes a check once it is in use. Because a check is a program and not a paragraph of instructions, it gives the same answer tomorrow, it can show its working, and it is written so the same rules can later run on a phone with no signal.