Blighty BoffinsWe cog your gears of innovation

research · reviewing

Independent validation of a supplier's defect-detection model

Somebody else's model, data it has never seen, and a number that will not please everyone. Short, unwelcome, occasionally decisive.

Budget

£18–30k

Duration

12 weeks

Closes

29 Aug 2026

Background

A manufacturer is midway through procuring a defect-detection system and has performance figures from the supplier obtained on the supplier's own dataset. This is normal and nearly meaningless. We have been asked to arrange an independent validation, and we would rather it were conducted by a third party than by us.

The work is to evaluate the model on labelled data from two production sites it has never seen, with the labelling done independently and disagreements between labellers recorded rather than resolved by fiat.

We expect performance to be materially worse than the tender figures, because it almost always is: models learn a site's lighting, fixturing and material lot. Establishing the size of that gap before a purchase is the entire value of the exercise.

Scope

  • Establish an independent labelling protocol and label data from two sites.
  • Evaluate the supplied model without retraining or tuning it.
  • Report performance per site and on the pooled set, with confidence intervals.
  • Assess calibration, not merely accuracy, and report the reliability of confidence scores.
  • Characterise the failure cases in terms a quality engineer can act on.

Constraints

The awkward facts, stated now rather than in week six.

  • The model is supplied as a black box; no retraining or architecture inspection is permitted
  • Labelling must be independent of both the supplier and the manufacturer's own inspectors
  • Base rates are heavily imbalanced, and accuracy alone will not be accepted as a metric
  • Both host sites must be visited; remote data collection is not sufficient

Deliverables

  • Labelling protocol with inter-labeller agreement statistics
  • Per-site and pooled performance report with confidence intervals
  • Calibration assessment of the model's reported confidence
  • A characterised set of failure cases with imagery
  • A plain recommendation on whether the tender figures are supported

Milestones

  1. Week 2Labelling protocol agreed; independence declarations exchanged
  2. Week 5Site visits complete and data collected
  3. Week 9Labelling complete with agreement statistics reported
  4. Week 12Evaluation report and recommendation delivered

Who should apply

  • University group or consultancy with machine learning evaluation experience
  • Genuine independence from the supplier under evaluation
  • Experience of imbalanced industrial datasets and appropriate metrics
  • Willingness to deliver a conclusion that ends a procurement

How applications will be judged

In roughly this order.

  • Demonstrable independence, declared explicitly
  • Proposed metrics and their appropriateness to a heavily imbalanced problem
  • Rigour of the labelling protocol, including treatment of disagreement
  • Prior experience of evaluations that produced an unwelcome answer

Terms

  • The report is delivered to the manufacturer and to the supplier simultaneously
  • The evaluating party may publish the methodology, anonymised, after twelve months
  • Labelled data remains the property of the host sites
  • No party may edit the conclusions; factual corrections only

Not accepting applications

Applications are under review and no further proposals are being accepted.

See what is open