13. How do we run a fair scouting platform trial before a transfer window?

Give suppliers the same players, questions, evidence deadline and incumbent baseline, then evaluate the answers rather than the demonstrations. The following Same-Shortlist Supplier Trial Kit is a proposed design; its duration and sample size are not industry benchmarks.

What should the trial include?

Use five to ten authorised player references spanning the intended roles and markets. Include a well-documented target, a less-covered target, a recent transfer and at least one difficult identity or missing-data case.

Ask each supplier to:

1. Assess the same role-specific strengths and limitations.
2. Provide supporting match evidence and counterexamples.
3. State what is unavailable or uncertain.
4. Produce the required club-facing report or export.
5. Explain which findings are additional to the supplied baseline.

A shortlist-only trial cannot fully test discovery. If finding previously unknown players matters, add a separate task using the same role brief, market boundaries and exclusions. Freeze the incumbent longlist before revealing supplier results.

What is a workable timetable?

Extend the trial if it does not include a meaningful delivery or correction cycle.

How should we score it?

Apply non-negotiables before weighted scoring: essential footage, permitted use, unresolved critical identity errors and required integration capability.

Illustrative weights:

Agree the weights before testing. Record not tested separately; do not silently redistribute its weight. Blind supplier identity where practical, vary task order and record vendor assistance.

For AI-enabled products, source verification and realistic pre-deployment testing also align with NIST’s voluntary generative-AI risk guidance. (nvlpubs.nist.gov)