# 13. How do we run a fair scouting platform trial before a transfer window?


**Give suppliers the same players, questions, evidence deadline and incumbent baseline, then evaluate the answers rather than the demonstrations.** The following Same-Shortlist Supplier Trial Kit is a proposed design; its duration and sample size are not industry benchmarks.

### What should the trial include?

Use five to ten authorised player references spanning the intended roles and markets. Include a well-documented target, a less-covered target, a recent transfer and at least one difficult identity or missing-data case.

Ask each supplier to:

1. Assess the same role-specific strengths and limitations.
2. Provide supporting match evidence and counterexamples.
3. State what is unavailable or uncertain.
4. Produce the required club-facing report or export.
5. Explain which findings are additional to the supplied baseline.

A shortlist-only trial cannot fully test discovery. If finding previously unknown players matters, add a separate task using the same role brief, market boundaries and exclusions. Freeze the incumbent longlist before revealing supplier results.

### What is a workable timetable?

| Proposed stage | Activity |
|---|---|
| Days 1–2 | Confirm scope, permissions, baseline and hard gates |
| Days 3–6 | Club users complete identical tasks; record assistance and time |
| Days 7–8 | Validate claims, inspect failures and test exports |
| Day 9 | Reconcile commercial scope and unresolved rights |
| Day 10 | Produce the decision memo and conditions for approval |

Extend the trial if it does not include a meaningful delivery or correction cycle.

### How should we score it?

**Apply non-negotiables before weighted scoring:** essential footage, permitted use, unresolved critical identity errors and required integration capability.

Illustrative weights:

| Criterion | Weight |
|---|---:|
| Additional supported intelligence | 30% |
| Coverage beyond minimum requirements | 25% |
| Accuracy and reproducibility | 15% |
| User workflow | 15% |
| Integration and portability | 10% |
| Fully loaded cost | 5% |

Agree the weights before testing. Record **not tested** separately; do not silently redistribute its weight. Blind supplier identity where practical, vary task order and record vendor assistance.

For AI-enabled products, source verification and realistic pre-deployment testing also align with NIST’s voluntary generative-AI risk guidance. ([nvlpubs.nist.gov](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf))