# Stage 5: How do you run a fair same-shortlist supplier trial?


**Evaluate the incumbent stack and each proposed addition on the same questions, evidence cut-off and acceptance criteria.** A small trial can reveal operational failures and useful additions; it cannot establish universal platform accuracy or predict future signings.

### Step 5.1: Build a deliberately varied sample

For a bounded starting design, use ten authorised player references and three questions per player. Include familiar candidates, uncertain cases, a recent transfer or identity challenge, thin histories and examples from each priority competition. Replace this design if it does not represent the purchasing decision.

For example:

| Task | Required output | What the evaluator checks |
|---|---|---|
| Role assessment | A supported judgement about a defined behaviour | Evidence quality, context and uncertainty |
| Contradiction search | The strongest evidence against the initial view | Whether the service challenges rather than merely confirms |
| Recruitment follow-up | A justified next action: watch, investigate, retain or stop | Whether the answer changes work the club should undertake |

A same-shortlist test assesses evaluation of known candidates. **It does not adequately test discovery.** If discovery is the purchasing objective, add a separate blind search exercise using a fixed role brief, eligible population and cut-off date. Have the club adjudicate candidate relevance without revealing its complete internal longlist.

### Step 5.2: Keep the baseline uncontaminated

Capture incumbent-only results before evaluators see challenger findings. Where practical, use separate assessors or rotate task order on equivalent cases to reduce learning effects. Log evaluator familiarity with each tool and provide comparable onboarding.

Test the configuration the club would buy:

- **Addition:** incumbent stack plus challenger, including joining and verification work.
- **Replacement:** proposed replacement configuration without the cancelled product.
- **Consolidation:** the exact retained bundle and rights.
- **Specialist service:** the delivered assessment plus the club’s review effort.

A demonstration of a challenger alone is not a sufficient replacement test if the planned workflow depends on other tools.

### Step 5.3: Separate self-service and assisted performance

Run a self-service round and, if relevant, a supplier-assisted round. Record assistance, support response and any manual work behind the scenes. Ask whether the production quote includes the same assistance.

Measure:

- Active evaluator time.
- Elapsed time to a usable answer.
- Completion and failure counts.
- Verification and correction time.
- Additional supported findings.
- Unresolved questions and new dependencies.

Report medians alongside ranges and failures. Do not describe a fast unsupported answer as a time saving.

### Step 5.4: Use a claim-level evidence ledger

Recommended columns:

`Task | Player ID | Claim | Source | Match/timestamp or record | Data/model version | Baseline already contained it? | Verified? | Contradictory evidence | Decision relevance | Reviewer | Review time`

Classify each finding as new source evidence, new interpretation, workflow improvement, repetition or unsupported. Count closed recruitment questions as well as individual claims; splitting one observation into five sentences should not increase the apparent value.

Ask the scouting lead: **“What would we do differently because of this finding?”** A legitimate answer can be that a previously uncertain judgement is now sufficiently supported to stop further work. Novelty alone is not the goal.

### How should you test role fit and league translation?

Require the supplier to explain its target, input data, role definitions, reference population, missing-data treatment and validation design. Ask for examples of failure, not just successful transfers.

For historical tests, require the information available at the historical decision date. Later revised data, future appearances or post-transfer outcomes must not leak into the inputs. If the supplier cannot reconstruct an as-of-date evaluation, label the exercise retrospective illustration—not a clean predictive test.

Predefine the outcome being tested. A ranking’s association with future minutes is not automatically evidence that it predicts tactical suitability, availability or transfer affordability. Ask for performance by relevant league, role and women’s population rather than accepting a pooled result.

### How should you compare SkillCorner with event data?

Test the additional question rather than assuming one replaces the other. SkillCorner’s open-data documentation distinguishes detected player positions from extrapolated positions through an `is_detected` field. For a movement-related task, preserve that distinction and ask how the proposed analysis handles inferred observations. ([skillcorner.github.io](https://skillcorner.github.io/opendata/))

For example, require an explanation of what a movement metric contributes beyond the incumbent assessment, then verify the relevant behaviour on authorised footage where feasible. Record cases in which the available camera view cannot resolve the question.

### How should you test AI scouting reports?

Treat fluency as presentation, not evidence. NIST’s Generative AI Profile identifies confidently incorrect content and data-provenance risks as matters requiring risk management. The following is a proposed scouting-specific implementation of that concern, not a NIST certification scheme. ([nvlpubs.nist.gov](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf?utm_source=openai))

1. Break the report into checkable factual claims and clearly labelled interpretations.
2. Open each cited source; verify the player, competition, season and relevant passage or event.
3. Include controlled cases where the evidence is missing or contradictory.
4. Check whether the system admits uncertainty rather than inventing an answer.
5. Repeat selected tasks and record model/version details where available.
6. Measure supported-claim rate, material-error count and human verification time.
7. Require human sign-off for decision-critical claims.

A citation to the correct website but the wrong player or season is not support. Do not send licensed data or confidential shortlist information to an external AI service until the intended processing is approved.

### How should you evaluate a regional scout?

Commission a bounded assignment before a broad retainer. Specify territory, player type, matches, observation method, report format, evidence references, delivery dates, conflicts and expense approval.

360 Scouting states that it uses Wyscout for video scouting and can report into a club’s database; its live coverage depends on scout location. This illustrates why an external report may add assessment capacity without adding a new underlying footage source. ([360scouting.com](https://360scouting.com/for-clubs/?utm_source=openai))

Require the report to separate observation from inference and third-party statements. Ask whether the assessor or firm represents, receives introductions fees from, or has another commercial relationship with relevant parties. Do not commission covert collection of medical information or unsupported personal allegations.