# Stage 4: How should a same-shortlist supplier trial work?


**Compare complete purchasing configurations on identical questions.** Test the incumbent stack against “incumbent plus candidate” when considering an addition, and against the proposed replacement configuration when considering cancellation.

A suggested sequence is two preparation days, five to ten working days of testing, and a decision review. Extend it when access, integration or coverage cannot be tested properly; these are planning allowances, not promises of completion.

### Use separate assessment and discovery tasks

A shared shortlist tests assessment. It does not, by itself, establish whether a supplier discovers useful candidates.

| Trial lane | Example assignment | Required output |
|---|---|---|
| Assessment | Assess the same candidates against a defined role requirement. | Supported findings, counter-evidence, clips or data references, and unresolved questions |
| Discovery | Find candidates meeting the same role, competition and evidence constraints. | Candidates not already on the baseline list, with eligibility for further review explained |
| Workflow | Produce a recruitment-meeting brief using the proposed configuration. | Final usable brief and a log of preparation, checking and support time |
| Integration | Deliver an authorised sample into the club’s test environment. | Reproducible import, identity mapping and exception report |
| Regional intelligence | Answer a territory-specific question not resolved by the existing stack. | Dated observations, source basis, uncertainty and conflict disclosures |

Do not force unlike services into identical technical tasks. A regional scout need not provide an API. They should still answer the same football question with evidence the club can inspect.

### What should the scorecard measure?

Use the following measures in the Same-Shortlist Supplier Trial Kit:

- **Additional supported findings:** decision-relevant conclusions unavailable from the baseline task.
- **Better-supported analysis:** improved interpretation of existing evidence, with assumptions visible.
- **Question resolution:** which predeclared uncertainties were answered, challenged or left open.
- **Time to an accepted answer:** include searching, cleaning, checking, support and report assembly.
- **Coverage exceptions:** missing evidence by competition, season and evidence type.
- **Verification failures:** incorrect identities, unsupported factual claims and broken evidence references.
- **User independence:** whether club staff can repeat the task without the demonstrator.

Separate three possible benefits: **new evidence, new analysis and lower effort**. A supplier may provide useful efficiency without new underlying data. Conversely, a new dataset may add no useful answer to the club’s question.

### Trial-control checklist

- [ ] Freeze questions, baseline outputs and essential acceptance criteria before reviewing results.
- [ ] Use the same evidence cutoff and equivalent task scope.
- [ ] Record evaluator experience and training time.
- [ ] Rotate testing order where practical to reduce repeated-task learning effects.
- [ ] Include a holdout task completed by club staff without supplier assistance.
- [ ] Record “not tested” rather than inventing a score.
- [ ] Give suppliers a defined opportunity to explain errors, then retest the correction.
- [ ] Preserve the original result alongside the corrected result.

**Do not let a weighted average conceal a failed essential requirement.** If the purchased workflow requires full-match footage or a particular usage right, its absence cannot be offset by a better interface.

**Exit condition:** the trial establishes a specific, reproducible benefit—or provides a documented reason not to buy.