Compare complete purchasing configurations on identical questions. Test the incumbent stack against “incumbent plus candidate” when considering an addition, and against the proposed replacement configuration when considering cancellation.
A suggested sequence is two preparation days, five to ten working days of testing, and a decision review. Extend it when access, integration or coverage cannot be tested properly; these are planning allowances, not promises of completion.
Use separate assessment and discovery tasks
A shared shortlist tests assessment. It does not, by itself, establish whether a supplier discovers useful candidates.
| Trial lane | Example assignment | Required output |
|---|---|---|
| Assessment | Assess the same candidates against a defined role requirement. | Supported findings, counter-evidence, clips or data references, and unresolved questions |
| Discovery | Find candidates meeting the same role, competition and evidence constraints. | Candidates not already on the baseline list, with eligibility for further review explained |
| Workflow | Produce a recruitment-meeting brief using the proposed configuration. | Final usable brief and a log of preparation, checking and support time |
| Integration | Deliver an authorised sample into the club’s test environment. | Reproducible import, identity mapping and exception report |
| Regional intelligence | Answer a territory-specific question not resolved by the existing stack. | Dated observations, source basis, uncertainty and conflict disclosures |
Do not force unlike services into identical technical tasks. A regional scout need not provide an API. They should still answer the same football question with evidence the club can inspect.
What should the scorecard measure?
Use the following measures in the Same-Shortlist Supplier Trial Kit:
- Additional supported findings: decision-relevant conclusions unavailable from the baseline task.
- Better-supported analysis: improved interpretation of existing evidence, with assumptions visible.
- Question resolution: which predeclared uncertainties were answered, challenged or left open.
- Time to an accepted answer: include searching, cleaning, checking, support and report assembly.
- Coverage exceptions: missing evidence by competition, season and evidence type.
- Verification failures: incorrect identities, unsupported factual claims and broken evidence references.
- User independence: whether club staff can repeat the task without the demonstrator.
Separate three possible benefits: new evidence, new analysis and lower effort. A supplier may provide useful efficiency without new underlying data. Conversely, a new dataset may add no useful answer to the club’s question.
Trial-control checklist
- Freeze questions, baseline outputs and essential acceptance criteria before reviewing results.
- Use the same evidence cutoff and equivalent task scope.
- Record evaluator experience and training time.
- Rotate testing order where practical to reduce repeated-task learning effects.
- Include a holdout task completed by club staff without supplier assistance.
- Record “not tested” rather than inventing a score.
- Give suppliers a defined opportunity to explain errors, then retest the correction.
- Preserve the original result alongside the corrected result.
Do not let a weighted average conceal a failed essential requirement. If the purchased workflow requires full-match footage or a particular usage right, its absence cannot be offset by a better interface.
Exit condition: the trial establishes a specific, reproducible benefit—or provides a documented reason not to buy.