# How should a same-shortlist supplier trial work?


**Run both an assessment test and a discovery test.** A shared shortlist is useful for comparing answers about known players; it cannot by itself show whether a supplier finds worthwhile candidates the club has not identified.

The protocol below is our proposed **Same-Shortlist Supplier Trial Kit**. The sample size and schedule are starting points to adapt, not validated universal thresholds.

### What should be fixed before the trial?

- One decision owner and a named signing deadline.
- The recruitment roles and priority markets.
- Essential evidence types and intended uses.
- The incumbent-stack baseline.
- Written authorisation for access, exports and evaluation.
- A manageable sample—for example, ten players across two roles and several priority competition-season combinations.
- Deliberate difficult cases, including incomplete histories or ambiguous identities.
- A common task sheet, response format and time allowance.

For an NWSL purchase, use women’s recruitment tasks throughout the acceptance sample. Do not substitute a men’s demonstration when a requested competition is unavailable.

### How do the trial stages compare?

| Stage | Assessment lane | Discovery lane | Control against misleading results |
|---|---|---|---|
| Baseline | Answer the questions using current entitlements | Produce the incumbent candidate list | Freeze the baseline before reviewing challenger outputs |
| Onboarding | Give users comparable orientation | Explain the same role constraints | Record onboarding effort separately |
| Execution | Assess the same players and questions | Request candidates within the same market, role and practical constraints | Separate self-service output from supplier-assisted consultancy |
| Verification | Inspect claims, clips and calculations | Check eligibility for the brief and supporting evidence | Use reviewers who did not produce the output where feasible |
| Repeat | Re-run selected tasks | Test a held-back role or market | Reduce familiarity and hand-picked-example effects |
| Decision | Identify additional answers and workflow changes | Identify supported candidates absent from the baseline | Record limitations, not just successes |

Use fresh players or counterbalanced task order where possible to reduce learning effects. Preserve exports, timestamps and model versions where authorised. Record “not tested” rather than assigning a confident score to an unavailable demonstration.

### What counts as incremental decision value?

For this desk’s proposed method, a finding qualifies only when it:

1. Is supported by inspectable evidence.
2. Was not already available in the frozen baseline answer.
3. Resolves a named uncertainty, changes a candidate’s assessment status, identifies a relevant candidate or materially improves an agreed workflow.

Keep different benefits separate:

- **New evidence:** an additional observation or dataset.
- **New interpretation:** a defensible inference from existing evidence.
- **Independent corroboration:** a genuinely separate check.
- **Workflow improvement:** comparable evidence delivered with less total effort.

Two dashboards reproducing the same upstream observation should not receive two independent-evidence credits. Conversely, a useful contradiction should not be penalised merely because it makes the decision less comfortable.

### Which measures belong in the trial workbook?

| Measure | Proposed definition | Guardrail |
|---|---|---|
| Questions resolved | Predefined questions answered to the agreed evidence standard | Do not count prose volume |
| Additional supported findings | Qualifying findings beyond the baseline | Deduplicate and classify evidence lineage |
| Usable coverage | In-scope completed matches satisfying requirements | Separate footage, events and tracking |
| Total answer time | Retrieval, analysis, checking and correction | Report onboarding separately |
| Identity quality | Correct, incorrect and unresolved matches | Do not hide false matches in a high overall join rate |
| Reproducibility | Ability to recreate the output from retained inputs and versions | Record changes and permitted retention limits |
| Adoption | Intended users complete the task without seller intervention | Distinguish usability from an assisted demonstration |
| Fully loaded burden | Cash commitment and staff capacity | Do not imply a return from future signings |