Run both an assessment test and a discovery test. A shared shortlist is useful for comparing answers about known players; it cannot by itself show whether a supplier finds worthwhile candidates the club has not identified.
The protocol below is our proposed Same-Shortlist Supplier Trial Kit. The sample size and schedule are starting points to adapt, not validated universal thresholds.
What should be fixed before the trial?
- One decision owner and a named signing deadline.
- The recruitment roles and priority markets.
- Essential evidence types and intended uses.
- The incumbent-stack baseline.
- Written authorisation for access, exports and evaluation.
- A manageable sample—for example, ten players across two roles and several priority competition-season combinations.
- Deliberate difficult cases, including incomplete histories or ambiguous identities.
- A common task sheet, response format and time allowance.
For an NWSL purchase, use women’s recruitment tasks throughout the acceptance sample. Do not substitute a men’s demonstration when a requested competition is unavailable.
How do the trial stages compare?
| Stage | Assessment lane | Discovery lane | Control against misleading results |
|---|---|---|---|
| Baseline | Answer the questions using current entitlements | Produce the incumbent candidate list | Freeze the baseline before reviewing challenger outputs |
| Onboarding | Give users comparable orientation | Explain the same role constraints | Record onboarding effort separately |
| Execution | Assess the same players and questions | Request candidates within the same market, role and practical constraints | Separate self-service output from supplier-assisted consultancy |
| Verification | Inspect claims, clips and calculations | Check eligibility for the brief and supporting evidence | Use reviewers who did not produce the output where feasible |
| Repeat | Re-run selected tasks | Test a held-back role or market | Reduce familiarity and hand-picked-example effects |
| Decision | Identify additional answers and workflow changes | Identify supported candidates absent from the baseline | Record limitations, not just successes |
Use fresh players or counterbalanced task order where possible to reduce learning effects. Preserve exports, timestamps and model versions where authorised. Record “not tested” rather than assigning a confident score to an unavailable demonstration.
What counts as incremental decision value?
For this desk’s proposed method, a finding qualifies only when it:
- Is supported by inspectable evidence.
- Was not already available in the frozen baseline answer.
- Resolves a named uncertainty, changes a candidate’s assessment status, identifies a relevant candidate or materially improves an agreed workflow.
Keep different benefits separate:
- New evidence: an additional observation or dataset.
- New interpretation: a defensible inference from existing evidence.
- Independent corroboration: a genuinely separate check.
- Workflow improvement: comparable evidence delivered with less total effort.
Two dashboards reproducing the same upstream observation should not receive two independent-evidence credits. Conversely, a useful contradiction should not be penalised merely because it makes the decision less comfortable.
Which measures belong in the trial workbook?
| Measure | Proposed definition | Guardrail |
|---|---|---|
| Questions resolved | Predefined questions answered to the agreed evidence standard | Do not count prose volume |
| Additional supported findings | Qualifying findings beyond the baseline | Deduplicate and classify evidence lineage |
| Usable coverage | In-scope completed matches satisfying requirements | Separate footage, events and tracking |
| Total answer time | Retrieval, analysis, checking and correction | Report onboarding separately |
| Identity quality | Correct, incorrect and unresolved matches | Do not hide false matches in a high overall join rate |
| Reproducibility | Ability to recreate the output from retained inputs and versions | Record changes and permitted retention limits |
| Adoption | Intended users complete the task without seller intervention | Distinguish usability from an assisted demonstration |
| Fully loaded burden | Cash commitment and staff capacity | Do not imply a return from future signings |