How should accuracy, AI reports and contextual models be tested?

Separate whether a record is correct from whether an interpretation is appropriate. A precisely calculated score can still fail the club’s intended comparison test.

What belongs in the Player Data Joinability Check?

Build an authorised, manually adjudicated identity sample that includes transfers, loans, similar names, diacritics, name changes and players represented in more than one team context.

Measure:

• Match precision: correct accepted links ÷ all accepted links.
• Link recall: correctly recovered links ÷ all true links established in the adjudicated sample.
• Unresolved rate: records requiring review ÷ records submitted.
• Repair burden: staff time to investigate and correct exceptions.

Require stable supplier IDs and an auditable crosswalk. Reject silent name-only merging. Keep historical team membership separate from current affiliation and retain the evidence used to resolve an ambiguous match.

Wyscout’s official API documentation provides versioning, field definitions, coordinate conventions and rate-limit information. SkillCorner’s official open-data documentation distinguishes detected player positions from extrapolated positions through an is_detected flag. These are concrete reasons to test semantics and quality flags, not merely whether a file loads. (apidocs.wyscout.com)

For an API trial, require a successful initial import, incremental refresh, correction replay, duplicate check, timeout/rate-limit handling and authorised export. Archive the schema and version used. Compare suppliers’ metrics only after documenting their definitions and transformations.

What belongs in the AI Scouting Report Evidence Check?

NIST’s Generative AI Profile identifies confidently false outputs and fabricated supporting logic or citations as confabulation risks. Treat a fluent scouting report as a set of claims to inspect, not as evidence of its own correctness. (nvlpubs.nist.gov)

For the proposed test:

1. Select checkable factual claims before reviewing their sources.
2. Classify each as supported, contradicted, unsupported or not verifiable.
3. Separate football judgement from factual claims and explicit predictions.
4. Verify that citations support the actual statement, player and timeframe.
5. Include cases where the correct response is to acknowledge insufficient evidence.
6. Record human verification time and whether the required questions were answered.

A report with no factual claims should not win by achieving a nominal zero-error rate. Judge task completion alongside accuracy. Escalate fabricated biographies, invented sources and wrong-player attributions through the critical-error gate.

What evidence should support role-fit or league-translation claims?

Request a model or methodology note covering inputs, population, target outcome, training/evaluation periods, missing-data handling and limitations. For predictive claims, ask for relevant out-of-time testing and evidence that information unavailable at the decision date was excluded.

Recommended challenge questions include:

• Does the comparison account for minutes, role and team context?
• Which competitions and player populations were used for validation?
• Is performance reported separately for the women’s competitions in scope?
• What happens when a player has little data or changes role?
• How sensitive is the shortlist to reasonable changes in weighting?
• When does the system decline to make a confident comparison?

Do not require a regional scout to present a machine-learning validation study. Require instead an explicit observation rubric, relevant match evidence, competing interpretations and a repeatable explanation of the recommendation.

Do not treat a supplier’s nationality field as proof of MLS roster status. MLS publishes specific domestic/international and acquisition rules; any eligibility-dependent conclusion should be checked by the club’s responsible football-operations staff against current rules and authorised records. (mlssoccer.com)