Success criteria for a signing, set before the outcome is known

The problem with retrospective standards

When a standard is chosen after the outcome is visible, it can be selected to support whatever conclusion the assessor already prefers. A competitor who produced little can be judged against contribution, and one who produced a great deal can be judged against consistency. Neither assessment is dishonest in intent, and both are unfalsifiable, because there was never a stated threshold that could have been missed.

This is why transfer verdicts feel decided in advance regardless of what happens, and why the same evidence supports opposite conclusions. Fixing the standard before the outcome removes the flexibility and makes the assessment mean something.

Writing the brief down

A recruitment decision is made to solve a problem, and that problem can be described in a sentence before anybody arrives. The brief might be to cover a specific weakness, to lower the average age of a group, or to provide depth for a congested period. Each of those briefs implies a different definition of success, and success against one is not evidence of success against another.

Publishing the brief is not usually possible from outside, and inferring it carefully is a reasonable substitute and rarely attempted. Even an inferred brief, stated openly, disciplines an assessment far more than an unstated assumption about what should have happened.

Depth signings and the wrong yardstick

A competitor recruited to provide cover is frequently judged against the standard of the competitor they were brought in to support. That comparison is guaranteed to be unfavourable, since the whole point of the arrangement was that one is better than the other. The relevant question is whether the drop-off during absence was smaller than it would otherwise have been, which is far harder to observe.

Because the correct comparison is difficult and the incorrect one is easy, the incorrect one dominates almost all public assessment. Naming the intended role at the outset is the cheapest available protection against that particular error.

Thresholds and honest uncertainty

A useful brief includes a rough sense of what an acceptable outcome looks like, expressed in terms of role and availability rather than exact figures. Setting it in words rather than numbers avoids false precision while still creating something that can be checked afterwards. It also forces the assessor to confront how wide the plausible range is, which is usually wider than confident predictions suggest.

A prediction with an acknowledged range is more useful than a point estimate, because it can be wrong in an informative way. Recording the range at the time is the only way to discover later whether the assessment process was any good.

Assessing the process rather than the case

Individual outcomes are noisy enough that a single result says almost nothing about the quality of the reasoning that produced it. Across many decisions, however, a pattern emerges, and that pattern is the only reliable evidence about whether an approach works. Keeping a record of pre-stated briefs and later outcomes converts a series of anecdotes into something that can actually be evaluated.

Clubs that do this internally have an advantage over commentary that judges each case fresh with a newly invented standard. The habit is transferable to any ranking exercise, which is why it appears wherever decisions have to be assessed under uncertainty.