A rating of strategic judgment or leadership capability compresses several decisions into one score․ The evaluator chooses which situations matter, interprets the person’s behavior and applies a standard․ Understanding those decisions helps a selection team judge what the score can support․
What an evaluator brings to a rating
The idiosyncratic rater effect is the contribution associated with an individual evaluator across their ratings․ Their scores may reflect personal standards, general impressions and their particular experience of the person being assessed․
Research on 360-degree performance ratings by Mount and colleagues found that individual raters accounted for important differences in ratings beyond the performance dimensions being assessed․ Hoffman and colleagues’ later study of multisource managerial ratings distinguished general performance, specific performance dimensions, individual rater effects and effects associated with the rater’s organizational position․
These findings concern how ratings vary across people and perspectives․ A share of rating variance is not a percentage of judgments that are wrong, nor a direct measure of an appointment’s risk․
Different perspectives can contain useful evidence
An evaluator’s distinctive view may include bias․ It may also contain information that others lack․ A leader can behave differently with peers and direct reports, or under different pressures․ Hoffman and colleagues explicitly discuss this possibility when interpreting rater-specific variance․
For a selection team, disagreement is therefore worth investigating․ Ask whether observers saw different behavior, used different standards or drew different conclusions from the same event․ An average score can conceal those distinctions․
Consider an illustrative reference discussion․ One colleague describes an executive as decisive because they resolved a stalled project․ Another describes them as dismissive because they closed discussion before a technical concern was examined․ Both accounts may relate to the same decision․ The useful work is to establish what happened, how the concern was handled and why that behavior matters for the proposed role․
Make the judgment traceable to behavior
Broad labels such as executive presence leave considerable room for interpretation․ A more specific inquiry connects the role’s demands to something an observer could describe․
If the appointment requires coordination across functions, ask about a situation in which priorities conflicted․ What did the executive do to understand the disagreement? How were responsibilities and trade-offs handled? What followed, and what was the executive’s own contribution?
When reviewing a rating or reference, examine four things․
- What action or interaction supports the judgment?
- What demands, constraints and opportunities shaped the behavior?
- What would effective leadership have looked like in that situation?
- How does the evidence connect to the responsibilities of this appointment?
These questions give the team a basis for discussing an interpretation and identifying missing evidence․ Clearer behavioral descriptions still require careful judgment and appropriate measurement validation․
Match the evidence to the decision
Feedback for development can reveal how people experience a leader and where a useful conversation should begin․ A consequential selection decision also needs evidence relevant to the future role, with enough precision for the intended use․
The suitability of a measure depends on its purpose, how it was developed and the evidence supporting its interpretation․ Behaviorally anchored scales, interviews, references and work samples each require scrutiny․ A structured format alone does not establish validity․
Sehm’s proposed capability-centered approach starts by defining what effective leadership would require in the role, then aligning the behavioral evidence and assessment methods with those priorities․ Organizations can develop that understanding with specialist support․
When a score becomes influential in an appointment discussion, ask the evaluator to explain the observation behind it․ Then identify which part of the interpretation the team can support and which part still needs evidence․
