Multi-Party AI Dialogue ── Corroboration for Others' Level, Divergence for Calibration

Two people watch the same person; one says "she's clearly senior level," the other "still mid-level at best." This happens all the time. They saw different scenes and stand in different places. Leave the measurement to one rater and that rater's blind spot becomes the result. This part is about binding several people's views into one number. The key is to corroborate, not to count heads; to take a weighted middle, not a plain average.

Why one rater cannot measure

The starting point is simple: "one person's view is always biased somewhere." Leave a health checkup to a single doctor and whatever that doctor missed stays missed. Work ability is the same. Knowledge and risk-detection are seen best by colleagues who reviewed alongside you; how you communicate and build trust is seen best by a manager or the field staff you gave feedback to. So no two witnesses carry equal value. For a given scene, a rater who never actually saw the behavior brings thin substance, however confident they sound.

That is why this design picks raters not by rank but by "which scene did they actually see." For each ability, it assigns at least two people who watched that scene closely. Variety of standing is not itself the goal. "Did they see the behavior" comes first, because no number of votes from people who did not see it changes the fact that they did not see it.

"Did they actually see it," and the weight of a vote

So for each rater we produce a number for how well they could see that ability. Call it observability o — in plain terms, "how clear the view was." The idea: were they positioned to see it, and can they actually recount the evidence (concrete events)? Multiply those two. So even a senior manager who cannot recount a single concrete event for that ability gets a low view score. Testimony, not rank, sets the weight.

The vote's weight is this "view" times the "certainty of the read." Certainty (confidence C), defined in earlier parts, comes from the amount of evidence, how well it fits together, and the clarity of view — how trustworthy that read is. As a formula, w = o × C: weight = view × certainty. Only when a person both saw it clearly (o) and gave coherent, sufficient evidence (C) does their vote become heavy. Like a photograph: only a shot that is in focus (o) and sharp across several frames (C) works as strong evidence.

How to bind colleagues' views — corroboration, not average or vote

This is the most important point of the part. When combining several colleagues' ratings, we use neither an average nor a plain majority vote. We use corroboration — the way a reporter won't run a story on one source alone, but only once several people independently confirm it.

In words: for each level (say "senior or above"), compute "the share of weight from people placing the person at that level or higher." Take the highest level whose share clears a preset corroboration line (θ). As a formula, for level ℓ compute the weighted share φ, and L_other (the outside view of the level) = the highest ℓ whose φ clears θ (W is the sum of colleague weights). You can skip the formula. The point: only a level backed by a majority of the weight gets through.

The corroboration line starts at 0.5. That equals "the weighted middle value (a weighted median)." The meaning is plain: one person rating high will not raise the level unless a majority of the weight backs that level. It rises only once corroborated. Why do this — an average lets one extreme high rating drag the whole upward; a vote treats a non-witness's ballot the same as a witness's. Corroboration prevents both. Raise the line above 0.5 and you demand stronger backing for higher levels — a more cautious stance.

The gap from self-rating — calibration, not capability

The subject also takes the same AI interview and produces their own level (L_self). The difference between this and the combined colleague level (L_other) is the gap (Δ). The formula is simple: Δ = self-rating − others' rating. Positive means they see themselves high (over-claiming), negative means low (modesty), near zero means accurate self-perception.

One line must never be crossed: this gap is not "capability." It is "how accurately you see yourself." So the source says flatly — "the gap is never added to or subtracted from the capability score; keep it in a separate column and read it as the quality of self-perception." We do not punish the self-inflater by lowering their level, nor reward the modest by raising it. The others' rating stays the others' rating; the gap sits beside it. Why — in a health checkup, the patient's self-report ("I feel fine") and the test results are recorded separately as a matter of course. Mix the two and neither stays trustworthy. Keeping the three layers — coordinate, capability, self-perception — unmixed to the end is the backbone of this design.

Agreement among raters — is it even measurable

If colleagues' ratings scatter all over, that ability is not yet measurable. When several referees in a sport disagree sharply, the first thing to suspect is not the player but that the rule reading isn't aligned — same idea here, so we check inter-rater agreement (Ag). The idea: "a one-step difference is forgiven." As a formula, take the share of pairs whose difference is within one band (Ag = pairs within ±1 band ÷ all pairs). Splitting between "senior" and "near-senior" is within interpretive range; splitting between "junior" and "senior" means people are reading the ruler differently.

The condition for finalizing a judgment (validity condition G0) requires three things at once: the total weight above a floor, agreement above a floor, and at least two colleagues with "view ≥ 0.7." Miss any one and the judgment is paused. Then a calibration meeting aligns how people read the ruler, and the measurement is redone. In the source's words, "low agreement is a sign the ruler is read differently; fix the standard before judging the person." Do not mistake low agreement for a problem with the person being rated. It is a problem with the measurers — you cannot blame the player when the referees disagree.

Different questions for the subject and for colleagues

The AI rephrases the same interview skeleton to fit who it is talking to. What stays constant: it asks no "what if" hypotheticals — only real past events. Why — people answer hypotheticals with ideals, and the actual behavior never surfaces. To the subject: "tell me one time you actually did ~," drawing out the basis of judgment (how broad a principle they reasoned from = abstraction α) and how far the action reached (scope σ). To a colleague: "tell me one time you saw this person do ~," carving out what that individual actually did from the whole team's outcome. Shared prohibitions for both: no leading, no hypotheticals, no asking several things at once, no asking for opinions — and always return an abstract word to concrete behavior ("Specifically? What did you do next?").

The full computation for one ability

Trace the source's procedure in words, like following a recipe. First the subject and each colleague are interviewed, derive coordinates from concrete events, and get a level (L). Next, sum the colleagues' weights; for each level take the highest one whose weighted share clears the corroboration line as the others' rating (L_other). Subtract this from the subject's rating to get the gap (Δ). From the colleagues, take the share of pairs that landed within one band to get agreement (Ag). If the validity conditions on weight, agreement, and number of clear-view raters fail, pause the judgment (needs calibration); if they hold, return a record bundling coordinate, others' rating, self-rating, gap, imbalance (b), and certainty (C). Only with this whole sequence does the level get fixed not as "a vague impression" but as "agreement with the evidence of actual behavior."

What the multi-rater design changes is the protagonist of measurement: from "one excellent rater" to "corroborated multiple testimonies." One person's blind spot is filled by another's observation, the corroboration line stops a runaway single vote, and inter-rater agreement exposes the measurers' own lack of preparation. The level thereby approaches a value that does not hinge on who happened to watch.

And the gap from self-rating sits quietly in its own column. It is neither a bonus nor a penalty on ability, but a record of how accurately a person sees themselves. As long as capability (the others' rating) and self-perception (the gap) are never mixed, the measurement stays an entrance to development, not an instrument for judging people.