The Expert’s Trap: Why Your Best Pilots Are Your Worst Examiners

Cognitive Bias in Assessment

The Expert’s Trap: Why Your Best Pilots Are Your Worst Examiners

Pushing the throttle forward, Miller felt the vibration of the Pratt & Whitney engines deep in his marrow, a sensation he had lived with for 22 years of line flying. He didn’t need to look at the gauges to know the EPR was stabilizing; he felt the aircraft breathe. But now, sitting in a plastic chair in a humid briefing room, that intuition was his greatest enemy. He was staring at a young first officer from a different hemisphere who had just butchered a mandatory read-back. Miller knew it was wrong. He could feel the error like a phantom limb, but when he opened his mouth to explain *why* the phrasing failed ICAO standards, all he could muster was: ‘You just need to sound more professional. You know, more like me.’

It was a catastrophic failure of pedagogy disguised as seniority. I’ve seen this scene play out in 52 different flight schools across 12 different countries, and it never gets less painful. We have this pervasive, dangerous myth in aviation that if you can fly the plane to Level 6 standards, you are automatically qualified to judge if someone else is doing the same. It is the ‘Curse of Knowledge’-a cognitive bias where once you have mastered a skill, you literally lose the ability to imagine what it is like not to possess it. You can’t see the steps anymore because you’re already at the top of the staircase.

The Empty Fridge: Looking for What Isn’t There

I’ve checked the fridge 2 times in the last hour, looking for a snack that I know isn’t there, which is a perfect metaphor for what these veteran pilots are doing. They are looking into the student’s performance for a linguistic structure they don’t have the tools to identify. They are hungry for a solution but keep opening the same empty door of ‘intuition.’

My friend Owen E.S., a sunscreen formulator who spends 12 hours a day measuring the viscosity of zinc oxide suspensions, once told me something that changed my perspective on this. He said that the most dangerous moment in the lab is when a chemist stops using the spectrometer and starts relying on the ‘feel’ of the cream. ‘Once you think you know the texture by heart,’ Owen said, ‘you stop seeing the microscopic separations that lead to a product failure 32 days later.’

– Owen E.S. (Metaphor Source)

In the stickpit, ‘feeling’ the language is a recipe for subjective, unfair, and ultimately dangerous assessments. A pilot who has spent 10002 hours in the air has internalised the cadence of air traffic control. To them, the language is no longer a set of grammatical rules; it is a background hum. When a student deviates, the expert hears ‘noise’ but can’t tell you if that noise is a failure of pronunciation, a lack of vocabulary, or a breakdown in functional language.

Expertise is a tomb where the ‘how-to’ goes to die.

Deconstructing the Six Pillars

We see this manifest in the way these accidental examiners treat the ICAO rating scale. They look at the six pillars-pronunciation, structure, vocabulary, fluency, comprehension, and interactions-and they treat them like a suggestion rather than a rubric. I once watched a check-pilot give a student a Level 2 in pronunciation simply because he didn’t like the student’s ‘attitude’ during the pre-flight briefing. There were 2 separate issues there, but the examiner lacked the analytical training to decouple his personal bias from the linguistic reality. He was a master of the sky, but a novice of the word.

Examiner Error Distribution (Conceptual Metrics)

Pronunciation

80% Score

Interactions

65% Score

Fluency

95% Score

This is why we see so much frustration in the industry. You have pilots with 12002 hours of experience who are told they aren’t ‘Level 6’ because they use slang or non-standard phraseology, and you have examiners who can’t explain the difference between a ‘mistake’ (a random slip) and an ‘error’ (a systematic lack of knowledge). Without the ability to deconstruct language, the examiner is just a person with a clipboard and a gut feeling.

The Opaque Assessor

It’s a contradiction I’ve lived myself. I used to pride myself on being a ‘tough’ assessor, thinking that my high standards for flying meant I had high standards for communication. I was wrong. I was just being opaque. I was asking students to hit a target I hadn’t even defined for myself.

It wasn’t until I realized that assessment is a technical skill as rigorous as an instrument approach that I started to actually help my students improve. You have to learn to see the zinc oxide, as Owen E.S. would say, rather than just feeling the cream.

1

Realization Point

The Glide Slope vs. The Sentence

There is a specific kind of arrogance that comes with 22 years of flawless landings. You start to believe that your eyes are calibrated to the truth. But language isn’t a glide slope. It doesn’t stay at 3.2 degrees. It’s messy, cultural, and deeply psychological. When an examiner tells a student to ‘just speak better,’ they are committing a form of professional malpractice. They are charging $272 for an exam and providing 0.2 percent of the value they should be.

SME + LE

The industry is slowly waking up to this gap. We are starting to realize that the ‘SME’ (Subject Matter Expert) needs to be paired with a ‘LE’ (Language Expert), or better yet, the SME needs to be trained to think like a linguist. This is exactly where organizations like Level 6 Aviation come into the picture. They recognize that the bridge between doing and assessing is built out of specific, teachable skills. You can’t just wish that bridge into existence with more flight hours. You have to study the anatomy of a sentence the same way you study the anatomy of a wing.

Measuring Souls with Broken Rulers

I remember one specific cadet, a brilliant flyer with 422 hours in small twins, who was nearly washed out of a major airline program because an examiner thought his English was ‘weak.’ When a trained rater actually looked at the transcripts, they found the student’s English was nearly perfect-he was just being quiet because the examiner was shouting at him. The examiner had confused a lack of confidence with a lack of competence. That is a 2-point swing on the ICAO scale that could have ended a career before it started.

We are measuring souls with broken rulers.

If we want to fix this, we have to stop treating the examiner role as a retirement home for senior captains. It’s not a reward for a long career; it’s a new career entirely. It requires a shift from the ‘what’ to the ‘why.’ A great examiner can tell you exactly which of the 6 descriptors a student is failing and provide a roadmap to fix it. They don’t say ‘sound more like me.’ They say, ‘Your use of complex grammatical structures is inconsistent when the workload increases, which is a hallmark of Level 4 proficiency.’ See the difference? One is a dismissal; the other is a diagnosis.

72%

Communication Failure in Accidents

The math doesn’t add up: Millions on simulators, pennies on communication training.

The Necessary Transformation

Even Owen E.S. knows that you can’t just mix the chemicals and hope for the best. You have to test the stability at 42 degrees Celsius, you have to check the pH, and you have to document every single variance. If we did that for sunscreen, why aren’t we doing it for the voices that guide 302 people through a thunderstorm at midnight?

We need to demand more from our examiners. We need to stop assuming that a gold braid on a sleeve translates to a silver tongue or a sharp analytical mind for language. The transition from the stickpit to the classroom is a climb that requires its own flight plan, its own fuel, and its own set of charts. Without them, we are just flying blind in a cloud of our own expertise, wondering why the ground is getting closer so fast.

The Shift in Mindset

Expert Mindset

Dismissal

“Sound like me”

Examiner Skill

Diagnosis

“Systematic lack of knowledge”

Is it possible that our greatest pilots are the ones most blinded by their own mastery? I think so. The more ‘natural’ a skill becomes, the more invisible its components are. To be a great examiner, you have to make the invisible visible again. You have to be willing to be a beginner in the world of linguistics so that you can guide your students out of it. It’s a humbling, difficult, and necessary transformation. And it’s one that the industry can no longer afford to ignore, lest we continue to fail the very people we are supposed to be lifting upskilling. After all, the sky doesn’t care how many years you’ve been flying if you can’t tell the person next to you what’s actually happening in a way they can understand.

The Path to Visible Mastery

📚

Study Linguistics

Learn the rubric, not just the results.

🔬

Deconstruct Errors

Separate slips from systematic knowledge gaps.

💡

Make Visible

Turn intuition into teachable steps.