- ATC communication failures are a leading safety risk: they are a contributing factor in a majority of incidents reported to NASA’s Aviation Safety Reporting System (ASRS), with readback errors exceeding 25% on transmissions carrying four or more instructions.
- Generic speech recognition simply can’t handle this domain: harsh radio acoustics, restricted jargon, and un-guessable local waypoints defeat general Automatic Speech Recognition (ASR), while multi-step pipelines are too slow and fragile for an offline cockpit.
- Zee breaks new ground with a single, unified architecture: we believe it is the first aviation speech model to fuse voice activity detection, speaker diarization, and transcription into one engine, pair that with a novel spatial-grounding technique for disambiguating local entities, and feed the output to a small language model that decomposes dense transmissions into clean, verifiable commands.
- This entire pipeline runs offline, on-device, with zero cloud dependency: the entire pipeline is within the cockpit’s strict compute and power limits. The payoff: 84% better accuracy than general-purpose speech models and a 58% cut in navigational context errors, paving the way for real-time readback validation.
TransmissionKSJC · clearance readbackSoCal Approach · DSNEE SIX arriva
Output
Plain text. Nothing downstream can check a readback against or act on.
Command extraction
Commands appear as each instruction completes.
Word times interpolated across the turn: this run’s own timestamps were not exported.
Zee saw an 84% error reduction compared to generic speech to text models. Same audio, two models: whisper-base has no aviation training, and both get the generic phraseology right. The gap is the vocabulary that only exists in one airspace, which Zee resolves from the aircraft’s position.
Air Traffic Control (ATC) still runs on the world’s oldest live communication medium: bandwidth-limited VHF AM radio, spoken in rapid, dense International Civil Aviation Organization (ICAO) phraseology on shared frequencies, where every instruction vanishes the moment it is heard. The cost is real, communication failures are a contributing factor in the majority of incidents reported to NASA’s ASRS. When a single transmission packs four or more instructions, readback error rates exceed 25%. Off-the-shelf speech recognition fails to close this gap: general-purpose models struggle with ATC’s harsh acoustics and restricted jargon, and cannot resolve location-specific waypoints that don’t exist in the audio signal alone. Furthermore, the chained pipeline which includes Voice Activity Detection (VAD), diarization, transcription, is too fragile and slow for a compute-constrained, offline cockpit.
This post goes deeper on the aviation speech capability of Zee, Archer’s proprietary aviation foundation model, built on our large-scale, expert-annotated corpus of ATC radio communications. We make three contributions: a single integrated speech model that replaces the traditional transcription pipeline, producing speaker-attributed transcripts directly; a spatial-grounding technique that uses location context to correctly transcribe local entities in any airspace; and a lightweight downstream model that turns each dense transmission into structured, verifiable commands.
The result: we’ve found that Zee outperforms general-purpose speech models on aviation audio by 84%, and spatial grounding cuts entity transcription errors by 58%. Best of all, it runs in real time, offline, and on-device. Every fleeting radio call becomes a permanent, scrollable, time-indexed record, and lays the foundation for real-time readback validation: catching a “six left” readback against a “six right” clearance before the aircraft ever commits.
Introduction
In February 1921, eighteen years after the Wright brothers’ first flight, the United States Postal Service attempted to fly mail across the continental US for the first time. Two planes flew westbound from New York City and two flew east from San Francisco.
Pilots navigated by a chain of bonfires lit on the ground. The two westbound planes were stopped by weather, one of the eastbound planes crashed near Nevada. The only remaining plane, flown by Jack Knight, fought through a snowstorm in the Midwest and signaled his nighttime landing for a refueling stop by circling a field in Iowa City until someone heard the plane and lit a flare on the runway. The mail reached New York in thirty-three hours and twenty minutes.
Why is this story so important? It began the age of communication between pilots and air traffic control. Today, a pilot’s cockpit responsibilities have expanded far beyond basic altimeters and spotting ground bonfires. Aviators must simultaneously scan complex avionics, manage crowded airspace, and decipher continuous ATC advisories across radio frequencies. This relentless, ever growing, cognitive load pushes human memory to its limits, making critical communication mistakes far from hypothetical.
Communication breakdowns over shared air traffic radio frequencies lie at the heart of some of the most fatal commercial aviation accidents: the recent tragedy at LaGuardia serves as a somber illustration. These communication failures represent a profound vulnerability in aviation safety, contributing as a factor in a majority of incidents reported to NASA’s ASRS, and roughly 30% in EUROCONTROL data. The consequences are tangible: nearly 40% of such errors lead directly to altitude deviations. On the ground, pilot deviations account for roughly 62% of all runway incursions, and missing or mistaken readbacks are among the communication errors that most often produce one.
Pilots face an overwhelming cognitive burden, and easing that mental load is precisely what we set out to do. However, the cockpit environment introduces a unique technical challenge for software and AI systems: the need for low-latency performance in a compute-constrained, offline environment. In the air, systems must function within rigorous constraints on memory, processing power, and energy consumption. We do not have the luxury of cloud-based server farms in flight.
To solve this, we developed Zee, our proprietary aviation foundation model (introduced in our first post on Zee), whose speech capability produces a real-time, time-indexed, scrollable transcript that transforms fleeting verbal commands into a permanent, verifiable record. Zee’s speech capability converts raw audio into a structured signal, modernizing aviation’s oldest communication medium, and runs offline and on device.
Why off-the-shelf ASR isn’t enough
Automatic speech recognition (ASR) has achieved near-human accuracy for transcribing podcasts and meetings, but it fails fundamentally when applied to the aviation domain. Air traffic control operates on a completely different acoustic channel, relying on bandwidth-limited, compressed AM radio that is dominated by squelch tones, propeller noise, and occasionally, stuck microphones.
Beyond the harsh acoustic environment, the ATC language itself poses a formidable barrier. Aviation radiotelephony is not conversational English; it is a highly restricted, specialized variant defined by the ICAO. It operates on a rigid set of standard phraseology, requiring a precise vocabulary and a strict syntactical order of information. Research indicates that a lack of familiarity with this specific jargon drastically reduces communication efficiency. Furthermore, veteran controllers issue clearances at a speed and density that would overwhelm an untrained listener. A single transmission lasting only a few seconds can pack a callsign, an altitude, a heading, a frequency change, and a traffic advisory. The formatting of these dense instructions heavily impacts a pilot’s ability to process them. For instance, whether a controller groups numbers (“forty-one hundred”) or sequences them (“four thousand one hundred”) alters how the data is encoded in real-time. Without the mental models of experienced pilots to “chunk” this information, it is highly likely that critical pieces of a clearance will be missed. As instruction complexity increases, pilot error rates also climb dramatically:
| Message complexity | Readback error rate | Impact on pilot comprehension |
|---|---|---|
| Simple (1 item) | 3.8% | Highly manageable, minimal errors. |
| Moderate (2-3 items) | 5-8% | Requires more retention, but largely manageable. |
| High (4+ items) | 25.7% | About 1 in 4 clearances are read back incorrectly. Compounding confusion becomes a major risk. |
| Extreme (13-16+ items) | 19% to 38% | Severe memory overload, often resulting in multiple errors per readback. |
This complexity is further compounded by location-specific language. ATC speech is dense with local waypoints, runways, and geographic references (e.g., “ATWIK,” “SoCal Approach,” or “exit at Bravo six”) that only hold meaning within their specific airspace or airport. A generic ASR model, having never seen these tokens grouped together, typically guesses or hallucinates the text it perceives. Ironically, the situation is worsened by human attempts at efficiency. To navigate congested frequencies and minimize their time on the radio, experienced controllers frequently deploy non-standard strategies, routinely omitting essential keywords related to frequencies, airspeeds, or headings. While intended to save time, these deviations from ICAO rules exacerbate the challenges of ATC radio.
Finally, standard ASR architectures struggle with the structural and hardware realities of the cockpit. A typical radio clip features a controller issuing an instruction followed immediately by a pilot reading it back. Off-the-shelf ASR models output an undifferentiated wall of text. Identifying “who said what” requires bolting on a secondary speaker diarization model, which introduces new failure modes and unacceptable latency. Controllers and pilots need transcripts that stream and update in real-time as they speak, not seconds after the fact. Furthermore, unlike terrestrial applications, the cockpit is an isolated environment with severe compute, memory, and energy constraints. Deploying speech recognition in this space requires small, robust, and quantized models that do not drain battery life or throttle device performance, making complex, chained ASR pipelines entirely unviable.
Solving this problem through machine learning requires more than a standard transcription model. It demands a specialized, low-latency system capable of filtering 1930s-era static, untangling heavily compressed jargon, and parsing non-standard human deviations, all within the strict hardware constraints of an aircraft cockpit.
The hard reality of transcribing ATC audio
Even with a specialized model architecture, the ultimate bottleneck in aviation ASR is high quality data. Aviation is a notoriously low-resource domain. Most existing human-labeled datasets are outdated, poorly annotated, and fail to reflect modern operations. After auditing every available transcription dataset, we realized none met our requirements. Annotating aviation data at scale is challenging: it’s noisy, fast-paced, and dense with jargon. It requires trained pilots and controllers to produce accurate transcriptions, and they require additional context such as flight state, airport layouts, operational status, and navigational fixes. We continue to be dedicated to our mission of collecting and annotating our own high quality, diverse, and large-scale ATC radio communications data.
Air Traffic ControllerSKYWEST FIFTY TWO TWELVE, I gotta change up the sequence a bit, I got a flow time to go out, and uh wait, uh just continue holding short of one right, I’ll give you a sequence here shortly.
Pilot AOkay uh short of one right SKYWEST FIFTY TWO TWELVE.
Air Traffic ControllerALASKA SIX FIFTY FIVE, left on Bravo, short of Echo.
Pilot BLeft on Bravo, short of Echo ALASKA SIX FIFTY FIVE.15.9 seconds, two aircraft, four turns.
This 15.9-second clip from the KSFO ground frequency perfectly illustrates the intensity of ATC communications. In less than 16 seconds, the controller delivers complex instructions to two different aircraft, with both pilots firing back their readbacks almost instantly. It’s significantly faster than the average American English speaker, who typically clocks in at 3.27 words per second. In audio sample 1, ATC and pilots converse at a rapid pace of 4.25 words per second, 30% faster. Even though the most experienced professionals are used to the relentless radio static and background noise, they still can’t rely on their ears alone to correctly identify fixes without location context and navigation chart
Air Traffic ControllerPILATUS FOUR FIVE X-RAY TANGO, ground clearance to Santa Rosa via the SUNOL ONE departure, SUNOL as filed, climb via the SID, expect maintain five thousand, expect five thousand as filed, departure one two one point three, squawk four two three two.A single transmission carrying a route, an altitude, a frequency, and a squawk.
Consider a real clearance delivery transmission out of San Jose International Airport (ICAO code: KSJC). Distinct identifiers like SUNOL or AMVIA aren’t random words, they are standardized aeronautical waypoints updated across terminal charts every 28 days. Transcribing this audio accurately requires more than a standard speech-to-text model. It requires a rich operational context. Without pinpointing the airport, tuning into the right frequency, and referencing active departure charts, local facility names like Santa Rosa and five-letter waypoint codes could turn into unintelligible speech or wrong transcriptions.
Ultimately, without that geographical context, even the most experienced ears struggle to map these identifiers to their true ground truth spelling. To solve this, we integrated localized context models and custom tooling into our labeling pipeline, ensuring high accuracy and scalable data quality.
Zee’s Speech Engine
Unfortunately, despite solid performance from both the VAD and diarization models individually, the pipeline broke down when chaining them together with our transcription model.
The issue stems from how context is lost across module boundaries. When the VAD model segments audio, it passes isolated “speech” clips downstream. If a speaker pauses mid-sentence, the VAD correctly splits on the silence, but the diarization model receives that snippet in a vacuum with no way to know it’s looking at a fragmented thought. By the time this truncated audio reaches the transcription engine, the model lacks the broader context required to produce an accurate transcript. We could only easily detect that these fragmented audio clips belonged to a single continuous thought after the transcription step had already failed.
Ultimately, we realized that passing audio through a chain of modular, isolated steps created cascading failure points. For this reason our foundation model Zee has a unified Speech Engine: combining VAD, diarization, and transcription into a single unified model. Zee’s Speech Engine model produces an integrated token stream that naturally captures both speaker roles and turn boundaries. Much like inserting punctuation within a sentence, the model embeds boundary markers directly during processing. This allows Zee to simultaneously transcribe audio and identify speakers seamlessly, without incurring extra computational overhead.
Injecting spatial awareness into audio models
Air traffic control communications contain highly localized vocabularies in airspaces around the world. Pilots flying around the San Francisco Bay hear references to local waypoints, procedures, and landmarks (localized entities) that would be unfamiliar to their colleagues flying in Chicago or New York. In order to safely navigate an airspace, pilots must be familiar with this local terminology. Human pilots often orient themselves with charts and software that map waypoints, procedures, and landmarks.
In theory, a large multimodal model might be able to ingest an image like the above chart and correctly transcribe localized entities, but the constraints of running inference on-device make this kind of model difficult to deploy.
The challenge for the transcription model is to hear a controller say “stick,” and understand that when heard near San Francisco, “stick” should be transcribed as “SSTIK,” while if heard in North Carolina, “stick” should be transcribed “STIKK.” Notably, the relevant signal needed to distinguish the two spellings is not contained in the audio. No transcription model could make this distinction without some way of ingesting spatial context.