How Apple's AFM 3 Solves On-Device Speech Comprehension

Spontaneous speech includes fillers, false starts, and missing punctuation. Turning audio into clean text used to mean cloud APIs or stiff on-device phonetic maps.

AFM 3 Core Advanced adds multimodal audio and dictation to Apple’s on-device foundation model line: sparse activation, semantic cleanup, and disfluency handling at roughly 1 billion active parameters drawn from a 20-billion-parameter flash-resident model. I build audio-first Mac and iOS apps (CoreRecord, LocalMemo). This piece follows the sparsity, flash routing, and quantization write-ups in the same research arc.

The disfluency dilemma

On-device dictation often prints what you said, fillers and restarts included. Mid-sentence corrections show up as raw text.

Cloud models gained ground by reading wider context, dropping fillers, and keeping intent. AFM 3 brings that language layer onto Apple Silicon without activating the full dense 20-billion-parameter model on every token.

The literal typist vs. the active listener

Older local speech-to-text behaves like a literal typist: acoustic phonetics hit the transcript letter-for-letter, with no read on what you meant to say.

AFM 3 Core Advanced behaves like an active listener. It pairs acoustic input with language comprehension, tracks sentence structure as you speak, strips disfluencies, and returns punctuated text. That context pass is the multimodal cleanup step.

What Apple describes for AFM 3 audio

Apple routes audio through the same multimodal path as text. The sparse expert layout (AFM 3 sparse architecture, Instruction-Following Pruning) keeps dictation near 1 billion active parameters per request instead of a dense 20B run.

In human evaluations for this generation, Apple reports gains in meaning capture, disfluency handling, and punctuation layout, including a 44.7% vs 17.6% overall quality win rate against their prior production dictation system in third-generation foundation model benchmarks.

Platform dictation is shifting from acoustic token mapping to language-aware output on the device.

How this connects to shipped apps

Research describes models. Production apps wire fixed Apple frameworks in separate layers.

CoreRecord on Mac

CoreRecord records microphone and system audio in one local session. Live transcription runs through SFSpeechRecognizer with on-device recognition enforced. Audio stays on the Mac.

Transcript lines with timestamps sit beside the .caf file on disk. Meeting summaries (CoreRecord AI tier) consume transcript text through Apple’s Foundation Models framework: Private Cloud Compute when you request it, with on-device SystemLanguageModel fallback. CoreRecord does not upload raw meeting audio for summarization.

AFM 3 multimodal dictation is where Apple is pushing platform speech intelligence. CoreRecord already splits the job: speech recognition for capture, foundation models on text when you ask for a summary.

LocalMemo on iPhone

LocalMemo transcribes voice offline, then runs Apple Foundation Models on the phone to structure notes and search the local library. Better platform speech plus on-device FM structuring supports fully offline voice workflows.

Conclusion

AFM 3 multimodal dictation gives on-device apps a path to cloud-grade cleanup without shipping audio off the phone or Mac.

Try CoreRecord for Mac meeting capture or LocalMemo for offline voice notes. See the App Catalog and writings hub for more Apple Silicon notes.

Sources: Introducing the Third Generation of Apple Foundation Models · Apple Machine Learning Research