audio.cpp: Cross-Platform Speech Recognition and TTS in Pure C++

Python libraries for speech work on a Mac are good until you need the same code to run somewhere else. mlx-audio is a fine example. It sits on Apple's MLX framework and makes inference on Apple Silicon quick, with ready-made support for dozens of speech-to-text and text-to-speech models. Its limit is portability: it's built for the Mac. audio.cpp takes the opposite route. It's an open-source speech inference engine written in plain C++ on the ggml runtime, with Metal, Vulkan and CUDA backends for fast GPU inference.

What it offers

• Runs a long list of popular open-source ASR and TTS models.
• Works on Windows, macOS and Linux.
• Uses the ggml stack, so you can pick CUDA, HIP/ROCm, Vulkan, Metal or plain CPU.
• Supports JSON pipelines for chaining steps into larger workflows.
• Bundles utilities for noise reduction, enhancement, resampling and STFT/ISTFT.

Models you can run

Text to speech covers Qwen3-TTS, VibeVoice, VoxCPM2, Fun-CosyVoice3, the IndexTTS-2 family, Higgs Audio v3, Fish Audio S2 Pro, Kokoro 82M, OmniVoice, Chatterbox, BreezeTTS 2 and Supertonic 3, among others.

Speech recognition includes two Qwen3-ASR sizes (0.6B and 1.7B), Fun-ASR-Nano, Higgs Audio v3 STT, VibeVoice ASR in normal and streaming forms, Nemotron 3.5 ASR Streaming and Voxtral-Mini-4B-Realtime.

Music generation is covered by MiniMax Music 3, ACE-Step 1.5 Turbo and the Small and Medium releases of Stable Audio 3, plus more.

Building it locally

Start by fetching the code:

git clone https://github.com/0xShug0/audio.cpp.git
cd audio.cpp

Then compile for whichever backend matches your hardware. On Linux with Vulkan:

scripts/build_linux.sh --backend vulkan --target audiocpp_cli --target audiocpp_server

On Linux with an NVIDIA GPU you'd use CUDA instead, either letting it detect the architecture or naming specific ones:

scripts/build_linux.sh --backend cuda --cuda-arch native --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cuda --cuda-arch "86;89" --target audiocpp_cli --target audiocpp_server

On a Mac, use the Metal script:

scripts/build_metal.sh --target audiocpp_cli

When the Metal build finishes, the audiocpp_cli binary lands in build/macos-metal-release/bin. That one CLI handles a wide set of tasks, chosen by name:

• Detection and analysis: vad finds voice activity, spk handles speaker embedding and recognition, align does forced alignment, and diar separates speakers.
• Core speech: asr transcribes, tts synthesizes, clon clones voices and vdes designs them.
• Transformation: vc converts voices, s2s goes speech to speech, svc converts singing voices and sep separates sources.
• Generation and transcription: gen produces music or sound from text or audio conditioning, and midi turns audio into symbolic MIDI events.

Getting models

A bundled Model Manager takes care of downloads, and it understands these commands:

list                list installable packages
info                show one package or family default
sizes               check package download sizes without downloading files
installed           report locally installed packages without network access
uninstall           remove only the files belonging to one installed package
clean-partial       remove abandoned staging downloads for one package
install             install one Hugging Face or ModelScope snapshot package

Start by listing what's available:

python3 tools/model_manager_v2.py list

The console prints the installable packages. To try both directions of speech, install Parakeet-TDT-0.6B-v3 for recognition and VoxCPM2 for synthesis:

# ASR
python3 tools/model_manager_v2.py install parakeet_tdt_q8_0
# TTS
python3 tools/model_manager_v2.py install voxcpm2_q8_0

Trying it out

For recognition, point the CLI at the Parakeet GGUF file and a WAV file:

build/macos-metal-release/bin/audiocpp_cli  \
  --task asr \
  --model models/Parakeet-TDT-0.6B-v3-GGUF/parakeet-tdt-0.6b-v3-q8_0.gguf \
  --audio lisa.wav

The console responds with the model family, task, mode, word-level timestamps with sample positions and confidence, and the final transcript text:

family=parakeet_tdt
task=asr
mode=offline
word_timestamps=[{"start_sample":0,"end_sample":14080,"word":"Breeze","confidence":0}, ...]
text_output=Breeze TTS2 is an open-weight text-to-speech model built for real-time interaction.

For synthesis with VoxCPM2, a basic voice clone takes a text string plus a reference recording:

build/macos-metal-release/bin/audiocpp_cli \
  --task tts \
  --model models/VoxCPM2-GGUF \
  --text "audio.cpp is running VoxCPM2 locally." \
  --voice-ref lisa.wav \
  --out voxcpm2.wav

A more precise "ultimate" clone adds the transcript of the reference clip as well, which helps the model match the voice more closely:

build/macos-metal-release/bin/audiocpp_cli \
  --task tts \
  --model models/VoxCPM2-GGUF \
  --text "audio.cpp is running VoxCPM2 locally." \
  --voice-ref lisa.wav \
  --reference-text "Breeze TTS 2 is an open-weight text-to-speech model built for real-time interaction." \
  --out voxcpm2.wav

Is it worth a look?

Next to mlx-audio, audio.cpp is leaner. It doesn't need a Python runtime or a pile of third-party packages. If you're building cross-platform AI features that depend on recognition or synthesis, test it. If transcription is all you need, take a look at transcribe.cpp, a related project covering 16 popular open-source ASR models and over 60 variants.