Vonage Audio Serializer for Pipecat

Overview

Pipecat is an open-source framework for building voice and multimodal conversational AI applications. It orchestrates AI services—such as speech-to-text, language models, and text-to-speech—alongside network transports and audio/video processing to produce low-latency, natural-sounding conversations.

The Vonage Audio Serializer for Pipecat converts audio frames between Pipecat's internal format and the audio format expected by the Vonage WebSocket connection. A Pipecat WebSocket transport owns the server connection and uses the serializer to encode outgoing frames and decode incoming data.

How It Works

Vonage routes audio to external services via managed WebSocket connections. The Vonage Audio Serializer acts as the protocol adapter between that WebSocket stream and Pipecat's internal audio pipeline:

  1. Your Pipecat application starts a WebSocket server transport configured with the Vonage serializer.
  2. Vonage opens a WebSocket connection to your server from a Video session through Audio Connector, or from a Voice call through an NCCO connect action.
  3. The WebSocket transport passes incoming data to the serializer, which converts it into the PCM frames Pipecat expects.
  4. Your pipeline processes the audio through its configured AI services and returns a response.
  5. The serializer converts outgoing frames to the audio format expected by the Vonage WebSocket connection, and the WebSocket transport sends the data to Vonage for playback.

Relationship to Other Vonage Pipecat Integrations

Vonage offers two separate Pipecat integrations that address different use cases:

Integration Transport Use case
Vonage Audio Serializer Audio WebSocket (Audio Connector / Voice NCCO) Audio-only AI pipelines for Voice or Video sessions
Vonage Video Transport for Pipecat WebRTC (Video Connector) Pipelines that also process or generate video, such as video avatars

Use the Audio Serializer when your pipeline only needs to process and return audio. Use the Vonage Video Transport for Pipecat when your pipeline also needs to work with video frames.

When to Use the Vonage Audio Serializer

  • Real-time voice AI assistants: Deploy an LLM-backed voice bot inside a Vonage Video session or on an inbound phone call.
  • Live transcription and translation: Pipe session audio through a transcription service and return translated speech to participants.
  • Call recording and analysis: Capture and analyze conversation content from Voice or Video calls in real-time.
  • Audio effects processing: Apply filtering, noise reduction, or other transformations to audio before it reaches participants.
  • Automated moderation: Detect and act on non-compliant or inappropriate speech as it occurs.

See Also