WebSockets in the Vonage Voice API

This guide explains how WebSockets integrate with the Vonage Voice API and help you build sophisticated real-time applications, such as AI-powered voice bots or live transcription services.

What Are WebSockets?

WebSockets are a communication protocol that provides a persistent, full-duplex connection between an application and the Vonage Voice platform.
Unlike HTTP, which requires a separate request for each exchange, a WebSocket maintains a single open connection over which messages can be sent in both directions at any time.

This is ideal for scenarios requiring low-latency, streaming data, such as sending and receiving audio packets in real time.

Why Are WebSockets Important for AI Connectors?

In AI voice applications, you often need to:

  • Receive live audio from a caller.
  • Transcribe that live audio.
  • Send synthesized audio responses back to the caller.
  • Send metadata for the session.
  • Exchange control signals dynamically.

WebSockets enable this by:

  • Streaming audio as binary packets.
  • Exchanging text messages as control commands or events.
  • Sending metadata at the beginning of the session.
  • Allowing near-instant interaction with AI services, such as speech recognition, NLU, or TTS engines.

Setting Up a WebSocket Server in Your Application

To connect Vonage to your WebSocket server (your application), you must:

  1. Deploy a WebSocket endpoint accessible over a secure (wss://) URL.
  2. Handle incoming connections initiated by Vonage.
  3. Process both binary messages (audio) and text messages (JSON commands/events).
  4. Optionally implement authentication or authorization logic.

Example (Node.js, ws):

const WebSocket = require('ws');
const server = new WebSocket.Server({ port: 8080 });

server.on('connection', socket => {
  console.log('Vonage connected');
  
  socket.on('message', message => {
    if (typeof message === 'string') {
      // Handle JSON commands or events
      console.log('Text message:', message);
    } else {
      // Handle binary audio
      console.log('Binary audio packet received');
    }
  });
  
  socket.on('close', () => {
    console.log('WebSocket disconnected');
  });
});

Using an NCCO to Establish the WebSocket Connection

To instruct Vonage to stream audio to your WebSocket server, configure an NCCO (Nexmo Call Control Object) action of type connect:

Example NCCO:

[
  {
    "action": "connect",
    "endpoint": [
      {
        "type": "websocket",
        "uri": "wss://your-server.example.com",
        "content-type": "audio/l16;rate=16000",
        "headers": {
          "custom-header": "value"
        }
      }
    ]
  }
]

Audio Format:

  • You control the audio format via content-type:
    • audio/l16;rate=24000: 16-bit linear PCM, 24kHz
    • audio/l16;rate=16000: 16-bit linear PCM, 16kHz (recommended for speech recognition).
    • audio/l16;rate=8000: 8kHz if needed.

Authenticating the WebSocket Connection

When Vonage connects to your WebSocket server, you may want to verify that the incoming connection is from Vonage. You can do this by configuring an authorization object on the WebSocket endpoint.

There are two supported authorization modes:

  • vonage: Vonage includes an Authorization header in the WebSocket opening handshake using the same JWT format used for signed webhooks (Bearer <JWT>). Your server should validate this JWT. See Signed webhooks for guidance on how to validate Vonage JWTs.
  • custom: Your application provides an Authorization header value which Vonage will send verbatim in the opening handshake. This lets you use your own authorization scheme and credentials.

Note: If authorization is omitted, set to null, or set to an empty object ({}), Vonage will not apply any authorization behavior for the WebSocket handshake.

See details in the NCCO Reference.

Bidirectional Interaction: Audio and Text Messages

From Vonage to Your Application:

  • Binary messages: Audio chunks captured from the caller.
  • Text messages: JSON events (e.g., connection opened, cleared, notifications).

From Your Application to Vonage:

  • Binary messages: Audio to play to the caller.
  • Text messages: Commands to control playback or request notifications.

This bidirectional flow enables:

  • Real-time transcription.
  • Playback of synthesized speech.
  • Control over playback buffers.
  • Event-driven interactions.

Parsing Vonage Packets (Binary vs JSON)

When your WebSocket server receives a message:

  • If the message is a Buffer or ArrayBuffer:
    • It’s audio data (raw PCM).
  • If the message is a string:
    • It’s a JSON-formatted control message.

Example JSON event:

{
    "event":"websocket:connected",
    "content-type":"audio/l16;rate=16000",
    "prop1": "value1",
    "prop2": "value2"
}

Always inspect the message type to route processing logic correctly.

Handling Incoming Binary Audio Packets

Binary messages contain raw PCM audio captured from the caller.

Key characteristics:

  • 16-bit signed little-endian PCM.
  • Sample rate defined by content-type (e.g., 16,000 Hz).
  • Each packet represents a short slice of audio (~20ms).

Typical processing:

  • Feed the audio into a speech recognition engine.
  • Buffer for later playback.
  • Save to disk for analysis.

Sending Binary Audio Packets to Vonage

To play audio back to the caller:

  1. Encode your audio as raw PCM.
  2. Match the sample rate and format specified in the NCCO.
  3. Send the audio data as binary WebSocket messages.

Important:
Vonage buffers incoming audio to play in order. This allows you to queue audio without gaps but requires buffer management, explained as follows.

How Audio Buffering Works

When you send binary audio packets:

  • Vonage buffers them internally.
  • WebSocket buffer size is 3072 packets which should be enough for around 60 seconds of audio.
  • Playback starts automatically.
  • Subsequent packets are queued.
  • You cannot interrupt playback mid-buffer without a control command.

This design ensures consistent playback without audio gaps.

Clearing the Audio Buffer (clear Command)

To immediately stop playback of buffered audio, send the clear command.

Outbound Command (from your application):

{
  "action": "clear"
}

` Effect:

  • All queued audio is discarded.
  • Playback stops immediately.

Inbound Acknowledgement (from Vonage platform):

{
  "event": "websocket:cleared"
}

Usage Scenario:
You need to interrupt playback to respond dynamically to the caller (e.g., after barge-in detection).

Notifying When Audio Ends (notify Command)

To get notified when the current audio buffer has finished playing, use the notify command.

Outbound Command (send after an audio payload that you want to know if it has finished playing):

{
  "action": "notify",
  "payload": {
    "customKey": "customValue"
  }
}

Behavior:

  • If audio is playing, Vonage sends an inbound notification to your application after playback ends.
  • If no audio is playing, the inbound notification is sent back to your application immediately.

Inbound Notification:

{
  "event": "websocket:notify",
  "payload": {
    "customKey": "customValue"
  }
}

Usage Scenario:
Synchronize your application logic (e.g., start recording or play a new prompt after the previous finishes).

JSON DTMF Messages

If any party on the call connected to the WebSocket sends a DTMF tone, this will trigger an event on the WebSocket. This event is a text message with a JSON payload, interleaved between the audio frames and having the following format:

{
  "event": "websocket:dtmf",
  "digit": "5",
  "duration": 260
}

You will receive one event for each keypress and each event will contain only one digit:

  • event allows you to identify it as a DTMF event.
  • digit contains the digit pressed 0-9, *, or #.
  • duration is the duration the key was pressed for in milliseconds; on most digital phone systems this will be a fixed length.

Listening to a Specific Participant in a Multiparty Conversation

When your application participates in a conversation with multiple call legs—such as a customer and an agent—you may want your WebSocket connection to only receive audio from a specific participant rather than the entire mixed audio.

This is called selective audio control, and it is achieved using the canHear and canSpeak properties of the NCCO conversation action.


Why Would You Use This?

  • Speech analytics: Capture only what the customer says, ignoring the agent.
  • Real-time transcription: Record the customer side for compliance.
  • Whisper prompts: Send audio only to the agent without the customer hearing.

How to Configure Selective Listening

To set this up:

  1. Create a named conversation (e.g., "customer_support").
  2. Connect the customer and agent call legs to the conversation.
  3. Add your WebSocket connection to the same conversation, specifying canHear and canSpeak as needed.

Example: WebSocket Listening to Customer Only

Below is an NCCO example where:

  • The customer is joined to the conversation.
  • The agent is joined to the conversation.
  • The WebSocket connects but only receives the customer’s audio.

Customer Leg NCCO:

[
  {
    "action": "conversation",
    "name": "support_room"
  }
]

Agent Leg NCCO:

[
  {
    "action": "conversation",
    "name": "support_room"
  }
]

WebSocket Leg NCCO:

[
  {
    "action": "conversation",
    "name": "support_room",
    "canHear": ["6a4d6af0-55a6-4667-be90-8614e4c8e83c"], // Customer leg ID
    "canSpeak": []
  }
]

How this works:

  • The WebSocket only hears the customer participant.
  • It does not send any audio back into the conversation (canSpeak is empty).
  • If you want it to inject audio (e.g., AI prompts) and play the audio to only a designated participant, you could include the participant call (leg) ID in canSpeak.
  • If you want it to inject audio (e.g., AI prompts) and play the audio to all participants, you do not include canSpeak parameter.

Handling WebSocket Disconnections and Fallback Options

Note: If your endpoint uses failover, reconnect, keepalive, or connectionTimeout, see WebSocket Failover, Reconnection, and Keepalive for how Vonage handles disconnections automatically.

When you use WebSockets with the Vonage Voice API, you don’t just rely on the WebSocket itself to know what’s happening.
Vonage also sends event callbacks to your eventUrl webhook. These HTTP POST requests provide authoritative information about call status and enable fallback behavior if the WebSocket connection fails.

This is important because simply observing the WebSocket closing doesn’t tell you why it closed. You need the event webhook to determine whether the disconnection was intentional or caused by an error.

Why Does This Matter?

When building production voice experiences—especially AI-powered or real-time ones—connections may drop unpredictably (e.g., server crashes, network timeouts).
To provide a graceful experience to the caller, you can implement fallback strategies such as playing a prompt, transferring to a human agent, or ending the call politely.

Webhook events give you a reliable mechanism to detect these situations and act accordingly.

How Vonage Notifies You of WebSocket Events

Whenever there is a significant change in the WebSocket connection status, Vonage sends an event webhook to your eventUrl.

Examples of relevant statuses:

  • unanswered: Vonage could not establish the WebSocket connection.
  • failed: The connection attempt failed.
  • disconnected: The WebSocket connection was dropped after being established.

Each event includes:

  • The uuid, identifying the call.
  • Timestamps.
  • Any custom headers you specified in the NCCO connect action.
  • The status field describing what happened.

Example Disconnected Event Payload

This event is sent when the WebSocket is disconnected after connection - whether due to an error or because your application closed it:

{
  "from": "442079460000",
  "to": "wss://example.com/socket",
  "uuid": "aaaaaaaa-bbbb-cccc-dddd-0123456789ab",
  "conversation_uuid": "CON-aaaaaaaa-bbbb-cccc-dddd-0123456789ab",
  "status": "disconnected",
  "timestamp": "2020-03-31T12:00:00.000Z",
  "headers": {
    "caller-id": "447700900123"
  }
}

How to Distinguish Intentional vs. Unintentional Disconnects

Important to understand:

  • When reconnection is not configured, a connection closed by your application or lost due to an error raises a disconnected event. With reconnect configured, an unclean close starts reconnection instead. disconnected is sent if the attempt budget is exhausted or your server closes the connection cleanly. By default, the event does not distinguish an intentional close from a connection failure.
  • If the websocket endpoint sets at least one of the resilience options (failover, connectionTimeout, keepalive, or reconnect), a disconnected event caused by your server closing the connection or by a connection loss includes cause. It also includes close_code if a Close frame was received, so close_code is absent after a network loss or a keepalive timeout. These fields are available even if you do not set reconnect.
  • To intentionally end the WebSocket session, for example when your voice bot has finished, either:
    • Close the WebSocket connection with code 1000. You still receive a disconnected event. If a resilience option is set, it includes cause: remote_closed and close_code: 1000, so you can recognize it as intentional.
    • Hang up the call leg via the Vonage Voice API. No disconnected event is sent, only completed. If you always end sessions this way, you receive disconnected only for unintentional failures.

Handling Failed Connections During Setup

Sometimes the WebSocket connection cannot be established in the first place (for example, if your server is offline).
You can configure your connect action to use synchronous event handling:

Example NCCO with synchronous eventType:

[
  {
    "action": "connect",
    "eventType": "synchronous",
    "eventUrl": [
      "https://example.com/events"
    ],
    "from": "447700900000",
    "endpoint": [
      {
        "type": "websocket",
        "uri": "wss://example.com/socket",
        "content-type": "audio/l16;rate=16000",
        "headers": {
          "caller-id": "447700900123"
        }
      }
    ]
  }
]

How it works:

  • If the connection attempt fails, Vonage immediately POSTs an event to your eventUrl.
  • The event status will be unanswered or failed.
  • You can respond with a new NCCO describing fallback behavior, such as playing a message or redirecting the call.

Example Failed Connection Event Payload

{
  "from": "442079460000",
  "to": "wss://example.com/socket",
  "uuid": "aaaaaaaa-bbbb-cccc-dddd-0123456789ab",
  "conversation_uuid": "CON-aaaaaaaa-bbbb-cccc-dddd-0123456789ab",
  "status": "unanswered",
  "timestamp": "2020-03-31T12:00:00.000Z",
  "headers": {
    "caller-id": "447700900123"
  }
}

Implementing Fallback Strategies

When you receive a webhook with status: disconnected, failed, or unanswered, you can:

  • Return a new NCCO in your webhook response to handle the fallback (e.g., play a prompt).
  • Allow the original NCCO to continue, if there are additional actions.
  • End the call, if no further action is specified.

Example fallback NCCO:

[
  {
    "action": "talk",
    "text": "We are unable to connect you at the moment. Please try again later."
  }
]

WebSocket Failover, Reconnection, and Keepalive

A websocket endpoint can include optional settings that let Vonage handle connection problems for you, so the call can continue without extra recovery logic in your application:

  • Failover: an ordered list of alternative URIs that Vonage tries if the primary uri cannot be reached, both when the call is set up and when reconnecting.
  • Automatic reconnection: Vonage re-establishes the connection if an established connection closes uncleanly.
  • Connection timeout and keepalive: settings that control how long a connection attempt can take and how quickly Vonage detects an unresponsive connection.

Important:

  • All of these settings are optional. An endpoint that uses none of them behaves as described in Handling WebSocket Disconnections and Fallback Options.
  • If an endpoint sets any of them, keepalive is turned on, and your event webhooks include additional WebSocket fields. See the Webhook Reference for details.
  • Failover and reconnection attempts are part of the original connection. They do not count as new connections against your CPS limit, so you no longer need a high CPS limit to handle many connections dropping at once, for example when one of your servers restarts.
  • You can use these settings wherever you can use a websocket endpoint: in an NCCO connect action, and in the to array of a POST /v1/calls request. For default values and allowed ranges, see the NCCO Reference.
  • Setting "eventType": "synchronous" on the connect action is optional. You only need it if you want to respond to a failed, unanswered, or disconnected event with a fallback NCCO.

The following NCCO connects a call to a WebSocket and sets all of the resilience options: failover, connectionTimeout, keepalive, and reconnect.

[
  {
    "action": "connect",
    "eventUrl": ["https://example.com/events"],
    "timeout": 20,
    "from": "447700900000",
    "endpoint": [
      {
        "type": "websocket",
        "uri": "wss://primary.example.com/socket",
        "content-type": "audio/l16;rate=16000",
        "headers": {
          "caller-id": "447700900123"
        },
        "authorization": {
          "type": "vonage"
        },
        "failover": [
          {
            "uri": "wss://secondary.example.com/socket"
          },
          {
            "uri": "wss://dr.example.net/socket",
            "authorization": {
              "type": "custom",
              "value": "Bearer eyJhbGciOi..."
            }
          }
        ],
        "connectionTimeout": 3,
        "keepalive": {
          "interval": 5,
          "timeout": 5
        },
        "reconnect": {
          "maxAttempts": 3,
          "initialDelay": 0.5,
          "maxDelay": 4,
          "cleanCloseCodes": [4000]
        }
      }
    ]
  }
]

Failover: Initial Connection

Add a failover array to the websocket endpoint to list up to 3 alternative URIs. If Vonage cannot connect to the primary uri when the call is set up, it tries these URIs in order.

The connection process works as follows:

  1. Vonage tries the primary uri.
  2. If that attempt fails, Vonage immediately tries each failover entry in order.
  3. When a URI connects, your eventUrl receives an answered event. The connected_uri field shows which URI Vonage connected to.
  4. If every URI fails, your eventUrl receives a failed or unanswered event. The attempts array lists the outcome for each URI that Vonage tried. You can respond with a fallback NCCO, as described in Handling Failed Connections During Setup.

For the fields of these events, see answered, failed, and unanswered in the Webhook Reference.

An attempt to a URI fails if:

  • It does not complete within connectionTimeout.
  • The connection is refused or cannot be made.
  • A TLS error occurs.
  • The handshake returns any response other than HTTP 101.

The failover entries follow these rules:

  • Each entry must contain a uri (ws:// or wss://) and can also contain headers and authorization. Any other fields are ignored.
  • An entry without headers uses the headers of the primary endpoint. An entry without authorization uses the authorization of the primary endpoint.
  • If an entry sets its own headers, they replace the primary endpoint's headers. The two are not merged.
  • content-type is always taken from the primary endpoint.

Note: The endpoint's connectionTimeout applies separately to each URI that Vonage tries. If every attempt times out, the initial connection can take up to (1 + number of failover entries) x connectionTimeout. For example, with three failover entries and the default timeout of 3 seconds, this is 4 x 3 = 12 seconds.

Automatic Reconnection

To turn on automatic reconnection, include the reconnect object in the websocket endpoint. Vonage reconnects only after an unclean close. If reconnect is not set, any close ends the WebSocket leg, as described in Handling WebSocket Disconnections and Fallback Options.

Clean and Unclean Closes

Clean Closes

Vonage does not reconnect after these:

  • Your server sends a WebSocket Close frame with status code 1000 (Normal Closure).
  • Your server sends a Close frame with no status code. Many libraries report this as 1005. Some libraries send this when you call close() with no arguments.
  • Your server sends a Close frame with a code you listed in reconnect.cleanCloseCodes.
  • Vonage closes the connection because the call or leg has ended. For example, the other party hangs up, the leg is ended through the Voice API, or the connect action's limit is reached.
Unclean Closes

Vonage reconnects after these if reconnect is set:

  • Your server sends a Close frame with any other code. Common examples are 1001 (Going Away), 1011 (Internal Error), 1012 (Service Restart), and 1013 (Try Again Later).
  • The connection drops without a Close frame, for example because of a network failure, a server crash, or a load balancer dropping the connection. Many libraries report this as 1006.
  • The connection becomes unresponsive, meaning no Pong arrives within keepalive.timeout. See Connection Timeout and Keepalive.
  • A TLS or protocol error occurs on an established connection.

Note: Vonage treats code 1001 (Going Away) as an unclean close on purpose. Servers commonly send it while they shut down or restart, which is when reconnecting is most useful.

How Reconnection Works

When an established connection closes uncleanly and reconnect is set, Vonage reconnects as follows:

  1. Vonage detects the unclean close, and your eventUrl receives a reconnecting event.
  2. Vonage waits for the backoff delay, then makes a reconnection attempt. In each attempt, Vonage:
    • Tries the URI that was last connected first.
    • Then tries the remaining URIs from the primary and failover list, in order.
    • Applies connectionTimeout separately to each URI.
  3. If no URI connects, Vonage waits for the next, longer backoff delay and makes another attempt, up to maxAttempts attempts.
  4. If an attempt succeeds:
    • Your eventUrl receives a reconnected event.
    • Your server receives the usual websocket:connected message, with the headers of the URI it connected to and a reconnect_attempt field.
  5. If all attempts fail:

The backoff delay works as follows:

  • Before the first attempt, Vonage waits initialDelay.
  • The wait doubles before each later attempt, up to maxDelay, with a random variation of up to 10%.
  • With the default values, Vonage waits about 0.5, 1, and 2 seconds before the three attempts.

After a successful reconnection, your server receives a websocket:connected message like this:

{
  "event": "websocket:connected",
  "content-type": "audio/l16;rate=16000",
  "caller-id": "447700900123",
  "reconnect_attempt": 1
}

See the Webhook Reference for the payload and all fields of the reconnecting, reconnected, and disconnected events.

Reconnection Attempt Budget

maxAttempts is a budget for the connection, not for each time the connection drops:

  • The budget is restored once a connection has stayed up for 30 seconds.
  • If a reconnected session drops again within 30 seconds, the attempts already used still count. The reconnect_attempt field in websocket:connected keeps counting in the same way.
  • If the budget is already used up, Vonage does not try again, and your eventUrl receives a disconnected event.

What Happens During Reconnection

While Vonage is reconnecting:

  • The call stays up. The leg uuid and conversation_uuid do not change.
  • Audio and DTMF events from the call during the gap are not delivered to your server.
  • Any audio your server had queued for playback before the connection closed is discarded. Playback starts fresh on the new connection. See How Audio Buffering Works.
  • Any notify markers in that queue are discarded with it. Your server receives neither websocket:notify for them nor websocket:cleared.
  • If the call ends, reconnection stops. You receive the normal end-of-call events, and no further WebSocket events.
  • The reconnecting and reconnected events are for information only. Vonage ignores any NCCO returned in response to them.

Ending a Session Intentionally

To end a WebSocket session on purpose, for example when your voice bot has finished, close the connection with code 1000. Vonage treats this as a clean close and does not reconnect. Your eventUrl receives a disconnected event with cause set to remote_closed, close_code set to 1000, and reconnect_attempts set to 0.

You can also hang up the leg through the Voice API, as before. In that case, Vonage sends completed but not disconnected.

Authorization Across Connection Attempts

When authorization is set to "type": "vonage", Vonage issues one token for the call. The same token is sent on every connection attempt: to the primary URI, to failover URIs, and on every reconnection. The token's iat claim is the time the call was set up, and its jti claim does not change between attempts.

Note: If your server rejects reused jti values, or tokens older than a given age, it will reject reconnection attempts.

Connection Timeout and Keepalive

Two settings control how Vonage deals with slow or unresponsive connections: connectionTimeout limits how long a connection attempt can take, and keepalive controls how quickly Vonage detects an unresponsive connection.

Connection Timeout

connectionTimeout is the maximum time allowed for one connection attempt to one URI. The time limit covers the whole process up to a completed WebSocket handshake:

  • Name resolution
  • TCP connection
  • TLS negotiation
  • The HTTP upgrade

Keepalive

On an established connection, Vonage uses keepalive to check that your server is still responding:

  1. Vonage sends a WebSocket Ping frame every keepalive.interval seconds.
  2. Vonage waits up to keepalive.timeout seconds for a Pong reply. Only a Pong counts. Other frames from your server do not.
  3. If no Pong arrives in time, Vonage treats the connection as unresponsive. This is an unclean close, so Vonage reconnects if reconnect is set.

keepalive is turned on automatically whenever the endpoint sets any of failover, connectionTimeout, keepalive, or reconnect. If you do not set keepalive, its default values are used.

Note: Most WebSocket server libraries reply to Ping frames automatically, as the WebSocket standard requires. You should not need to change your code to support keepalive.

Sizing Your Timeouts

After an unclean close, the time a caller may hear silence is the sum of the time to detect the close, the backoff delays, and the connection attempts.

For the example NCCO above, the worst case, where every attempt to every URI times out, is about 40 seconds:

  • Detection of an unresponsive connection: 10 seconds (keepalive.interval + keepalive.timeout).
  • Backoff delays: 3.5 seconds (0.5 + 1 + 2).
  • Connection attempts: 27 seconds (3 attempts x 3 URIs x 3 seconds).

A refused connection, where the server is up but nothing is listening, fails within milliseconds, so recovery is usually much faster. If the host does not answer at all, each attempt waits the full connectionTimeout.

Choose values so that the worst case is acceptable for your callers.

Connecting to AI Engines

Find Vonage sample applications to connect to popular AI engines below: