WebSockets in the Vonage Voice API
This guide explains how WebSockets integrate with the Vonage Voice API and help you build sophisticated real-time applications, such as AI-powered voice bots or live transcription services.
What Are WebSockets?
WebSockets are a communication protocol that provides a persistent, full-duplex connection between an application and the Vonage Voice platform.
Unlike HTTP, which requires a separate request for each exchange, a WebSocket maintains a single open connection over which messages can be sent in both directions at any time.
This is ideal for scenarios requiring low-latency, streaming data, such as sending and receiving audio packets in real time.
Why Are WebSockets Important for AI Connectors?
In AI voice applications, you often need to:
- Receive live audio from a caller.
- Transcribe that live audio.
- Send synthesized audio responses back to the caller.
- Send metadata for the session.
- Exchange control signals dynamically.
WebSockets enable this by:
- Streaming audio as binary packets.
- Exchanging text messages as control commands or events.
- Sending metadata at the beginning of the session.
- Allowing near-instant interaction with AI services, such as speech recognition, NLU, or TTS engines.
Setting Up a WebSocket Server in Your Application
To connect Vonage to your WebSocket server (your application), you must:
- Deploy a WebSocket endpoint accessible over a secure (wss://) URL.
- Handle incoming connections initiated by Vonage.
- Process both binary messages (audio) and text messages (JSON commands/events).
- Optionally implement authentication or authorization logic.
Example (Node.js, ws):
const WebSocket = require('ws');
const server = new WebSocket.Server({ port: 8080 });
server.on('connection', socket => {
console.log('Vonage connected');
socket.on('message', message => {
if (typeof message === 'string') {
// Handle JSON commands or events
console.log('Text message:', message);
} else {
// Handle binary audio
console.log('Binary audio packet received');
}
});
socket.on('close', () => {
console.log('WebSocket disconnected');
});
});
Using an NCCO to Establish the WebSocket Connection
To instruct Vonage to stream audio to your WebSocket server, configure an NCCO (Nexmo Call Control Object) action of type connect:
Example NCCO:
[
{
"action": "connect",
"endpoint": [
{
"type": "websocket",
"uri": "wss://your-server.example.com",
"content-type": "audio/l16;rate=16000",
"headers": {
"custom-header": "value"
}
}
]
}
]
Audio Format:
- You control the audio format via
content-type:audio/l16;rate=24000: 16-bit linear PCM, 24kHzaudio/l16;rate=16000: 16-bit linear PCM, 16kHz (recommended for speech recognition).audio/l16;rate=8000: 8kHz if needed.
Authenticating the WebSocket Connection
When Vonage connects to your WebSocket server, you may want to verify that the incoming connection is from Vonage. You can do this by configuring an authorization object on the WebSocket endpoint.
There are two supported authorization modes:
vonage: Vonage includes anAuthorizationheader in the WebSocket opening handshake using the same JWT format used for signed webhooks (Bearer <JWT>). Your server should validate this JWT. See Signed webhooks for guidance on how to validate Vonage JWTs.custom: Your application provides anAuthorizationheader value which Vonage will send verbatim in the opening handshake. This lets you use your own authorization scheme and credentials.
Note: If authorization is omitted, set to null, or set to an empty object ({}), Vonage will not apply any authorization behavior for the WebSocket handshake.
See details in the NCCO Reference.
Bidirectional Interaction: Audio and Text Messages
From Vonage to Your Application:
- Binary messages: Audio chunks captured from the caller.
- Text messages: JSON events (e.g., connection opened, cleared, notifications).
From Your Application to Vonage:
- Binary messages: Audio to play to the caller.
- Text messages: Commands to control playback or request notifications.
This bidirectional flow enables:
- Real-time transcription.
- Playback of synthesized speech.
- Control over playback buffers.
- Event-driven interactions.
Parsing Vonage Packets (Binary vs JSON)
When your WebSocket server receives a message:
- If the message is a Buffer or ArrayBuffer:
- It’s audio data (raw PCM).
- If the message is a string:
- It’s a JSON-formatted control message.
Example JSON event:
{
"event":"websocket:connected",
"content-type":"audio/l16;rate=16000",
"prop1": "value1",
"prop2": "value2"
}
Always inspect the message type to route processing logic correctly.
Handling Incoming Binary Audio Packets
Binary messages contain raw PCM audio captured from the caller.
Key characteristics:
- 16-bit signed little-endian PCM.
- Sample rate defined by
content-type(e.g., 16,000 Hz). - Each packet represents a short slice of audio (~20ms).
Typical processing:
- Feed the audio into a speech recognition engine.
- Buffer for later playback.
- Save to disk for analysis.
Sending Binary Audio Packets to Vonage
To play audio back to the caller:
- Encode your audio as raw PCM.
- Match the sample rate and format specified in the NCCO.
- Send the audio data as binary WebSocket messages.
Important:
Vonage buffers incoming audio to play in order. This allows you to queue audio without gaps but requires buffer management, explained as follows.
How Audio Buffering Works
When you send binary audio packets:
- Vonage buffers them internally.
- WebSocket buffer size is 3072 packets which should be enough for around 60 seconds of audio.
- Playback starts automatically.
- Subsequent packets are queued.
- You cannot interrupt playback mid-buffer without a control command.
This design ensures consistent playback without audio gaps.
Clearing the Audio Buffer (clear Command)
To immediately stop playback of buffered audio, send the clear command.
Outbound Command (from your application):
{
"action": "clear"
}
` Effect:
- All queued audio is discarded.
- Playback stops immediately.
Inbound Acknowledgement (from Vonage platform):
{
"event": "websocket:cleared"
}
Usage Scenario:
You need to interrupt playback to respond dynamically to the caller (e.g., after barge-in detection).
Notifying When Audio Ends (notify Command)
To get notified when the current audio buffer has finished playing, use the notify command.
Outbound Command (send after an audio payload that you want to know if it has finished playing):
{
"action": "notify",
"payload": {
"customKey": "customValue"
}
}
Behavior:
- If audio is playing, Vonage sends an inbound notification to your application after playback ends.
- If no audio is playing, the inbound notification is sent back to your application immediately.
Inbound Notification:
{
"event": "websocket:notify",
"payload": {
"customKey": "customValue"
}
}
Usage Scenario:
Synchronize your application logic (e.g., start recording or play a new prompt after the previous finishes).
JSON DTMF Messages
If any party on the call connected to the WebSocket sends a DTMF tone, this will trigger an event on the WebSocket. This event is a text message with a JSON payload, interleaved between the audio frames and having the following format:
{
"event": "websocket:dtmf",
"digit": "5",
"duration": 260
}
You will receive one event for each keypress and each event will contain only one digit:
eventallows you to identify it as a DTMF event.digitcontains the digit pressed0-9,*, or#.durationis the duration the key was pressed for in milliseconds; on most digital phone systems this will be a fixed length.
Listening to a Specific Participant in a Multiparty Conversation
When your application participates in a conversation with multiple call legs—such as a customer and an agent—you may want your WebSocket connection to only receive audio from a specific participant rather than the entire mixed audio.
This is called selective audio control, and it is achieved using the canHear and canSpeak properties of the NCCO conversation action.
Why Would You Use This?
- Speech analytics: Capture only what the customer says, ignoring the agent.
- Real-time transcription: Record the customer side for compliance.
- Whisper prompts: Send audio only to the agent without the customer hearing.
How to Configure Selective Listening
To set this up:
- Create a named conversation (e.g.,
"customer_support"). - Connect the customer and agent call legs to the conversation.
- Add your WebSocket connection to the same conversation, specifying
canHearandcanSpeakas needed.
Example: WebSocket Listening to Customer Only
Below is an NCCO example where:
- The customer is joined to the conversation.
- The agent is joined to the conversation.
- The WebSocket connects but only receives the customer’s audio.
Customer Leg NCCO:
[
{
"action": "conversation",
"name": "support_room"
}
]
Agent Leg NCCO:
[
{
"action": "conversation",
"name": "support_room"
}
]
WebSocket Leg NCCO:
[
{
"action": "conversation",
"name": "support_room",
"canHear": ["6a4d6af0-55a6-4667-be90-8614e4c8e83c"], // Customer leg ID
"canSpeak": []
}
]
How this works:
- The WebSocket only hears the
customerparticipant. - It does not send any audio back into the conversation (
canSpeakis empty). - If you want it to inject audio (e.g., AI prompts) and play the audio to only a designated participant, you could include the participant call (leg) ID in
canSpeak. - If you want it to inject audio (e.g., AI prompts) and play the audio to all participants, you do not include
canSpeakparameter.
Handling WebSocket Disconnections and Fallback Options
Note: If your endpoint uses failover, reconnect, keepalive, or connectionTimeout, see WebSocket Failover, Reconnection, and Keepalive for how Vonage handles disconnections automatically.
When you use WebSockets with the Vonage Voice API, you don’t just rely on the WebSocket itself to know what’s happening.
Vonage also sends event callbacks to your eventUrl webhook. These HTTP POST requests provide authoritative information about call status and enable fallback behavior if the WebSocket connection fails.
This is important because simply observing the WebSocket closing doesn’t tell you why it closed. You need the event webhook to determine whether the disconnection was intentional or caused by an error.
Why Does This Matter?
When building production voice experiences—especially AI-powered or real-time ones—connections may drop unpredictably (e.g., server crashes, network timeouts).
To provide a graceful experience to the caller, you can implement fallback strategies such as playing a prompt, transferring to a human agent, or ending the call politely.
Webhook events give you a reliable mechanism to detect these situations and act accordingly.
How Vonage Notifies You of WebSocket Events
Whenever there is a significant change in the WebSocket connection status, Vonage sends an event webhook to your eventUrl.
Examples of relevant statuses:
unanswered: Vonage could not establish the WebSocket connection.failed: The connection attempt failed.disconnected: The WebSocket connection was dropped after being established.
Each event includes:
- The
uuid,identifying the call. - Timestamps.
- Any custom
headersyou specified in the NCCOconnectaction. - The status field describing what happened.
Example Disconnected Event Payload
This event is sent when the WebSocket is disconnected after connection - whether due to an error or because your application closed it:
{
"from": "442079460000",
"to": "wss://example.com/socket",
"uuid": "aaaaaaaa-bbbb-cccc-dddd-0123456789ab",
"conversation_uuid": "CON-aaaaaaaa-bbbb-cccc-dddd-0123456789ab",
"status": "disconnected",
"timestamp": "2020-03-31T12:00:00.000Z",
"headers": {
"caller-id": "447700900123"
}
}
How to Distinguish Intentional vs. Unintentional Disconnects
Important to understand:
- When reconnection is not configured, a connection closed by your application or lost due to an error raises a
disconnectedevent. Withreconnectconfigured, an unclean close starts reconnection instead.disconnectedis sent if the attempt budget is exhausted or your server closes the connection cleanly. By default, the event does not distinguish an intentional close from a connection failure. - If the
websocketendpoint sets at least one of the resilience options (failover,connectionTimeout,keepalive, orreconnect), adisconnectedevent caused by your server closing the connection or by a connection loss includescause. It also includesclose_codeif a Close frame was received, soclose_codeis absent after a network loss or a keepalive timeout. These fields are available even if you do not setreconnect. - To intentionally end the WebSocket session, for example when your voice bot has finished, either:
- Close the WebSocket connection with code
1000. You still receive adisconnectedevent. If a resilience option is set, it includescause: remote_closedandclose_code: 1000, so you can recognize it as intentional. - Hang up the call leg via the Vonage Voice API. No
disconnectedevent is sent, onlycompleted. If you always end sessions this way, you receivedisconnectedonly for unintentional failures.
- Close the WebSocket connection with code
Handling Failed Connections During Setup
Sometimes the WebSocket connection cannot be established in the first place (for example, if your server is offline).
You can configure your connect action to use synchronous event handling:
Example NCCO with synchronous eventType:
[
{
"action": "connect",
"eventType": "synchronous",
"eventUrl": [
"https://example.com/events"
],
"from": "447700900000",
"endpoint": [
{
"type": "websocket",
"uri": "wss://example.com/socket",
"content-type": "audio/l16;rate=16000",
"headers": {
"caller-id": "447700900123"
}
}
]
}
]
How it works:
- If the connection attempt fails, Vonage immediately POSTs an event to your
eventUrl. - The event
statuswill beunansweredorfailed. - You can respond with a new NCCO describing fallback behavior, such as playing a message or redirecting the call.
Example Failed Connection Event Payload
{
"from": "442079460000",
"to": "wss://example.com/socket",
"uuid": "aaaaaaaa-bbbb-cccc-dddd-0123456789ab",
"conversation_uuid": "CON-aaaaaaaa-bbbb-cccc-dddd-0123456789ab",
"status": "unanswered",
"timestamp": "2020-03-31T12:00:00.000Z",
"headers": {
"caller-id": "447700900123"
}
}
Implementing Fallback Strategies
When you receive a webhook with status: disconnected, failed, or unanswered, you can:
- Return a new NCCO in your webhook response to handle the fallback (e.g., play a prompt).
- Allow the original NCCO to continue, if there are additional actions.
- End the call, if no further action is specified.
Example fallback NCCO:
[
{
"action": "talk",
"text": "We are unable to connect you at the moment. Please try again later."
}
]
WebSocket Failover, Reconnection, and Keepalive
A websocket endpoint can include optional settings that let Vonage handle connection problems for you, so the call can continue without extra recovery logic in your application:
- Failover: an ordered list of alternative URIs that Vonage tries if the primary
uricannot be reached, both when the call is set up and when reconnecting. - Automatic reconnection: Vonage re-establishes the connection if an established connection closes uncleanly.
- Connection timeout and keepalive: settings that control how long a connection attempt can take and how quickly Vonage detects an unresponsive connection.
Important:
- All of these settings are optional. An endpoint that uses none of them behaves as described in Handling WebSocket Disconnections and Fallback Options.
- If an endpoint sets any of them,
keepaliveis turned on, and your event webhooks include additional WebSocket fields. See the Webhook Reference for details. - Failover and reconnection attempts are part of the original connection. They do not count as new connections against your CPS limit, so you no longer need a high CPS limit to handle many connections dropping at once, for example when one of your servers restarts.
- You can use these settings wherever you can use a
websocketendpoint: in an NCCOconnectaction, and in thetoarray of aPOST /v1/callsrequest. For default values and allowed ranges, see the NCCO Reference. - Setting
"eventType": "synchronous"on theconnectaction is optional. You only need it if you want to respond to afailed,unanswered, ordisconnectedevent with a fallback NCCO.
The following NCCO connects a call to a WebSocket and sets all of the resilience options: failover, connectionTimeout, keepalive, and reconnect.
[
{
"action": "connect",
"eventUrl": ["https://example.com/events"],
"timeout": 20,
"from": "447700900000",
"endpoint": [
{
"type": "websocket",
"uri": "wss://primary.example.com/socket",
"content-type": "audio/l16;rate=16000",
"headers": {
"caller-id": "447700900123"
},
"authorization": {
"type": "vonage"
},
"failover": [
{
"uri": "wss://secondary.example.com/socket"
},
{
"uri": "wss://dr.example.net/socket",
"authorization": {
"type": "custom",
"value": "Bearer eyJhbGciOi..."
}
}
],
"connectionTimeout": 3,
"keepalive": {
"interval": 5,
"timeout": 5
},
"reconnect": {
"maxAttempts": 3,
"initialDelay": 0.5,
"maxDelay": 4,
"cleanCloseCodes": [4000]
}
}
]
}
]
Failover: Initial Connection
Add a failover array to the websocket endpoint to list up to 3 alternative URIs. If Vonage cannot connect to the primary uri when the call is set up, it tries these URIs in order.
The connection process works as follows:
- Vonage tries the primary
uri. - If that attempt fails, Vonage immediately tries each
failoverentry in order. - When a URI connects, your
eventUrlreceives anansweredevent. Theconnected_urifield shows which URI Vonage connected to. - If every URI fails, your
eventUrlreceives afailedorunansweredevent. Theattemptsarray lists the outcome for each URI that Vonage tried. You can respond with a fallback NCCO, as described in Handling Failed Connections During Setup.
For the fields of these events, see answered, failed, and unanswered in the Webhook Reference.
An attempt to a URI fails if:
- It does not complete within
connectionTimeout. - The connection is refused or cannot be made.
- A TLS error occurs.
- The handshake returns any response other than HTTP
101.
The failover entries follow these rules:
- Each entry must contain a
uri(ws://orwss://) and can also containheadersandauthorization. Any other fields are ignored. - An entry without
headersuses theheadersof the primary endpoint. An entry withoutauthorizationuses theauthorizationof the primary endpoint. - If an entry sets its own
headers, they replace the primary endpoint'sheaders. The two are not merged. content-typeis always taken from the primary endpoint.
Note: The endpoint's connectionTimeout applies separately to each URI that Vonage tries. If every attempt times out, the initial connection can take up to (1 + number of failover entries) x connectionTimeout. For example, with three failover entries and the default timeout of 3 seconds, this is 4 x 3 = 12 seconds.
Automatic Reconnection
To turn on automatic reconnection, include the reconnect object in the websocket endpoint. Vonage reconnects only after an unclean close. If reconnect is not set, any close ends the WebSocket leg, as described in Handling WebSocket Disconnections and Fallback Options.
Clean and Unclean Closes
Clean Closes
Vonage does not reconnect after these:
- Your server sends a WebSocket Close frame with status code
1000(Normal Closure). - Your server sends a Close frame with no status code. Many libraries report this as
1005. Some libraries send this when you callclose()with no arguments. - Your server sends a Close frame with a code you listed in
reconnect.cleanCloseCodes. - Vonage closes the connection because the call or leg has ended. For example, the other party hangs up, the leg is ended through the Voice API, or the
connectaction'slimitis reached.
Unclean Closes
Vonage reconnects after these if reconnect is set:
- Your server sends a Close frame with any other code. Common examples are
1001(Going Away),1011(Internal Error),1012(Service Restart), and1013(Try Again Later). - The connection drops without a Close frame, for example because of a network failure, a server crash, or a load balancer dropping the connection. Many libraries report this as
1006. - The connection becomes unresponsive, meaning no Pong arrives within
keepalive.timeout. See Connection Timeout and Keepalive. - A TLS or protocol error occurs on an established connection.
Note: Vonage treats code 1001 (Going Away) as an unclean close on purpose. Servers commonly send it while they shut down or restart, which is when reconnecting is most useful.
How Reconnection Works
When an established connection closes uncleanly and reconnect is set, Vonage reconnects as follows:
- Vonage detects the unclean close, and your
eventUrlreceives areconnectingevent. - Vonage waits for the backoff delay, then makes a reconnection attempt. In each attempt, Vonage:
- Tries the URI that was last connected first.
- Then tries the remaining URIs from the primary and failover list, in order.
- Applies
connectionTimeoutseparately to each URI.
- If no URI connects, Vonage waits for the next, longer backoff delay and makes another attempt, up to
maxAttemptsattempts. - If an attempt succeeds:
- Your
eventUrlreceives areconnectedevent. - Your server receives the usual
websocket:connectedmessage, with theheadersof the URI it connected to and areconnect_attemptfield.
- Your
- If all attempts fail:
- Your
eventUrlreceives adisconnectedevent and the NCCO continues. - You can respond with a fallback NCCO, as described in Implementing Fallback Strategies.
- Your
The backoff delay works as follows:
- Before the first attempt, Vonage waits
initialDelay. - The wait doubles before each later attempt, up to
maxDelay, with a random variation of up to 10%. - With the default values, Vonage waits about 0.5, 1, and 2 seconds before the three attempts.
After a successful reconnection, your server receives a websocket:connected message like this:
{
"event": "websocket:connected",
"content-type": "audio/l16;rate=16000",
"caller-id": "447700900123",
"reconnect_attempt": 1
}
See the Webhook Reference for the payload and all fields of the reconnecting, reconnected, and disconnected events.
Reconnection Attempt Budget
maxAttempts is a budget for the connection, not for each time the connection drops:
- The budget is restored once a connection has stayed up for 30 seconds.
- If a reconnected session drops again within 30 seconds, the attempts already used still count. The
reconnect_attemptfield inwebsocket:connectedkeeps counting in the same way. - If the budget is already used up, Vonage does not try again, and your
eventUrlreceives adisconnectedevent.
What Happens During Reconnection
While Vonage is reconnecting:
- The call stays up. The leg
uuidandconversation_uuiddo not change. - Audio and DTMF events from the call during the gap are not delivered to your server.
- Any audio your server had queued for playback before the connection closed is discarded. Playback starts fresh on the new connection. See How Audio Buffering Works.
- Any
notifymarkers in that queue are discarded with it. Your server receives neitherwebsocket:notifyfor them norwebsocket:cleared. - If the call ends, reconnection stops. You receive the normal end-of-call events, and no further WebSocket events.
- The
reconnectingandreconnectedevents are for information only. Vonage ignores any NCCO returned in response to them.
Ending a Session Intentionally
To end a WebSocket session on purpose, for example when your voice bot has finished, close the connection with code 1000. Vonage treats this as a clean close and does not reconnect. Your eventUrl receives a disconnected event with cause set to remote_closed, close_code set to 1000, and reconnect_attempts set to 0.
You can also hang up the leg through the Voice API, as before. In that case, Vonage sends completed but not disconnected.
Authorization Across Connection Attempts
When authorization is set to "type": "vonage", Vonage issues one token for the call. The same token is sent on every connection attempt: to the primary URI, to failover URIs, and on every reconnection. The token's iat claim is the time the call was set up, and its jti claim does not change between attempts.
Note: If your server rejects reused jti values, or tokens older than a given age, it will reject reconnection attempts.
Connection Timeout and Keepalive
Two settings control how Vonage deals with slow or unresponsive connections: connectionTimeout limits how long a connection attempt can take, and keepalive controls how quickly Vonage detects an unresponsive connection.
Connection Timeout
connectionTimeout is the maximum time allowed for one connection attempt to one URI. The time limit covers the whole process up to a completed WebSocket handshake:
- Name resolution
- TCP connection
- TLS negotiation
- The HTTP upgrade
Keepalive
On an established connection, Vonage uses keepalive to check that your server is still responding:
- Vonage sends a WebSocket Ping frame every
keepalive.intervalseconds. - Vonage waits up to
keepalive.timeoutseconds for a Pong reply. Only a Pong counts. Other frames from your server do not. - If no Pong arrives in time, Vonage treats the connection as unresponsive. This is an unclean close, so Vonage reconnects if
reconnectis set.
keepalive is turned on automatically whenever the endpoint sets any of failover, connectionTimeout, keepalive, or reconnect. If you do not set keepalive, its default values are used.
Note: Most WebSocket server libraries reply to Ping frames automatically, as the WebSocket standard requires. You should not need to change your code to support keepalive.
Sizing Your Timeouts
After an unclean close, the time a caller may hear silence is the sum of the time to detect the close, the backoff delays, and the connection attempts.
For the example NCCO above, the worst case, where every attempt to every URI times out, is about 40 seconds:
- Detection of an unresponsive connection: 10 seconds (
keepalive.interval+keepalive.timeout). - Backoff delays: 3.5 seconds (0.5 + 1 + 2).
- Connection attempts: 27 seconds (3 attempts x 3 URIs x 3 seconds).
A refused connection, where the server is up but nothing is listening, fails within milliseconds, so recovery is usually much faster. If the host does not answer at all, each attempt waits the full connectionTimeout.
Choose values so that the worst case is acceptable for your callers.
Connecting to AI Engines
Find Vonage sample applications to connect to popular AI engines below: