Live Captions Translation (Private Beta)

Use Live Captions Translation to translate real-time captions into different languages for participants in a Vonage Video session.

Private Beta

Note: Live Captions Translation is provided as a private beta feature. Contact us to enable Live Captions Translation for your project.

The private beta is available for selected customers to preview and evaluate the feature and provide feedback. In order to incorporate feedback and adapt the product to customer needs, it may be necessary to make breaking changes that affect the APIs and customer code. Please be aware that it may be necessary to modify code written during the private beta phase after the product is made generally available.

This page includes the following sections:

Private Beta
Overview
Session-level translation
Publisher-level input language
Subscriber-level translation
Supported languages
Notes

Overview

Live Captions Translation extends the Live Captions feature by adding real-time translation of captions. With Live Captions, audio is transcribed in the spoken language. With Live Captions Translation, those captions can be automatically translated into a different language for participants.

Before using Live Captions Translation, make sure you are familiar with the Live Captions feature, including how to start captions using the REST API and how to subscribe to captions in the client SDKs.

Translation can be enabled at two levels:

Session-level translation — Set a translation language for all participants when starting captions via the server-side API.
Subscriber-level translation — Individual subscribers choose their own translation language using the client SDKs, independently of the session-level setting.

These two levels can be used together or independently.

Session-level translation

When calling the Start Captions REST API (POST /v2/project/<apiKey>/captions), you can include an optional targetLanguageCode parameter in the request body:

{
  "sessionId": "<session-id>",
  "token": "A valid OpenTok token with the role set to moderator",
  "languageCode": "en-US",
  "targetLanguageCode": "es-ES",
  "maxDuration": 1800,
  "partialCaptions": true
}

targetLanguageCode (String) — (Optional) The BCP-47 language code for the target translation language (see the supported languages list). When set, all participants in the session will receive translated captions in the specified language instead of the original transcribed language.

Behavior:

If targetLanguageCode is not set (or omitted), captions work as normal — participants receive captions in the original spoken language defined by languageCode.
If targetLanguageCode is set, all participants in the session receive captions translated into the specified language by default.
A subscriber who sets their own translation language (see Subscriber-level translation) will receive captions in their chosen language, overriding the session-level targetLanguageCode.

Publisher-level input language

Publishers can specify the language they are speaking using the client SDKs, allowing the captioning service to correctly interpret their audio. This is useful in multilingual sessions where different publishers may speak different languages. The publisher-level input language can be changed at any time during a session.

You can configure the publisher input language using any of the Vonage Video API client SDKs, including Web (OpenTok.js), iOS, Android, Windows, and Linux. Below are reference examples for each platform.

Setting the input language (Web)

By default, the captions input language is unset (null), letting the server use the languageCode defined when starting captions. To change the input language for a specific publisher, call setCaptionsInputLanguage with a valid BCP-47 language code. Pass null to revert to the default.

// Example BCP-47 language code for captions input language
const captionsLanguageCode = 'fr-FR'

// Set the publisher's captions input language
try {
    await publisher.setCaptionsInputLanguage(captionsLanguageCode)
} catch (err) {
    console.error(`Failed to set captions input language: ${err.message}`)
}

// Pass null to revert to the default language
try {
    await publisher.setCaptionsInputLanguage(null)
} catch (err) {
    console.error(`Failed to revert captions input language: ${err.message}`)
}

Getting the current input language (Web)

Use getCaptionsInputLanguage to check the publisher's currently configured input language. It returns the selected BCP-47 language code, or null if no input language has been set.

const captionsLanguage = publisher.getCaptionsInputLanguage()
if (captionsLanguage) {
    console.log(`Captions input language is set to: ${captionsLanguage}`)
} else {
    console.log('No input language set; using the session default.')
}

Setting the input language (Android)

Use the setCaptionsInputLanguage() method of the PublisherKit object to set the input language. Pass null to revert to the default language.

// Set the publisher's captions input language
publisher.setCaptionsInputLanguage("fr-FR");

// Revert to the default language
publisher.setCaptionsInputLanguage(null);

Getting the current input language (Android)

Use the getCaptionsInputLanguage() method to check the publisher's currently configured input language. It returns the selected BCP-47 language code, or null if no input language has been set.

String captionsLanguage = publisher.getCaptionsInputLanguage();
if (captionsLanguage != null) {
    Log.d(TAG, "Captions input language is set to: " + captionsLanguage);
} else {
    Log.d(TAG, "No input language set; using the session default.");
}

Setting the input language (iOS)

Set the captionsInputLanguage property of the OTPublisherKit object to the desired BCP-47 language code. Set it to nil to revert to the default language.

// Set the publisher's captions input language
publisher.captionsInputLanguage = @"fr-FR";

// Revert to the default language
publisher.captionsInputLanguage = nil;

Getting the current input language (iOS)

Read the captionsInputLanguage property to check the publisher's currently configured input language. It returns the selected BCP-47 language code, or nil if no input language has been set.

NSString *captionsLanguage = publisher.captionsInputLanguage;
if (captionsLanguage) {
    NSLog(@"Captions input language is set to: %@", captionsLanguage);
} else {
    NSLog(@"No input language set; using the session default.");
}

Setting the input language (Windows)

Set the CaptionsInputLanguage property of the Publisher object to the desired BCP-47 language code. Set it to null to revert to the default language.

// Set the publisher's captions input language
publisher.CaptionsInputLanguage = "fr-FR";

// Revert to the default language
publisher.CaptionsInputLanguage = null;

Getting the current input language (Windows)

Read the CaptionsInputLanguage property to check the publisher's currently configured input language. It returns the selected BCP-47 language code, or null if no input language has been set.

string captionsLanguage = publisher.CaptionsInputLanguage;
if (captionsLanguage != null)
{
    Console.WriteLine($"Captions input language is set to: {captionsLanguage}");
}
else
{
    Console.WriteLine("No input language set; using the session default.");
}

Setting the input language (Linux)

Use the otc_publisher_set_captions_input_language function to set the publisher's input language. Pass NULL to revert to the default language.

// Set the publisher's captions input language
otc_publisher_set_captions_input_language(publisher, "fr-FR");

// Revert to the default language
otc_publisher_set_captions_input_language(publisher, NULL);

Getting the current input language (Linux)

Use otc_publisher_get_captions_input_language to check the publisher's currently configured input language. It returns the selected BCP-47 language code, or NULL if no input language has been set.

const char* captionsLanguage = otc_publisher_get_captions_input_language(publisher);
if (captionsLanguage != NULL) {
    printf("Captions input language is set to: %s\n", captionsLanguage);
} else {
    printf("No input language set; using the session default.\n");
}

Subscriber-level translation

Subscribers can choose their own translation language using the client SDKs, and can change it at any time during a session. This works independently of the session-level targetLanguageCode.

You can configure subscriber-level translation using any of the Vonage Video API client SDKs, including Web (OpenTok.js), iOS, Android, Windows, and Linux. Below are reference examples for each platform.

Setting the translation language (Web)

The default captions language is determined by the languageCode (or targetLanguageCode) set in the Start Captions API. To change the translation language for a specific subscriber, call setCaptionsTranslationLanguage with a valid BCP-47 language code. Pass null to revert to the default language.

// Example BCP-47 language code for captions translation
const captionsLanguageCode = 'es-ES'

// Set the subscriber's captions translation language
try {
    await subscriber.setCaptionsTranslationLanguage(captionsLanguageCode)
} catch (err) {
    console.error(`Failed to set captions translation language: ${err.message}`)
}

// Pass null to revert to the default language
try {
    await subscriber.setCaptionsTranslationLanguage(null)
} catch (err) {
    console.error(`Failed to revert captions translation language: ${err.message}`)
}

Getting the current translation language (Web)

Use getCaptionsTranslationLanguage to check the subscriber's currently selected translation language. It returns the selected BCP-47 language code, or null if no translation language has been set.

const captionsLanguage = subscriber.getCaptionsTranslationLanguage()
if (captionsLanguage) {
    console.log(`Captions translation language is set to: ${captionsLanguage}`)
} else {
    console.log('No translation language set; using the session default.')
}

Setting the translation language (Android)

Use the setCaptionsTranslationLanguage() method of the SubscriberKit object to set the translation language. Pass null to revert to the default language.

// Set the subscriber's captions translation language
subscriber.setCaptionsTranslationLanguage("es-ES");

// Revert to the default language
subscriber.setCaptionsTranslationLanguage(null);

Getting the current translation language (Android)

Use the getCaptionsTranslationLanguage() method to check the subscriber's currently selected translation language. It returns the selected BCP-47 language code, or null if no translation language has been set.

String captionsLanguage = subscriber.getCaptionsTranslationLanguage();
if (captionsLanguage != null) {
    Log.d(TAG, "Captions translation language is set to: " + captionsLanguage);
} else {
    Log.d(TAG, "No translation language set; using the session default.");
}

Setting the translation language (iOS)

Set the captionsTranslationLanguage property of the OTSubscriberKit object to the desired BCP-47 language code. Set it to nil to revert to the default language.

// Set the subscriber's captions translation language
subscriber.captionsTranslationLanguage = @"es-ES";

// Revert to the default language
subscriber.captionsTranslationLanguage = nil;

Getting the current translation language (iOS)

Read the captionsTranslationLanguage property to check the subscriber's currently selected translation language. It returns the selected BCP-47 language code, or nil if no translation language has been set.

NSString *captionsLanguage = subscriber.captionsTranslationLanguage;
if (captionsLanguage) {
    NSLog(@"Captions translation language is set to: %@", captionsLanguage);
} else {
    NSLog(@"No translation language set; using the session default.");
}

Setting the translation language (Windows)

Set the CaptionsTranslationLanguage property of the Subscriber object to the desired BCP-47 language code. Set it to null to revert to the default language.

// Set the subscriber's captions translation language
subscriber.CaptionsTranslationLanguage = "es-ES";

// Revert to the default language
subscriber.CaptionsTranslationLanguage = null;

Getting the current translation language (Windows)

Read the CaptionsTranslationLanguage property to check the subscriber's currently selected translation language. It returns the selected BCP-47 language code, or null if no translation language has been set.

string captionsLanguage = subscriber.CaptionsTranslationLanguage;
if (captionsLanguage != null)
{
    Console.WriteLine($"Captions translation language is set to: {captionsLanguage}");
}
else
{
    Console.WriteLine("No translation language set; using the session default.");
}

Supported languages

Live Captions Translation uses the same BCP-47 language codes as Live Captions. See the supported languages list for the full list of available languages.

Notes

Live Captions Translation requires Live Captions to be active for the session. You must start captions using the Start Captions API before translation is available.
Setting a subscriber-level translation language overrides the session-level targetLanguageCode for that subscriber only. Passing null reverts the subscriber to the default behavior (session-level targetLanguageCode if set, otherwise the original languageCode).
For the base Live Captions feature documentation — including architecture, publishing captions, subscribing to captions, status webhooks, and known issues — see the Live Captions guide.