Media Processor (Web)

Use the Vonage Media Processor library to apply custom transformations to published video.

You can use the Publisher.setAudioMediaProcessorConnector() and Publisher.setVideoMediaProcessorConnector() methods to apply custom audio and video transformers that use the Vonage Media Processor library.

There are three ways to implement transformers:

• Basic: Use the Publisher.applyVideoFilter() method to easily apply the pre-defined background blur or background replacement filter. This method implements these filters using the Vonage Media Library, requiring only a few configuration settings:

• Moderate: You can use transformers from the Vonage ML transformers library.

See thebackground image replacement and background blur sample apps.

• Advanced: You can write your own transformer class that implements the Transformer class:

The Vonage Media Processor library, available on npm, facilitates the use of the Insertable Streams API to perform transform operations on video and audio tracks.

The library includes two key classes:

• MediaProcessor — Controls transformers and includes a setTransformers() method that lets you set transformers.
• MediaProcessorConnector — Accepts a MediaProcessor object in its constructor and handles input and output tracks. The OpenTok.js implementation handles the tracks. You only have to provide the MediaProcessorConnector instance.

To use a MediaProcessorConnector to connect a transformer to a publisher's audio or video, use the Publisher.setAudioMediaProcessorConnector() and Publisher.setVideoMediaProcessorConnector() methods.

Publisher.setVideoMediaProcessorConnector() method

The Publisher.setVideoMediaProcessorConnector() method applies a video transformer to a published stream:

Publisher.setAudioMediaProcessorConnector() method

The Publisher.setAudioMediaProcessorConnector() applies an audio transformer to a published stream:

Client requirements

The Insertable Streams API used by the Publisher.setVideoMediaProcessorConnector() and Publisher.setAudioMediaProcessorConnector() methods is only supported in recent versions of Chrome, Electron, Opera, and Edge. It is not supported in other (non-Chromium-based) browsers or on iOS. You can check if the client supports this feature by calling the OT.hasMediaProcessorSupport() method.

Transformers require adequate processor support. Even in supported browsers, transformers may not be stable when background processes limit available processing resources. Use Chrome (desktop or Android) for optimal results.

For the best user experience, ensure laptop/desktop computers meet these minimum requirements:

• CPU — Intel Core i5 (with 4 cores or higher) or Apple Silicon M1 (or higher)
• 8G RAM (or more)
• Dedicated GPU (for example, NVIDIA)

Some operating systems may offer multiple battery settings or modes to conserve energy (for example, to extend laptop battery life) by throttling CPU performance. This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend setting your operating system to use best performance mode (or to to not use low-power mode) in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of the transformation.

More about transformers and sample code

For a more information, see the Vonage Media Processor library documentation.

The video-api-web-samples repo includes the following sample apps using the Vonage Media Processor library to apply transformations to streams published in OpenTok.js:

• Basic-Audio-Transformer — Shows how to apply an audio transformer using a web worker and a separate thread for the web worker.
• Basic-Video-Transformer — Shows how to apply a video transformer using a canvas, a web worker, and a separate thread for the web worker.
• Shows how to use a basic transformer to zoom the publisher.
• Moderate-Background-Blur — Shows how to apply a video transformer from the @vonage/ml-transformers module.

There are additional samples that use the Vonage Media Transformers library here.

Other user interface customization options

For other available options for customizing the user interface, see the Customizing the UI guide.