Applying filters and effects to published audio and video
You can apply filters and effects to published audio and video in the following ways:
- Publishing a MediaStreamTrack object and applying effects to the object
- Applying a background blur or background replacement filter
- Applying the advanced noise suppression filter
- Using the Vonage Media Processor library to apply custom transformations
You can apply filters and effects to subscribed audio and video by using the Vonage Media Processor library to apply transformations.
These are described in the sections that follow.
Publishing a MediaStreamTrack object and applying effects to the object
You can use an audio MediaStream track or a video MediaStream track as the source audio or video for a published stream. Using this feature, you can apply filters and effects, such as background blur or background replacement, to the published audio or video.
You can use the OT.getUserMedia() method to get a reference to a MediaStream that uses the camera selected by the user. You can then use the video MediaStreamTrack obtained from that MediaStream object as the source for a Video element. You can then add that Video element to an HTML Canvas element, apply filters or effects to the canvas, and use the filtered video MediaStreamTrack object obtained from the canvas as the video source for a published stream. For an example, see the Stream-Filter sample opentok-web-samples repo on GitHub.
You can use the OT.getUserMedia() method to get a reference to a MediaStream that uses the microphone selected by the user. You can then use the audio MediaStreamTrack obtained from that MediaStream object as the as the audioSource when calling OT.initPublisher(). You can then create an AudioContext object and call its createMediaStreamSource() object, passing in the MediaStream object to create a MediaStreamAudioSourceNode object. You can then apply audio filters to the MediaStreamAudioSourceNode object, which will result in the filters being applied to the published stream.
Applying a background blur or background replacement filter
Use the Publisher.applyVideoFilter() method to apply a video filter for the publisher. You can apply a background blur filter or a background image filter. A publisher can have only one (or zero) filters applied at one time. When you set a new filter, a previously set filter is removed.
The following code applies a background blur filter to a publisher's video:
The following code applies a background replacement filter to a publisher's video:
You can also apply a video filter by setting the videoFilter option when calling the OT.initPublisher() method:
Important:
- Video filters require adequate processor support. Even in supported browsers, transformers may not be stable when background processes limit available processing resources. See Client requirements.
- The background image is resized to match the dimensions of the publisher's video. For best results, use an image that has the same dimensions (or at least the same aspect ratio) as the published video. Set the
mirroroption tofalsewhen calling theOT.initPublisher()method to have the background image appear in the correct orientation (not mirrored) in the publisher's page.
For details, see the reference documentation for Publisher.applyVideoFilter().
This method is incompatible with the Publisher.setVideoMediaProcessorConnector() method. Calling this method after Publisher.setVideoMediaProcessorConnector(connector) returns a promise that is rejected with an error, with the name property of the error set to 'OT_NOT_SUPPORTED'. You can remove the connector by calling Publisher.setVideoMediaProcessorConnector(null).
You can also set the initial video filter by setting a videoFilter option when calling OT.initPublisher() method.
To clear the video filter, call the Publisher.clearVideoFilter() method.
Notes:
- Video filters are only supported in recent versions of Chrome, Electron, Opera, and Edge. They are not supported in other (non-Chromium-based) browsers or on iOS. You can check if the client supports this feature by calling the
OT.hasMediaProcessorSupport()method. - The Vonage Media Library includes transformers that provide more options than are provided with the background blur and background image filters provided by the
Publisher.applyVideoFilter()method. Also, you can use the Vonage Media Library to write your own custom audio and video transformers. See the Using the Vonage Media Processor developer guide.
Applying the advanced noise suppression filter
Note: This is a beta feature.
Advanced noise suppression is an audio filter that greatly reduces the background noise around the publisher. This can be useful when a participant is in a crowded or noisy environment, like in a cafe, or if white noise is present, like from an HVAC system. It can work separately or in conjunction with the noiseSuppression property set when calling the OT.initPublisher method.
Use the Publisher.applyAudioFilter() method to apply an audio filter for the publisher. You can apply an advanced noise suppression filter. A publisher can have only one (or zero) filters applied at one time. When you set a new filter, any previously set filter is removed. This is a beta feature.
The following code applies an advanced noise suppression audio filter to a publisher's audio:
You can also apply the advanced noise suppression audio filter by setting the audioFilter option when calling the OT.initPublisher() method:
Important: Audio filters require adequate processor support. Even in supported browsers, transformers may not be stable when background processes limit available processing resources. See Client requirements.
For details, see the reference documentation for Publisher.applyAudioFilter().
This method is incompatible with the Publisher.setAudioMediaProcessorConnector() method. Calling this method after Publisher.setAudioMediaProcessorConnector(connector) returns a promise that is rejected with an error, with the name property of the error set to 'OT_NOT_SUPPORTED'. You can remove the connector by calling Publisher.setAudioMediaProcessorConnector(null).
To clear the advanced noise suppression filter, call the Publisher.clearAudioFilter() method.
Notes:
- Audio filters are only supported in recent versions of Chrome, Electron, Opera, and Edge. They are not supported in other (non-Chromium-based) browsers or on iOS. You can check if the client supports this feature by calling the
OT.hasMediaProcessorSupport()method. - When using
Publisher.applyAudioFilter(), assets are downloaded from Vonage servers. If you prefer to self-host these assets, please refer to this blog post for information on hosting, and for using thePublisher.setAudioMediaProcessorConnector()method to apply the advanced noise suppression filter with self-hosted assets.
Using the Vonage Media Processor library to apply custom transformations
See the Using the Vonage Media Processor developer guide.
Adjusting audio and video
Learn more about manipulating audio and video of Vonage Video API streams for your web application. Publish only video or audio, adjust the frame rate, and more.