Media Processor (iOS)

You can use pre-built transformers in the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video.

You can use the OTPublisherKit.audioTransformers and OTPublisherKit.videoTransformers properties to apply audio and video transformers to a stream.

The Vonage Video iOS SDK includes two ways to implement transformers:

• Moderate — For video, you can apply the background blur video transformer included in the Vonage Media Library. See Applying a video transformer from the Vonage Media Library.

• Advanced — You can create your own custom video transformers and custom audio transformers.

Client requirements

The transformers from the Vonage Media Library are supported on Apple A11 Bionic chipset and above, on the following device-operating system combinations:

Test on other devices to check for support.

Transformers require adequate processor support. Even on supported devices, transformers may not be stable when background processes limit available processing resources. The same limitations may apply with custom media transformers in addition to transformers from the Vonage Media Library.

iOS may throttle CPU performance to conserve energy (for example, to extend battery life). This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend disabling low-power mode in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of the transformation.

Vonage Media Library integration

Due to significant increased size when integrating Vonage Media Library into SDK, from OpenTok SDK v2.27.4 the Media Transformers are available via the opt-in Vonage Media library. This library needs to explicitly be added to the project.

The Vonage Media Library was initially embedded in OpenTok SDK. If your OpenTok SDK version is older than 2.27.4, move directly to Applying a video transformer from the Vonage Media Library and Applying an audio transformer from the Vonage Media Library.

The Vonage Media Library is available as the Pod "VonageClientSDKVideoTransformers", for use with CocoaPods.

If a call to [OTVideoTransformer initWithName:properties:] or [OTAudioTransformer initWithName:properties:] is made without loading the library, the transformer returned will be null. An exception will be raised with the following error code 0x0A000006 - OTC_MEDIA_TRANSFORMER_OPENTOK_TRANSFORMERS_LIBRARY_NOT_LOADED.

Applying a video transformer from the Vonage Media Library

Use the [OTVideoTransformer initWithName:properties:] method to create a video transformer that uses a named transformer from the Vonage Media Library.

Two transformers are supported:

• Background blur. For this filter, set the name parameter to "BackgroundBlur". And set a properties parameter to a JSON string. The format of the JSON is "{"radius":"None"}". Valid values for the radius property are "None", "High", and "Low". If you set the radius property to "Custom", add a custom_radius property to the JSON string: "{"radius":"Custom","custom_radius":"value"}" (where custom_radius is a positive integer defining the blur radius).

• Background replacement. For this filter, set the name parameter to "BackgroundReplacement". And set a properties parameter to a JSON string. The format of the JSON is "{"image_file_path":"path/to/image"}", where image_file_path is the absolute file path of a local image to use as virtual background. Supported image formats are PNG and JPEG.

Creating a custom video transformer

Create a class that implements the OTCustomVideoTransformer protocol. Implement the `[OTCustomVideoTransformer transform:]` method, applying a transformation to the `OTVideoFrame` object passed into the method. The `[OTCustomVideoTransformer transform:]` method is triggered for each video frame:

Then set the OTPublisherKit.videoTransformers property to an array that includes the object that implements the OTCustomVideoTransformer interface:

You can combine the Vonage Media library transformer (see the previous section) with custom transformers or apply multiple custom transformers by adding multiple PublisherKit.VideoTransformer objects to the ArrayList used for the OTPublisherKit.videoTransformers property.

Applying a video transformer from the Vonage Media Library

Use the [OTVideoTransformer initWithName:properties:] method to create a video transformer that uses a named transformer from the Vonage Media Library.

Two transformers are supported:

• Background blur. For this filter, set the name parameter to "BackgroundBlur". And set a properties parameter to a JSON string. The format of the JSON is "{"radius":"None"}". Valid values for the radius property are "None", "High", and "Low". If you set the radius property to "Custom", add a custom_radius property to the JSON string: "{"radius":"Custom","custom_radius":"value"}" (where custom_radius is a positive integer defining the blur radius).

• Background replacement. For this filter, set the name parameter to "BackgroundReplacement". And set a properties parameter to a JSON string. The format of the JSON is "{"image_file_path":"path/to/image"}", where image_file_path is the absolute file path of a local image to use as virtual background. Supported image formats are PNG and JPEG.

Creating a custom audio transformer

Create a class that implements the OTCustomAudioTransformer protocol. Implement the [OTCustomAudioTransformer transform:] method, applying a transformation to the OTAudioData object passed into the method. The [OTCustomAudioTransformer transform:] method is triggered for each audio frame. The following example applies a simple amplitude limiter on the audio:

Then set the OTPublisherKit.audioTransformers property to an array that includes the object that implements the CustomAudioTransformer interface:

You can apply multiple custom transformers by adding multiple OTPublisherKit.AudioTransformer objects to the ArrayList used for the OTPublisherKit.audioTransformers property.

Clearing video transformers for a publisher

To clear video transformers for a publisher, set the OTPublisherKit.videoTransformers property to an empty array.

Clearing audio transformers for a publisher

To clear audio transformers for a publisher, set the OTPublisherKit.audioTransformers property to an empty array.

Sample app

See this sample at the opentok-ios-sdk-samples repo on GitHub.