Custom Video Rendering

Overview

The Vonage Video API allows for modifications to be made to the video renderer to be used in Android and iOS applications.

This how-to will go over:

• Make modifications to the video renderer in your Vonage Video Android application
• Make modifications to the video renderer in your Vonage Video iOS application

Android

Before you start

This how-to shows the code required to make minor modifications to the video renderer used by a Subscriber object.

You can also use the same techniques to modify the video renderer used by a Publisher object (though this example only illustrates a custom renderer for a subscriber).

The code for this section is available in the Basic-Video-Renderer-Java project of the opentok-android-sdk-samples repo. If you haven't already, you'll need to clone the repo into a local directory. On the command line, run:

Open the Basic-Video-Renderer-Java project in Android Studio to follow along.

Exploring the Code

In this example, the app uses a custom video renderer to display a inverted color version of the video.

InvertedColorsVideoRenderer is a custom class that extends the BaseVideoRenderer class (defined in the Android SDK). The BaseVideoRenderer class lets you define a custom video renderer to be used by a Vonage Video publisher or subscriber:

The InvertedColorsVideoRenderer() constructor sets a renderer property to a GLSurfaceView object. The app uses this object to display the video using OpenGL ES 2.0. The renderer for this GLSurfaceView object is set to a MyRenderer object. MyRenderer is a custom class that extends GLSurfaceView.Renderer, and it is used to render the video to the GLSurfaceView object:

The onFrame() method of the video renderer is inherited from the BaseVideoRenderer class. The BaseVideoRenderer.onFrame() method is called when the publisher (or subscriber) renders a video frame to the video renderer.

The InvertedColorsVideoRenderer implementation of this method, it takes the frame's image buffer (YUV representation of the frame), passes it to the displayFrame method of the MyRenderer object and calls the requestRender() method of the GLSurfaceView object:

To render the video frames, the renderer class uses OpenGL shaders. In this sample shader produces the inverted color effect, more precisely this is achieved by this line which is inside the fragmentShaderCode string:

iOS

Before you start

This how-to shows the code required to make minor modifications to the video renderer used by an OTPublisher object.

You can also use the same techniques to modify the video renderer used by an OTSubscriber object (though this example only illustrates a custom renderer for a publisher).

The code for this section is in the Basic Video Renderer project of the opentok-ios-sdk-samples repo, so if you haven't already, you'll need to clone the repo into a local directory — this can be done using the command line:

Change directory to the Basic Video Renderer project:

Then install the Vonage Video dependency:

Open the project in Xcode to follow along.

Exploring the Code

In this example, the app uses a custom video renderer to display a black-and-white version of the OTPublisher object's video.

In the main ViewController, after initializing the OTPublisher object, the videoRender property of the OTPublisher object is set to an instance of OTKBasicVideoRender:

OTKBasicVideoRender is a custom class that implements the OTVideoRender protocol (defined in the iOS SDK). This protocol lets you define a custom video renderer to be used by a Vonage Video publisher or subscriber.

The [OTKBasicVideoRender init:] method sets a _renderView property to a UIView object. This is the UIView object that will contain the view to be rendered (by the publisher or subscriber). In this sample, the UIView object is defined by the custom OTKCustomRenderView class, which extends UIView:

The OTKCustomRenderView class includes methods (discussed later) that convert a video frame to a black-and-white representation.

The [OTVideoRender renderVideoFrame:] method is called when the publisher (or subscriber) renders a video frame to the video renderer. The frame an OTVideoFrame object (defined by the iOS SDK). In the OTKCustomRenderView implementation of this method, it takes the frame and passes it along to the [renderVideoFrame] method of the OTKCustomRenderView object:

The [OTKCustomRenderView renderVideoFrame] method iterates through the pixels in the plane, adjusts each pixel to a black-and-white value, adds the value to a buffer. It then writes the buffer to a CGImageRef representing the view's image, and calls [self setNeedsDisplay] to render the image view:

See also

• Vonage Video API Android SDK reference
• Vonage Video API iOS SDK reference