Adjusting audio and video — Android

You can make audio and video adjustments to published and subscribed streams:

Publishing audio or video only

You can toggle audio and video on or off, by calling the setPublishAudio() and setPublishVideo() methods of the Publisher object (passing in a Boolean value). For example, the following code turns audio off:

mPublisher.setPublishAudio(false);

Publishing in a voice-only session

To set up a voice-only session, call the videoTrack() method of the Publisher.Builder object, passing in false, when you instantiate each Publisher object in the session. For example, the following code creates a publisher for a voice-only session:

mPublisher = new Publisher.Builder(context)
  .name("Bob's video")
  .videoTrack(false)
  .build();

Selecting the camera used by a publisher

You can cycle between cameras used to publish a stream by calling the cycleCamera() method of the Publisher object:

mPublisher.cycleCamera();

If you are using a custom video capturer, use the BaseVideoCapturer.CaptureSwitch interface to define the behavior of the Publisher.cycleCamera() method.

Mirroring the local display of a Publisher's video

You can set the mirrorInLocalRender property of the CaptureSettings object returned by calling the getCaptureSettings() method of the capturer property of a PublisherKit object. Set this to true to have the publisher's locally rendered video mirrored or to false to not have it mirrored. You may want to mirror video when using a front-facing camera.

You can also set the mirrorInLocalRender property of the CaptureSettings object when creating a custom video capturer.

This setting only affects the rendered video in the publisher's client application. It has no effect on the video in subscribing clients.

The built-in capturer of a Publisher has the locally rendered video mirrored for when using a front-facing camera, but not when using a rear-facing camera.

Detecting when the publisher's camera changes

When the camera used by the Publisher changes, the changedCamera(Publisher publisher, int newCameraId)) method of the Publisher.CameraListener object is called.

@Override
public void changedCamera(publisher, newCameraId) {
    //The publisher's camera changed.
}

Setting the resolution and frame rate for a video

You can set the resolution and frame rate for a publisher's stream by calling the resolution() and frameRate() methods of the Publisher.Builder object when you instantiate the Publisher object:

mPublisher = new Publisher.Builder(context)
  .name("Bob's video")
  .resolution(CameraCaptureResolution.HIGH)
  .frameRate(CameraCaptureFrameRate.FPS_30)
  .build();

Note that in sessions that use the OpenTok Media Router (sessions with the media mode set to routed), lowering the frame rate proportionally reduces the bandwidth the stream uses. However, in sessions that have the media mode set to relayed, lowering the frame rate does not reduce the stream's bandwidth.

Note: See the 1080p developer guide for considerations about using 1080p resolution.

Enabling the camera torch (flashlight)

Call the Publisher.setCameraTorch() method to enable or disable the camera flashlight (torch) while capturing video:

Publisher publisher = new Publisher.Builder(MainActivity.this).build();
  // ...
  publisher.setCameraTorch(true);

The default value is false (the flashlight is not used).

Note not all device cameras have a camera flashlight. If the active camera does not support camera flashlight functionality (for example, the front camera typically does not support it), calling this method has no effect.

Setting the camera zoom

Call the Publisher.setCameraZoomFactor() method to set the camera zoom ratio (factor):

Publisher publisher = new Publisher.Builder(MainActivity.this).build();
  // ...
  publisher.setCameraZoomFactor(0.7);

The ratio/factor value ranges from 0.5 to the maximum zoom factor. Values between 0.5 and 1.0 represent ultra-wide-angle (zoom out), and values between 1.0 and the maximum zoom factor represent zooming in. The actual zoom factor applied is clamped to the range supported by the active camera's configuration — if the camera does not support ultra-wide-angle, zoom factors set below 1.0 will not take effect and no zoom will be applied. For values over the maximum zoom factor supported by the camera, the zoom factor will be set to the maximum supported value.

Using a custom video capturer

The Publisher classes implement a standard video capturer that uses video directly from the device's camera. You can use the PublisherKit classes to implement a custom video capturer.

The OpenTok Android SDK includes a BaseVideoCapturer class. Override this class to create a custom video capturer. Call the capturer() method of the Publisher.Builder object when you instantiate the Publisher object:

// Use a custom video capturer.
// CustomVideoCapturer extends BaseVideoCapturer:
mCapturer = new CustomVideoCapturer();

mPublisher = new Publisher.Builder(context)
  .name("Bob's video")
  .capturer(mCapturer)
  .build();

Or, if you are using the PublisherKit class to define a custom publisher implmentation, you can set the capturer property of a class that extends the PublisherKit class:

// Use a custom video capturer.
// CustomVideoCapturer extends BaseVideoCapturer:
mCapturer = new CustomVideoCapturer();
mPublisher.capturer = mCapturer;

The getCaptureSettings() method of the BaseVideoCapturer class returns the settings of the video capturer, including the frame rate, width, height, video delay, and video format for the capturer. Your video capturer object can override this method to define your custom video capturer's settings, as in the following:

@Override
public CaptureSettings getCaptureSettings() {
    // Set the preferred capturing size
    configureCaptureSize(640, 480);

    CaptureSettings settings = new CaptureSettings();
    settings.fps = mCaptureFPS;
    settings.width = mCaptureWidth;
    settings.height = mCaptureHeight;
    settings.format = NV21;
    settings.expectedDelay = 0;
    return settings;
}

Call the startCapture() method of the video capturer object to start capturing video from the custom video capturer.

The BaseVideoCapturer class also includes a provideByteArrayFrame(byte[] data, int format, int width, int height, int rotation, boolean mirrorX) and provideIntArrayFrame(int[] data, int format, int width, int height, int rotation, boolean mirrorX). You call these methods to provide a frame of video as a either a byte array or as an array of integers.

For an example, see the OpenTokVideoCapturer class in the OpenTokHelloWorld sample application.

Use a custom video capturer to publish a screen-sharing stream. For more information, see Screen-sharing.

You can also use a custom video renderer for videos.

Using a custom audio driver

By default, the OpenTok Android SDK uses the device microphone to capture audio to transmit to a published stream. And it uses the and speakers (or headphones) to play back audio from a subscribed stream. Create a class that extends the BaseAudioDevice, defined in the OpenTok Android SDK:

public class CustomAudioDevice extends BaseAudioDevice

Then call the AudioDeviceManager.setAudioDevice() method to set the app to use the an instance of the audio device class for capturing and rendering audio samples:

AudioDeviceManager.setAudioDevice(customAudioDevice);

The BaseAudioDevice class has a getAudioBus() method that returns a AudioBus object, defined in the OpenTok Android SDK. The AudioBus has a readRenderData() method for reading audio samples from subscribed audio streams. And it has a a writeCaputureData() method for supplying audio samples to the publisher's audio stream.

For examples, see the Basic-Audio-Driver and Advanced-Audio-Driver samples in the vonage-video-android-sdk-samples repo on github).

Applying audio and video tranformations to published streams

You can use pre-built transformers from the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video. See this topic.

Tuning audio quality

To fine tune audio quality for a publisher stream the following approaches can be used:

You adjust the audio bitrate for a publisher stream by calling the audioBitrate() method of the PublisherKit.Builder object you use to instantiate a Publisher.
The desired bitrate for the published audio, in bits per second. The supported range of values is 6,000 - 510,000. (Invalid values are ignored.) Set this value to enable high-quality audio (or to reduce bandwidth usage with lower-quality audio). If you do not set this option, OpenTok automatically assigns an audio bitrate for the stream.

The following are recommended settings:

  • 8,000 - 12,000 for narrowband (NB) speech
  • 16,000 - 20,000 for wideband (WB) speech
  • 28,000 - 40,000 for full-band (FB) speech
  • 48,000 - 64,000 for full-band (FB) music

Setting the audio output mode for a voice session

By default, the OpenTok Android SDK uses the device loudspeaker (instead of the headset speaker) for playing audio. This is preferable for apps that include both video and audio. However, in a voice-only session, it is preferable to have the audio played back using the headset speaker. You can have the app do this by making the following call:

AudioDeviceManager.getAudioDevice().setOutputMode(
    OutputMode.Handset);

Reducing audio bandwidth with Opus DTX

Opus DTX (Discontinuous Transmission) is an audio codec that can reduce the bandwidth usage when a participant is not speaking. This can be useful in large sessions with many audio participants. You enable Opus DTX by calling the enableOpusDtx() method of the PublisherKit.Builder object you use to instantiate a Publisher. For more information, see this Vonage Video API knowledge base article.

Setting the audio volume for a subscriber

You can individually set the audio volume for each subscriber by calling the Subscriber.setAudioVolume() method.

Subscribing to audio or video only

By default, a Subscriber object is initialized to subscribe to audio and video, if they are available.

You can toggle audio on or off by calling the setSubscribeToAudio() method of the Subscriber object:

mSubscriber.setSubscribeToAudio(false); // audio off
mSubscriber.setSubscribeToAudio(true);  // audio on

You toggle video on or off by calling the setSubscribeToVideo() method of the Subscriber object:

mSubscriber.setSubscribeToVideo(false); // video off
mSubscriber.setSubscribeToVideo(true);  // video on

Note however that you can only subscribe to audio or video if the client publishing the stream includes audio or video. For example, calling subscribeToVideo(false) will have no effect if the client publishing the stream is publishing audio only.

Detecting whether a stream has audio or video

You can check the hasAudio() and hasVideo() methods of a Stream object to see if it has audio or video

if (!mStream.hasAudio()) {
	// You may want to adjust the user interface
}
if (!mStream.hasVideo()) {
	// You may want to adjust the user interface
}

For example, when you subscribe to a stream, you may want to adjust the user interface based on whether the stream has audio or video. For example, you may want to indicate to the user whether a stream has audio or not; or you may not want to display a subscriber's view if a stream does not have video.

Setting publisher video degradation preference

When bandwidth or CPU are limited, the video engine can reduce the frame rate and/or resolution of the publisher video to adjust to the limitations.

The degradation preference is a configuration setting that allows tweaking this behaviour.

Developers are responsible for defining and applying degradation preferences according to their use case and requirements.

It works as a preference, meaning that the video engine will try to follow the selected preference, but there're no guarantees on the result.

This preference can behave differently when using scalable video, where the video engine can decide to remove layers instead of degrading the resolution.

Content Hint and degradation preference serve different but related purposes: Content Hint describes the type of content being transmitted, while degradation preference controls the encoding strategy. When you set a Content Hint, the video engine automatically determines the appropriate degradation preference (for example, "text" automatically selects to maintain resolution). An explicitly set degradation preference will override this automatic selection.

You can set this value using any of the provided DegradationPreference values:

Not set

mPublisher.setDegradationPreference(DegradationPreference.DegradationPreferenceNotSet);

The default value. The video engine will decide the optimal degradation preference.

Maintain frame rate and resolution

mPublisher.setDegradationPreference(DegradationPreference.DegradationPreferenceMaintainFrameRateAndResolution);

The video engine will try to keep the frame rate steady and not reduce the resolution.

Maintain frame rate

mPublisher.setDegradationPreference(DegradationPreference.DegradationPreferenceMaintainFrameRate);

The video engine will try to keep the frame rate steady but might reduce resolution if necessary.

Maintain resolution

mPublisher.setDegradationPreference(DegradationPreference.DegradationPreferenceMaintainResolution);

The video engine will not reduce the resolution but might reduce the frame rate if necessary.

Balanced

mPublisher.setDegradationPreference(DegradationPreference.DegradationPreferenceBalanced);

The video engine will try to reach a balance between reducing resolution and frame rate when necessary.