Adjusting audio and video — iOS (Swift)

You can make audio and video adjustments to published and subscribed streams:

Publishing audio or video only

You can toggle audio and video on or off, by setting the publishAudio and publishVideo properties. For example, the following code turns audio off:

publisher.publishAudio = false;

Publishing in a voice-only session

To set up a voice-only session, set the videoTrack property of the OTPublisherSettings object to false when you initialize each OTPublisher object in the session. For example, the following code creates a publisher for a voice-only session:

let publisherSettings = OTPublisherSettings()
publisherSettings.videoTrack = false
let publisher = OTPublisher(delegate: self, settings: publisherSettings)

Selecting the camera used by a publisher

You can specify (or change) the camera used to publish a stream:

publisher.cameraPosition = .front // front camera
publisher.cameraPosition = .back // back camera

Note that the cameraPosition property is not available in the OTPublisherKit class. If you are using the OTPublisherKit class to implement a custom video capturer, you can define the camera used in the custom video capturing code.

Detecting when the publisher's camera changes

When the camera used to publish a stream changes, the publisher delegate sends an OTPublisher publisher(_:didChangeCameraPosition:) message:

func publisher(_ publisher: OTPublisher, didChangeCameraPosition position: AVCaptureDevice.Position) {
	if position == .front {
		// front camera
	} else {
		// back camera
	}
}

Setting the resolution and frame rate for a video

You can set the resolution and frame rate for a publisher's stream by setting the cameraResolution and cameraFrameRate properties of the OTPublisherSettings object you pass into the OTPublisher init(delegate:settings:) method:

let publisherSettings = OTPublisherSettings()
publisherSettings.cameraResolution = .high
publisherSettings.cameraFrameRate = .rate30FPS
let publisher = OTPublisher(delegate: self, settings: publisherSettings)

Note that in sessions that use the OpenTok Media Router (sessions with the media mode set to routed), lowering the frame rate proportionally reduces the bandwidth the stream uses. However, in sessions that have the media mode set to relayed, lowering the frame rate does not reduce the stream's bandwidth.

Note: See the 1080p developer guide for considerations about using 1080p resolution.

Enabling the camera torch (flashlight)

Use the cameraTorch publisher property to enable or disable the camera flashlight (torch) while capturing video:

let publisherSettings = OTPublisherSettings()
publisherSettings.cameraTorch = true
// ...
let publisher = OTPublisher(delegate: self, settings: publisherSettings)

The default value is false (the flashlight is not used).

Note not all device cameras have a camera flashlight. If the active camera does not support camera flashlight functionality (for example, the front camera typically does not support it), calling this method has no effect.

Setting the camera zoom

Use the cameraZoomFactor publisher property to set the camera zoom ratio (factor):

let publisherSettings = OTPublisherSettings()
publisherSettings.cameraZoomFactor = 0.7;
// ...
let publisher = OTPublisher(delegate: self, settings: publisherSettings)

The ratio/factor value ranges from 0.5 to the maximum zoom factor. Values between 0.5 and 1.0 represent ultra-wide-angle (zoom out), and values between 1.0 and the maximum zoom factor represent zooming in. The actual zoom factor applied is clamped to the range supported by the active camera's configuration — if the camera does not support ultra-wide-angle, zoom factors set below 1.0 will not take effect and no zoom will be applied. For values over the maximum zoom factor supported by the camera, the zoom factor will be set to the maximum supported value.

Using a custom video capturer

The OTPublisher class captures video from a camera on the iOS device. You can use the custom video stream API to define a custom video capturer, using the OTPublisherKit and OTVideoCapture classes. For an example, see the "Project 2: Let's Build OTPublisher" sample in the samples directory of the OpenTok iOS SDK (or at the opentok-ios-sdk-samples-swift repo on github).

You can also use a custom video renderer for videos.

Using a custom audio driver

By default, the OpenTok iOS SDK uses the device microphone to capture audio to transmit to a published stream. And it uses the speakers (or headphones) to play back audio from a subscribed stream. Call the OTAudioDeviceManager setAudioDevice(_:) method to set up a custom audio driver for capturing and rendering audio samples:

let myAudioDevice = MyAudioDevice()
OTAudioDeviceManager.setAudioDevice(myAudioDevice)

Call OTAudioDeviceManager setAudioDevice(_:) before you initialize an OTSession object. Call the OTAudioDevice setAudioBus(_:) method to set an audio bus object. This object, defined by the OTAudioBus protocol, has a OTAudioBus writeCaptureData(_:numberOfSamples:) method and a OTAudioBus readRenderData(_:numberOfSamples:) method. These methods are called when audio samples are made available for writing to the publisher's audio stream or for reading from the subscribed streams.

For an example, see the "Project 7: External Audio Device" sample in the samples directory of the OpenTok iOS SDK (or at the opentok-ios-sdk-samples repo on GitHub).

Applying audio and video tranformations to published streams

You can use pre-built transformers from the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video. See this topic.

Tuning audio quality

To fine tune audio quality for a publisher stream the following approaches can be used:

Audio bitrate as a publisher settings

The desired bitrate for the published audio, in bits per second. The supported range of values is 6,000 - 510,000. (Invalid values are ignored.) Set this value to enable high-quality audio (or to reduce bandwidth usage with lower-quality audio). If you do not set this option, OpenTok automatically assigns an audio bitrate for the stream.

The following are recommended settings:

  • 8,000 - 12,000 for narrowband (NB) speech
  • 16,000 - 20,000 for wideband (WB) speech
  • 28,000 - 40,000 for full-band (FB) speech
  • 48,000 - 64,000 for full-band (FB) music

Reducing audio bandwidth with Opus DTX

Opus DTX (Discontinuous Transmission) is an audio codec that can reduce the bandwidth usage when a participant is not speaking. This can be useful in large sessions with many audio participants. You enable Opus DTX by setting the enableOpusDtx property of the OTPublisherKitSettings object you pass into the OTPublisherKit(delegate:, settings:) constructor (when creating a Publisher). For more information, see this Vonage Video API knowledge base article.

Setting the audio volume for a subscriber

You can individually set the audio volume for each subscriber by setting the OTSubscriberKit.audioVolume property.

Subscribing to audio or video only

By default, a Subscriber object is initialized to subscribe to audio and video, if they are available.

When you subscribe to a stream, you can specify whether to initially subscribe to audio or video (if they are available). For example, the following code subscribes to the audio stream only:

subscriber = OTSubscriber(stream: stream, delegate: self)
subscriber.subscribeToAudio = false

After you create a Subscriber object, you toggle audio on or off by setting the subscribeToAudio property of the Subscriber object:

subscriber.subscribeToAudio = false // audio off
subscriber.subscribeToAudio = true // audio on

You toggle video on or off by setting the subscribeToVideo property of the Subscriber object:

subscriber.subscribeToVideo = false // video off
subscriber.subscribeToVideo = true // video on

Note however that you can only subscribe to audio or video if the client publishing the stream includes audio or video. For example, setting subscribeToVideo to false will have no effect if the client publishing the stream is publishing audio only.

Detecting whether a stream has audio or video

By default, a Subscriber object plays back both audio and video (if they are available). You can check if a stream has audio or video (if the publisher of the stream is streaming audio or video) by checking the hasAudio and hasVideo properties of the Stream object:

if !stream.hasAudio {
	// You may want to adjust the user interface
}
if !stream.hasVideo {
	// You may want to adjust the user interface
}

For example, when you subscribe to a stream, you may want to adjust the user interface based on whether the stream has audio or video. For example, you may want to indicate to the user whether a stream has audio or not; or you may not want to display a subscriber's view if a stream does not have video.

Setting publisher video degradation preference

When bandwidth or CPU are limited, the video engine can reduce the frame rate and/or resolution of the publisher video to adjust to the limitations.

The degradation preference is a configuration setting that allows tweaking this behaviour.

Developers are responsible for defining and applying degradation preferences according to their use case and requirements.

It works as a preference, meaning that the video engine will try to follow the selected preference, but there're no guarantees on the result.

This preference can behave differently when using scalable video, where the video engine can decide to remove layers instead of degrading the resolution.

Content Hint and degradation preference serve different but related purposes: Content Hint describes the type of content being transmitted, while degradation preference controls the encoding strategy. When you set a Content Hint, the video engine automatically determines the appropriate degradation preference (for example, "text" automatically selects to maintain resolution). An explicitly set degradation preference will override this automatic selection.

You can set this value using any of the provided OTDegradationPreference values:

Not set

publisher.degradationPreference = .notSet;

The default value. The video engine will decide the optimal degradation preference.

Maintain frame rate and resolution

publisher.degradationPreference = .maintainFrameRateAndResolution;

The video engine will try to keep the frame rate steady and not reduce the resolution.

Maintain frame rate

publisher.degradationPreference = .maintainFrameRate;

The video engine will try to keep the frame rate steady but might reduce resolution if necessary.

Maintain resolution

publisher.degradationPreference = .maintainResolution;

The video engine will not reduce the resolution but might reduce the frame rate if necessary.

Balanced

publisher.degradationPreference = .balanced;

The video engine will try to reach a balance between reducing resolution and frame rate when necessary.