Live Captions — Vonage Video iOS SDK (Swift)

The OpenTok iOS SDK includes methods for iOS clients to publish and receive captions in an OpenTok session.

Live captioning must be enabled at the session level via the REST API.

Live captioning is only supported in routed sessions.

This topic includes the following sections:

Publishing live captions

An OpenTok publisher can start or stop publishing real-time live captions by setting the publishCaptions property of the OTPublisherKit object:

If the publisher does not include an audio track, the [PublisherKit publisher:didFailWithError:] message is sent, with the code property of the error set to OTPublisherMissingAudioTrack.

The Vonage iOS SDK does not support a publisher receiving events for its own captions. To render the speaker’s own captions, create a hidden subscriber (to the publisher’s stream) to listen for the caption events. (See the next section.)

Subscribing to live captions

A subscriber may start or stop receiving captions by setting the subscribeToCaptions property of the OTSubscriberKit object:

subscriber.subscribeToCaptions = true

You can call this method regardless of whether the publisher of the stream is currently publishing live captions. The subscriber will start receiving captions data once the publisher begins publishing captions.

To stop receiving captions, set the property to false:

subscriber.setSubscribeToCaptions = false

The OTSubscriberKitCaptionsDelegate(_:subscriber:caption:isFinal:) message is sent when a subscriber to a stream receives captions:

// OTSubscriberKitDelegate callbacks:
func subscriber(_ subscriber: OTSubscriberKit, caption text: NSString, isFinal final: Bool) {
  // Display caption text in UI.
}

You can set up a key-value observer for the hasCaptions property of an OTStream object to see when the stream has captions enabled and disabled.

//add observer on a stream of interest
let hasCaptionsObservation: NSKeyValueObservation = subscriber.stream.observe(\.hasCaptions, options: [.old, .new]) { object, change in
	guard let oldValue = change.oldValue else { return }
	guard let newValue = change.newValue else { return }
	print("KVO change for \(subscriber.stream.streamId) hasCaptions: \(newValue)")
	//manipulate GUI element
}

Receiving your own live captions

The Vonage Video API does not support a publisher receiving events for its own captions. To render the speaker's own captions, create a hidden subscriber (to the local publisher's stream) to listen for caption events. Do not display this subscriber's view in the UI, and do not subscribe to its audio (to avoid echo). You can then add the captions text to the UI.

The Publisher by default does not publish captions, so set the publishCaptions property of the publisher to true:

    publisher = OTPublisher(delegate: self, settings: pubSettings!)
    publisher?.publishCaptions = true

Subscribe to the stream in the OTPublisherKitDelegate(publisher, streamCreated) method:

func publisher(_ publisher: OTPublisherKit, streamCreated stream: OTStream) {
	// Add view
	let pubView = self.publisher?.view
	if(pubView != nil) // audio only publisher
	{
		ViewUtils.addViewFill(pubView!, rootView: self.publisherView!)
	}

	let subscriber = OTSubscriber(stream: stream, delegate: self)
	subscriber.subscribeToAudio = false
	subscriber.subscribeToVideo = true // typically should be false, but due to an existing bug in 2.25.3, it is set to true. Future versions will fix the issue.
	subscriber.subscribeToCaptions = true
	subscriber.captionsDelegate = self

	do {
		try self.session?.subscribe(subscriber)
	} catch let error {
		print("Error subscribing: \(error)")
	}

	selfPublisherStreamId = stream?.streamId
}

As noted above, you must subscribe to video to receive captions. (This will be fixed in a future version.)

This example stores the publisher's stream ID in a selfPublisherStreamId variable, used here to prevent its view from being added to the UI:

func subscriberDidConnect(toStream subscriberKit: OTSubscriberKit) {
	if subscriber.stream.streamId == selfPublisherStreamId {
    	return // Don't show the self subscriber view.
	}

	// Show other subscribers in the UI
	if let subsView = subscriber?.view {
		subsView.frame = CGRect(x: 0, y: kWidgetHeight, width: kWidgetWidth, height: kWidgetHeight)
		view.addSubview(subsView)
	}
}

The subscriber has the transcribed text (for your published stream) in the following delegate:

func onSubscriberCaptionText(_ subscriber: OpaquePointer!, caption_text: String!) {
	// Adjust UI to show captions.
	DispatchQueue.main.async {
		if let s = self.views[subscriber] {
			s.updateCaptionText(caption_text)
		}
	}
}