Post Processing Composed Archives

Use the archive post-processing tool to process a collection of Vonage Video API archives, such ones that have been generated by the automatic archive feature.

Vonage Video API archives are time-limited. Part of a session might not be recorded in a single archive. To avoid losing a part of the session, use the auto-archiving feature Auto-archive, which enables the generation of several files to cover recording of very long sessions. Alternatively, use the Simultaneous-archives feature to start a second recording prior to ending the previous recording. In both cases, these archives are generated with a short overlap between the end of one and the beginning of the next.

If you prefer to have a single video file containing the entire session, you can use our Post Processing Composed Archives tool, which merges recordings and attempts to make the transition as perceptually smooth as possible. You can use this tool to generate a single but longer file that contains media from several shorter archive files. You can also use this tool with video recordings that are not created as Vonage Video archives, as long as some media is overlapping and codecs configuration are present.

This guide presents a brief explanation of how the post processing tool determines the overlapping period and how it produces the merge between two archives.

Overview

To deliver long recordings, our proposal detects overlapping media information between two consecutive archives by finding the video frame and audio sample that represent the beginning and the end of the repeated data. The tool presented here uses decision rules to generate long archives with this redundancy removed. It is resilient to the existence of black frames, highly redundant information, highly static content, glitches, and audio-video desynchronizations.

To discover the overlapping area, the archives post-processing logic only needs to analyze a small portion of both files, some seconds before the ending of the first archive and the beginning of the second.

The input archives must have identical codec properties, otherwise the tool will fail with an error. Besides, it requires the ffmpeg and ffprobe softwares to be installed in the user’s path, and they must have been compiled with the required audio and video parameters listed here:

  • Composed Archive Video Params
    • codec: H264
    • profile: Constrained Baseline
    • pixel format: YUV 420p
    • level: 4.1
    • FPS: 25
    • PTS timescale: 90000
  • Composed Archive Audio Params
    • codec: AAC LC
    • sample rate: 48Khz
    • channels: 1
  • Container
    • type: mp4

Content type

An archive might only contain audio, video or both. Regarding the overlapping content: the video motion can be extremely static or highly dynamic, and the audio signal can present potent noise or long silence periods. Apart from this, our tool can face other challenging contents for example, a publisher stops publishing during the overlapping period, translating to some black frames presence into the final archive, or when the media flow had packet loss when it was being recorded, giving rise to green decoder glitches on the video side or choppy audio.

Note please that both audio and video overlapping time ranges are evaluated independently.

Algorithm

Audio-only Archives

As the human auditory system perceives frequencies logarithmically, our audio algorithm logarithmically assesses the frequency content of the ending and the beginning parts of, respectively, the first and the second audio signals that change over time.

Once the time-frequency transformation is conducted, a sliding window with a fixed size is used to partition the processed signals and to iteratively compute their separate correlations.

We provide two different solutions according to the number of iterations carried out during the sliding window computation of the algorithm:

  • Deep-search: this option can be executed if -s or --deep-search is added to the running command.
  • Partial-search: default algorithm.

Video-only Archives

For analysing the video frames overlapping, several metrics have been implemented. Each of them are based upon the idea that the minimum value these algorithms return, the higher the similarity is between frames from both videos. Similar to audio, the post-processing code slides frame to frame the second video portion, which corresponds to the first number of seconds specified by the user as assessment period, over the corresponding frame from the same number of seconds at the ending part of the first video.

The code provides the option to render graphical results. In such case, the graph will show a green area that depicts the frames highest similarity relationship of the first and second archives where the overlapping takes place. Additionally, a 45 degrees line is also marked in blue reflecting the most similar frames are consecutively ordered, respectively. This correlation is later converted to timing data to obtain the overlapping period for video.

Please note that some framewise pre-processing is always applied before computing any of our four algorithms.

Audio+Video Archives

In the case of having archives with both audio and video, our tool finds separately the overlapping period for each of them. Next, it evaluates the intersection and makes different decisions according to: a) if the intersection is null; b) if the audio and video intersections are not similar enough; and c) when both overlaps are very similar. For a) and b) audio overlap data is prioritized as audio information is usually more legitimate for cases in which the video data might present a highly static trend. Finally, for the c) situation both overlapping periods are passed to the merge mechanism in which each, audio and video intervals, is used when merging each media track.

Other useful material

Our developer documentation contains easy to understand explanations and ready to use code. Please refer to the README files in https://github.com/opentok/archive-post-processing and https://github.com/opentok/archive-post-processing/tree/main/examples/archive-stitcher.

References

  • [Auto-archive]: Vonage property, "Auto-archiving. Vonage Video API archiving lets you record, save and retrieve sessions." Link: auto-archive.

  • [Simultaneous-archives]: Vonage property, "Simultaneous archiving. Vonage Video API archiving lets you record, save and retrieve sessions." Link: simultaneous-archives.