The Video Video Media Processor API is now available on all supported native platforms (iOS, Android, Windows, and macOS), allowing you to add custom transformations to your published video.
In the past, we have spoken about how we can use this functionality in the JavaScript SDK to create effects like background blur and apply virtual background but let's take a look at how we can now create a custom transformer in Android and iOS to add an overlay image to the video feed.
Alongside the release of this feature, we also have a new developer guide showing you how to make use of the Media Processor to create a background blur effect as well as other custom transformations. However, let's take a look at how we can create a custom effect in Android and iOS.
Before you get started, make sure that you have followed getting started guide and have a basic video chat application setup with a created Publisher object.
Create a class that implements the PublisherKit.CustomVideoTransformer interface. Implement the PublisherKit.CustomVideoTransformer.onTransform(BaseVideoRenderer.Frame frame) method. The PublisherKit.CustomVideoTransformer.onTransform(BaseVideoRenderer.Frame frame) method is triggered for each video frame. In the implementation of the method, apply a transformation to the frame object passed into the method:
public class MyCustomTransformer implements PublisherKit.CustomVideoTransformer {
@Override
public void onTransform(BaseVideoRenderer.Frame frame) {
// Your custom transformation
}
}If we wanted to add, for example, an image on top of the video to act as a logo or watermark we could do:
public class MyCustomTransformer implements PublisherKit.CustomVideoTransformer {
public Bitmap resizeImage(Bitmap image, int width, int height) {
return Bitmap.createScaledBitmap(image, width, height, true);
}
@Override
public void onTransform(BaseVideoRenderer.Frame frame){
// Obtain the Y plane of the video frame
ByteBuffer yPlane = frame.getYplane();
// Get the dimensions of the video frame
int videoWidth = frame.getWidth();
int videoHeight = frame.getHeight();
// Calculate the desired size of the image
int desiredWidth = videoWidth / 8; // Adjust this value as needed
int desiredHeight = (int) (image.getHeight() * ((float) desiredWidth / image.getWidth()));
// Resize the image to the desired size
image = resizeImage(image, desiredWidth, desiredHeight);
int logoWidth = image.getWidth();
int logoHeight = image.getHeight();
// Location of the image (center of video)
int logoPositionX = videoWidth * 1/2 - logoWidth; // Adjust this as needed for the desired position
int logoPositionY = videoHeight * 1/2 - logoHeight; // Adjust this as needed for the desired position
// Overlay the logo on the video frame
for (int y = 0; y < logoHeight; y++) {
for (int x = 0; x < logoWidth; x++) {
int frameOffset = (logoPositionY + y) * videoWidth + (logoPositionX + x);
// Get the logo pixel color
int logoPixel = image.getPixel(x, y);
// Extract the color channels (ARGB)
int logoAlpha = (logoPixel >> 24) & 0xFF;
int logoRed = (logoPixel >> 16) & 0xFF;
// Overlay the logo pixel on the video frame
int framePixel = yPlane.get(frameOffset) & 0xFF;
// Calculate the blended pixel value
int blendedPixel = ((logoAlpha * logoRed + (255 - logoAlpha) * framePixel) / 255) & 0xFF;
// Set the blended pixel value in the video frame
yPlane.put(frameOffset, (byte) blendedPixel);
}
}
}
}Then pass the object that implements the PublisherKit.CustomVideoTransformer interface into the PublisherKit.setVideoTransformers() method:
MyCustomTransformer transformer = new MyCustomTransformer();
ArrayList<publisherkit.videotransformer> videoTransformers = new ArrayList<>();
videoTransformers.add(transformer);
mPublisher.setVideoTransformers(videoTransformers);
</publisherkit.videotransformer>You can combine the Vonage Media library transformer (see the previous section) with custom transformers or apply multiple custom transformers by adding multiple PublisherKit.VideoTransformer objects to the ArrayList passed into the PublisherKit.setVideoTransformers() method.
Then pass the object that implements the PublisherKit.CustomAudioRransformer interface into the PublisherKit.setAudioTransformers() method:
MyCustomAudioTransformer transformer = new MyCustomAudioTransformer();
ArrayList<publisherkit.videotransformer> audioTransformers = new ArrayList<>();
audioTransformers.add(transformer);
mPublisher.setAudioTransformers(audioTransformers);
</publisherkit.videotransformer>You can apply multiple custom transformers by adding multiple PublisherKit.AudioTransformer objects to the ArrayList passed into the PublisherKit.setAudioTransformers() method.
Before you get started, make sure that you have followed getting started guide and have a basic video chat application setup with a created OTPublisher object.
Create a class that implements the OTCustomVideoTransformer protocol. Implement the [OTCustomVideoTransformer transform:] method, applying a transformation to the OTVideoFrame object passed into the method. The [OTCustomVideoTransformer transform:] method is triggered for each video frame:
@interface CustomTransformer : NSObject <otcustomvideotransformer>
@end
@implementation CustomTransformer
- (void)transform:(nonnull OTVideoFrame *)videoFrame {
// Your custom transformation
}
@end
</otcustomvideotransformer>If we wanted to add, for example a image on top of the video to act as a logo or water mark we could do:
@interface CustomTransformer : NSObject <otcustomvideotransformer>
@end
@implementation CustomTransformer
- (void)transform:(nonnull OTVideoFrame *)videoFrame {
UIImage* image = [UIImage imageNamed:@"Vonage_Logo.png"];
uint32_t videoWidth = videoFrame.format.imageWidth;
uint32_t videoHeight = videoFrame.format.imageHeight;
// Calculate the desired size of the image
CGFloat desiredWidth = videoWidth / 8; // Adjust this value as needed
CGFloat desiredHeight = image.size.height * (desiredWidth / image.size.width);
// Resize the image to the desired size
UIImage *resizedImage = [self resizeImage:image toSize:CGSizeMake(desiredWidth, desiredHeight)];
// Get pointer to the Y plane
uint8_t* yPlane = [videoFrame getPlaneBinaryData:0];
// Create a CGContext from the Y plane
CGContextRef context = CGBitmapContextCreate(yPlane, videoWidth, videoHeight, 8, videoWidth, CGColorSpaceCreateDeviceGray(), kCGImageAlphaNone);
// Location of the image (in this case right bottom corner)
CGFloat x = videoWidth * 4/5;
CGFloat y = videoHeight * 1/5;
// Draw the resized image on top of the Y plane
CGRect rect = CGRectMake(x, y, desiredWidth, desiredHeight);
CGContextDrawImage(context, rect, resizedImage.CGImage);
CGContextRelease(context);
}
@end
</otcustomvideotransformer>Then set the OTPublisherKit.videoTransformers property to an array that includes the object that implements the OTCustomVideoTransformer interface:
CustomTransformer* logoTransformer;
logoTransformer = [customTransformer alloc];
OTVideoTransformer *myCustomTransformer = [[OTVideoTransformer alloc] initWithName:@"logo" transformer:logoTransformer];
NSMutableArray * myVideoTransformers = [[NSMutableArray alloc] init];
[myVideoTransformers addObject:myCustomTransformer];
_publisher.videoTransformers = [[NSArray alloc] initWithArray:myVideoTransformers];You can combine the Vonage Media library transformer (see the previous section) with custom transformers or apply multiple custom transformers by adding multiple PublisherKit.VideoTransformer objects to the ArrayList used for the OTPublisherKit.videoTransformers property.
Looking for a more complete sample to test out and experiment with? We have you covered! Sample applications showcasing the media processor are available for the following platforms:
Let us know what you're building with Vonage Video Media Processor. Chat with us on our Vonage Community Slack or send us a message on X, formerly known as Twitter.
){kind=small}
