
While rendering videos on-device is standard for many mobile and desktop apps, developers often hesitate to do it in the browser, and with some reason. Browsers do have limitations, but they're more capable than many assume. You can still render up to an hour of video, and avoiding costly servers for rendering and replication is a major win.

Here are the top 5 myths we’ve heard most often.

<div style={{ textAlign: "center"}}>
#### Myth #1: Browser video editing is slow and clunky
</div>

It's important to know that modern browsers can utilize Web Codecs APIs for hardware-accelerated encoding and decoding. This means they leverage dedicated CPU and GPU hardware accelerated abilities to speed up the process. [Web Codecs API](https://caniuse.com/?search=Web%20Codecs) is widely supported across browsers, with some exceptions for AudioDecoder in Safari, and it continues to improve. At Rendley, we took in consideration the Safari limitation and designed our system to overcome it.

Additionally, WebAssembly is commonly used in this space, offering excellent low-level memory control. In most cases, rendering is faster than real-time, though it can vary based on video resolution, bitrate, and hardware capabilities.

<div style={{ textAlign: "center"}}>
#### Myth #2: Videos cannot be longer than 5 minutes
</div>

This is false! While there is a browser limitation of 2GB\* per file (because arrays can have a maximum length of int32), this usually translates to about an hour of Full HD video encoded with H.264. We really hope this will change in the future, but still, 2GB is more than enough for plenty of use cases.

\*The maximum file size depends on the browser, for instance, for Chromium browsers it is 2GB, Safari 4GB and Firefox 8GB.

<div style={{ textAlign: "center"}}>
#### Myth #3: You have to keep the browser tab open for rendering
</div>

This is mostly true for projects that use a media player to render videos. Browsers tend to optimize background tasks (like media playback) to maintain performance and save power, which can disrupt the player. However, there is an alternative method, which is decoding frames, drawing them onto a canvas, and then encoding the final result. It works well in the background and avoids the limitations of the media player approach. Rendley SDK is using the later approach, decoding and drawing each frame at maximum speed, leaving a very small memory footprint.

<div style={{ textAlign: "center"}}>
#### Myth #4: It’s just for basic trimming
</div>

Not true! If you implement the video editing process on a WebGL canvas, you can do far more than basic editing. You can apply advanced effects, filters, and transitions that work seamlessly. You could also use a Canvas2D, but it would be far less performant due to the fact you would have to loop over each frame and pixel and do it while using the CPU. For instance, our SDK supports keyframe animations, LUTs for advanced filters, even After Effects compositions as Lottie files.

<div style={{ textAlign: "center"}}>
#### Myth #5: The final video might look different from what was created
</div>

On the contrary, what you see in the editor is what you get in the final output. When rendering occurs on a server, you have to remap the changes that user did in the editor and it’s essential to match the user’s creation pixel for pixel. Rendering on the client-side, however, simplifies this process and ensures that the output matches exactly what was created during editing.

If you still consider that rendering on the server is more suitable for you, we have an [open source implementation](https://github.com/rendleyhq/video-rendering-server) that runs the same SDK Engine we have on the client-side, but on the server side.


Think browsers can't handle video rendering? Think again, we’re busting the top 5 myths.

5 Myths About Rendering Videos in Browser (Debunked)


Every time we mention rendering videos in the browser, we get raised eyebrows and concerns about performance. This is for a good reason, mainly because, for a long time, browsers were not capable of doing it and had to use the CPU for decoding and encoding frames.

This bottleneck made companies willing to implement a video editing solution to create two separate systems: a playback mechanism that would run in the browser, allowing users to trim/split videos, apply effects, add text clips, etc., visually, and a backend mechanism that would take the same structure created on the web and recreate it on a server to render it at high speeds.

Such an approach, although it has a faster rendering time, has disadvantages, such as:

- All the clips should be uploaded to a server for the backend to be able to create the composition.
- All the filters/effects/transitions should also be created through a custom rendering solution.
- The backend should download the clips to create the composition.
- Rendering several videos simultaneously can be very computationally intensive, requiring autoscaling solutions.
- The generated video should be stored on the server for the interface to download the rendered output.
- Users typically upload videos larger than 10MB, so you have to consider network connectivity.
- Cannot create a product that works well on slow networks or offline.
- You have to ensure the composition created on the backend matches the one the user created.
- Working with text is challenging and requires consideration of font families and styles.
- Server providers charge for traffic as well, so you have to keep this in consideration.

## Browser Rendering Steps

Generally, rendering the video in the browser involves several steps:

![Rendering steps](/images/posts/rendering-videos-browser-using-webcodecs-api/rendering-steps.webp "Rendering steps")

1. Capture the frame.
2. Encode the frame into a video stream.
3. Repeat until all the frames are captured.
4. Mux the video and audio streams into a container.

The most time-consuming steps in this flow are capturing and encoding the frames.

A simple implementation of capturing video frames could be seeking an [HTMLVideoElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement), rendering the texture into the canvas, and taking a screenshot of the whole composition. The seeking in this case will always try to locate the [I-Frame](<https://en.wikipedia.org/wiki/Video_compression_picture_types#Intra-coded_(I)_frames/slices_(key_frames)>) to paint the result, which is a headache for the browser, even if you increase the seeking time by 1/30 (the FPS) seconds. It's also worth mentioning that switching the tabs while rendering happens will pause the video player and cause the final video to have missing sections because browsers have underlying optimizations to ensure good performance.

Regarding encoding, the common solution would be to use the WASM build of [FFmpeg](https://ffmpeg.org/). However, it is not possible to start an encoding process and just append the new frames into the stream because the WASM build does not support it at the moment. You would have to temporarily store all the frames in memory (not a good idea) or store a chunk of them, transform them into an encoded video, and repeat the process.

There is a great video from Christopher Chedeau explaining more about the pipeline that goes into encoding a video and the limitations of the browser:

<YouTube id="46_9YsHHUEo" />

<br />

## WebCodecs API

The good news is that browsers have started adding support for hardware-accelerated video/audio/image encoders and decoders that solve the issues mentioned above - this set of APIs is known as the [WebCodecs API](https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API).

For the capturing part, the seeking can now be replaced with decoding the frame using the [VideoDecoder API](https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder). This is much faster than seeking because the browser doesn’t have to look for [I-Frames](<https://en.wikipedia.org/wiki/Video_compression_picture_types#Intra-coded_(I)_frames/slices_(key_frames)>), [B-Frames](<https://en.wikipedia.org/wiki/Video_compression_picture_types#Bi-directional_predicted_(B)_frames/slices_(macroblocks)>), and [P-Frames](<https://en.wikipedia.org/wiki/Video_compression_picture_types#Predicted_(P)_frames/slices>) to reconstruct the image. Instead, it can just retrieve what has changed and recreate the texture from that.

Encoding can be done through the [VideoEncoder API](https://developer.mozilla.org/en-US/docs/Web/API/VideoEncoder), which enables appending the video frames directly into the stream without having to store the frames temporarily.

## WebCodecs Considerations

There are things to be aware of when deciding to implement a rendering mechanism with WebCodecs. Here are some hiccups we faced during the process:

- There are no built-in APIs for muxing and demuxing. You have to either use FFmpeg or an alternative library.
- You need additional tooling for identifying the codec string when initializing the decoder. It should follow the pattern described [here](https://aomediacodec.github.io/av1-isobmff/#codecsparam).
- Sometimes, the decoder tells you if it supports a specific configuration only after decoding a few packets, so you have to handle it gracefully.
- WebCodecs is [not supported](https://caniuse.com/?search=webcodecs) in all browsers yet (e.g., Firefox).
- The logs are not always explicit and require you to have additional tooling for debugging.
- When encoding videos, you have to make sure that the devices you are using support the specific resolution. For instance, a 7-year-old Android phone might not be able to encode a video larger than Full HD, and you have to account for that. Also, the encoder doesn’t inform you of this.
- Canvas works with RGB/RGBA color format, but in many cases, the video frames coming from VideoDecoder will be in YUV, requiring you to handle the conversion.
- You can’t get the media/stream info from the video and should use ffmpeg/mp4box.
- You have to identify and handle rotated videos yourself.
- There is a limited number of codecs supported.

## Conclusion

It is possible to render videos in the browser and do it in a performant manner that, in many cases, replaces the need for a server. But the process of developing such a system is overly complicated and involves a lot of hacking and getting around browser's limitations.

Over the last year, our team has been working on creating a **Video Editing SDK** meant to take care of all the issues and limitations mentioned above, while giving you full access to build the flow and interface you want. Check it out at [rendley.com](https://rendley.com/)


Learn how to render videos directly in the browser using the WebCodecs API, overcoming performance bottlenecks and simplifying video processing.

Showing posts from sdk tag

5 Myths About Rendering Videos in Browser (Debunked)

Rendering Videos in the Browser Using WebCodecs API