Responsive media on the web (ok, mostly responsive video)
As the web has evolved, building sites with media rich content has gotten more and more complex. We (as developers) are expected to build sites that are media-rich and engaging - but at the same time load quickly and work on many different platforms. This is a very hard balancing act that has been the focus of the web performance community...well since the beginning of the web.
In this post, we’ll focus on how media can be efficiently delivered on the web - we’ll focus mostly on video, but will use images as examples, as there has been a lot of work to make images fast for all devices and screens.
Images
Images are a logical place to start when considering responsive media - I mean you could just consider them a single frame video!
When it comes to image delivery on the web, there are a lot of HTML native approaches to deliver images as small as possible - and properly sized for the device. Let’s use some examples with the <picture> tag to walk through how you can properly serve images responsively. Here’s the example from the MDN page:
<picture>
<source srcset="/media/cc0-images/surfer-240-200.jpg"
media="(min-width: 800px)">
<img src="/media/cc0-images/painted-hand-298-332.jpg" alt="" />
</picture>
There are 2 images in the picture tag. The <img src> is the fallback for browsers that do not support the picture tag, and is the “image of last resort” in case the <source> images do not fit the criteria of the browser or space on the screen.
The <source> attribute describes alternative images that can be displayed. In the example above - there is a different image for screens where the screen width is over 800 pixels. The idea is that a “big image” can be delivered to big screens, and a “small image” for mobile or smaller screens. We can use the picture tag with even more complex methods:
<picture>
<source media="(min-width: 1200px)" src="hero-large.webp 1200w" type="image/webp">
<source media="(min-width: 1200px)" src="hero-large.jpg 1200w" type="image/jpg">
<source media="(min-width: 800px)" src="hero-medium.webp 800w" type="image/webp">
<source media="(min-width: 800px)" src="hero-medium.jpg 800w" type="image/jpg">
<source src="hero-small.webp" type="image/webp">
<source src="hero-small.jpg" type="image/jpg">
</picture>
In this example, we have 3 different versions of the same image - ‘hero-large’, ‘hero-medium’ and ‘hero-small’. They are offered in 2 different formats: webp and jpg. While this is can lead to a great (and fast) experience for your customers, Iit does mean 6 different versions of each image must be created.
When the browser encounters the picture tag - it iterates through the options, and will use the first version of the image that meets all the criteria. If the browser supports webp, and the screen is over 1200 pixels wide, the first image will be used. If the screen is under 800px, and the browser does not support webp, the ‘hero-small.jpg’ will be used.
This is incredible - your website now supports 3 different screen sizes and 2 different image formats. Of course, this means either the developers or the content management system must create 6 versions of each image that is uploaded - an extra development step, and potentially a lot more storage usage.
Video
The video tag works in similar ways to the image tag when it comes to formats:
<video autoplay muted controls>
<source src=”hero.webm” type=”video/webm”>
<source src=”hero.mp4” type = “video/mp4”>
</video>
If the browser can handle webm videos - this format will be used, otherwise it will fallback to the mp4 (98.1% support in browsers today - and the last 1.9% don't support video at all).
In a soon to be published article, HTTP Archive data finds that using multiple source files is not used widely - less than 10% of video tags use this feature (on 7.6M webpages tested). Even worse, of those that use multiple source files ~75% lead with the mp4, negating the presence of the other versions of the file (since the browser uses the first format it knows how to use, and all browsers know how to use a mp4.) This implies that only 2.5% of the video tags are correctly utilizing the source attribute to deliver different versions of the same video.
No media queries
There are no media queries for video. There once were media queries in the video specification, but they were removed, and support for media queries left Chrome in version 34 (In November 2021 Chrome is version 95). So, the pattern we used to make images responsive will not work for video.
Why were they removed?
When it comes to images, generating 3-4 different types of image can be tricky - which formats? Which sizes? What is the process to generate - is it automated, or do you need to build a specialized upload process? If your site has hundreds of images - you may now have thousands of images to store and deliver.
Video suffers from the same issues. But videos - being generally more than one frame long - make the problem more immediate. Re-encoding a video takes much longer than reformatting an image - so the time to generate the files can be on the order of minutes (as opposed to a second or 2 for images).
If you are using a CMS - it must know how to take an uploaded video, and turn it into several different sizes - with a couple of different formats. If you are not using a CMS, guess what - it falls on you to learn how to re-encode the different sizes and formats for each video.
Video doesn't work the same way
When you add an image to a webpage - it stays where it is. It’s static. If you download a huge image on a poor connection - it takes longer, but the page will still load.
Video is not static. But since the <video> tag is markup, it cannot change what is rendered on the page.
Let’s consider a couple of use cases:
You have a <video> embedded on your page, and it renders at 600px wide. Your page uses logic to choose and download a video source file that is optimized for a 600px space (saving a lot of data over the 'desktop' 1920x1080 video.)
- What happens when the user presses the “fullscreen” button?
- What if your customers have a chromecast, and they cast it to their 60” TV?
The video sent to full screen/TV will be 600px wide, and will look pixelated. Even with media queries - there is no way to switch the video input without stopping the first download, and starting a new download. If you are 15 minutes into the mp4 - the browser would have to have logic to figure out where that time is in the video. Needless to say, this is not a good user experience.
- What happens when you start a video, but then you move to an area with lower bandwidth?
The video download will not be able to keep up with the video playback, and the video will stall.
Responsive video: JavaScript player
Rather than create a myriad of different sizes and file types in the <video> tag, the solution is adaptive bitrate streaming (ABS), supported by a JavaScript player sitting on top of the <video> tag. We’ve described how ABS streaming works in several posts. Esentially several different sizes of the video are created, and the video is delivered in short segments or chunks.
The JavaScript player sits atop the video tag, and works as a controller for the video - monitoring both the screen size (and changes in screen size) and the network conditions. When a change occurs that could affect the video playback, the player is able to make changes midstream to the video. Since each chunk defines a timestamp in the video - switching midstream between different video qualities is a simple exercise for the player.
With ABS, jumping from 600p wide to full screen, or casting to a TV just works. When the netwrork conditions deterioriate or improve, the player can adapt the playback quality to optimzie the experience for the viewer.
It is totally understandable to be hesitant in adding a new JavaScript framework to your page - we are so bloated in JS today (that's an entirely different topic) - but for a few hundred KB, the JS player enhances the video playback experience on all devices. With a video that is specifically sized for mobile devices - the savings in video delivery is very likely to exceed the cost of the JS incurred on video setup.
Conclusion
The video tag is amazing, and will continue to be the backbone of video on the web. To make the video tag more responsive to various screen sizes, there are 2 possible approaches - reading media queries, or using a JavaScript controller to power adaptive bitrate video.
The media query approach solves one problem - but raises several more that impact the user experience. The JavaScript player adds additional page weight, but delivers the video responsively, in a network-aware way. We at api.video believe that adaptive bitrate video is the future of video, and the added weight of the JavaScript player to the video tag is a small tradeoff in exchange for providing stellar video experiences on the web.
