video api

video industry

Normal

small_emilien.jpeg

thumbnail_emilien.jpeg

emilien.jpeg

Emilien Gobillot

Video trends

**The comeback of specialized hardware and its impact on the video industry**

Over the past few decades, the trend in computing has been toward general-purpose computing systems that can run a wide variety of applications. This has led to the widespread use of [CPUs and GPUs](https://www.intel.com/content/www/us/en/products/docs/processors/cpu-vs-gpu.html). However, there has been a recent resurgence of interest in specialized hardware, which is designed specifically to perform one or a few tasks extremely efficiently. In this blog post, we will explore the reasons behind this trend and some of the critical examples of specialized hardware being used in the video industry today.

**Why is specialized hardware making a comeback in computing?**

Specialized hardware in the computing industry refers to computer components or devices designed for a specific purpose, such as application-specific integrated circuits (ASICs)  or systems-on-a-chip (SoCs) . These hardware components are optimized for their specific task, resulting in faster and more efficient performance than general-purpose hardware.

Several factors have contributed to the recent resurgence of specialized hardware in computer science:

* **High efficiency : Why we want it quick and ressource-efficient**

One key driver is the increasing demand for high-performance solutions. Artificial intelligence, machine learning and video processing are domains where CPUs are far from being the right solution. Regarding AI workloads, training, and running models are more efficient on GPU and even better on specialized hardware like Tensor processing units (TPUs). Google has been the first to design TPUs. These ASICs (Application Specific Integrated Circuits) outperform the CPUs.

Why are CPUs outperformed ? In early 2023 commodity servers come with 1 or 2 sockets and maximum 96 cores per socket for the x86 microarchitecture (128 for ARM). On the other hand, a server with a mid range CPUs can receive several ASICs/TPUs on the PCIe ports. Those boosted systems outperforms by a lot the CPUs based system. The TPUv4, in terms of Tflops is 2000x faster than a laptop according to Google.

The benefits of a specialized hardware is that it provides better performance, higher density and a lower energy consumption than the classic CPUs.

In most cases, TPU/ASIC > GPU > CPU on the 3 criteria : performance, density, and energy savings.

* **Total Cost of Ownership (TCO): Why It's Worth the cost**

Specialized hardware exhibits smaller size, higer speed, and greater electrical efficiency. As a result the density increases and the number of computing units decreases. At the end, the TCO is improved. After the switch to ASICs based VCUs (Video coding unit), Youtube is [reported](https://www.semianalysis.com/p/google-new-custom-silicon-replaces) to have saved millions of CPUs ! At this scale you also save data centers 💰

**How specialized hardware is changing the game in the video industry**

In the video industry, one of the most computationally intensive processes is video transcoding, especially for high-quality videos. Transcoding decodes a video from one codec and encodes it to another, such as from H.264 to H.265. This process can also involve changing other video parameters like resolution, bit rate, frame rate, etc. This has led to a need for fast and efficient video transcoding solutions without sacrificing the video quality output.

Transcoding is a service api.video provides in addition to hosting and delivery. Via an API call, a video is uploaded, and then the URL of the video is returned immediately. Note that high resolution and new codecs increase the computational cost.

Similarly to Artificial Intelligence (AI), several types of hardware are commonly used in video transcoding applications:

1. ASICs (Application Specific Integrated Circuits): Video ASICs are specialized hardware accelerators designed specifically for video processing tasks, such as video encoding/ decoding / transcoding. They are sometimes called VPU (Video Processing Unit) or VCU (Video coding Unit). They significantly improve performance over traditional CPUs, particularly for high-resolution or high-frame-rate video. Alphabet, the mother company of Google and YouTube has designed its own ASICs : codename Argos. For a system with 20 VCUs, the TCO has decreased by a 33x factor for VP9 encoding, and throughput increased by a 100x factor!

large_untitled.png

medium_untitled.png

small_untitled.png

thumbnail_untitled.png

untitled.png

Facebook/ Meta made the same move to specific hardware. They partnered with silicon vendors to [design custom ASICs](https://engineering.fb.com/2019/03/14/data-center-engineering/accelerating-infrastructure/).

The electric consumption of an ASIC is around a few watts.

2. FPGAs (Field-programmable gate arrays): FPGAs are programmable hardware devices that can be customized for specific applications, including video transcoding. They can provide significant performance improvements over traditional CPUs for certain types of video transcoding tasks, such as simultaneously encoding or decoding multiple video streams. Twitch is using an [FPGA from Xilinx](https://blog.twitch.tv/en/2018/12/19/how-v-p9-delivers-value-for-twitch-s-esports-live-streaming-35db26f6322f/).

The electric consumption is around 25 W for a PCIe card

3. Graphics processing units (GPUs): GPUs are another type of hardware. This hardware is more generic than FPGA and ASIC, but specialized video encoders/decoders are added to these cards. Nvidia was the first company to build a datacenter GPU for transcoding. Intel enters the market in late 2022.

The electric consumption is generally around 70W.

Regarding energy consumption ASICs > (are better than) FPGAs > GPU > CPU. This order is the same for the transcoding speed.

**Reducing the carbon footprint of video transcoding : api.video’s approach**

At api.video we are really focused on the energy consumption of our transcoding pipeline. We want to minimize the ecological impact of video transcoding while keeping the transcoding speed high. The infrastructure team has performed multiple benchmarks, and here are the results.

large_image-2.png

medium_image-2.png

small_image-2.png

thumbnail_image-2.png

image-2.png

Those numbers are based on public information (marketing sheet) for the GPUs and FPGAs and real benchmarks for the CPU and ASICs. The quality of the video's output could change the results slightly, but we clearly see a winner regarding power efficiency. In a future blog post, we will share precise benchmarks about the ASICs vs. CPU battle with you. Stay tuned.

Along this article, we have seen the resurgence of dedicated hardware in computer science is driven by a growing need for high-performance computing, particularly in fields like artificial intelligence, machine learning, and video transcoding. Dedicated hardware can provide better performance and energy efficiency than traditional general-purpose computing units, making it an attractive option for many applications. As computing continues to evolve, we will likely see more specialized hardware being developed for specific tasks, driving innovation and progress in the field. 

Our video API allows you to add videos to your products while prioritizing the reduction of your carbon footprint. [Join us](https://api.video/) and take part of the transition to an environmentally responsible video infrastructure.

large_23mar23-1.png

medium_23mar23-1.png

small_23mar23-1.png

thumbnail_23mar23-1.png

23mar23-1.png

The comeback of specialized hardware

codec

transcoding

large_logo_avatar.jpg

medium_logo_avatar.jpg

small_logo_avatar.jpg

thumbnail_logo_avatar.jpg

logo_avatar.jpg

api.video team

Have you ever uploaded a video on YouTube and wondered how your video is magically available in 5+ different qualities and 5+ different sizes? It’s not magic ✨ it’s *video transcoding*. Well, it’s kind of magic 🌟

In this article, we’ll tell you everything you want to know about video transcoding — from what it is to its importance in video streaming and [video APIs](https://api.video/). With that said, let’s get started! 🚀

## The difference between video encoding and video transcoding 

Encoding and transcoding sound very similar but, in fact, are two very different processes. To learn about video transcoding, let’s first describe video encoding.

### What is video encoding? 

[Video encoding](https://api.video/blog/video-trends/guide-to-video-encoding-video-codec) digitally compresses raw video from a digital or analog camera. You almost never have to worry about video encoding as your device encodes all videos automatically. The video you recorded of your dog this morning, and now stored on your phone, is thanks to video encoding. The other video you shared with your friend using WhatsApp wouldn't be possible without video encoding.

Encoding is necessary for creating manageable file sizes.

### What is video transcoding?

Video transcoding re-compresses an encoded file which means that video transcoding cannot happen without an initial video encoding.

Transcoding, compresses a file from one digital [codec](https://api.video/blog/product-updates/every-video-format-codec-and-container-explained) to another. Being able to play videos across a wider range of devices, or to improve playback experience are two of the many reasons that we transcode videos today.

An example of video transcoding is to transcode an encoded AV1 video to an H264 video codec. Why do that you may ask? AV1 codec, despite being an efficient codec, is not supported on a wide range of devices. H264 codec on the other hand is supported on almost all of the devices out there.

large_transcoding-1-1.png

medium_transcoding-1-1.png

small_transcoding-1-1.png

thumbnail_transcoding-1-1.png

transcoding-1-1.png

## Types of video transcoding

When transcoding a video file, some data is lost, which is what we call lossy—the greater the compression, the smaller the file size and the lower the quality. Let’s now look at three main types of video transcoding.

### 1. Lossy-to-lossy

It occurs when one type of lossy data is converted into a different kind of lossy data. Having a file of lower quality before transcoding means the quality will drop even further during the process. Reducing the bitrate for use on portable devices is one of the justification for this transcoding method.

### 2. Lossless-to-lossy

This transcoding reduces quality loss relative to lossy-to-lossy methods while creating files with manageable sizes for mobile devices. To make good use of this transcoding technique, you will need to save archives of files that have been compressed losslessly.

### 3. Lossless-to-lossless

This type of transcoding involves using improved compression to shrink a file without altering its quality. This transcoding method saves quality when switching between formats, but the generated files are sometimes too large to transfer to mobile devices.

## Why is video transcoding used?

The use of video transcoding is based on requirements. Here we can look at some of the top reasons for using video transcoding:

- **Adaptive bitrate streaming (ABR streaming)** - ABR streaming ensures the best video experience on a wide range of devices depending on your internet connection’s speed and stability. Transcoding is a crucial intermediate step to get the highest quality streaming output based on HLS or DASH. For example, video playback on Youtube could be switching between 1080p and 720p versions of the video stream, depending on your device and connection. This leads to the fewest buffering instances and higher-quality playbacks.
- **Limited storage capacity** - The word cloud is one of the biggest tricks played by marketers. It allows us to imagine a world with an infinite storage capacity for data. This is far from reality. Cloud is some hardware located in a room somewhere. No matter where you store your videos, cloud or not, you are using certain storage space. Transcoding is the key to maximizing your storage capacity, resulting in lower carbon footprints. Yes, actual carbon footprints.
- **Quality of user experience** - Transcoding is about more than just reducing a file size. Transcoding is also used to improve the playback experience.
- **Multiple target devices, browsers, and applications** - Transcoding is used for video compatibility amongst different devices, browsers, and applications.

## Who uses video transcoding?

Many individuals and businesses use video transcoding. Here are some concrete examples of users among the many who use video transcoding.

- Video streaming APIs for businesses and developers such as [api.video](https://api.video/). Video transcoding is an integral element of an adaptive video streaming workflow to get video content ready for a delivery protocol — like the current industry standard, HLS or DASH — that can play on as many devices as possible.
- Video-On-Demand (VOD) services such as Netflix, Hulu, and Youtube use video transcoding on large scales.
- Content Management Systems (CMS) targeting multiple devices with a variety of video formats and sizes.
- As a video editor, you may prefer to work on the highest quality video codec, and once you're done editing, you could transcode your video to other codecs for delivering to a wider range of devices.

## Video APIs and video transcoding

As one of the more important users of video transcoding, video APIs are beneficial for businesses, both big and small, to create and host on-demand videos. 

[High-quality video APIs](https://api.video/) such as api.video can change your video into HTTP Live Streaming (HLS) for smooth streaming at high speeds anywhere in the world without you needing to worry about doing the process yourself. For the highest possible video quality on any connection, it handles the nitty-gritty of transcoding in different qualities for ABR streaming over HLS while keeping the best playback quality on a wide range of devices.

## Final thoughts

New gadgets are being introduced every day. This means more cameras with greater capturing capacities and more screens with higher display qualities.  At the same time, the video world is booming, and more and more businesses need to reach out to users with all sorts of devices and bandwidth. Transcoding is not magic, but it certainly has a magical aspect to it. It makes it possible for us to connect through high-quality videos.

💡 Hopefully, now you know more about transcoding and why it’s used in our industry. 

Would you like to start your video journey today? 🎬 Go ahead and start testing in the [api.video](https://dashboard.api.video/register) dashboard for free.

large_29nov22.png

medium_29nov22.png

small_29nov22.png

thumbnail_29nov22.png

29nov22.png

Video transcoding: What, why, and how?

Modern technology has been helpful for an increasingly fast and simple method of consuming various types of video media, whether it’s a YouTube video, our favorite shows on Netflix, or even catching up with friends across the globe on Zoom. Despite the relative ease of doing anything related to video, a few complex processes take place behind the scenes, away from public view. One such process is video encoding.

Video encoding basically makes the world go ‘round in regards to anything related to video and video content. Thanks to the technologies behind the video encoding process, it has become much easier for people and companies to share video content on the internet. But what exactly is video encoding, and what happens during the process? We’ll take a closer look at this and much more in this handy guide to video encoding.

## What is video encoding?

large_poster-1_2.png

medium_poster-1_2.png

small_poster-1_2.png

thumbnail_poster-1_2.png

poster-1_2.png

In a nutshell, video encoding occurs when you take raw video footage, either from a digital camera or an analog film camera, and digitally compress it to make it a more manageable file size. Video encoding happens daily in both professional and personal capacities, and many people don’t realize it.

But, without encoding, services like Netflix and YouTube would not exist – or, at least, not in the same way that it does today. Without it, we couldn’t send videos of our beloved pets to our friends and family as easily as we can today.

## Why is video encoding used?

There are a variety of reasons why video encoding is used, but all factors point to the single function of video encoding – **compression**.

This is done to ensure optimal performance and compatibility with a selected suite of software and hardware, be it for DVD/Blu-ray playback, mobile viewing, online video streaming, or general video editing. A file's video and audio information undergo transformation and compression during the encoding process, all in accordance with the requirements of the selected encoding standard.

On a professional level, the ability to compress raw video into more manageable sizes makes it easier to transmit video content such as training, promotion, presentation, and so on over the internet. 

On a personal level, video encoding makes it much easier to send videos taken from your phone directly to friends and family. Most modern-day personal devices have built-in video encoding capabilities to allow for this to happen without a second thought.

To put things in perspective, uncompressed or unencoded video can have a data rate that’s as low as 3,000 Mbps and as high as 12,000 Mbps, depending on the length and quality of the video recorded. 

**Without encoding, it would take up to days for anyone to not just transmit video, but to be able to consume it, too.**

## Differences between video encoding and video transcoding

You need to be familiar with two more [essential terms related to video encoding](https://api.video/what-is/) – transcoding and codecs.

Transcoding relates to the conversion from one digital encoding format to another. It takes the already encoded video file, decompresses it, and then alters it before re-encoding or compressing it into a different codec. Codecs are what allow encoding to occur in the first place. They are software or hardware that gives us the ability to compress video to make them more easily stored and delivered tightly.

While you may hear “encoding” and “transcoding” used interchangeably, the two terms actually mean different things. Putting it succinctly, encoding is the initial process of compressing a video from a source – such as a camera – into a codec, while transcoding takes an already encoded video file to convert it into a different codec.

## Examples of video codecs

In order to understand video encoding, you must first be familiar with two interrelated terms: [video codecs and video formats](https://api.video/blog/product-updates/every-video-format-codec-and-container-explained). These also tend to be used interchangeably, but much like the terms encoding and transcoding, they refer to two different things.

Video formats are the type of files in which the compressed video/audio is stored. Also known as a “container”, you can tell what a video format is by looking at the file’s extension — the three to four characters that sit after a period in a file name. Here are some of the most common video formats used today:

* MP4
* FLV
* MOV
* MKV
* LXF
* MXF
* AVI
* WebM
* QT

On the other hand, **the codec is short for “Coder / Decoder” (compression / decompression)** and relates to the video compression algorithms that allow encoding to happen. Over the years, codecs have gone through multiple iterations as it coincides with the increasing sophistication and rise in the quality of filmmaking. 

If you’re a content creator, you might be aware of some of the video codecs that exist today:

- **H.264 – also known as AVC, this is the most widely-used video codec available on every platform and mobile device.**
- H.265 – also known as HEVC, stands for High-Efficiency Video Coding, and it’s slowly overtaking H.264 as the standard. This codec can compress video much more efficiently than its predecessor.
- **VP9 – is an open-source, royalty-free codec developed by Google and is considered the direct competitor of H.265. The video compression codec VP9 was created to reduce file sizes, making streaming videos simpler.**
- **AV1 – is another royalty-free, open-source codec released in 2019 by the non-profit Alliance for Open Media. It was developed to become the successor to Google’s VP9 and contend with HEVC more directly.**

## How do different technologies handle video encoding

As you may have suspected, there are a majority of ways that different technologies handle the video encoding process. Let’s take a look at the most common technologies that use video and how video encoding works on each of them:

### Video editing software

When it comes to video content creation, a majority of people will use one of two video editing software — Adobe Premiere Pro or Apple’s Final Cut Pro. Both of these video editing software handle video encoding in very similar ways: when exporting the finished video.

Depending on which device the video content is recorded on, it will already be encoded and formatted according to the settings of the device. For those using iPhones recording 1080p video at 25 frames per second (FPS), for example, the iPhone will automatically encode the video using H.264 and contain it in a .mov file format. The video editing software will “open up” the container to get to the encoded, compressed video file within so the editing process can begin.

After editing the footage is complete, it will need to be exported, which typically concerns video encoding. The edited raw media taken from a recording device such as a camera will be encoded via the codec of the person’s choosing before being put into a .mp4 container once they’ve hit “export”. As we’ve mentioned before, the most common codec used in video content creation is H.264 for HD videos, with 4K videos using H.265. 

### Websites

Different websites utilize different techniques to handle video encoding – with a majority of the major sites like YouTube, Netflix, and so on using their own technologies to handle the process. A majority of social media websites allow you to upload video files directly onto the site to share, although most of them require the files to be saved in specific containers. 

These websites will take your video file and transcode it before using their own systems and serves to re-encode it to match multiple display resolutions. This process will happen regardless of your video file's original codec and settings. Different websites have different processes, but companies that don’t have the capacity to build their own video encoding systems can use a third-party application programming interface (or API) to handle this process. These APIs act as [dedicated video encoder](https://api.video/encoding/) that allows companies to add on-demand videos to their websites more efficiently.

Using [api.video](http://api.video) as an example, all of the videos submitted to the platform will be transformed into H.264 codec in an MP4 container. This may be done using transmuxing and/or transcoding.

We expect the video bitrate to be lower than 16 Mbps for up to 1080p resolution and 50 Mbps for up to 4K resolution. We accept any bitrates, however, we will scale them down to the limits mentioned because anything above is usually challenging for most viewers’ connection to watch.

### API

The ability to generate, store, and distribute video is made possible via a video API. [api.video](http://api.video)’s solution can be tailored to suit any specific need. 

The backend process of video on demand (VOD) is exceptionally complex, even if you’re a software programmer specializing in video. It is necessary to encode, package, and typically transcode the data before sending it to any of several possible destinations. All of these things, and more, may be done with minimal fuss using api.video.

Our API basically keeps the complexities of streaming media hidden from view, so you can rapidly develop apps with as much control as you need. This saves time and effort so developers can concentrate on making the product.

## Conclusion

As you can see through this video encoding guide, a lot of work goes into making the video you watch and the video you share possible. Without video encoding compressing raw video footage, it could take days to upload and transmit video content over the internet and perhaps even longer to be able to download and enjoy them. 

You can integrate your videos in minutes with api.video. [Contact us](https://api.video/contact) for more information.

large_poster-3.png

medium_poster-3.png

small_poster-3.png

thumbnail_poster-3.png

poster-3.png

A Guide to Video Encoding & Video Codecs

medium_screen-shot-2021-01-26-at-11-48-47-am.jpg

small_screen-shot-2021-01-26-at-11-48-47-am.jpg

thumbnail_screen-shot-2021-01-26-at-11-48-47-am.jpg

screen-shot-2021-01-26-at-11-48-47-am.jpg

Erikka Innes

4K has been around just long enough for some of us to want the next best thing and when it comes to resolution, we may be approaching our limit. That might sound strange at first, if you've heard about Roger Clark's project to find out how many megapixels the human eye can see. By his calculations, you can see up to 576 megapixels. 8K only creates 33.17 megapixel images, so what gives? We should be able to see a lot more. 

The answer in the way the human eye processes visual information. The way it's designed, you can only see with the kind of clarity required for a 576 megapixel image with a tiny part of your eye called the fovea. 
It lies at the center of your visual field and covers about 1% of the retina. Amazingly, about 50% of the part of your brain that's for visual processing focuses solely on details received from the fovea! What is more, in the very best of lighting conditions for your eye, with someone with the very best vision, that person can only see about 1/60th of a degree with their fovea. This is the equivalent of a dot .07 inches across and 20 feet away! 


## Okay, So Can We See 8K? 

If that's how we see, can our eyes detect 8K? Do we need 8K if it's just a tiny part of your eyeball that can really see with the incredible level of detail you'd need to enjoy 8K in an image? The answer is...it depends. If you're going to sit at home on the sofa and stare at a TV, then no, probably not. According to Paul Maguire, you'd have to sit less than 4.2 feet away from your 65 inch TV or 3.6 feet away from your 55 inch TV to see all the detail in 4K. For anything greater than 4K you'd have to sit less than 3 feet away. Which is liable to irritate any other viewers in the room that wanted to see the screen around you. This doesn't mean 8K resolution isn't useful. 

## Benefits of 8K

8K is potentially beneficial for recording video. If you have 8K quality, you can zoom in on parts of your shots and still have 4K quality or better. It can allow someone shooting to have more choices from every shot because the quality will still be very high. 

8K is also great creating intensely realistic, immersive experiences. If you have a wrap around TV or a planetarium, or virtual reality goggles, 8K is amazing. It allows for the rendering of softer, more realistic edges. It increases the realness of a scene because the human eye can't see any pixelation. 


## Conclusion

For virtual reality, 8K will probably become the standard. As for becoming the standard for streaming video, probably not for a while. It's not necessary in many cases, and at this time it's also difficult to stream at that level and know it will arrive at its destination as 8K. Someone streaming in 8K would likely end up outputting to lower resolutions based on the wide range of viewing devices users have. 
api.video upload your videos up to 4K, with fast encoding and a worldwide delivery. You can [test it for free](https://dashboard.api.video/register) and start building your video project or check [our Community](https://community.api.video/).


Can the human eye see 8K resolution and when is 8K most useful?

large_can-the-human-eye-see-8k-resolution-and-when-is-8k-most-useful.png

medium_can-the-human-eye-see-8k-resolution-and-when-is-8k-most-useful.png

small_can-the-human-eye-see-8k-resolution-and-when-is-8k-most-useful.png

thumbnail_can-the-human-eye-see-8k-resolution-and-when-is-8k-most-useful.png

can-the-human-eye-see-8k-resolution-and-when-is-8k-most-useful.png

What kinds of video experiences are best with 8K?

live - create

doug.jpg

twitter-brands.svg

Doug Twitter

Doug Sillars

Frequently asked question: How many live streams can I use simultaneously with api.video? 
Answer: How many cameras do you have?

That sounds like a snarky response (and perhaps it is *a little bit* snarky). Some platforms - like YouTube Live or Twitch - allow just one live stream per account at a time. But, with api.video there is no such limitation - as long as you have cameras (and available bandwidth), you can stream to your heart’s content from every single one of them.

## Why would I stream more than once?

There’s no reason to limit your viewers to just one view - stream from every angle (or stream from every angle that you have a camera for!).  Or if your client needs video from a number of locations at once - api.video can support you.  Here are some example use cases:


Are you running a multi track conference?  Stream each room from the same api.video account. Add a hallway cam to show the social interactions.  
Do you support more than one client?  Don’t drive yourself crazy scheduling to fill the one slot - let everyone have their own live stream(s).
Sporting event - multiple angles.  Does the coach want a stream from the opposite side?  From behind the goal/end-zone? From the top of the stadium?

## Create your live streams


The first step in having multiple live streams is to create your live streams.  You can create as many streams as you’d like. 

Creating [live streams](https://api.video/blog/tutorials/live-stream-tutorial) is easy with our [API](https://docs.api.video/reference/post_live-streams).  

Once you’ve created your live stream you (or your clients/users) can re-use each livestream over and over (like a Space-X rocket). 

* Stop the stream after each conference talk, and restart before the next. If you have ```record:true``` set for the live stream, you’ll get VOD versions of each video saved into your account.  Each recording will have the livestream name, and the time (in GMT). 
* Assign live stream(s) to your clients, and let them use them to their heart’s content. No more work is needed on your part.
* Name each livestream for easy recovery of the videos:  “50 yard line”  “Client X” “Conference Fizz 2021: Buzz track”




## A wildly absurd demo 


As an example of using live concurrent live streams, I hooked up all the cameras I could find (and that were charged up) to my laptop/wifi. In the example below, I was able to run 5 concurrent live streams - showing the capabilities of api.video, but also pushing my computer (and my internet connection) to their limits:
<image>



medium_257523051420257536349489364058826396261704n.jpg

small_257523051420257536349489364058826396261704n.jpg

thumbnail_257523051420257536349489364058826396261704n.jpg

257523051420257536349489364058826396261704n.jpg

* [https://Livestream.a.video](https://Livestream.a.video) - using an USB webcam (mounted on monitor)
[https://Record.a.video](https://Livestream.a.video) - using Mac webcam
[OBS](https://api.video/blog/no-code/start-and-record-a-live-stream-with-obs) - using Iriun and a phone - this app connects your phone to a computer via Wi-Fi (on right of photo on a stand)
[Android livestream](https://api.video/blog/tutorials/livestream-from-your-android-phone) - phone using Larix Broadcaster to stream (leaning on laptop)
[Pi livestream](https://api.video/blog/tutorials/video-streaming-with-a-raspberry-pi) - my raspberry Pi camera (pointed out the window)





This was a fun demo to try out, but it did push my laptop to the limit. What it does show is that with a few computers & cameras and an api.video account, it is really easy to simultaneously live stream as many views, angles, rooms as you desire.



large_concurrent-livestreams.jpg

medium_concurrent-livestreams.jpg

small_concurrent-livestreams.jpg

thumbnail_concurrent-livestreams.jpg

concurrent-livestreams.jpg

How many live streams can I run at once?

## The benefits of a cloud video platform
The cloud is collections of servers in data centers that run software, maintain databases and serve all kinds of content to all the corners of the world. As we make it easier to create and share video by adding technologies like fast 5G networks and cheaper camera equipment, more and more people are creating video content. To handle the demand, most video streaming platforms use the cloud. Netflix famously started the move to the cloud after a corrupted database prevented them from mailing anyone their DVDs for 3 days! YouTube uses the cloud, and so does api.video. There are many benefits to providing video via the cloud. 

## 1. You can use a network that can handle video streaming and heavy traffic
A major benefit to using a cloud video platform is that it's up to someone else to handle the technical details of video streaming and coping with heavy traffic. You can just add video to your company portal, or build your own viewing interface or whatever you like.

## 2. Lower cost
It's less money for your business or project if you don't have to store your video content, manage the servers with the content, or hire an IT team. Instead, you pay for a video service, like api.video, and everything is stored and managed for you. 

## 3. Video focused content delivery network
If you choose a cloud video platform like api.video, you get to use our private content delivery network. Because api.video handles only video content, the network is optimized to provide the best experience when working with video. Many companies that work with video have their own networks for their content, including Netflix - their network is called Netflix Open Connect. 

## 4. Continuity
When you store content yourself, you have to have a plan for what happens if part of your system goes down. If you use a cloud service for your video, this is handled for you. By storing it in the cloud, it can be replicated and maintained in multiple locations if need be, so that your video is always there when you need it. 

## 5. Mobility
If your video content is stored in the cloud, it's easier to access. You can stream content at home or a coffee shop, anywhere there's internet, you'll be able to get to your videos and interact with them. 

## api.video and the cloud
At api.video, all our services are in the cloud. We use our own Content Delivery Network (CDN) to ensure fast, reliable uploading, transcoding, and delivery. 

large_top-5-benefits-of-a-cloud-video-platform-1.jpg

medium_top-5-benefits-of-a-cloud-video-platform-1.jpg

small_top-5-benefits-of-a-cloud-video-platform-1.jpg

thumbnail_top-5-benefits-of-a-cloud-video-platform-1.jpg

top-5-benefits-of-a-cloud-video-platform-1.jpg

small_yvanmichel.png

thumbnail_yvanmichel.png

yvanmichel.png

Yvan Michel

All companies in the video streaming sector have the same goal-- guarantee the best watching experience with efficient cost control for ingest, transcoding, and delivery. It is not an easy task and requires a lot of considerations on the Product and Engineering sides.

Measuring the Quality of Experience (QoE) is one of the methods widely used to assist this process. In this blog post, we will go through the QoE principle and functioning by relying on common use cases and metrics.

## Principle

As soon as someone uses a website or an app, stakeholders need feedback in order to learn about the strengths and weaknesses of what they've built. The measurements performed for the QoE purpose are based on this same principle and are seamless for the end-user.

The QoE consists of collecting and sending users subjective data to better understand how the product is used to make the right decisions. In this case, knowing more about how videos are consumed by viewers through a first-party player.

It helps to get more insights about questions like "What will be the effects of reducing the 1080p video bitrate on the end-user engagement?" and "Is it a good approach to preload the video on the start-screen to boost its loading time?"

NOTE: QoE data must comply with local privacy policies like [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation) or [CCPA](https://en.wikipedia.org/wiki/California_Consumer_Privacy_Act). No data that breaks these laws or disrespects user privacy shall be stored without the user's explicit consent.

## Functioning

The player performs periodic measures related to the video playback but also computes metrics from its own runtime context like the [FCP](https://web.dev/fcp/), its loading time, and more. It then forwards all this info to a dedicated backend for later processing and analysis.

large_qos-and-qoe-workflow.jpg

medium_qos-and-qoe-workflow.jpg

small_qos-and-qoe-workflow.jpg

thumbnail_qos-and-qoe-workflow.jpg

qos-and-qoe-workflow.jpg

The objective is to gather enough relevant data about the "Where", the "Who" and the "How". The "Where" and the "Who" are quite important but are more "context" oriented. The "How" really refers to the QoE.

### The "Where"

Locate where videos are watched to know the distribution of the audience:

- The `continent` → *Is North America, the main continent in terms of audience?*
- The `country` → *How many videos are watched in France?*

GeoIP Databases allow retrieving the `continent` and the `country` based on the user IP.

### The "Who"

Identify the profile of devices used to watch the videos:

- The `device_type` → *Are CTVs representing a significant part of the audience in Asia?*
- The `platform` and the `browser` family or version → *When was the latest version of macOS released? What is the part of IE11 to clean legacy code?*

Even if it is not 100% trustable because it tends to be deprecated and it can be overridden, the `User-Agent` HTTP header exposes this info and provides, at least, good hints. There are some alternatives like [UA-CH](https://web.dev/user-agent-client-hints/).

For a given visit, having a `user_session` with the number of `views` helps to confirm that after consecutive video playbacks, the experience remains stable and great.

### The "How"

Understand the video consumption behaviours:

- The `rebuffering_time` → *Why is the engagement so low in Asia?*
- The `watch_time` → *What is the best moment to trigger an ad-break?*
- The `abr_max_rendition` → *Is the 720p more played when increasing the default player dimensions?*
- The `playback_error` → *Why are macroblocks appearing on Safari/desktop?*

There are a lot of factors that can influence video playback. Rebuffering is part of them but is the consequence of underlying problems: slow delivery, low user bandwidth, etc.

> A viewer who experienced a rebuffer delay that equals or exceeds 1% of the video duration played 5.02% less of the video in comparison to a similar viewer who experienced no rebuffering.
> 
> Source from [Akamai](https://people.cs.umass.edu/~ramesh/Site/HOME_files/imc208-krishnan.pdf)

But what is rebuffering? A `rebuffering` describes the moment when the video playback stopped because of an insufficient amount of downloaded video bytes *(buffering)* without any user interactions. Seeking or switching manually to another video rendition is not considered to be `rebuffering`.

In other words, `rebuffering` is an unexpected buffering lasting *N* milliseconds causing the video playback to stop.

The `watch_time` is a duration representing the time a user has watched a video. It can be associated with the `rebuffering_time` to compute the `rebuffering_precentage` during a `user_session`.

ABR (Adaptive Bitrate) algorithms choose the best video renditions according to user network conditions. The `abr_max_rendition` describes the highest video rendition reached when ABR is enabled.

Finally, the `playback_error` indicates which errors have occurred during the video playback to explain sudden disruptions or degradations within the network, transmuxing, or decoding.

## Conclusion

We've had a sneak peek at how QoE can be useful to drive operational decisions. Combined with the [Quality of Service](https://en.wikipedia.org/wiki/Quality_of_service) *(QoS)*, this data constitutes a powerful set of valuable information to continuously ensure that video playback is optimal.

At [api.video](https://api.video), we are committed to providing the best watching experience for all, no matter the location or the device. In another blog post, we will go through other side usages based on QoE data especially for observability to detect proactively issues and assess impact in case of outages. Stay tuned!

## Resources

- [https://www.akamai.com/blog/performance/enhancing-video-streaming-quality-for-exoplayer-part-1-quality-of-user-experience-metrics](https://www.akamai.com/blog/performance/enhancing-video-streaming-quality-for-exoplayer-part-1-quality-of-user-experience-metrics)

Benefits of measuring the video playback QoE

large_qoe-1.jpg

medium_qoe-1.jpg

small_qoe-1.jpg

thumbnail_qoe-1.jpg

qoe-1.jpg

The benefits of measuring the video playback QoE

## Responsive media on the web (ok, mostly responsive video)


As the web has evolved, building sites with media rich content has gotten more and more complex.  We (as developers) are expected to build sites that are media-rich and engaging - but at the same time load quickly and work on many different platforms.  This is a very hard balancing act that has been the focus of the web performance community...well since the beginning of the web.

In this post, we’ll focus on how media can be efficiently delivered on the web - we’ll focus mostly on video, but will use images as examples, as there has been a lot of work to make images fast for all devices and screens.


## Images

Images are a logical place to start when considering responsive media - I mean you could just consider them a single frame video!

When it comes to image delivery on the web, there are a lot of HTML native approaches to deliver images as small as possible - and properly sized for the device.  Let’s use some examples with the ```<picture>``` tag to walk through how you can properly serve images responsively. Here’s the example from the MDN page:

```
<picture>
    <source srcset="/media/cc0-images/surfer-240-200.jpg"
            media="(min-width: 800px)">
    <img src="/media/cc0-images/painted-hand-298-332.jpg" alt="" />
</picture>
```
	
There are 2 images in the picture tag. The ```<img src>``` is the fallback for browsers that do not support the picture tag, and is the “image of last resort” in case the ```<source>``` images do not fit the criteria of the browser or space on the screen.

The ```<source>``` attribute describes alternative images that can be displayed. In the example above - there is a different image for screens where the screen width is over 800 pixels.  The idea is that a “big image” can be delivered to big screens, and a “small image” for mobile or smaller screens. We can use the picture tag with even more complex methods:  

```
<picture>
    <source media="(min-width: 1200px)" src="hero-large.webp 1200w" type="image/webp">
    <source media="(min-width: 1200px)" src="hero-large.jpg 1200w" type="image/jpg">
    <source media="(min-width: 800px)" src="hero-medium.webp 800w" type="image/webp">
    <source media="(min-width: 800px)" src="hero-medium.jpg 800w" type="image/jpg">
    <source src="hero-small.webp" type="image/webp">
    <source src="hero-small.jpg" type="image/jpg">
</picture>
```
	
In this example, we have 3 different versions of the same image - ‘hero-large’, ‘hero-medium’ and ‘hero-small’.  They are offered in 2 different formats: webp and jpg. While this is can lead to a great (and fast) experience for your customers, Iit does mean 6 different versions of each image must be created.

When the browser encounters the picture tag - it iterates through the options, and will use the first version of the image that meets all the criteria.  If the browser supports webp, and the screen is over 1200 pixels wide, the first image will be used.  If the screen is under 800px, and the browser does not support webp, the ‘hero-small.jpg’ will be used.

This is incredible - your website now supports 3 different screen sizes and 2 different image formats.  Of course, this means either the developers or the content management system must create 6 versions of each image that is uploaded - an extra development step, and potentially a lot more storage usage.

	
## Video

The video tag works in similar ways to the image tag when it comes to formats:

```
<video autoplay muted controls>
	<source src=”hero.webm” type=”video/webm”>
	<source src=”hero.mp4” type = “video/mp4”>
</video>
```
	
If the browser can handle webm videos - this format will be used, otherwise it will fallback to the mp4 (98.1% support in browsers today - and the last 1.9% don't support video at all).

In a soon to be published article, HTTP Archive data finds that using multiple source files is not used widely - less than 10% of video tags use this feature (on 7.6M webpages tested).  Even worse, of those that use multiple source files ~75% lead with the mp4, negating the presence of the other versions of the file (since the browser uses the first format it knows how to use, and all browsers know how to use a mp4.) This implies that only 2.5% of the video tags are correctly utilizing the source attribute to deliver different versions of the same video.

### No media queries

There are no media queries for video.  There *once were* media queries in the video specification, but they were removed, and support for media queries left Chrome in version 34 (In November 2021 Chrome is version 95).  So, the pattern we used to make images responsive will not work for video.


### Why were they removed?

When it comes to images, generating 3-4 different types of image can be tricky - which formats?  Which sizes? What is the process to generate - is it automated, or do you need to build a specialized upload process?  If your site has hundreds of images - you may now have thousands of images to store and deliver.

Video suffers from the same issues.  But videos - being generally more than one frame long - make the problem more immediate.  [Re-encoding](https://api.video/what-is/video-encoding) a video takes much longer than reformatting an image - so the time to generate the files can be on the order of minutes (as opposed to a second or 2 for images). 

If you are using a CMS - it must know how to take an uploaded video, and turn it into several different sizes - with a couple of different formats.  If you are not using a CMS, guess what - it falls on you to learn how to re-encode the different sizes and formats for each video.

### Video doesn't work the same way

When you add an image to a webpage - it stays where it is.  It’s static.  If you download a huge image on a poor connection - it takes longer, but the page will still load.

Video is not static.  But since the ```<video>``` tag is markup, it cannot change what is rendered on the page.  

### Let’s consider a couple of use cases:

You have a ```<video>``` embedded on your page, and it renders at 600px wide.  Your page uses logic to choose and download a video source file that is optimized for a 600px space (saving a lot of data over the 'desktop' 1920x1080 video.)  

1. What happens when the user presses the “fullscreen” button?  
2. What if your customers have a chromecast, and they cast it to their 60” TV?  

The video sent to full screen/TV will be 600px wide, and will look pixelated. Even with media queries - there is no way to switch the video input without stopping the first download, and starting a new download.  If you are 15 minutes into the mp4  - the browser would have to have logic to figure out where that time is in the video.  Needless to say, this is not a good user experience.

3. What happens when you start a video, but then you move to an area with lower bandwidth?  

The video download will not be able to keep up with the video playback, and the video will stall.



## Responsive video: JavaScript player

Rather than create a myriad of different sizes and file types in the ```<video>``` tag, the solution is adaptive bitrate streaming (ABS), supported by a JavaScript player sitting on top of the ```<video>``` tag. We’ve described how [ABS streaming](https://api.video/what-is/adaptive-bitrate-streaming) [works](https://api.video/blog/video-trends/responsive-video-delivery) in  several posts. Esentially several different sizes of the video are created, and the video is delivered in short segments or chunks.  

The JavaScript player sits atop the video tag, and works as a controller for the video - monitoring both the screen size (and changes in screen size) and the network conditions.  When a change occurs that could affect the video playback, the player is able to make changes midstream to the video.  Since each chunk defines a timestamp in the video - switching midstream between different video qualities is a simple exercise for the player.

With ABS, jumping from 600p wide to full screen, or casting to a TV *just works*.  When the netwrork conditions deterioriate or improve, the player can adapt the playback quality to optimzie the experience for the viewer.

It is totally understandable to be hesitant in adding a new JavaScript framework to your page - we are so bloated in JS today (that's an entirely different topic) - but for a few hundred KB, the JS player enhances the video playback experience on all devices.  With a video that is specifically sized for mobile devices - the savings in video delivery is very likely to exceed the cost of the JS incurred on video setup.


## Conclusion

The video tag is amazing, and will continue to be the backbone of video on the web.  To make the video tag more responsive to various screen sizes, there are 2 possible approaches - reading media queries, or using a JavaScript controller to power adaptive bitrate video.

The media query approach solves one problem - but raises several more that impact the user experience.  The JavaScript player adds additional page weight, but delivers the video responsively, in a network-aware way.  We at api.video believe that adaptive bitrate video *is* the future of video, and the added weight of the JavaScript player to the video tag is a small tradeoff in exchange for providing stellar video experiences on the web.





thumbnail_diseno-web-responsive-design.jpeg

diseno-web-responsive-design.jpeg

Responsive media: the right approach for images is not the right approach for video

small_anthony.png

thumbnail_anthony.png

anthony.png

Anthony Dantard

Customers are "divinely discontent" and their expectations are constantly on the rise. Thanks to services like YouTube & Netflix, we all expect blazingly fast load times and smooth video playback for streaming on-demand video. What might be less obvious is that, whether we're aware of it or not, that expectation is slowly transferring to real-time video communication and live streaming applications as well.

At api.video, we're obsessed with delivering the best developer and viewer experience for our customers. So it's no surprise that we spend a lot of time thinking about and keeping up-to-date on video protocol development. Each video protocol has its strengths and weaknesses. Each is better suited for different applications. 

In this post, we outline four of the major low-latency protocols, talk about the pros and cons of each, and give our commentary on where we think the future of low latency streaming is going. We'll talk about the following protocols in depth in this post:

- WebRTC
- LL-HLS
- LL-DASH
- HESP

# WebRTC
The WebRTC protocol supports both audio and video streams for real-time bi-directional communication. It can be used for ingestion and distribution with an end-to-end latency between 300ms - 600ms depending on network quality and the distance between users. The protocol really took off in 2011 when Google bought Global IP Solutions, the company that originally developed the protocol, and gave the project significantly more human and financial resources. 

Ten years later, WebRTC has become the standard for real-time video communication on the web. The components that WebRTC is based on are accessible via a JavaScript API maintained by the W3C and the IETF, allowing users to live stream directly to a web browser without installation of any third-party tool.

In theory, WebRTC allows the establishment of a video conference system between 2 browsers by connecting them to each other peer-to-peer. However, in practice the architecture of computer networks (internet, corporate, and domestic) can greatly complicate matters.

large_webrtcprotocolstack.png

medium_webrtcprotocolstack.png

small_webrtcprotocolstack.png

thumbnail_webrtcprotocolstack.png

webrtcprotocolstack.png

### Finding your way through the internet labyrinth

In some cases, we are not directly connected to the internet and we are behind several network layers such as a NAT, a firewall, or a proxy. To get around these problems, WebRTC allows you to use the ICE (Interactive Connectivity Establishment) protocol. It allows you to find the most direct path to make two machines communicate and cross the different network layers. For this, a STUN / TURN server is necessary in order to obtain the external address of a user and relay the traffic if a direct connection is not possible.

The WebRTC protocol requires, in addition to its own multimedia server infrastructure, a STUN / TURN server infrastructure to operate communications in a production environment with real massive use.

## To further confuse the issue

The protocol specification requires that all data exchanged between peers must be encrypted (video, audio, data application). Thus, the protocol embeds several security protocols to fill the gaps of the UDP protocol when it's used. 

WebRTC uses SDP (Session Description Protocol) to expose some properties of the peer-to-peer connection like the types of media to be exchanged and their associated codecs, the network transports, and some bandwidth information.

Respectively DTLS (Datagram Transport Layer Security) is involved to generate Self-Signed certificates, for the negotiation of the secrets used for encrypting media data, and for secure transport of application data between peers. SRTP (Secure Real-Time Transport) is used for the transport of video and audio streams. The last security protocol involved is SCTP (Stream Control Transport Protocol), which is used for the exchange of application data.

### The challenges of WebRTC scalability

Several network architectures and topologies can be implemented to operate communications with the WebRTC protocol, each responding to a specific use case, with advantages and drawbacks specific to each topology. We will not dig into topology details in this article, but be aware that SFU (Selective Forwarding Unit) is perhaps the most popular architecture in modern WebRTC applications today.

The main challenge with this protocol is the scalability on a planetary scale that is cost effective. We are not saying that WebRTC can't be scaled worldwide, it was true a few years ago at its early stage but it's not true anymore thanks to the great improvements and work bring by some great companies to improve topology. 

As WebRTC is a mashup of several protocols that each have to be learned and mastered by engineers rendering it very complex. It's difficult for most companies to implement it in their infrastructure at a global scale in a cost effective way. That's why only a few companies have succeeded at making it fully scalable today. They do so by mixing SFU and MCU (MultiPoint Conferencing Unit) topologies, then putting it on the edge.

## Low Latency HTTP ABR streaming protocols

In contrast to WebRTC, which is a peer to peer protocol that, in theory does not require a central server to establish communication between two machines, HTTP-based streaming protocols require the use of a server and communication over standard HTTP. They are based on the very foundations of the web.

### HLS/LL-HLS

The HTTP Live Streaming (HLS) protocol can be used to deliver live and on-demand content streams to global-scale audiences and was introduced by Apple in 2009 with scalability in mind rather than latency. While the average latency faced by users is around 15s, the HLS latency can reach between 30s and 1 minute. Even with performant infrastructure and optimized packaging and player configuration, one can expect latencies of around 6s, which is just too high for real-time live video use cases. However it's still the most popular ABR streaming protocol. Its success is mainly due to its great compatibility with a wide range of devices and browsers and supports advanced features like closed captions, ads, and content protection with AES encryption or DRM.  

Although the protocol can also be used to ingest video, it's normally only used for distribution and then paired with the RTMP protocol for ingestion. Below is a typical live streaming architecture involving both the RTMP and HLS protocol.

large_rtmphlslivestreamingworkflow.png

medium_rtmphlslivestreamingworkflow.png

small_rtmphlslivestreamingworkflow.png

thumbnail_rtmphlslivestreamingworkflow.png

rtmphlslivestreamingworkflow.png

Without digging into the details of how HLS works (see our blog [articles](https://api.video/blog/video-trends/what-is-hls-video-streaming-and-how-does-it-work) to learn more on this topic), below is a simple schema describing how playlists and media segmentation gives HLS the ability to do adaptive bitrate streaming (ABS).

Diagram of an HLS playlist and segmentation

large_hlsplaylistsmediasegmentation.png

medium_hlsplaylistsmediasegmentation.png

small_hlsplaylistsmediasegmentation.png

thumbnail_hlsplaylistsmediasegmentation.png

hlsplaylistsmediasegmentation.png

So how did HLS evolve to support lower latencies? In the beginning, HLS only worked with MPEG-TS container formats associated with the H.264/AVC video codec. In 2016, support for fragmented MP4 (FMP4) was added and this led to support for the CMAF format and compatibility with DASH distribution. One year later, HLS added support for H.265/HEVC (FMP4 only), which then allowed for a significant reduction in bandwidth usage.

Thus, Low-Latency HLS, an extension of the base HLS protocol, was finalized in April 2020 and enables low-latency video and live streaming by utilizing partial segments and on-the-fly transfer of those segments while maintaining HLS's inherent scalability. On the first draft of this extension, the protocol required the support of "HTTP/2 Push" which made it difficult to implement. Unsurprisingly, as major CDN vendors chose not to support this feature, killing the ability to scale LL-HLS, as none of the CDN players could cache the contents. Fortunately, Apple listened to community and industry concerns and came back with a new revision of the protocol extension, that removed the need for "HTTP/2 Push," but kept the need of relying on the "HTTP/2" protocol. They merged the spec into HLS making it fully backward compatible with the HLS specification and that also means that all features covered by HLS can be retrieved in LL-HLS. 

LL-HLS works practically the same way as HLS, but makes some important changes to reduce the latency in the packaging process. Below are the most important updates to the specification which make low-latency possible while preserving scalability and ability to do ABR streaming:

- **Partial Segments**: A segment is divided into Partial Segments or "Parts" of a few milliseconds which are referenced in the Media Playlist. These have a very short cache lifetime in contrast to the parent segment which can have a longer lifespan on the CDN. Once a complete segment is generated, its associated Partial Segments can be removed from the media playlist to reduce the bandwidth footprint.
- **Preload hints**: The media playlist can have a “Preload Hints” tag, which allows the player to know that new partial segments are coming so that the player can start to request them and the server will respond to this request as soon as the data is available.
- **Block Playlist Reload**: This feature avoids media playlist polling between the player and the server by adding query parameters to the request which inform the server that the player wants the response only when the playlist includes a new segment or partial segment.
- **Playlist Delta Updates**: Thanks to the new "EXT-X-SKIP" tag, a player can request just the updated part of the Media Playlist and save the transfer cost of the data that it already has.
- **Rendition Reports**: The "EXT-X-RENDITION-REPORT" tag added for each rendition line indexed in the Primary Manifest (aka: Master Manifest) provides some useful information for the Player to operate ABR such as the last sequence number and the last partial segment available for each rendition.

large_ll-hls-workflow-and-media-segmentation.png

medium_ll-hls-workflow-and-media-segmentation.png

small_ll-hls-workflow-and-media-segmentation.png

thumbnail_ll-hls-workflow-and-media-segmentation.png

ll-hls-workflow-and-media-segmentation.png

With current public cloud providers you can expect to reach an end-to-end latency in the range of 2s with appropriate Group of Pictures , partial fragments, and reasonable buffer size. While this is still much slower than what WebRTC can achieve, it involves significantly less complexity and is easier to implement in your existing live streaming architecture. You can also push the protocol to its limits by changing some settings and achieve a latency of around 1s, but this comes at the expense of stability and the quality of the playback experience.

### DASH/LL-DASH

DASH stands for Dynamic Adaptive Streaming over HTTP and was born in April 2012 by the work of the Moving Picture Experts Group (MPEG) in order to create an industry-standard alternative to the HLS protocol owned by Apple. It works similar to HLS in that it's based on the preparation of content at different quality levels and divides each video quality into several chunks indexed in manifest files to enable ABR streaming. 

It also supports content protection with encryption through Common Encryption standard (CENC) which makes it compatible with all common DRM systems.

The main difference between LL-HLS is that LL-DASH is completely agnostic of the codecs used for encoding video and audio streams. Unfortunately this codec freedom comes with the cost that LL-DASH is not compatible with Apple devices relying on iOS, including Apple TV.

In 2017, LL-DASH provided the necessary changes to the standardized protocol to decrease latency to 2s. Behind the scenes, it relied on the Common Media Application Format (CMAF). CMAF offers the ability to play with some useful HTTP features that can be used to decrease latency a lot. The two features that allow Low Latency for DASH with CMAF are "Chunked Encoding" and "Chunked Transfer Encoding" — both part of the HTTP/1.1 spec (prohibited in HTTP/2) that respectively allow you to break a segment into smaller chunks of a few milliseconds that can be sent to the distribution layer once encoded and this is where the "Chunked Transfer Encoding" feature take the relay to allow to distribute encoded chunks on the fly. However to operate this kind of delivery process, the whole distribution stack has to support the chunked transfer encoding feature from the origin right through to the CDN.

Diagram of LL-HLS workflow and media segmentation

large_dashll-dash2.png

medium_dashll-dash2.png

small_dashll-dash2.png

thumbnail_dashll-dash2.png

dashll-dash2.png

### HESP

The High Efficiency Stream Protocol (HESP) is another ABR HTTP based streaming protocol and is the most recent one of all. It was created by THEOPlayer through the HESP Alliance, whose main mission is to work on the standardization of the protocol and its promotion.

HESP has been developed with the aim of solving several of the main constraints of other HTTP streaming protocols. It's goals are:

- To reach ultra low latency(<500ms) while preserving scalability (i.e. it can still work with major CDN vendors)
- To reduce the bandwidth required during the streaming
- To reduce zapping times (zapping is the time it takes to switch between streams, think of it as changing the channels on cable television)

These three performance metrics have a direct impact on the user experience of the audience of a live stream. Given the extremely low latency, it can also be a viable alternative to WebRTC. The main drawback of HESP is that it's a proprietary commercial protocol and–for commercial use–it's quite expensive. Let's dig a bit into a technical overview of how it works.

The main difference with HESP in comparison to other low-latency protocols is that it relies on two streams rather than one. Before we go into the details of how these streams allow us to reach sub-second of latency, let's talk a bit about the different types of frames that are required when streaming video:

In video compression there are several types of frames:

- IDR-Frame (aka: Key Frame)
- I-Frame
- P-Frame
- B-Frame

Let's start with the I-Frame, you will better understand the others. An I-Frame contains the whole image and is coded without reference to any other frame except itself.

The key frame (or IDR-Frame) is a special I-Frame where no frame after the IDR frame can reference any frame before it. That being said, all IDR-Frames are I-Frames but not the other way around. The key frame is used by any player to be able to start a playback.

The P-Frame holds only the changes in the image from the previous one.

The B-Frame holds the changes between the current frame and both the preceding and following frames.

Now that you know the role of the different kinds of frames composing a video, we can go back to the streams that compose the HESP protocol:

- The first stream is called the "Initialization Stream" and contains only key frames.
- The second one is called the "Continuation Stream" and is like a normal encoded stream, meaning that it can contain all types of frames depending on the encoding parameters and defined profiles for maximum performance or maximum compatibility.

The Initialization Stream is used only for the start of the playback or when you slide on the video timeline to change the position of the playback. Because this stream contains only key frames, the decoder behind the player is able to instantly decode the frame and so start(or restart) the playback of the live event. Once the first frame is pulled from the first stream and decoded, the player switches automatically to the second stream to continue video playback. This is because key frames are complete images, so they have a high cost in terms of bandwidth. By switching to the second stream, the player falls back to a regular live streaming bandwidth footprint and profits from the CDN capability for scaling the audience and reducing the load to the origin.

As for DASH/LL-DASH, HESP works also with CMAF-CTE for packaging and distribution. It inherits from all the features such as encryption, DRM support, subtitles and captions, advertisements, and more.

large_hesp-workflow.png

medium_hesp-workflow.png

small_hesp-workflow.png

thumbnail_hesp-workflow.png

hesp-workflow.png

This protocol sounds revolutionary on paper, right? It does respond to many issues, it's true. But as with other protocols–and everything else in life–nothing is perfect and therefore it brings its own set of drawbacks and limitations.

The first drawback is that, as the protocol works with two streams which are encoded in sync, the encoding and storage cost is higher than the other HTTP based streaming protocols.

The second drawback is that even if it relies on CMAF-CTE packaging and distribution, the packager has to be updated to handle the two streams instead of one in the encoding and distribution steps.

Finally, as with the packager, the player must also be updated in order to be able to play with the HESP protocol and handle all of its subtleties.

And finally, last but not least, you have to pay royalties for any commercial use which inherently limits its adoption.

### So which protocol is the best?

The quick answer is none. What a helpful answer for making a choice, right?

The more complete answer is that it will depend heavily on the use case that you want to cover and the time and resources (both financial and human) you have at your disposal.

If your needs is to offer to your customers and audience the ability to live stream with a reasonable amount of latency (6s-15s), while remaining cost effective , we recommend you use HLS and/or DASH protocols as they can scale easily to millions of viewers. You can find many open-source libraries and streaming platforms that have implemented them very well. Our preferred choice, is HLS, as it has the largest footprint in terms of adoption and support across browsers and devices. It can be played virtually anywhere. (If you want to test it, you can try our live streaming service for free just by creating an account [here](https://dashboard.api.video/register)).

In the case that latency is critical for your business, you should look at low latency or even ultra low latency protocols. If you only needs mono-directional live streaming capability with a latency in the range of ~2s (e.g. for sport events, concerts, e-learning with students listening a teacher and can ask questions in a chat) and you don't have a big budget, you should look at LL-HLS and/or LL-DASH protocols.

For the remaining use cases where more than 1s of latency is not acceptable, you don't have many options left. It's either going to be WebRTC or HESP.

If you're a non-profit organization but need to serve a large audience or just don't want to have to build super complex infrastructure and don't have a lot of cash in your pocket, take a look at the HESP protocol.

In case you need bi-directional video communication, WebRTC has proven its ability to do it well, but if building an in-house infrastructure isn't part of your core business, you'll probably want to rely on a provider who already successfully built such an infrastructure in a scalable fashion. Be aware that the likelihood of having to pay a high cost is quite high. Since HESP Alliance's royalty fees are client-based and offer different packages to meet the fees, if you have a reasonable audience this can also be a good option for limited budget.

Finally, if money is not a concern, our choice would be HESP because it involves much less work compared to an implementation of WebRTC and is compatible with a wide range of devices and of browsers.

At [api.video](https://api.video) we deeply believe that HTTP-based Low or Ultra Low Latency streaming protocols will win this battle in the end. The reasons are quite simple, these protocols involve much less work to be implemented in a legacy infrastructure, profit from more support across devices and browsers, can scale more easily than WebRTC, and–as they don't have prohibitive licensing fees–they can be adopted by a larger number of companies. 

This is why we're building end-to-end Edge video infrastructure based on HLS. We're confident that, in the end, HLS will be the single streaming protocol that will meet all video streaming use cases in the near future.

Ultra low latency for live streaming video

large_add-a-subheading.png

medium_add-a-subheading.png

small_add-a-subheading.png

thumbnail_add-a-subheading.png

add-a-subheading.png

The future of [ultra] low-latency video streaming

Top 7 causes of streaming video buffering issues

Nobody likes waiting for buffering. In [one survey](https://www.streamingmedia.com/Articles/News/Online-Video-News/Streaming-Video-Services-Are-Getting-Worse-63-See-Buffering-122408.aspx), 63% of streaming viewers went so far as to say buffering issues were one of the most serious problems they encountered when trying to watch video. Slow buffering can take you out of the moment in an important part of a TV show, film, or game. Sometimes it can end your interest in watching something altogether if the buffering keeps happening with long delays. The way buffering is supposed to work, is your device streams ahead of what you’re watching so that there’s always more video ready for you to see. If everything goes according to plan, you’ll seamlessly be able to watch your video content. But when it doesn’t go to plan, what’s the cause? The source of your buffering problem can come from a few different places.

The easiest way to start diagnosing your buffering problem is to check your internet speed. 
If your internet is slow, you’re going to have buffering problems. There’s a number of reasons your internet could be slow, so it’s best to start with a test. You can choose from any of these free online speed tests: [Internet Speed Test Sites](https://www.lifewire.com/internet-speed-test-sites-2626177). Should you determine that your problem is internet speed, you can then start looking at what’s slowing you down. 

## 1. You’re using wifi

Wifi is great, but sometimes it’s a lot slower than Ethernet. [Ethernet can transmit data at up to 10 Gbps per second, or higher, depending on the service you buy and your equipment. Wifi on the other hand, maxes out at around 6.9 Gbps or less.](https://blog.rottenwifi.com/wifi-vs-ethernet-speed/) If you’re having trouble with buffering, you might consider connecting your playback device using an Ethernet cable if it’s possible. This is a secure, fast solution that might solve your buffering problem. If you can’t set up your device to work with an Ethernet cable (for example if you’re using your mobile phone, or sometimes a device like a Roku or Fire stick), we’ll go over additional ways you can solve buffering issues.

## 2. Too many devices on your system or too many applications running

If you connect the maximum number of devices to your system, or you have too many devices trying to do something at once, this can slow down your network. Then you’ll get buffering problems on what you’re watching. A simple solution is to turn off some of the devices while you’re watching TV or a movie. If you’re not watching on your TV, but using a computer instead, then you might try closing applications and windows. The less your computer has to process, the more it can focus on delivering you a seamless video experience. 

## 3. Your internet plan

Some internet plans don’t provide speeds that are suitable for seamless video viewing or gaming. Check to see what you have, and then use the speed test we talked about earlier to make sure you’re getting what you were promised. If everything seems under the speed offered and you’ve diagnosed everything else, you can consider a different provider. You can also upgrade your plan if you are getting what you were promised, but the internet speed isn’t fast enough for your needs.

## 4. Old technology 

Some people love updating their computers and gadgets and routers all the time. If you’re not one of these people, you might have old computers, TVs, or an old router. Old computers and TVs are less efficient, but the real show stopper here would most likely be an old router. Old routers can cause interference with Wifi connectivity and slow down your internet speeds. So, if you can, try to keep your router up-to-date. It also helps to keep your devices up-to-date. If you can’t, then sometimes you can just update your graphics card and that will help. 

## 5. Change video resolution

If you don’t mind a lower quality video, you can try changing the resolution for your content. Choosing a lower level of quality will make it easier for the video content to be delivered as part of a seamless viewing experience. If you are using a small screen, it’s difficult to tell the difference between 720p and 1080p, so you won’t be missing much. As a general rule of thumb, the smaller your viewing screen, the lower the quality can be. Try different resolutions to see what works for your eyes and your system. 

## 6. Check your VPN 

If you added a VPN to your system, this slows performance because your data must be encrypted then decrypted at a remote server. Something you can do, is test your VPN to figure out if it’s impacting video delivery on your devices. 

## 7. Check your browser settings

If you are viewing from a browser on a computer, the problem could be your web browser. Clearing the cache, removing cookies and browsing history can speed everything up and improve your video viewing experience. 

## api.video and buffering

api.video helps avoid buffering issues by providing blazing fast transcoding and delivery via HLS. HLS uses adaptive bitrate, meaning your customers get the best quality video at the best speed for their devices. api.video also helps you and your customers avoid buffering by using its own, video-focused Content Delivery Network (CDN) and edge servers so that content can be delivered to users as quickly as possible from somewhere local to them. If you’re looking to add video to your website, platform, or application, api.video offers a complete, flexible end-to-end solution for VOD, live streaming, transcoding, hosting, delivery, and usage analytics. Sign up for an [api.video trial account](https://api.video) to try it out. 

large_top-7-causes-of-streaming-video-buffering-issues-2.jpg

medium_top-7-causes-of-streaming-video-buffering-issues-2.jpg

small_top-7-causes-of-streaming-video-buffering-issues-2.jpg

thumbnail_top-7-causes-of-streaming-video-buffering-issues-2.jpg

top-7-causes-of-streaming-video-buffering-issues-2.jpg

Featured article

# api.video blog

Blog

Follow the api.video Blog to explore tutorials, video use cases, and learn about on-demand and live-streaming video updates.