api.video lets you quickly and securely deliver on-demand and live-stream videos directly from your website, software, or app. 

api.video

Glossary-Lossy-compression.svg

## What is lossy compression? 
Lossy compression is a way of compressing data where you reduce the file size by removing data in the file. Another name for it is irreversible compression, because you can't return to the original image, video, or audio that you started out with. In discussions, you will often hear either lossy vs. lossless compression or irreversible and reversible compression. (Though the latter terms are more common if you are looking at medical data.) 

Simple diagram of lossy compression, boxes going from original to compressed to restored.

large_lossy.png

medium_lossy.png

small_lossy.png

thumbnail_lossy.png

lossy.png

## What are some popular lossy compression techniques?
The most common lossy data compression techniques are transform coding and predictive coding. Generally speaking, transform coding involves taking samples of images or sound, chopping them into small segments, then quantizing them. Quantization is the process of determining what parts of an image you can discard or consolidate with minimal loss in quality. The most common lossy compression algorithm is the discrete cosign transform (DCT), which was first published by Nasir Ahmed and K.R. Rao in 1974. Predictive coding involves taking previous or following decoded data to predict the current sample of audio or an image frame. 

The goal of lossy compression is to be able to reconstruct data in a way where you don't notice that some of the original data is missing. Many popular [video compression standards](https://api.video/blog/video-trends/the-history-of-video-compression-starts-in-1929/) are lossy, for example H.264, H.265, AV1 and VP9 to name a few.

As the algorithms for compression improve, videos are able to be compressed at a greater rate, with less perceived loss of quality.  This is due to the improved quantization math that is run in each generation of video compression.  This improved compression allows for larger dimension video to be transmitted without drastic bandwidth improvements.  For example, the compression between VP9 and AV1 (a generational shift in the same platform) reduces the file size by nearly 50%.

While lossy compression can help video be transmitted more rapidly and take up less space, this type of format suffers from generation loss. Over time if you continue to compress and decompress your file using lossy compression methods you will decrease the quality of your file over time. Best practice is to keep a back up of your video files using a lossless compression method so you can preserve a high quality version of your file to use when needed.

## api.video and lossy compression 
At api.video, we transcode all content into HLS, which is the industry standard for streaming your video in the highest quality possible across the internet. HLS uses H.264 video compression, which is lossy in a way that you won't notice. 

Glossary-Machine-Learning.svg

## What is Machine Learning?

### Definition and Fundamental Concepts

Machine Learning (ML) is a subset of [Artificial Intelligence](https://api.video/what-is/artificial-intelligence/) that focuses on the development of algorithms and statistical models that enable computer systems to improve their performance on a specific task through experience. In the context of video technology, ML algorithms can learn from and make predictions or decisions based on video data without being explicitly programmed for each specific scenario.

## Types of Machine Learning in Video Processing
Machine Learning in video processing typically falls into three main categories:
1. Supervised Learning: Models learn from labeled video data to make predictions or classifications.
2. Unsupervised Learning: Algorithms identify patterns in unlabeled video data.
3. Reinforcement Learning: Systems learn optimal actions through trial and error in a video-based environment.  

&nbsp;  

These approaches enable a wide range of applications in video analysis, editing, and generation.

## Key Machine Learning Techniques for Video
Several ML techniques are particularly relevant to video technology:
- Convolutional Neural Networks (CNNs): Excel at analyzing spatial information in video frames, crucial for object recognition and scene understanding.
- Recurrent Neural Networks (RNNs): Effective for processing sequential data, allowing analysis of temporal aspects in videos.
- Generative Adversarial Networks (GANs): Used for generating new video content or enhancing existing footage.
- Transfer Learning: Allows models trained on one video task to be repurposed for another, saving time and computational resources.

## Applications in Video Technology
Machine Learning has transformed numerous aspects of video technology:
- Video Classification and Tagging: ML models can automatically categorize videos based on content, genre, or style.
- Object Detection and Tracking: Algorithms can identify and follow specific objects or people across video frames.
- Video Summarization: ML techniques can create concise summaries of longer videos, extracting key moments or scenes.
- Video Enhancement: ML models can upscale resolution, reduce noise, or even colorize black and white footage.
- Content-Based Recommendation Systems: ML algorithms analyze viewing habits and video content to suggest relevant videos to users.
- Automated Video Editing: ML-powered tools can assemble rough cuts, suggest edits, or even generate entire video sequences based on learned patterns.

## Challenges and Considerations
While powerful, Machine Learning in video technology faces several challenges:
- Data Requirements: ML models often require large amounts of diverse, high-quality video data for training.
- Computational Intensity: Processing and analyzing video data can be computationally expensive, especially for real-time applications.
- Interpretability: Many ML models, particularly deep learning ones, operate as "black boxes," making it difficult to understand their decision-making process.
- Bias and Fairness: ML models can inadvertently learn and perpetuate biases present in their training data.

## The Future of Machine Learning in Video
As ML techniques continue to advance, we can expect several exciting developments in video technology:
- Real-Time Processing: Improvements in hardware and algorithms will enable more sophisticated ML operations to be performed in real-time, even on mobile devices.
- Multimodal Learning: Future ML models will likely integrate learning from visual, audio, and textual data more seamlessly, enabling more comprehensive video understanding.
- Personalized Content Creation: ML could enable highly personalized video content, dynamically adapting to individual viewer preferences and contexts.
- Advanced Video Synthesis: We may see ML models capable of generating highly realistic video content from minimal input, potentially revolutionizing areas like visual effects and virtual production.  

&nbsp;  

As Machine Learning continues to evolve, it will undoubtedly play a crucial role in shaping the future of video technology, offering new possibilities for content creation, analysis, and user experience while also presenting new challenges and ethical considerations for the industry to address.

Glossary-Manifest-file.svg

## What is a video manifest file?

A video manifest gets its name from one of the definitions of a manifest. According to the Merriam-Webster dictionary: "a list of passengers or an invoice of cargo for a vehicle (such as a ship or plane)"

In the context of video, a manifest file is a text-based document that includes details about the media stream. It serves as a guide for the media player, storing information on how to fetch and play the video or audio content.

In [**HTTP Live Streaming (HLS)**](https://api.video/what-is/http-live-streaming-hls/) video streaming, manifest files have the extension `m3u8`.

## HLS streaming

A video stream can have multiple manifest files: a single primary playlist, which is historically called a 'master' playlist, and multiple media playlists.

## Primary manifest

The primary manifest is the first file that a player requests when video playback begins. This file lists all of the available streaming versions of the video that are available. There are 2 lines per version, and they typically look like this:

```
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4400000,RESOLUTION=3840x2160, CODECS="avc1.42e030,mp4a.40.2"
/processing_video/2160/manifest.m3u8
```

The first line of this example manifest stores information about the media stream:
- requires a bandwidth of 4.4 MBPS,
- has 4k video resolution (3840x2160),
- requires codecs describes the codecs `avc1.42e030` or `mp4a.40.2`, so the player can find a compatible version for a web browser.

&nbsp;  

The second line is a path to where this specific version of the video can be found. Notice that this path points to another `m3u8` manifest file, which is the media manifest file. The primary manifest has links that point to each streaming version’s media manifest.

## Media manifest

This file lists all of the video segments available for the video stream.  Media manifests generally have this structure:

```
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-ALLOW-CACHE:NO
#EXT-X-TARGETDURATION:4
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-KEY:METHOD=AES-128,URI="aes.key",IV=0x07B1D7401B75CA2937353EC4550FEE5D
#EXTINF:4.000000,
0000.ts
#EXTINF:4.000000,
0001.ts
#EXTINF:4.000000,
0002.ts
#EXTINF:4.000000,
0003.ts
#EXTINF:4.000000,
0004.ts
#EXTINF:0.120000,
0005.ts
#EXT-X-ENDLIST

```

In this example, each segment is 4 seconds long (EXTINF), and there is a link to the file.

## Usage in players
The video player uses the primary mainfest to decide which quality version of the video should be played. The player makes this decision based on these factors:
- the available network speed
- the size of the playback window

&nbsp;  

If any of these parameters change, the player can adjust the stream to a different quality version. In this case the player requests the media manifest of that version and immediately switches to the media files at a given time in the video playback to ensure uninterrupted playback.

## How api.video uses manifest files
At api.video, every video you upload is encoded as an HLS video stream, with up to 6 different qualities / bitrates. For every upload, the API responds with 3 links:
- the primary manifest
- the player URL
- an `iframe` that you can use for embedding

&nbsp;  

The video player and the `iframe` embed links use the primary manifest to begin playback. With just a simple upload, all your HLS streaming and manifest needs are taken care of!

Glossary-Metadata.svg

## What is a Metadata?

In the context of digital video, metadata refers to the additional information that is embedded within a video file, providing valuable data about the content, its creation, and its intended use. This metadata can enhance the organization, searchability, and overall management of video assets across various platforms and workflows.

## Understanding Video Metadata
Video metadata encompasses a wide range of information that can be associated with a video file, including:
1. **Descriptive Metadata**: This includes details about the video's content, such as the title, description, keywords, tags, and categorization.
2. **Technical Metadata**: This covers information about the video's technical specifications, including the codec, bit rate, resolution, frame rate, and duration.
3. **Administrative Metadata**: This data provides details about the video's creation, such as the author, production date, copyright information, and licensing details.
4. **Structural Metadata**: This type of metadata describes the internal organization and structure of the video, such as chapters, scenes, or specific points of interest.
5. **Contextual Metadata**: This includes information about the video's use or purpose, such as its intended audience, language, or accessibility features (e.g., closed captions, audio descriptions).  

&nbsp;  

Video metadata plays a crucial role in various aspects of video management and distribution, including:
- **Search and Discovery**: Metadata enables users to easily find and retrieve relevant video content based on various search criteria.
- **Workflow Automation**: Metadata can be used to automate tasks like transcoding, captioning, or asset management within video production and post-production workflows.
- **Analytics and Reporting**: Metadata provides valuable insights into video performance, audience engagement, and other important metrics.
- **Personalization and Recommendation**: Contextual metadata can be used to personalize video recommendations and enhance the user experience.

## Leveraging Video Metadata
To effectively leverage video metadata, it's important to follow established standards and best practices. This includes using industry-recognized metadata schemas, such as Dublin Core, IPTC, or EXIF, which provide a common vocabulary and structure for describing video content.

By consistently capturing and maintaining accurate video metadata, content creators, media organizations, and video platforms can unlock the full potential of their video assets, improving discoverability, accessibility, and overall video management efficiency.  

&nbsp;  

At api.video, we recognize the importance of video metadata, see our [blog article](https://api.video/blog/tutorials/video-metadata/) about it! Our platform provides comprehensive support for video metadata, allowing you to easily add, manage, and leverage this valuable information to enhance the discoverability, organization, and distribution of your video content.  Whether you're a content creator, a media company, or a video platform operator, unlock the full potential of your video assets with metadata with api.video. [Sign up Now](https://dashboard.api.video/register)!

## What are AI Models?

In the context of video and AI technology, models refer to mathematical representations or frameworks designed to process, analyze, or generate video content. These models are typically computer programs that have been trained on large datasets to perform specific tasks related to video processing. They serve as the backbone of various AI-powered video applications, ranging from content analysis to video generation.

## Types of Models in Video AI

Several types of models are commonly used in video AI:

1. Predictive Models: Forecast future frames or events in a video sequence.
2. Classification Models: Categorize videos or elements within videos into predefined classes.
3. Generative Models: Create new video content or modify existing footage.
4. Detection Models: Identify and locate specific objects or features within video frames.
5. Segmentation Models: Divide video frames into meaningful segments or regions.

These models often leverage [neural networks](https://api.video/what-is/neural-networks/) and other [machine learning](https://api.video/what-is/machine-learning/) techniques to achieve their objectives.

## Key Components of Video AI Models

Video AI models are typically composed of several key components that work in concert to process and generate video content. At the core is the model's architecture, which defines the overall structure, such as convolutional layers for spatial analysis or recurrent layers for temporal processing. The model's parameters are the adjustable values that are fine-tuned during the training process to optimize performance. A loss function serves as a measure of how well the model is performing its intended task, while an optimizer algorithm is responsible for adjusting the model's parameters to minimize this loss.

Crucial to the model's operation are the input and output layers, which serve as interfaces for receiving video data and producing the desired results. These interconnected components, each playing a vital role, form the foundation of sophisticated Video AI systems capable of tackling a wide range of video-related tasks and applications.

## Applications in Video Technology

Models play a crucial role in various video technology applications:
- Video Compression: Models can learn efficient representations of video data, enabling more effective compression techniques.
- Content Analysis: AI models can automatically tag, categorize, and describe video content, facilitating easier search and organization.
- Visual Effects: Generative models can create or enhance visual effects, such as style transfer or object removal.
- Video Editing: Models can assist in tasks like automatic color grading, scene detection, or even suggesting edits based on content and style.
- Quality Enhancement: Models can upscale resolution, reduce noise, or correct artifacts in video footage.
- Personalized Content Delivery: Recommendation models can analyze user preferences and video characteristics to suggest relevant content.

## Challenges and Considerations

Developing and implementing models for video AI presents several challenges:
- Computational Complexity: Processing high-resolution video data often requires significant computational resources.
- Data Requirements: Many models need large amounts of diverse, high-quality video data for effective training.
- Real-Time Performance: Achieving low-latency performance for real-time video applications can be challenging, especially on consumer devices.
- Interpretability: Complex models, particularly deep learning models, can be difficult to interpret, raising concerns about transparency and accountability.
- Bias and Fairness: Models can inadvertently learn and perpetuate biases present in their training data, leading to unfair or discriminatory outcomes.

## The Future of Models in Video AI

As technology advances, we can anticipate exciting developments in video AI models. Future models will likely integrate analysis of visual, audio, and textual data more seamlessly, enabling more comprehensive video understanding. Advancements in model compression and efficient architectures will enable more powerful models to run on edge devices. Self-supervised learning approaches could reduce the need for large amounts of labeled data, making it easier to train models for specific video tasks.

Additionally, adaptive models that can dynamically adjust their architecture or parameters based on input video characteristics or task requirements, as well as the development of techniques to make complex video AI models more interpretable and transparent, are promising areas of future progress in this rapidly evolving field.

As models continue to evolve, they will drive innovation in video technology, enabling more sophisticated analysis, editing, and generation capabilities. However, addressing challenges related to efficiency, fairness, and ethical use will be crucial in realizing the full potential of these models in the video industry.

Glossary-Loop.svg

## What is the video loop attribute in HTML5?
**Loop** is an attribute used in the HTML5 video tag.  When the loop attribute is added to the video, it tells the player to restart the video when it completes, causing a loop in the video.  When the loop tag is added to a video, the video will loop indefinitely, there is no way to define just one or two loops of the video.
 
When combined with [muted](https://api.video/what-is/muted/) and [autoplay](https://api.video/what-is/autoplay/) the effect is similar to that of an animated GIF - with the advantage that typically video files are smaller than an animated GIF.

## Loop with api.video

**Video**: 

For a single video, you can add #loop to the end of the player url, and the video will loop:

Or you can add the loop parameter when you use the player SDK:
 
```
   player.loadConfig({
      id: "<VIDEO_ID>",
      hideTitle: true,
      loop: true,
   });
 
```
 
**Player**: 
 
If you have many videos that you'd like to have loop, you can create a video player that defaults to looping videos, and assign the videos you wish to loop to that player by initializing ```forceLoop``` to true:
 
```
curl --request POST \
  --url https://ws.api.video/players \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data '
{
  "enableApi": true,
  "enableControls": true,
  "forceAutoplay": false,
  "hideTitle": false,
  "forceLoop": true
}
'
```   

&nbsp;  

Learn more about our [video player](https://api.video/product/player/), fully customizable, with prefect playback quality.

## What is video latency? 
Video latency is a measure of how long it takes between when a video stream is requested and when transfer of the video stream starts. Latency is used when you are working with a live stream. In the smallest unit of measurement, its how long it takes one frame of video to move from where it's being captured by a camera to the player you want it to appear on. Sometimes this is referred to as glass-to-glass.


## When is low video latency most important?
There are some situations where latency might not matter as much. For example, if you're watching a playback of an event, latency might not be a big deal, you can wait a few moments to get started watching. But if you're having a conversation, it might become difficult to have your conversation if there's drag between video of you and the other participants reaching one another. The most common use cases where you'd want low latency are:

* **Two-way communication** - meetings and broadcasts where you want to speak with your audience real time.
* **Betting and auctions** - anything where people need to place a bid or make a bet, you're going to need everything to be as up-to-date and realtime as possible.
* **Video games** - you need to be able to respond to what's happening in your game real-time.

## What causes latency?
When live streaming, your latency is affected by many things: 

* **Device** - you are limited by what your device is capable of.
* **Distance to travel** - the further your video is from its destination, the greater the chance of latency.
* **Video codec and vide size** - bigger files take longer to transmit, and different codecs will handle the files at different speeds, resulting in higher or lower latency.
* **Bandwidth** - the higher the bandwidth the faster you can stream video, resulting in lower latency.

## How is latency categorized?
You may have heard about different types of streaming described with different latency terms like standard, low and others. But how much latency occurs for each type? Here's a quick list: 

* **Standard broadcast latency** - 5-18 seconds, TV and some VOD online have this much latency. 
* **Low latency** - 1-5 seconds, cable TV and some VOD have this much latency.
* **Ultra-low latency** - Less than 1 second, when you have this much latency, you're at the start of what can be used for real-time streaming. It's suitable for two-way communication, though you will sometimes notice a pause if you're trying to do something complex, like play music together across a two-way connection. 
* **Real-time latency** - Milliseconds, you can't tell that there's a delay in most cases. This can still be noticeable for things that require precision, again the best example would be if you're trying to do something like play music, where the timing needs to be close to perfect for everyone to be together. 

## Video latency and api.video
At api.video, enjoy [low latency live streaming](https://api.video/product/live-stream/), stream your videos with a latency of 3 seconds in ideal conditions.




Glossary-Latency.svg

small_hourglass-620397640.jpg

thumbnail_hourglass-620397640.jpg

hourglass-620397640.jpg

Glossary-Large-Language-Model.svg

## What is Large Language Model (LLM)?

A Large Language Model (LLM) is a type of artificial intelligence system designed to understand, process, and generate human-like text based on vast amounts of training data. In the context of video and AI, LLMs play a crucial role in understanding and generating text-based content related to video, such as scripts, captions, and metadata. These models use [deep learning](https://api.video/what-is/deep-learning/) techniques to process and produce language at a scale and complexity that mimics human-like understanding.

## How Large Language Models Work
LLMs are based on complex neural network architectures, typically using transformer models. They are trained on enormous datasets, often containing billions of words from diverse sources. Key aspects of their functionality include:
1. Tokenization: Breaking down input text into smaller units (tokens) for processing.
2. Attention Mechanisms: Allowing the model to focus on relevant parts of the input when generating output.
3. Contextual Understanding: Grasping the nuances and context of language beyond simple word-for-word translation.  

&nbsp;  

In video applications, LLMs can process textual information associated with video content, enabling sophisticated analysis and generation tasks.

## Applications in Video Production and AI
Large Language Models (LLMs) have several important applications in the video and AI industry, spanning various aspects of content creation and management. In the realm of creative development, LLMs can assist in script generation, producing dialogue or entire storylines based on given prompts or themes. They also enhance accessibility by processing audio transcripts to generate accurate and context-aware captions for videos. For content management, LLMs excel at video summarization, creating concise text summaries useful for cataloging or quick reference.

These models contribute to improved searchability and organization through content tagging and metadata generation. In terms of user interaction, LLMs enable sophisticated natural language interfaces, allowing for intuitive voice commands and interactions with video editing software and AI-powered production tools. Furthermore, when integrated with video analysis AI, LLMs facilitate cross-modal understanding by providing text descriptions of visual content, effectively bridging the gap between visual and textual information. This diverse range of applications demonstrates the versatility and potential of LLMs in revolutionizing various aspects of video production and AI-driven content creation.

## Challenges and Limitations
While powerful, LLMs face several challenges in the video and AI domain:
1. Context Limitation: LLMs may struggle with very long-form content or maintaining consistency over extended narratives.
2. Bias and Accuracy: Models can perpetuate biases present in their training data, potentially leading to unfair or inaccurate representations in video content.
3. Computational Resources: Running and fine-tuning LLMs requires significant computational power, which can be costly.
4. Integration with Visual AI: Seamlessly combining language understanding with visual processing remains a complex challenge.

## Future of Large Language Models in Video and AI

The future of LLMs in video and AI looks promising:
1. Multimodal Models: Future LLMs may integrate more closely with visual and audio processing, enabling more holistic understanding of video content.
2. Real-time Script Adaptation: LLMs could enable dynamic script adjustments during live productions based on real-time feedback or changing conditions.
3. Enhanced Creative Assistance: More sophisticated models could serve as creative partners in the video production process, offering ideas and solutions to complex narrative challenges.
4. Improved Localization: Advanced LLMs could provide more nuanced and culturally appropriate translations and adaptations of video content for global audiences.
5. AI Directors: Combined with other AI technologies, LLMs could potentially guide entire video productions, making high-level creative decisions based on narrative understanding.  

&nbsp;  

As Large Language Models continue to evolve, their integration with video technology is likely to create new possibilities for content creation, analysis, and interaction, potentially transforming the landscape of video production and consumption.

## What is inter-frame compression? 
Inter-frame compression is a type of compression that is based on what happens across frames. By taking advantage of the fact that sometimes very little changes from frame-to-frame, you can store less information and reduce the overall size of a video. A simple example of interframe compression is taking one frame and saving the entire frame. This frame becomes a keyframe that is used as a reference.

Moving forward in a video, an interframe compression algorithm would only make note of changes that differ from the keyframe. This is all that's stored. Later, when the video is decompressed, all that has to be added is the changes that occur in the video from the keyframe you are comparing against. If there is a dramatic change in the video, for example a new scene begins, then a new keyframe would be selected and used for the next block of frames as the reference point. More keyframes means smoother playback, but a bigger video file. 

## I, P and B Frames 
In the simple example I went over in the earlier section I talked about keyframes. There are several types of frames that are common in video compression algorithms that use inter-frame compression. These are called I, P and B frames. 

### What are I-Frames?
I-frame is short for intra-coded frame. For this type of frame, compression techniques are limited to those which act only upon that frame. These frames are also referred to as keyframes, because these are the frames that are completely stored as-is and used as reference points for the other frames in the video. 

### What are P-Frames? 
P-frame is short for prediction frame. With inter-frame prediction, the frame is divided into blocks. The encoder then tries to find blocks that match or are similar to those of a previously encoded frame (which would be an I-frame or keyframe).

For blocks that match, then the block is encoded by pointing to the reference block it matches. This pointer can later be used to restore that matching block to the frame. In cases where the block doesn't match exactly the encoder computes the difference and this information is saved for use when decoding later. The P-frame must have an I-frame to reference in order to perform the calculations and block matching.

### What are B-Frames?
A B-frame does something similar to what P-frames are doing, however these are predicted frames that can be created based on information in either direction. B-frames are predicted from information contained in P and I-frames. 

## S-Frames
S-frame is short for switching frame. These are relatively new, and are part of the AV1 codec. It's a P-frame that only uses an I-frame, all other information must be provided by the video stream. This type of frame is used to allow Adaptive Bit Rate (ABR) to stream smaller segments of frames. A segment can start with an S-frame copied from a previous I-frame and then the decoder knows it can decode from this frame without worrying about whether it references data that comes before this frame.

Something unique about an S-frame is it can be a different resolution from the original I-frame based on what's available for streaming. If you're interested in learning more about S-frames and how they're used in AV1 to improve video compression, check out this talk: [Tarek Amara - S-Frame in AV1: Enabling better compression for low latency live streaming](https://www.youtube.com/watch?v=o5sJX6VA34o). If you're interested in learning more about the history of video compression, check out [The History of Video Compression Standards, From 1929 Until Now](https://api.video/blog/video-trends/the-history-of-video-compression-starts-in-1929/).

Video on demand

Live streaming

Analytics

Video infrastructure

Video player

AI features

See all features

Online learning & Corporate training

Social & Entertainment

Marketplace & E-commerce

Communication & UGC

Generative AI

Quickstart guides

Documentation

Ecosystem catalog

Clients & SDKs

Community

Blog

Demos

Customer success stories

Help center

Lossy compression

What is lossy compression?

What are some popular lossy compression techniques?

api.video and lossy compression