api.video lets you quickly and securely deliver on-demand and live-stream videos directly from your website, software, or app. 

api.video

Glossary-Embed.svg

## What is embedding?
Embedding is the process of adding external content such as links, videos, images, code samples, GIFs, and interactive objects like buttons to a website.  The most common way to embed content on the web is through an iframe.

The iframe allows a developer to insert code from a second source on a webpage in a safe way - the code runs in a different context and cannot hijack or pollute the code on the page.

## Embedding a video
When a video is uploaded to api.video, a videoId is assigned to the video, and a player URL is created that looks like:

```
https://embed.api.video/vod/vi4blUQJFrYWbaG44NChkH27
```

This url can be pasted into the browser and played, but since it has a JavaScript player and other dependencies, it cannot be simply added to a webpage. For that reason, every API response form api.video also includes the embeddable iframe url:  

```
<iframe src="https://embed.api.video/vod/vi4blUQJFrYWbaG44NChkH27" width="100%" height="100%" frameborder="0" scrolling="no" allowfullscreen="true"></iframe>
```

The iframe tells the browser that it can go to 100% the width and height of the container, will fit the size (no scrolling) and can go full screen (so you can see the whole video).

Glossary-FFmpeg.svg

## What is FFmpeg?

[FFmpeg](https://ffmpeg.org/) stands for Fast Forward Moving Picture Experts Group. It is a free and open source software project that offers many tools for video and audio processing. It's designed to run on a command line interface, and has many different libraries and programs to manipulate and handle video files. FFmpeg is available for use in many different programming languages. With FFMPEG, you can decode, encode, transcode, mux, demux, stream, filter and play most types of video and audio.

large_ffmpeg-logo-wine.png

medium_ffmpeg-logo-wine.png

small_ffmpeg-logo-wine.png

thumbnail_ffmpeg-logo-wine.png

ffmpeg-logo-wine.png

## FFmpeg commands

FFmpeg offers three main command line tools:

- **FFmpeg** - FFmpeg is a powerful command-line tool used for converting audio and video formats. It can handle a wide range of multimedia files and streams. This tool is capable of performing various functions such as capturing real-time input from hardware devices (like webcams and audio interfaces) or software sources (like screen capture or stream links), extracting audio from video files, merging multiple videos, adding watermarks, and more.
- **FFplay** - FFplay is a simple media player that uses the Simple DirectMedia Layer (SDL) library and FFmpeg libraries to play multimedia content. It's a straightforward tool to quickly play audio and video files from the command line. While it lacks the advanced playback features of more sophisticated media players, it's useful for testing and debugging media files directly within a command-line environment.
- **FFprobe** - FFprobe a tool for displaying details about your media in text, csv, xml or json. It extracts detailed metadata and technical information from audio and video files, such as codec details, frame rate, bit rate, resolution, duration, and more. The output can be displayed in various formats, including plain text, CSV (Comma-Separated Values), XML (eXtensible Markup Language), or JSON (JavaScript Object Notation), making it easy to parse and use in scripts or other applications.

&nbsp;

**Did you know?** The FFMPEG logo is based on the zigzag scan used in entropy encoding for images and video.

## What is the FFMPEG Project?

The FFMPEG project was started by Farice Bellard under the pseudonym Gérard Lantau in 2000 and was then led by Michael Niedermayer from 2004 until 2015. Due to internal disagreements about how the project was managed, from 2011 on, FFMPEG was split into two forks – FFMPEG and libav. Both projects were nearly identical in terms of what they contained, but distributors would have to choose which to use. Today, libav is reintegrated with FFMPEG. A new release of FFMPEG is published every roughly three months.

&nbsp;  

Most video programs include FFMPEG as a part of the video processing pipeline (**[api.video](https://api.video/)** included). Part of what makes FFMPEG so popular is its inclusion of codecs for the encoding and/or decoding of most audio and video file formats, allowing it to be used for transcoding of all types of media files into a single common format.

## Examples with FFMPEG

Some examples in our sample applications include **[livestream.a.video](https://livestream.a.video/)** and **[record.a.video](https://record.a.video/)**. These two web applications record the camera using webRTC protocols. To stream on the web with api.video, this needs to be converted to RTMP. This is done using FFMPEG on an intermediate NodeJS server:

```
	var ops = [
				'-i','-',
        '-preset', 'ultrafast', '-tune', 'zerolatency', 
			  '-r', framerate, '-g', framerate*2, '-keyint_min',keyint_min, 
     				'-c:a', 'aac', '-b:a','44k', '-ar', 44100, 
			        '-f', 'flv', socket._rtmpDestination		
		
		];
	
		ffmpeg_process=spawn('ffmpeg', ops);
```

The operators take the incoming video stream, and using the ```ultrafast``` preset, and the ```zerolatency``` tuning, convert the video to flv (RTMP flash format), and send the file to the rtmpDestination variable.

In [watermark.a.video](https://watermark.a.video) the livestream further adds an image as a watermark:

```
var ops = [
				'-i', socket._watermarkUrl,
				'-i','-',
				'-preset', 'ultrafast', '-tune', 'zerolatency', 
			  '-r', framerate, '-g', framerate*2, '-keyint_min',keyint_min, 
				'-filter_complex', '[1:v][0:v] overlay=25:25',
     		'-c:a', 'aac', '-b:a','44k', '-ar', 44100, 
			  '-f', 'flv', socket._rtmpDestination		
		
		];
```

In this case, there are 2 inputs, the video, and the image (watermarkUrl).  the filter_complex places the 0 input on top of the 1 input at x,y (25, 25).

These are just 2 small examples of the power that ffmpeg can harness with your videos.

&nbsp;

_api.video enables you to quickly and securely deliver on-demand and live-stream videos directly from your website, software, or app. [Sign-up](https://dashboard.api.video/register) for a free sandbox account to try out api.video’s video API, or [talk to our team](https://meetings.hubspot.com/apivideo/talk-to-us) to get a demo._ 

Glossary-frames-per-second-fps.svg

## What is Video Framerate?

**Framerate**: how frequently the images in a video update. Each image in a video is a frame,and the images update many times per second. The framerate is measured in **frames per second (fps)** and the FPS tells you how many images appear in that second.

In the 'flip book" image above, each page is a frame, and the 'video' updates as fast as the user is able to flip through the pages.  The higher the framerate, the more 'fluid' the motion will appear.

giphy-downsized.gif

## What are common framerates used today?

* Film is typically recorded at 24 fps. 
* TV is typically 29.97 FPS or 59.94 FPS.  The values were initially 30/60 - based on the common electrical frequency of 60Hz, but an artifact in black and white televisions was removed by lowering the frame rate by 0.1%, and as a result, these framerates are still common today.
* Video games are typically played at 60 fps.  With fast motion, it is important to not have any jank or lag in the visuals, and it was found that 60 FPS was best for gaming.
* Cartoons: Cartoons are often 'inked' at 8 ("on threes") or 12 ("on twos") frames per second, and then repeated on screen multiple times to reach the 24 fps expected by viewers. The term "on threes" comes from how often each frame is repeated to get to the 24 fps that is ultimately broadcast.

## Human perception

Until recently, it was assumed that humans cannot see faster than 60 fps.

Most TVs and monitors refresh at 60 Hz (and since Hz has the unit 'per second', you can think of that as 60 refreshes per second).  Some gaming monitors refresh at 90 or even 120 Hz.

A [recent study at MIT](https://mollylab-1.mit.edu/sites/default/files/documents/FastDetect2014withFigures.pdf) showed the brain can process an image in as low as 13ms which comes to ~ 75 fps, indicating that our eye and brain can process motion at a faster rate, if only subconsciously.

## FPS at api.video

Each video uploaded to api.video is encoded into multiple sizes for streaming, and we have optimized the framerate for each stream. The frames per second varies by the size of the video, and as the video dimensions get larger, so does the fps of the video:

* 240p: 18 fps
* 360p: 24 fps
* 480p: 24 fps
* 720p: 24 fps
* 1080p: 30 fps
* 4k:    60 fps





Glossary-Gamma-correction.svg

## What is gamma correction? 

Definition: Gamma correction is a crucial nonlinear operation in digital imaging and video processing that adjusts the luminance (brightness) of images or video content. It consists of two main processes: gamma encoding and gamma decoding. This technique optimizes how brightness is represented in digital media and displayed on screens, accounting for differences between how cameras capture light and how human eyes perceive it.

Graphic depicting perceived linear brightness vs. physical linear brightness

gamma-correction-brightness.png

## Key Aspects:

### Purpose:
- Ensures accurate brightness representation on digital displays
- Optimizes data storage and transmission by focusing on perceptually important information

### Human vs. Camera Vision:
- Cameras have a linear response to light intensity
- Human vision has a nonlinear response, more sensitive to changes in darker areas
- Gamma correction bridges this gap, making digital images appear more natural to human eyes

### Encoding Process:
- Redistributes tonal information to match human perception
- Allocates more data to shadow details and less to highlights
- Compensates for the overemphasis on brightness in camera sensors

### Decoding Process:
- Adjusts stored digital information for proper display on various screens
- Accounts for different display technologies and their characteristics

### Practical Applications:
- Improves overall image quality and detail visibility
- Ensures consistent appearance across different devices and displays
- Essential for accurate color reproduction in digital media

### Considerations:
- Different displays may interpret gamma values differently
- Requires careful balancing to preserve important visual information while optimizing data usage

Gamma correction is fundamental in digital imaging, ensuring that what we see on our screens closely matches our natural visual perception, despite the limitations of both capture and display technologies.

## What is Generative AI?

Generative AI refers to artificial intelligence systems capable of creating new content, including images, text, and most relevantly for our context, video. In the realm of video production and processing, Generative AI algorithms can produce, manipulate, and enhance visual content in ways previously unimaginable. These systems learn patterns from existing data and use that knowledge to generate new, original content that didn't exist before.

## How Generative AI Works in Video
Generative AI in video applications typically relies on [deep learning](https://api.video/what-is/deep-learning/) techniques, particularly advanced neural networks. These networks are trained on vast datasets of video content, learning to understand and replicate patterns in motion, color, texture, and even narrative structure. The [algorithms](https://api.video/what-is/algorithm/) powering Generative AI in video often use architectures like Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs).

For instance, in video generation, the AI might be trained on thousands of hours of footage, learning to predict how scenes typically unfold. It can then use this knowledge to generate entirely new video sequences, complete with realistic motion and consistent visual elements.

## Applications in Video Production
Generative AI is revolutionizing various aspects of video production:
1. Video Creation: AI can generate entire video sequences from text descriptions or rough storyboards, saving time in pre-production and conceptualization.
2. Video Editing: Generative AI can automate tasks like color grading, scene transitions, and even suggest edits based on content and style preferences.
3. Visual Effects: AI can generate complex visual effects, create realistic 3D models, or even digitally age or de-age actors.
4. Video Upscaling: Generative techniques can enhance low-resolution footage, adding detail and improving quality.
5. Content Personalization: AI can generate personalized video content for individual viewers, adapting storylines or product placements in real-time.

## Challenges and Limitations
Despite its potential, Generative AI in video faces several challenges. The computational demands of generating high-quality video require significant processing power, which can be costly and time-consuming. While improving rapidly, AI-generated video can sometimes fall into the "uncanny valley," appearing almost but not quite realistic. The ability to generate realistic fake videos raises important ethical concerns about misinformation and consent.

Additionally, questions arise about who owns the rights to AI-generated content and how it relates to the training data, complicating issues of copyright and ownership. These challenges highlight the complex landscape in which Generative AI for video is developing, balancing technological advancements with practical, ethical, and legal considerations.

## Future of Generative AI in Video
The future of Generative AI in video is promising and potentially transformative:
1. Real-time Video Generation: As processing power increases, we may see AI generating high-quality video in real-time, enabling new forms of interactive entertainment.
2. Automated Video Production: Entire video productions could be generated from scripts, reducing the need for physical sets and potentially democratizing video creation.
3. Enhanced Personalization: Videos could adapt in real-time to viewer preferences, creating truly personalized viewing experiences.
4. Integration with [Large Language Models](https://api.video/what-is/large-language-model/): This could lead to more sophisticated narrative generation and even AI-driven directorial decisions.  

&nbsp;  

As [Generative AI](https://api.video/use-cases/generative-ai/) continues to evolve, it's poised to redefine what's possible in video production, opening new avenues for creativity and efficiency in the industry.

Glossary-DVR.svg

## What is a DVR?

A DVR, or "Digital Video Recorder", is a device or software that enables users to record live television programs and store them digitally for later viewing. This technology has revolutionized the way people consume media, as it provides them with the ability to pause, rewind, and skip through live TV, effectively giving them greater control over their viewing experience.

One of the key features of a DVR is its recording capability. Users can record their favorite shows, movies, or any other video content to the device's internal hard drive, which can typically store hundreds of hours of recorded content depending on the model and storage capacity. This "time-shifting" functionality allows individuals to watch programs at their convenience, rather than being limited to the original broadcast schedule.

In addition to recording, DVRs often have the ability to pause and rewind live television. This can be particularly useful for taking a break during a program or rewinding to re-watch a specific segment.

Moreover, many DVRs are equipped with the capability to automatically skip over commercial breaks, providing users with a more seamless and uninterrupted viewing experience.

The scheduling feature of a DVR is another valuable aspect, as it enables users to plan and set recordings in advance, ensuring they never miss a favorite show or movie. This level of control and flexibility has contributed to the growing popularity of DVRs among television viewers.

cctv-nvr-surveillance-camera-systems.jpeg

At api.video we allow instant rewind of live streams, and we call this our DVR feature.  When a stream is live, you can rewind the live stream up to 5 minutes backwards to watch 'nearly live' content. Perhaps you missed what was said, and want to make sure you understand the speaker.  Or perhaps there was a sports play that you'd like to watch again.  Our DVR feature allows your users to rewind the live stream and see the action again!

## What is dithering?

Dithering is a technique you can use to make a lower bit-depth blend colors by introducing random noise to the image in strategic locations. When is dithering useful? - When you need to represent an image with lower [bit depth](https://api.video/what-is/bit-depth/) than you captured it at, or when you need to use a low bit depth in general. Sometimes a lower bit depth doesn't have enough colors to express the image well. In such cases, you'll see sharp lines or bands of color in places where the color shifts from one shade to another. This makes the image look less realistic, and lower quality. 

Here's an example:

Dithering is used in many different codecs to improve efficiency and storage of image and video. This technique is not limited to video - it's also used on the audio portion of video codecs and it can be applied to images as well.

See how the high bit depth on the left image (24-bit) allows for a more seamless transition between gradients of color. However in the right image (8-bit), you see sharp delineation between the bands of color. By dithering, or applying random noise to these areas, you can soften the delineation. For images, you can often create an approximation of a color by choosing the right colors from a palette and placing them next to each other. The human eye will see the colors mixed together as the desired color even though the palette doesn't contain this color. For example, say you have a color palette that doesn't have purple as a choice. You could approximate purple by placing very tiny red and blue pixels next to each other in the pattern of a checkerboard. If fine enough, your eye won't notice it's created with red and blue, and simply see purple as desired. 

Sometimes a limited color palette is ideal because the image will take less space. Remember that for bit depth, for each pixel enough space has to be available for whatever choice you might make. So in 8-bit color, a space that can handle up to 256 values is stored per pixel. In 10-bit color, a space that can handle up to 1024 values is stored per pixel. Sometimes it's not efficient to store the higher quality option, and this is where dithering is useful. 

Here's an image showing how dithering works. You can see that at first, with big patches of red and blue, you see the colors separately. But as the patches become smaller and smaller, you see more and more purple! 

small_cctv-nvr-surveillance-camera-systems.jpeg

thumbnail_cctv-nvr-surveillance-camera-systems.jpeg

Glossary-Dithering.svg

thumbnail_bit-depth.png

bit-depth.png

thumbnail_dithering-red-blue-samples.png

dithering-red-blue-samples.png

Glossary-Deep-Learning.svg

## What is Deep Learning?
Deep learning is a subset of machine learning that employs artificial neural networks with multiple layers, designed to mimic the human brain's structure and function. In the context of video and AI, deep learning algorithms can process and analyze vast amounts of visual data, enabling sophisticated tasks in video production, editing, and analysis. These neural networks learn hierarchical representations of data, with each layer building upon the previous one to recognize increasingly complex patterns and features in video content.

## Fundamentals of Deep Learning in Video Processing
At its core, deep learning for video relies on convolutional neural networks (CNNs) and recurrent neural networks (RNNs). CNNs excel at spatial analysis, detecting features like edges, textures, and objects within individual video frames. RNNs, on the other hand, are crucial for understanding temporal relationships, allowing the system to track motion and changes over time across multiple frames.

These neural networks are trained on large datasets of video content, learning to recognize patterns and make predictions. The training process involves feeding the network with labeled data, allowing it to adjust its internal parameters through backpropagation. As the network processes more data, it refines its ability to recognize and generate complex video features.

## Applications in Video Technology
Deep learning has revolutionized numerous aspects of video technology. Some key applications include:

1. Video Compression: Reducing file sizes while maintaining visual quality.
2. Video Enhancement: Upscaling low-resolution footage and reducing noise.
3. Object Recognition and Tracking: Enabling automated content moderation and intelligent surveillance.
4. Special Effects: Facilitating seamless integration of CGI elements and real-time face swapping.
5. AI-Powered Editing: Automating cutting, arranging, and even generating video content.
6. Content Analysis: Extracting metadata and understanding video context for better searchability and recommendation systems.  

&nbsp;  

These capabilities are pushing the boundaries of what's possible in video production, allowing for more creative and efficient workflows.

## Challenges and Considerations
Despite its power, deep learning in video faces several challenges. The computational requirements for processing high-resolution video in real-time can be substantial, often necessitating specialized hardware. There's also the issue of interpretability – the complex nature of deep neural networks can make it difficult to understand exactly how they arrive at their outputs, which can be problematic in sensitive applications.

Data quality and quantity remain crucial factors. Deep learning models require vast amounts of diverse, high-quality video data to train effectively and avoid biases. Ensuring this data is representative and ethically sourced is an ongoing challenge in the field.

## The Future of Deep Learning in Video
As deep learning techniques continue to advance, we can expect even more transformative applications in video technology. Researchers are exploring ways to make these models more efficient and capable of learning from smaller datasets. The integration of deep learning with other AI technologies, such as [Large Language Models](https://api.video/what-is/large-language-model/), promises to enable more holistic understanding and generation of video content.

Future developments may lead to fully automated video production systems capable of generating entire films or TV shows based on high-level creative direction. We might also see advancements in personalized video experiences, where content adapts in real-time based on viewer emotions or preferences, all powered by sophisticated deep learning [algorithms](https://api.video/what-is/algorithm/).

## What is a discrete cosine transform (DCT)?

A discrete cosine transform is a type of Fourier transform that's used to compress video. It transforms a digital signal into the sum of its trigonometric functions. A digital signal is a wave, so you can figure out the signal's amplitude over time and turn that into a list of coefficients where each one represents a function that's used to get part of the original wave. 

The question is, how do you pull apart the wave so you can compress it into a set of coefficients representing different functions? And the answer is, by using the discrete cosine transform! Cosine functions are used rather than sine functions, because it's more efficient - you need less cosine functions to describe a signal. In general, a DCT transform takes a vector of length n that contains amplitudes, and returns a new vector of length n with all the coefficients for n cosine functions used to represent the signal. You can encode this in an n x n matrix, where each row represents a cosine function of a different frequency. 

## DCT and block compression
Because an n x n matrix is used in a DCT transform, this technique is often called block compression. You're taking a set of DCT blocks and compressing them. Blocks can be of different sizes, ranging from 4x4 to 32x32 pixels, with the most common being 8x8. 

## How DCT works
This is a very high level explanation of how DCT works. There are a few steps. The frame in your video is divided into blocks of pixels. Then each row of pixels is represented as a cosine wave. This cosine wave can be represented as a row in a matrix. You create a matrix of the size of your block, with a cosine function for each row or wave. Next, you figure out the coefficients for each of the waves in your matrix. This is accomplished through matrix multiplication. The end result is your completed matrix, which you can now compress. You take the K most significant cosine waves and save them. Say you have a matrix that is  So you will lose some data, but if you've chosen correctly, you'll be able to decompress your matrix and have the most important data intact. When you decompress, you add enough zeroes to your compressed matrix to recreate the original size of the matrix you compressed down from. You can then use an inverse DCT equation to get back your compressed data. 

## DCT today
The DCT compression technique has been around since the 70s. Today it's still the most popular algorithm for compressing video and appears in every codec, with various optimizations depending on the codec. 

Video on demand

Live streaming

Analytics

Video infrastructure

Video player

AI features

See all features

Online learning & Corporate training

Social & Entertainment

Marketplace & E-commerce

Communication & UGC

Generative AI

Quickstart guides

Documentation

Ecosystem catalog

Clients & SDKs

Community

Blog

Demos

Customer success stories

Help center

Embed

What is embedding?

Embedding a video