api.video lets you quickly and securely deliver on-demand and live-stream videos directly from your website, software, or app. 

api.video

Glossary-frames-per-second-fps.svg

## What is Video Framerate?

**Framerate**: how frequently the images in a video update. Each image in a video is a frame,and the images update many times per second. The framerate is measured in **frames per second (fps)** and the FPS tells you how many images appear in that second.

In the 'flip book" image above, each page is a frame, and the 'video' updates as fast as the user is able to flip through the pages.  The higher the framerate, the more 'fluid' the motion will appear.

giphy-downsized.gif

## What are common framerates used today?

* Film is typically recorded at 24 fps. 
* TV is typically 29.97 FPS or 59.94 FPS.  The values were initially 30/60 - based on the common electrical frequency of 60Hz, but an artifact in black and white televisions was removed by lowering the frame rate by 0.1%, and as a result, these framerates are still common today.
* Video games are typically played at 60 fps.  With fast motion, it is important to not have any jank or lag in the visuals, and it was found that 60 FPS was best for gaming.
* Cartoons: Cartoons are often 'inked' at 8 ("on threes") or 12 ("on twos") frames per second, and then repeated on screen multiple times to reach the 24 fps expected by viewers. The term "on threes" comes from how often each frame is repeated to get to the 24 fps that is ultimately broadcast.

## Human perception

Until recently, it was assumed that humans cannot see faster than 60 fps.

Most TVs and monitors refresh at 60 Hz (and since Hz has the unit 'per second', you can think of that as 60 refreshes per second).  Some gaming monitors refresh at 90 or even 120 Hz.

A [recent study at MIT](https://mollylab-1.mit.edu/sites/default/files/documents/FastDetect2014withFigures.pdf) showed the brain can process an image in as low as 13ms which comes to ~ 75 fps, indicating that our eye and brain can process motion at a faster rate, if only subconsciously.

## FPS at api.video

Each video uploaded to api.video is encoded into multiple sizes for streaming, and we have optimized the framerate for each stream. The frames per second varies by the size of the video, and as the video dimensions get larger, so does the fps of the video:

* 240p: 18 fps
* 360p: 24 fps
* 480p: 24 fps
* 720p: 24 fps
* 1080p: 30 fps
* 4k:    60 fps





Glossary-Gamma-correction.svg

## What is gamma correction? 

Definition: Gamma correction is a crucial nonlinear operation in digital imaging and video processing that adjusts the luminance (brightness) of images or video content. It consists of two main processes: gamma encoding and gamma decoding. This technique optimizes how brightness is represented in digital media and displayed on screens, accounting for differences between how cameras capture light and how human eyes perceive it.

Graphic depicting perceived linear brightness vs. physical linear brightness

small_gammacorrectionbrightness.png

thumbnail_gammacorrectionbrightness.png

gamma-correction-brightness.png

## Key Aspects:

### Purpose:
- Ensures accurate brightness representation on digital displays
- Optimizes data storage and transmission by focusing on perceptually important information

### Human vs. Camera Vision:
- Cameras have a linear response to light intensity
- Human vision has a nonlinear response, more sensitive to changes in darker areas
- Gamma correction bridges this gap, making digital images appear more natural to human eyes

### Encoding Process:
- Redistributes tonal information to match human perception
- Allocates more data to shadow details and less to highlights
- Compensates for the overemphasis on brightness in camera sensors

### Decoding Process:
- Adjusts stored digital information for proper display on various screens
- Accounts for different display technologies and their characteristics

### Practical Applications:
- Improves overall image quality and detail visibility
- Ensures consistent appearance across different devices and displays
- Essential for accurate color reproduction in digital media

### Considerations:
- Different displays may interpret gamma values differently
- Requires careful balancing to preserve important visual information while optimizing data usage

Gamma correction is fundamental in digital imaging, ensuring that what we see on our screens closely matches our natural visual perception, despite the limitations of both capture and display technologies.

Glossary-Generative-AI.svg

## What is Generative AI?

Generative AI refers to artificial intelligence systems capable of creating new content, including images, text, and most relevantly for our context, video. In the realm of video production and processing, Generative AI algorithms can produce, manipulate, and enhance visual content in ways previously unimaginable. These systems learn patterns from existing data and use that knowledge to generate new, original content that didn't exist before.

## How Generative AI Works in Video
Generative AI in video applications typically relies on [deep learning](https://api.video/what-is/deep-learning/) techniques, particularly advanced neural networks. These networks are trained on vast datasets of video content, learning to understand and replicate patterns in motion, color, texture, and even narrative structure. The [algorithms](https://api.video/what-is/algorithm/) powering Generative AI in video often use architectures like Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs).

For instance, in video generation, the AI might be trained on thousands of hours of footage, learning to predict how scenes typically unfold. It can then use this knowledge to generate entirely new video sequences, complete with realistic motion and consistent visual elements.

## Applications in Video Production
Generative AI is revolutionizing various aspects of video production:
1. Video Creation: AI can generate entire video sequences from text descriptions or rough storyboards, saving time in pre-production and conceptualization.
2. Video Editing: Generative AI can automate tasks like color grading, scene transitions, and even suggest edits based on content and style preferences.
3. Visual Effects: AI can generate complex visual effects, create realistic 3D models, or even digitally age or de-age actors.
4. Video Upscaling: Generative techniques can enhance low-resolution footage, adding detail and improving quality.
5. Content Personalization: AI can generate personalized video content for individual viewers, adapting storylines or product placements in real-time.

## Challenges and Limitations
Despite its potential, Generative AI in video faces several challenges. The computational demands of generating high-quality video require significant processing power, which can be costly and time-consuming. While improving rapidly, AI-generated video can sometimes fall into the "uncanny valley," appearing almost but not quite realistic. The ability to generate realistic fake videos raises important ethical concerns about misinformation and consent.

Additionally, questions arise about who owns the rights to AI-generated content and how it relates to the training data, complicating issues of copyright and ownership. These challenges highlight the complex landscape in which Generative AI for video is developing, balancing technological advancements with practical, ethical, and legal considerations.

## Future of Generative AI in Video
The future of Generative AI in video is promising and potentially transformative:
1. Real-time Video Generation: As processing power increases, we may see AI generating high-quality video in real-time, enabling new forms of interactive entertainment.
2. Automated Video Production: Entire video productions could be generated from scripts, reducing the need for physical sets and potentially democratizing video creation.
3. Enhanced Personalization: Videos could adapt in real-time to viewer preferences, creating truly personalized viewing experiences.
4. Integration with [Large Language Models](https://api.video/what-is/large-language-model/): This could lead to more sophisticated narrative generation and even AI-driven directorial decisions.  

&nbsp;  

As [Generative AI](https://api.video/use-cases/generative-ai/) continues to evolve, it's poised to redefine what's possible in video production, opening new avenues for creativity and efficiency in the industry.

Glossary - Cloud video streaming (2).svg

## What is GPT (Generative Pre-trained Transformer) ?

GPT, which stands for Generative Pre-trained Transformer, is a type of [Large Language Model](https://api.video/what-is/large-language-model/) based on the transformer architecture. While primarily designed for natural language processing tasks, GPT models have significant implications for video technology, particularly in areas where text generation or understanding is crucial.

## How GPT Works
GPT models use a transformer architecture, which includes:
1. Self-attention mechanisms
2. Feed-forward neural networks
3. Positional encodings
4. Layer normalization

&nbsp;  

These components allow the model to process and generate text by understanding context and relationships between words.

## Applications in Video Technology

Although GPT is primarily a text model, it has several applications in video technology. In the pre-production phase, GPT can assist content creators by generating scripts or dialogue for videos. It can also enhance accessibility by creating detailed captions or subtitles through video captioning. For efficient content management, GPT can produce text summaries of video content, which are useful for cataloging or quick reference. The model's capabilities extend to metadata generation, where it can create descriptive tags and categories for video content based on textual descriptions or transcripts.

Furthermore, GPT can power natural language interfaces, enabling voice commands and intuitive interactions for video editing software. These diverse applications demonstrate how GPT, despite its text-based nature, can significantly contribute to various aspects of video production and management.

## Challenges and Considerations

Integrating GPT in video technology presents several challenges:
- Context Limitation: GPT may struggle with very long-form content or maintaining consistency over extended narratives.
- Visual Understanding: As a text-based model, GPT lacks direct visual comprehension, requiring integration with other AI systems for full video understanding.
- Computational Resources: Running and fine-tuning GPT models requires significant computational power.
- Bias and Accuracy: Like other AI models, GPT can perpetuate biases present in its training data.

## Future of GPT in Video Technology

As GPT and similar models evolve, we can anticipate exciting developments in their application to video technology. Future versions may achieve closer multimodal integration with visual processing, enabling a more holistic understanding of video content. This could lead to innovations such as real-time script adaptation during live productions based on immediate feedback.

As GPT technology continues to advance, its integration with video production and processing tools is likely to create new possibilities for content creation, analysis, and interaction, potentially transforming various aspects of the video industry.

## What is the H.261 video codec?
The H.261 codec is the first codec in the H.26x family. Today, it's still the most widely used international compression standard for video conferencing (though it isn't the main part of video conferencing anymore - it's used only for backwards compatibility these days). It was developed by the International Telecommunications Union and released in 1988. There were other codecs before this one, for example H.120, but this was the first one to really be usable. All the other H.26 series of codecs use the design principles that first appeared in this codec. 

## What's unique about the H.261 vieo codec - macroblocks
The H.261 codec is notable for being the first video codec to use macroblocks. Each macroblock in H.261 consisted of a 16x16 array of luma samples and two corresponding 8x8 arrays of chroma samples using 4:2:0 sampling and YCbCr color space.

This codec uses 8x8 discrete cosine transform (DCT) to reduce spatial redundancy. Something unique about this standard is it only specifies how to decode video. So all kinds of different motion estimation algorithms and other concepts were applied to compress and encode the video. Encoders can pre-process in any way they want and decoders can post-process any way they want so long as they decode the video according to the standard. A popular technique still employed today was deblocking filtering. Because this codec uses blocks of pixels to do compression and because it's the first of its kind, the blocks end up looking...well blocky. So to offset that, people use tricks like deblocking filtering or dithering to try to reduce the appearance of the blocks in the final decompressed video. 

H.261 as people know it today is very different from how it was back in 1988 -- it went through several revisions to make it a complete standard. One of the more unique features added to it was a backwards-compatible way to transmit still images. To achieve this, images were subsampled horizontally and vertically, dividing the picture into four sub pictures that were sent sequentially to form the complete picture. 

## H.261 codec today
The H.261 codec is becoming obsolete with newer members of this codec family. However, it's still one of the most popular international standards for video conferencing, so you're likely to see it in use a bit longer while everything converts to using newer codecs. And you'll likely see the basics of the design for H.261 in the rest of the family in the use of DCT and macroblocking. 



Glossary-FFmpeg.svg

## What is FFmpeg?

[FFmpeg](https://ffmpeg.org/) stands for Fast Forward Moving Picture Experts Group. It is a free and open source software project that offers many tools for video and audio processing. It's designed to run on a command line interface, and has many different libraries and programs to manipulate and handle video files. FFmpeg is available for use in many different programming languages. With FFMPEG, you can decode, encode, transcode, mux, demux, stream, filter and play most types of video and audio.

ffmpeg-logo-wine.png

## FFmpeg commands

FFmpeg offers three main command line tools:

- **FFmpeg** - FFmpeg is a powerful command-line tool used for converting audio and video formats. It can handle a wide range of multimedia files and streams. This tool is capable of performing various functions such as capturing real-time input from hardware devices (like webcams and audio interfaces) or software sources (like screen capture or stream links), extracting audio from video files, merging multiple videos, adding watermarks, and more.
- **FFplay** - FFplay is a simple media player that uses the Simple DirectMedia Layer (SDL) library and FFmpeg libraries to play multimedia content. It's a straightforward tool to quickly play audio and video files from the command line. While it lacks the advanced playback features of more sophisticated media players, it's useful for testing and debugging media files directly within a command-line environment.
- **FFprobe** - FFprobe a tool for displaying details about your media in text, csv, xml or json. It extracts detailed metadata and technical information from audio and video files, such as codec details, frame rate, bit rate, resolution, duration, and more. The output can be displayed in various formats, including plain text, CSV (Comma-Separated Values), XML (eXtensible Markup Language), or JSON (JavaScript Object Notation), making it easy to parse and use in scripts or other applications.

&nbsp;

**Did you know?** The FFMPEG logo is based on the zigzag scan used in entropy encoding for images and video.

## What is the FFMPEG Project?

The FFMPEG project was started by Farice Bellard under the pseudonym Gérard Lantau in 2000 and was then led by Michael Niedermayer from 2004 until 2015. Due to internal disagreements about how the project was managed, from 2011 on, FFMPEG was split into two forks – FFMPEG and libav. Both projects were nearly identical in terms of what they contained, but distributors would have to choose which to use. Today, libav is reintegrated with FFMPEG. A new release of FFMPEG is published every roughly three months.

&nbsp;  

Most video programs include FFMPEG as a part of the video processing pipeline (**[api.video](https://api.video/)** included). Part of what makes FFMPEG so popular is its inclusion of codecs for the encoding and/or decoding of most audio and video file formats, allowing it to be used for transcoding of all types of media files into a single common format.

## Examples with FFMPEG

Some examples in our sample applications include **[livestream.a.video](https://livestream.a.video/)** and **[record.a.video](https://record.a.video/)**. These two web applications record the camera using webRTC protocols. To stream on the web with api.video, this needs to be converted to RTMP. This is done using FFMPEG on an intermediate NodeJS server:

```
	var ops = [
				'-i','-',
        '-preset', 'ultrafast', '-tune', 'zerolatency', 
			  '-r', framerate, '-g', framerate*2, '-keyint_min',keyint_min, 
     				'-c:a', 'aac', '-b:a','44k', '-ar', 44100, 
			        '-f', 'flv', socket._rtmpDestination		
		
		];
	
		ffmpeg_process=spawn('ffmpeg', ops);
```

The operators take the incoming video stream, and using the ```ultrafast``` preset, and the ```zerolatency``` tuning, convert the video to flv (RTMP flash format), and send the file to the rtmpDestination variable.

In [watermark.a.video](https://watermark.a.video) the livestream further adds an image as a watermark:

```
var ops = [
				'-i', socket._watermarkUrl,
				'-i','-',
				'-preset', 'ultrafast', '-tune', 'zerolatency', 
			  '-r', framerate, '-g', framerate*2, '-keyint_min',keyint_min, 
				'-filter_complex', '[1:v][0:v] overlay=25:25',
     		'-c:a', 'aac', '-b:a','44k', '-ar', 44100, 
			  '-f', 'flv', socket._rtmpDestination		
		
		];
```

In this case, there are 2 inputs, the video, and the image (watermarkUrl).  the filter_complex places the 0 input on top of the 1 input at x,y (25, 25).

These are just 2 small examples of the power that ffmpeg can harness with your videos.

&nbsp;

_api.video enables you to quickly and securely deliver on-demand and live-stream videos directly from your website, software, or app. [Sign-up](https://dashboard.api.video/register) for a free sandbox account to try out api.video’s video API, or [talk to our team](https://meetings.hubspot.com/apivideo/talk-to-us) to get a demo._ 

## What is embedding?
Embedding is the process of adding external content such as links, videos, images, code samples, GIFs, and interactive objects like buttons to a website.  The most common way to embed content on the web is through an iframe.

The iframe allows a developer to insert code from a second source on a webpage in a safe way - the code runs in a different context and cannot hijack or pollute the code on the page.

## Embedding a video
When a video is uploaded to api.video, a videoId is assigned to the video, and a player URL is created that looks like:

```
https://embed.api.video/vod/vi4blUQJFrYWbaG44NChkH27
```

This url can be pasted into the browser and played, but since it has a JavaScript player and other dependencies, it cannot be simply added to a webpage. For that reason, every API response form api.video also includes the embeddable iframe url:  

```
<iframe src="https://embed.api.video/vod/vi4blUQJFrYWbaG44NChkH27" width="100%" height="100%" frameborder="0" scrolling="no" allowfullscreen="true"></iframe>
```

The iframe tells the browser that it can go to 100% the width and height of the container, will fit the size (no scrolling) and can go full screen (so you can see the whole video).

large_ffmpeg-logo-wine.png

medium_ffmpeg-logo-wine.png

small_ffmpeg-logo-wine.png

thumbnail_ffmpeg-logo-wine.png

Glossary-Embed.svg

Glossary-DVR.svg

## What is a DVR?

A DVR, or "Digital Video Recorder", is a device or software that enables users to record live television programs and store them digitally for later viewing. This technology has revolutionized the way people consume media, as it provides them with the ability to pause, rewind, and skip through live TV, effectively giving them greater control over their viewing experience.

One of the key features of a DVR is its recording capability. Users can record their favorite shows, movies, or any other video content to the device's internal hard drive, which can typically store hundreds of hours of recorded content depending on the model and storage capacity. This "time-shifting" functionality allows individuals to watch programs at their convenience, rather than being limited to the original broadcast schedule.

In addition to recording, DVRs often have the ability to pause and rewind live television. This can be particularly useful for taking a break during a program or rewinding to re-watch a specific segment.

Moreover, many DVRs are equipped with the capability to automatically skip over commercial breaks, providing users with a more seamless and uninterrupted viewing experience.

The scheduling feature of a DVR is another valuable aspect, as it enables users to plan and set recordings in advance, ensuring they never miss a favorite show or movie. This level of control and flexibility has contributed to the growing popularity of DVRs among television viewers.

cctv-nvr-surveillance-camera-systems.jpeg

At api.video we allow instant rewind of live streams, and we call this our DVR feature.  When a stream is live, you can rewind the live stream up to 5 minutes backwards to watch 'nearly live' content. Perhaps you missed what was said, and want to make sure you understand the speaker.  Or perhaps there was a sports play that you'd like to watch again.  Our DVR feature allows your users to rewind the live stream and see the action again!

## What is dithering?

Dithering is a technique you can use to make a lower bit-depth blend colors by introducing random noise to the image in strategic locations. When is dithering useful? - When you need to represent an image with lower [bit depth](https://api.video/what-is/bit-depth/) than you captured it at, or when you need to use a low bit depth in general. Sometimes a lower bit depth doesn't have enough colors to express the image well. In such cases, you'll see sharp lines or bands of color in places where the color shifts from one shade to another. This makes the image look less realistic, and lower quality. 

Here's an example:

Dithering is used in many different codecs to improve efficiency and storage of image and video. This technique is not limited to video - it's also used on the audio portion of video codecs and it can be applied to images as well.

See how the high bit depth on the left image (24-bit) allows for a more seamless transition between gradients of color. However in the right image (8-bit), you see sharp delineation between the bands of color. By dithering, or applying random noise to these areas, you can soften the delineation. For images, you can often create an approximation of a color by choosing the right colors from a palette and placing them next to each other. The human eye will see the colors mixed together as the desired color even though the palette doesn't contain this color. For example, say you have a color palette that doesn't have purple as a choice. You could approximate purple by placing very tiny red and blue pixels next to each other in the pattern of a checkerboard. If fine enough, your eye won't notice it's created with red and blue, and simply see purple as desired. 

Sometimes a limited color palette is ideal because the image will take less space. Remember that for bit depth, for each pixel enough space has to be available for whatever choice you might make. So in 8-bit color, a space that can handle up to 256 values is stored per pixel. In 10-bit color, a space that can handle up to 1024 values is stored per pixel. Sometimes it's not efficient to store the higher quality option, and this is where dithering is useful. 

Here's an image showing how dithering works. You can see that at first, with big patches of red and blue, you see the colors separately. But as the patches become smaller and smaller, you see more and more purple! 

Video on demand

Live streaming

Analytics

Video infrastructure

Video player

AI features

See all features

Online learning & Corporate training

Social & Entertainment

Marketplace & E-commerce

Communication & UGC

Generative AI

Quickstart guides

Documentation

Ecosystem catalog

Clients & SDKs

Community

Blog

Demos

Customer success stories

Help center

Framerate/frames per second (fps)

What is Video Framerate?

What are common framerates used today?

Human perception

FPS at api.video