api.video lets you quickly and securely deliver on-demand and live-stream videos directly from your website, software, or app. 

api.video

Glossary-OTT.svg

## What is Over the top (OTT)?

Over the Top (OTT) describes how a video can be delivered to customers directly over the internet.  The term derives from the delivery "on top" of basic internet protocols, bypassing the traditional providers of video: TV, cable and satellite. OTT video streaming has revolutionised the way movies and shows are watched by consumers.

There are many OTT providers, including Netflix, Hulu, Disney+ and more.  Many "traditional broadcasters have created their own OTT services (HBO Max, Peacock (NBC), etc). 

OTT delivery has led to new viewing strategies by customers. Now that shows are available on demand, consumers are no longer bound to a specific time or day of the week for a TV show. As a result, viewing patterns have changed in a way that is more convenient to the viewer. Further, the concept of "binging" a show - watching an entire season in the course of a few hours or days has also arisen from OTT video streaming.

## OTT at api.video

api.video hosts and streams video on demand.  If you have OTT content that you want delivered quickly and efficiently worldwide, we can host and deliver that content for you.  We also offer "private videos'' that only allow authenticated users access to the movies, and prevent the sharing of the video urls.


Glossary - Poster.svg

## What is the video poster HTML5 attribute? 
The video poster is the HTML5 attribute for the video tag that defines the image that appears in front of a video before it starts playing.

If no poster tag is defined in the video tag, the browser will display the first frame of the video. Since many videos begin with a black screen, this is not visually appealing, so a different screenshot from the video, or perhaps an unrelated image is used.

A list of different live streams, two are news, one is Rick Astley, one is Doug

large_screenshot-2021-06-14-at-13-30-50.png

medium_screenshot-2021-06-14-at-13-30-50.png

small_screenshot-2021-06-14-at-13-30-50.png

thumbnail_screenshot-2021-06-14-at-13-30-50.png

poster-example.png

The poster tag points to an image: 

```
poster=https://example.com/poster.jpg
```

There is no way to serve alternative formats with the poster tag, so typically a format that is supported in many browsers (like jpg or png) are used.  

&nbsp;  

> Note: The poster image is downloaded before the video, so if your video is going to autoplay, this will block the video from loading.  Additionally, creating a poster of the first frame of the video is redundant, as the player will do this anyway, and this results in that frame being downloaded twice.

## Poster at api.video
- The poster tag is called *thumbnail* at api.video.  
- The default thumbnail is selected from the "middle frame" of the video (example: the 15 minute point of a 30 minute video).

## Changing the thumbnail
- Set a frame from the video: The [pick a thumbnail](https://docs.api.video/reference#patch_videos-videoid-thumbnail) endpoint sends a timestamp where the image should be extracted from the video and set as the poster.
- [Upload a Thumbnail](https://docs.api.video/reference#post_videos-videoid-thumbnail): Upload a jpg image to be used as the video thumbnail.


Glossary-Preload.svg

## What is the preload attribute in HTML5?
Preload is an attribute of the HTML5 video tag.  The preload attribute lets the developer advise the browser how much of the video should be preloaded prior to viewing.  There are 3 values for the preload attribute:
- None: preload=none tells the browser that there is a video, but to download 0 bytes of the file.  This clearly saves data, but at the cost of a delayed video startup - there is no video stored locally on the device.
- metadata: preload=metadata tells the browser to download a percentage of the video (typically 2-5% of the file) to get metadata (size, bitrate, etc. from the video, as well as have a bit of teh video ready for immediate playback.  **IF preload is not set, the default setting is metadata**
- auto: preload=auto tells the browser to download the entire video.  This is the most data intensive, but results in full playback of the video when the video playback has started.  *If the preload attribute is not assigned a value, the auto selection is used*  

&nbsp;  

> Note that ```preload=auto``` is a suggestion that the entire file can be downloaded, but the browser may have a different algorithm and download a larger percentage than ```preload=metadata``` instead of the entire file.  

&nbsp;  

The autoplay attribute takes precedence over preload. For example, in the following video tag:
```
<video src=myvideo.mp4 preload=none autoplay>
```
The browser will download the entire video.

## Preload at api.video
Preload is not typically used with streaming video, as the video player can adapt playback based on  network conditions - giving a more nuanced approach to video download.

Glossary-Rate-limit.svg

## What is an API rate limit?
An API rate limit is a threshold placed on the number of API calls that can be placed over a period of time.  For example: you may only mayke 3 login attempts an hour, or make 150 calls to Twitter every 5 minutes.

API services have upper limits on the number of requests that can be handled every second (millisecond).  If one user (or several users) begin taking a substantial portion of the available bandwidth of the API - the API will fail to work for others,  By placing limits on individual users, API developers can protect their infrastructure from too many requests.

Rate limits can also affect bandwidth. For those in the USA, your "unlimited 4G data" mobile data plan is typically 25 or 40 GB at 4G speeds, and then you are rate limited - your plan is throttled to 2g for the rest of the month.

ratelimit.png

## Rate limits at api.video

We currently have no rate limits on our API, but we do monitor for accounts that are using excessive numbers of calls, and will work with that developer to mitigate their usage.  We are considering rate limits on video uploads, but these are still under development.

### Rate limits on video playback

There are no limits on video playback. If there was ever an unprecedented global "viral video" at api.video, we would be able to handle the load.  On the first request from our backend, api.video would deliver the video to our global video CDN, where it would be cached.  The delivery from our CDN will be fast an efficient, and since it is abstracted from our infrastructure - will not impact and other video playback from api.video.

## What is RTMP?
RTMP stands for Real Time Messaging Protocol and for many years, it's been the standard for streaming video over the internet. What really put RTMP on the map was Adobe's Flash Player. Flash became so popular that browsers and encoders everywhere rushed to support it, which included support for RTMP. Because of this, today most encoders and media servers can support RTMP. That's why it's still the standard for streaming video and live streaming. 

The RTMP streaming protocol is TCP-based (Transmission Control Protocol) and designed to maintain constant, low-latency connections between a video player and server. The design allows RTMP to provide smooth streaming for viewers. And because RTMP uses TCP, it's very reliable. TCP uses a three way handshake when transporting data - the client contacts the server, the server accepts and responds, then the client acknowledges the response. At this point a session is opened, and RTMP can start streaming video. 

RTMP was popular up through the early 2010s, but then came the rise of HLS (created in 2009) and adaptive bitrate streaming. Adaptive bitrate streaming allows you to optimize content based on available connectivity and viewer devices. This type of streaming requires the use of HTTP-based protocols. So today, RTMP is used to move between servers, but when it's time to deliver content to the viewer, the last mile of delivery usually uses HLS and adaptive bitrate delivery. 

RTMP will likely continue to be popular because so many encoders and servers still support it, but over time, other protocols will replace it. If you want more details about the benefits and drawbacks of RTMP, check out api.video's blog post: [What is RTMP and Why Do We Use it for Live Streaming?](https://api.video/blog/video-trends/what-is-rtmp-and-why-do-we-use-it-for-live-streaming/)

For Live streaming, check this article [comparing RTMP to SRT](https://api.video/blog/video-trends/srt-vs-rtmp/).

Glossary-neural-networks.svg

## What are Neural Networks?

Neural Networks are a fundamental component of [deep learning](https://api.video/what-is/deep-learning/), inspired by the structure and function of the human brain. In the context of video technology, neural networks process vast amounts of visual data to perform tasks such as object recognition, scene understanding, and even video generation. These networks consist of interconnected nodes (neurons) organized in layers, which work together to analyze and interpret complex patterns in video content.

## Types of Neural Networks in Video Processing
Several types of neural networks are particularly relevant to video technology:
1. Convolutional Neural Networks (CNNs): Excel at processing grid-like data, making them ideal for analyzing individual video frames.
2. Recurrent Neural Networks (RNNs): Specialize in sequential data, allowing them to process temporal aspects of video.
3. Long Short-Term Memory Networks (LSTMs): A type of RNN that can capture long-term dependencies in video sequences.
4. Generative Adversarial Networks (GANs): Used for generating new video content or enhancing existing footage.

## How Neural Networks Process Video Data
Neural networks process video data through a series of transformations:
- Input Layer: Receives raw video data, typically as a sequence of frames.
- Hidden Layers: Apply various operations (e.g., convolutions, pooling) to extract features from the input.
- Output Layer: Produces the final result, which could be a classification, a generated frame, or other video-related information.  

&nbsp;  

During training, the network adjusts its internal parameters through backpropagation, minimizing the difference between its predictions and the actual desired outputs.

## Applications in Video Technology
Neural networks have revolutionized numerous aspects of video technology. They enable video classification, automatically categorizing videos based on content, genre, or style. Object detection and tracking algorithms can identify and follow specific objects or people across video frames. Neural networks also play a crucial role in video enhancement, allowing for upscaling resolution, reducing noise, or even colorizing black and white footage.

Additionally, they are employed in video generation, creating new video content based on learned patterns or textual descriptions. Neural networks have even been applied to video compression, developing more efficient algorithms for video storage and streaming. Finally, facial recognition techniques leverage neural networks to identify and analyze faces in video footage for various applications.

## Challenges and Considerations
While powerful, neural networks in video technology face several challenges:
- Computational Requirements: Training and running complex neural networks on video data can be computationally intensive.
- Data Hunger: Neural networks often require large amounts of diverse, high-quality video data for effective training.
- Interpretability: The complex nature of neural networks can make it difficult to understand their decision-making processes.
- Overfitting: Networks may become too specialized to their training data, performing poorly on new, unseen video content.

## The Future of Neural Networks in Video
As neural network technologies continue to advance, we can expect several exciting developments. Improvements in hardware and network architectures will enable more sophisticated neural network operations to be performed in real-time, even on mobile devices. Advancements in 3D and 4D convolutions will allow networks to better understand spatial and temporal relationships in video.

Future networks will likely integrate the processing of visual, audio, and textual data more seamlessly, enabling more comprehensive video understanding through multimodal approaches. 

Additionally, self-supervised learning techniques could reduce the need for large amounts of labeled data, making it easier to train networks for specific video tasks.

Neural networks will undoubtedly offer new possibilities for content creation, analysis, and user experience. However, addressing their challenges related to efficiency, interpretability, and ethical use will be key to realizing their full potential in the video industry.

## What is Natural Language Processing?

Natural Language Processing (NLP) is a branch of [Artificial Intelligence](https://api.video/what-is/artificial-intelligence/) that focuses on the interaction between computers and human language. In the context of video technology, NLP plays a crucial role in understanding, generating, and manipulating text and speech associated with video content. It bridges the gap between human communication and computer understanding, enabling more intuitive and powerful video-related applications.

## Key Components of NLP in Video Technology
NLP can play a vital role in video technology. It involves several key components that enable a wide range of video applications. Speech recognition converts spoken words in videos into text, while text analysis helps understand the meaning and context of written or transcribed content. Sentiment analysis determines the emotional tone of language used in videos. Machine translation facilitates cross-language accessibility by translating video content between languages. Text generation creates human-readable descriptions or summaries based on video content or metadata.

These NLP capabilities enable a wide range of video applications, from automated captioning and subtitling to advanced video search and personalized recommendations, making them an integral part of the video technology landscape.

## NLP Techniques Relevant to Video
Several NLP techniques are particularly relevant to video technology:
- Tokenization: Breaking down text or speech into individual words or subwords.
- Part-of-Speech Tagging: Identifying grammatical components in sentences.
- Named Entity Recognition: Identifying and classifying named entities (e.g., people, places, organizations) in text.
- Semantic Analysis: Understanding the meaning and relationships between words and phrases.
- Language Modeling: Predicting the probability of sequences of words, crucial for generating coherent text.

## Applications in Video Technology
NLP has transformed numerous aspects of video technology:
- Automatic Captioning and Subtitling: Generating accurate text captions for video content, including speaker identification and timing alignment.
- Content Summarization: Creating concise text summaries of video content, useful for video indexing and quick reference.
- Video Search and Retrieval: Enabling natural language queries to find specific moments or topics within large video libraries.
- Script Analysis and Generation: Assisting in screenplay writing or generating dialogue for animated characters.
- Voice-Controlled Video Editing: Allowing editors to use natural language commands to manipulate video content.
- Metadata Generation: Automatically creating descriptive tags and categories for video content based on audio and visual analysis.

## Challenges and Considerations
While powerful, NLP in video technology faces several notable challenges. Adapting to various languages and cultural contexts in global video content requires robust multilingual and multicultural understanding. Maintaining context retention over long video sequences or across multiple scenes can also prove difficult.

Additionally, accurately processing the informal language, colloquialisms, and slang often used in videos presents a significant hurdle. Integrating language understanding with the visual elements in the video is another key challenge. Importantly, ensuring the responsible use of NLP technologies, especially in sensitive content analysis, raises crucial privacy and ethical concerns that must be addressed. Overcoming these multifaceted challenges will be crucial as NLP continues to play an increasingly integral role in video technology.

## The Future of NLP in Video
As NLP techniques continue to advance, we can expect several exciting developments in video technology:
- Multimodal NLP: Closer integration of language processing with visual and audio analysis for more comprehensive video understanding.
- Real-Time Translation and Dubbing: Instant, high-quality translation and lip-synced dubbing of video content into multiple languages.
- Advanced Conversational Interfaces: More natural and context-aware voice control for video editing and navigation systems.
- Emotion-Aware Content Creation: NLP models that can understand and generate emotionally nuanced scripts or captions.
- Personalized Content Adaptation: Tailoring video content, including dialogue and text elements, to individual viewer preferences and comprehension levels.  

&nbsp;  

As Natural Language Processing continues to evolve, it will play an increasingly crucial role in enhancing the creation, distribution, and consumption of video content. By bridging the gap between human language and video technology, NLP will enable more intuitive, accessible, and personalized video experiences across various platforms and applications.

Glossary-Natural-Language-Processing.svg

Glossary-Muted.svg

## What is the muted attribute in HTML5?
The muted attribute is used in the HTML5 video player to tell the browser that playback should begin without audio.  When the attribute is omitted (or set to false), the video will begin playback with audio.

The muted attribute is most often paired with [autoplay](https://api.video/what-is/autoplay/). Many browsers will not autoplay a video with audio playback, so the combination of  ```muted autoplay``` will ensure that videos will autoplay in most browsers.

If you intend to have a silent video, you might want to consider removing the audio track from the video itself, rather than send the audio to each viewer and have the browser hide the audio.

1200px-muteicon-svg.png

### Muted with api.video

* [PlayerSDK](https://docs.api.video/sdks/player/apivideo-player-sdk): you can set the video to muted on loading:

```
   player.loadConfig({
      id: "<VIDEO_ID>",
      muted: true,
      hideControls: true,
   });
```


## What is a Multimodal AI

Multimodal AI refers to artificial intelligence systems that can process and integrate information from multiple types of input or "modalities." In the context of video technology, this typically involves combining visual data with other forms of input such as audio, text, or even sensor data. Multimodal systems aim to achieve a more comprehensive understanding of content by leveraging the complementary information provided by different modalities.

## Key Modalities in Video AI
In video technology, multimodal AI often deals with the following modalities:
1. Visual: The actual video frames or images
2. Audio: The soundtrack, including speech, music, and sound effects
3. Text: Subtitles, captions, or associated metadata
4. Temporal: The sequential nature of video frames
5. Contextual: Additional information like user data or environmental sensors

## Multimodal Techniques in Video Processing
Several techniques are crucial for multimodal video processing:
- Cross-modal Learning: Training models to understand relationships between different modalities.
- Fusion Strategies: Methods for combining information from multiple modalities, such as early fusion, late fusion, or hybrid approaches.
- Attention Mechanisms: Techniques that allow the model to focus on relevant parts of each modality.
- Transfer Learning: Applying knowledge gained from one modality to improve performance in another.

## Applications in Video Technology
Multimodal AI has transformed various aspects of video technology:
- Content Understanding: Providing more accurate and nuanced interpretation of video content by considering multiple information sources.
- Enhanced Search and Retrieval: Enabling more precise video search based on combinations of visual, audio, and textual queries.
- Accessibility Features: Improving automatic captioning and audio description by leveraging multiple modalities.
- Video Summarization: Creating more comprehensive and accurate video summaries by considering both visual and audio content.
- Emotion Recognition: Analyzing facial expressions, voice tone, and speech content for more accurate emotion detection in videos.
- Virtual Assistants: Powering more sophisticated AI assistants that can understand and respond to multimodal video-related queries.

## Challenges and Considerations
Multimodal AI in video technology faces a range of challenges, including data alignment to ensure proper synchronization and integration of data from different modalities, scalability issues due to the increased computational requirements of processing multiple data streams, modality imbalance where one modality may be more informative or reliable than others, interpretability concerns around understanding how the model integrates and weighs information from different modalities, and privacy and ethical considerations stemming from the increased potential for invasive analysis when combining multiple data sources.

Overcoming these hurdles will be crucial as multimodal AI continues to evolve and become more widely adopted in video applications.

## The Future of Multimodal AI in Video
As multimodal AI techniques continue to advance, we can expect several exciting developments:
- More Seamless Integration: Improved methods for fusing information from different modalities, leading to more natural and comprehensive video understanding.
- Expansion of Modalities: Incorporation of new types of data, such as biometric or environmental information, into video analysis.
- Personalized Experiences: Multimodal systems that can adapt to individual users' preferences and behaviors across different modalities.
- Advanced Content Creation: AI-powered tools that can generate or edit videos based on multimodal inputs, such as combining text, sketches, and voice commands.

&nbsp;  

Real-time Multimodal Processing: Faster and more efficient algorithms enabling sophisticated multimodal analysis for live video applications.

As multimodal AI continues to evolve, it promises to revolutionize how we create, analyze, and interact with video content, offering richer, more contextual, and more personalized video experiences across various platforms and applications.

Video on demand

Live streaming

Analytics

Video infrastructure

Video player

AI features

See all features

Online learning & Corporate training

Social & Entertainment

Marketplace & E-commerce

Communication & UGC

Generative AI

Quickstart guides

Documentation

Ecosystem catalog

Clients & SDKs

Community

Blog

Demos

Customer success stories

Help center

OTT (over the top)

What is Over the top (OTT)?

OTT at api.video