Inter-frame compression

What is inter-frame compression?

Inter-frame compression is a type of compression that is based on what happens across frames. By taking advantage of the fact that sometimes very little changes from frame-to-frame, you can store less information and reduce the overall size of a video. A simple example of interframe compression is taking one frame and saving the entire frame. This frame becomes a keyframe that is used as a reference.

Moving forward in a video, an interframe compression algorithm would only make note of changes that differ from the keyframe. This is all that's stored. Later, when the video is decompressed, all that has to be added is the changes that occur in the video from the keyframe you are comparing against. If there is a dramatic change in the video, for example a new scene begins, then a new keyframe would be selected and used for the next block of frames as the reference point. More keyframes means smoother playback, but a bigger video file.

I, P and B Frames

In the simple example I went over in the earlier section I talked about keyframes. There are several types of frames that are common in video compression algorithms that use inter-frame compression. These are called I, P and B frames.

What are I-Frames?

I-frame is short for intra-coded frame. For this type of frame, compression techniques are limited to those which act only upon that frame. These frames are also referred to as keyframes, because these are the frames that are completely stored as-is and used as reference points for the other frames in the video.

What are P-Frames?

P-frame is short for prediction frame. With inter-frame prediction, the frame is divided into blocks. The encoder then tries to find blocks that match or are similar to those of a previously encoded frame (which would be an I-frame or keyframe).

For blocks that match, then the block is encoded by pointing to the reference block it matches. This pointer can later be used to restore that matching block to the frame. In cases where the block doesn't match exactly the encoder computes the difference and this information is saved for use when decoding later. The P-frame must have an I-frame to reference in order to perform the calculations and block matching.

What are B-Frames?

A B-frame does something similar to what P-frames are doing, however these are predicted frames that can be created based on information in either direction. B-frames are predicted from information contained in P and I-frames.

S-Frames

S-frame is short for switching frame. These are relatively new, and are part of the AV1 codec. It's a P-frame that only uses an I-frame, all other information must be provided by the video stream. This type of frame is used to allow Adaptive Bit Rate (ABR) to stream smaller segments of frames. A segment can start with an S-frame copied from a previous I-frame and then the decoder knows it can decode from this frame without worrying about whether it references data that comes before this frame.

Something unique about an S-frame is it can be a different resolution from the original I-frame based on what's available for streaming. If you're interested in learning more about S-frames and how they're used in AV1 to improve video compression, check out this talk: Tarek Amara - S-Frame in AV1: Enabling better compression for low latency live streaming. If you're interested in learning more about the history of video compression, check out The History of Video Compression Standards, From 1929 Until Now.