The five design principles applied to every software video codec

Software video codecs are very diverse. Each one is carefully optimized depending on what the codec designers hope to achieve. For example codecs used when capturing video will probably try to retain as much data as possible. However a codec that's going to be used to compress video for conferencing might decrease video quality and increase transmission speed to lower latency. While they may have diverse use cases, one thing software codecs have in common is the design principles applied to their creation. There are five principles applied to every software codec.

Principle 1: Maximize allocation of processing resources based on your codec use case

As I just mentioned, different codecs may hope to achieve different levels of efficiency across their elements based on what they'll be used for. For software codecs, processing resources and available transmission bit rate are what limit the design. Each design is about considering trade-offs, if you put more resources into one area of functionality in a codec, it will become more efficient, but then less resources are available for other areas of functionality. There are four main areas a codec will allocate resources to:

• Encoded frame rate - You'd refer to the use case for your codec, say it's video conferencing, and then try to maximize allocation of resources to efficiently achieve the standard 12-15 frames per second that's expected for desktop video conferencing. If you have a unique, efficient way of increasing quality, maybe you would be able to increase the frames per second. At a minimum though, you'd want to achieve the standard.
• Frame size - Frame size is what it sounds like, how many pixels wide and tall each frame of your video is. Depending on what level of quality or detail you want to offer with your codec, you'd want to maximize frame size as much as possible. Your use case for the codec would dictate how you choose to maximize frame size.
• Peak coded bit rate - The coded bit rate is how many bits are used per second to represent video. While higher coded bit rates mean the video is less compressed, they can be useful if you want to be able to store or transfer video at a high quality. High bit rate is hard on processors, so it's a decision that must be made carefully. Again, depending on your use case you might want to maximize the peak coded bit rate, or focus your resources on other areas that are more important for your codec.
• Video quality - Video quality is tied to bit rate, so depending on use case, strategies will vary. Quality and computational complexity have a complicated relationship, but within any codec there are many places where you can trade off quality for decreased computational complexity.

Principle 2: Minimize latency whenever possible

No matter the codec, the less the delay, the better. Delays are less important for some codecs, but to be avoided at all costs for other codecs - like those used for video conferencing.

Principle 3: Minimize compiled code and storage requirements

Not all systems using software video codecs will have a lot of available memory, so codec designs take this into account. Whenever possible, they cut down on the amount of code that needs to be compiled and the amount of storage needed for the video codec to function.

Principle 4: Make your platform code robust and portable

The more places a codec can easily be used, the greater the chance it will be adopted everywhere. Easy-to-use means the code's been well tested so the codec isn't buggy, and the codec is easy to use on many different platforms.

Principle 5: Provide a flexible API

Make your codec developer friendly by providing an easy-to-use, well documented API.

Resources

1. Chapter 13 of Video Codec Design - http://lad.dsc.ufcg.edu.br/mpeg/VCD/Ch13.pdf
2. Codec - https://en.wikipedia.org/wiki/Codec
3. A Comprehensive Guide to Video Codec Design - https://www.semiconductorstore.com/blog/2014/A-Comprehensive-Guide-to-Video-Codec-Design/866/
4. Thumbnail by Photo by Diego PH on Unsplash