api.video
SearchCtrl K

Features

Developers

Use FFMPEG to resize and overlay two videos, and then live stream them

August 23, 2021 - Doug in ffmpeg

Overlaying two videos with FFMPEG

FFMPEG is a free and open source software project that offers many tools for video and audio processing. It's designed to run on a command line interface, and has many different libraries and programs to manipulate and handle video files.

The Question

How can I best overlay 2 live camera feeds with FFMPEG to produce one live stream to broadcast at api.video? I’m using a raspberry PI, and having a lot of latency issues.

We then began working on the FFMPEG command that he was using to build the livestream.

For testing purposes, I replaced the two livestreams with videos stored locally on my computer. My goal is to create a video that looks like this: Die Hard with an Indiana Jones (and the Last Crusade) overlay.

My command began with:

ffmpeg  -i DieHard.mp4 -i Indianajones.mp4

The -i tells ffmpeg that these are input files, and then the filename. If you were using a livestream, or a remote video you could use a url.

The end of the ffmpeg command looks like:

-f flv rtmp://broadcast.api.video/s/23df9302-5897-4fca-8b27-0eb39ea67ab2

This says to FFMPEG, save the video output as flv and send it to my livestream endpoint. When video begins arriving at this endpoint, your api.video stream will begin broadcasting whatever video is being sent. At this point, it will just take the first input (DieHard.mp4) and stream that file - ignoring Indiana Jones. I need to do some work to add in the Indiana Jones video...

If you want to test locally, you can change the output to:

test.mp4

To save the video locally to your computer.

Initial command

ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 test.mp4

This takes in the 2 videos, outputs just DieHard.mp4 and saves it into test.mp4. We want to overlay the Indiana Jones video.

The first thing we’ll do is scale the Diehard video to 1080p (it is 1080p, but this will ensure that all future videos fit correctly), and we will give the output of this transformation the name ‘main’

 -filter_complex '[0:v]scale=-1:1080[main]’

This command says - take the video from input 1, and resize it to -1:1080. The 2nd value is pretty obvious - make the video 1080 high. The ‘-1’ on the horizontal axis says make the height 1080, and scale the width so that the video is not stretched in any way. If the input video is 16:9,the output will be 1920x1080. If the input video is 4:3, the output will be 1440x1080.

Let’s resize Indiana Jones to 360p (again using the ‘-1’ to ensure that the video is not stretched), and name the resized video “overlay”. We place a semicolon between the two commands:

 -filter_complex '[0:v]scale=-1:1080[main];[1:v]scale=-1:360[overlay]’

Now we have 2 correctly sized videos, let’s set the overlay on top of the main video. In creating the overlay, we could hard code the coordinates, but we can also use variables. In this case, main_w and main_h are for the main video width and height, and overlay_w and overlay_h are assigned to the overlay video.

We'll place the overlay in the bottom right - 25 pixels from the right and 25 pixels from the bottom. To do this, we’ll subtract 25 and the dimension of the overlay from the dimension of the main video. The command will look like:

[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)

The [main] video is used as the main video, and the [overlay] as the overlay, and then we use the variables to place the overlay video.

The full command now looks like:

ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 -filter_complex '[0:v]scale=-1:720[main];[1:v]scale=-1:240[overlay];[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)' test.mp4

And the video looks exactly how we’d like it to look! The audio, however, is from the main video: Die Hard. We can remove the audio (making the video silent) by only exporting the video from the output video. To do this, we can use the map parameter. We’ll name the overlay filter out and then just map the first track (track 0) to the output:

ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 
-filter_complex '[0:v]scale=-1:720[main];[1:v]scale=-1:240[overlay];[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)[out]' -map '[out]'0:0  test.mp4

This results in a silent movie. We can programmatically pick which audio track (track 1 in both cases) with a second -map command:

  • -map 0:1 selects the Die Hard audio (the original 0 index input, track 1)
  • -map 1:1 selects Indiana jones audio (the original 1 index input, track 1)

Here are the final 2 commands (with the stream parameters added in:

ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 -filter_complex '[0:v]scale=-1:720[main];[1:v]scale=-1:240[overlay];[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)[out]' -map '[out]'0:0 -map 0:1 -f flv rtmp://broadcast.api.video/s/23df9302-5897-4fca-8b27-0eb39ea67ab2

ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 -filter_complex '[0:v]scale=-1:720[main];[1:v]scale=-1:240[overlay];[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)[out]' -map '[out]'0:0 -map 1:1 -f flv rtmp://broadcast.api.video/s/23df9302-5897-4fca-8b27-0eb39ea67ab2

Here’s what the final result looks like (Die Hard audio version):

In order to create his live stream, our developer simply changed the input from 2 movie trailers to his 2 live streams, and it all just worked.

Conclusion

As Indiana Jones famously did not say: “What is FFMPEG? Fortune and Glory kid. Fortune and Glory.”

And we are pretty sure John McClain never said “Yippiee Ki Yay FFMPEG!”

But that is because neither of them ever got the chance to play with the tooling that allowed us to create the video above.

In this post, we’ve talked through how to scale the dimensions of multiple input videos, how to overlay them, and then how to apply an audio track to the videos. Finally, the video created by ffmpeg is streamed to api.video for live streaming consumption by your viewers.

Doug

Head of Developer Relations

Connect your users with videos

Create a free account today