Normal

doug.jpg

twitter-brands.svg

Doug Twitter

Doug Sillars

Tutorials

ffmpeg

# 🎶🎶🎶 Now here's the story... 🎶🎶🎶

As a child, the Brady Bunch was a regular part of my afternoon. It wasn't a new show, but every day after school there were cartoons and the Brady Bunch reruns on TV.

For those not from the USA (it was interesting that at api.video, it was just the Americans who knew the show), the story revolves around a single dad with 3 boys who marries a single mom with 3 girls. They hired a wacky housekeeper, and together the nine characters were the Brady Bunch. Years of laugh track hijinks followed their adventures.

## The opening

The image above is a short GIF, but each character is looking around and smiling at the other members of the Brady clan. It's an iconic US television look, and I thought it would be fun to re-create a "Brady Bunch" video using FFMPEG to interlace 9 different videos into that familiar 9-grid pattern.


> A huge Thank You! to Anne-Sophie and Cedric for recording all of our colleagues (and of course to my api.video colleagues for going along with this fun project).

## Starting my FFMPEG command

After pitching this idea to the team, I woke up a few days later to a number of videos shared in a Slack channel. Woo hoo! Here we go!

When composing an FFMPEG command, the first step is to input all of the files you will be using. We'll do a **-filter_complex** to resize and build the video layout, and then output the video in **done.mp4**.

```
ffmpeg -i sorin.mp4 -i olia.mp4	-i maria.mp4  -i josephine.mp4 
-i yohann.mp4 -i anne-sophie.mp4 -i brady_erikka.mov 
-i cedric.mov -i pierre.mov  -i T3.mov  -i fancy.mov 
-i romain.mov	-i jb.mov 
-filter_complex '<we'll get here in a second>' 
done.mp4

```

For the filter complex, we'll generate a 4K video.   4K is 3840x2160, and is super convenient as it allows us to place 9 1280x720 videos into the "Brady grid":




large_screenshot-2021-09-10-at-08-46-09.png

medium_screenshot-2021-09-10-at-08-46-09.png

small_screenshot-2021-09-10-at-08-46-09.png

thumbnail_screenshot-2021-09-10-at-08-46-09.png

screenshot-2021-09-10-at-08-46-09.png

So, each of the videos needs to be 1280x720.  And here is where we came across our first complication.  Half of the videos were recorded in landscape mode:


large_screenshot-2021-09-10-at-09-04-25.png

medium_screenshot-2021-09-10-at-09-04-25.png

small_screenshot-2021-09-10-at-09-04-25.png

thumbnail_screenshot-2021-09-10-at-09-04-25.png

screenshot-2021-09-10-at-09-04-25.png

..but half were recorded in portrait mode....

large_screenshot-2021-09-10-at-09-05-07.png

medium_screenshot-2021-09-10-at-09-05-07.png

small_screenshot-2021-09-10-at-09-05-07.png

thumbnail_screenshot-2021-09-10-at-09-05-07.png

screenshot-2021-09-10-at-09-05-07.png

The landsacpe videos are all the right 16:9 proportion - but are 1080p - so will have to be rescaled to 1280x720.

The portrait videos are 1080x1920, so we'll have to do a crop,and then slightly upscale them.  But we're using FFMPEG, so this will be a snap!

large_screenshot-2021-09-10-at-09-08-16.png

medium_screenshot-2021-09-10-at-09-08-16.png

small_screenshot-2021-09-10-at-09-08-16.png

thumbnail_screenshot-2021-09-10-at-09-08-16.png

screenshot-2021-09-10-at-09-08-16.png

### Building the palette

We want our video to be 4k, so we first input a video and resize  it to 4K. It doesn't matter which video, since we will draw on top of it. I'm using the center video - the 5th video (index 4) as the background. It is resized to 4K, and named main.

```
[4]scale=3840:2160[main]
```

### Cropping inside the filter_complex

For each video, we can build something like this:

```
[0:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[0];
```

Let's walk through what this command does:

```[0:v]``` - grab the video from input 0 (thats sorin.mp4). 

```crop=in_w:in_w*9/16:0:in_h*0.2```  - the crop command, which has 4 parameters video width: video height:x start: y start
* video width: in_w - take the width of the input video
* video height: in_w\*9/16 - take the width, and multiply by 9/16 to keep the video  in the 9:16 format. (this will do nothing to the landscape videos, but will crop the portrait ones).
* x_start: 0.  Grab from the far left of the video
* y_start: in_h\*0.2 Get the height of the video, and begin the crop 20% down. The percentage can be modified slightly on each video - as some of our colleagues are taller than others.

Now we've cropped each video.  The landscape videos will be mostly unchanged, but the portrait videos will be cropped into a landscape view around the face.

```scale=1280:720[0]```  Resize the 16:9 video to 720p. 

```[0]```. Rename this modified video [0].  This is a variable that we can call later for placement.

We have to do this for each video, so our **-filter_complex** begins growing:

```
[4]scale=3840:2160[main];
[0:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[0];
[1:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[1];
[2:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[2];
[3:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[3];
[4:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[4];
[5:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[5];
[6:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[6];
[7:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[7];
[8:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[8];
[9:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[9];
[10:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[10];
[11:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[11];
[12:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[12];
[13:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[13];
[14:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[14];
[15:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[15]

```

It took a few tests to lay out each person the way I wanted - mostly changing the **in_h** multiplier to properly center the face.

### Layout of the videos

The next bit of the **filter_complex** lays out each of the videos into the grid.

```
[main][0]overlay=0:0[main0]
```

This tells FFMPEG to take the "main" 4K video, and overlay video "0" at (0,0).  Rename this intermediate video "main0". We now repeat this for each video, updating the variables and the (x,y) coordinates for each video.

Only one video will remain at the center (index 4) - the video of our CEO Cedric.


```
[main][0]overlay=0:0[main0];
[main0][1]overlay=1280:0[main1];
[main1][2]overlay=2560:0[main2];
[main2][3]overlay=0:720[main3];
[main3][4]overlay=1280:720[main4];
[main4][5]overlay=2560:720[main5];
[main5][6]overlay=0:1440[main6];
[main6][7]overlay=1280:1440[main7];
[main7][8]overlay=2560:1440[main8];
[main8][9]overlay=0:0[main9];
[main9][10]overlay=1280:0[main10];
[main10][11]overlay=2560:0[main11];
[main11][12]overlay=0:720[main12];
[main12][13]overlay=2560:720[main13];
[main13][14]overlay=0:1440[main14];
[main14][15]overlay=1280:1440[main15];
[main15][16]overlay=2560:1440
```

Now, those of you with keen eyes will notice that video 9 will sit on top of video 0 (10 on top of 1, etc.). I have more videos to add to the grid of 9 than places for them.  We'll fix that in our next step.

### Video timings

Each video has a slightly different length. When a video reaches an end, the video freezes, while the rest continue playing. I'd like all the videos to move for the entire clip. Looking at all of them, the shortest video is 9 seconds long. (Thanks to Pierre for defining our video's length 🤣).

We want each video to be 9 seconds long. We can fix this by changing the input parameter for each video:

```-t 9 -i  doug.mp4```

```-t 9``` says, "grab only the first 9 seconds of doug.mp4." Adding this to each video - now the overall video is exactly 9 seconds long!

### More video timing - desynchronizing the movements

Having watched these videos a lot, everyone moved their heads in the same pattern of motions, at approximately the same rate. When they are all combined into one video, the results were eerily robotic. Usually, you want all your videos in sync, but in this case, we want to actually artificially de-synchronize our videos a bit.  

To do this, we can grab 9 seconds of each video - but from a different start point for each video:

```
 -ss 5 -t 9 -i olia.mp4
```

```-ss 4``` start offset of 4 seconds
```-t 9``` 9 seconds of video

This command tells FFMpeg: "input 9 seconds of Olia.mp4, starting at 5 seconds into the video,". As long as we are careful and the SS + t values are not longer than the video's length, we'll be fine (I created a spreadsheet). So we input different -ss values for each video.

### Timing part 3: More than 9 videos

As noted, we have more than 9 videos. We could have just stopped at 9, but I wanted to include everyone. The easiest way to do that is to have two rounds of people in the 9 grid. We'll put Cedric in the middle for 18 seconds (his video was over 20s, so we are ok), and array 2 sets of 8 people around him.

```
 -ss 5 -t 9 -i erikka.mp4
 
 -itsoffset 9 -t 9 -i romain.mov
```

Erikka's video will be 9 seconds long (starting at time 5 from her video), and will be inserted into the final video at t=0.  Romain's video will be 9 seconds long as well, but the ```-itsoffset 9``` parameter says "Begin adding this video 9 seconds into the video."

So the result is Erikka is on the video for time 0-9s, and at t=9, Romain's video replaces her for t=9-18.

At this point, I'm super happy with the video. but it is just missing something... and that is (of course) a text overlay.

## The api.video Bunch

There is a [free Brady Bunch font](https://www.dafont.com/brady-bunch.font).  I downloaded it, and placed it in the same directory as the videos.  When you add text, you specify the location, the size and the font. I'll have 3 lines of text. The comma seperate each line, but for readability, I'll add a few lines between each one:

```
drawtext=enable:fontfile=BradBunR.ttf:fontsize=400:borderw=10:bordercolor=white:text='The':x=(3840)/2-200:y=(2160)/2-600,

drawtext=enable:fontfile=BradBunR.ttf:fontsize=400:borderw=10:bordercolor=white:text='api.video':x=(3840)/2-400:y=(2160)/2-400,

drawtext=enable:fontfile=BradBunR.ttf:fontsize=400:borderw=10:bordercolor=white:text='Bunch':x=(3840)/2-200:y=(2160)/2+200
```


* ```drawtext=enable``` Turn on the drawing (I guess)
* ```fontfile=BradBunR.ttf``` Load in the Brady Bunch font!
* ```fontsize=400``` It is a 4K video, so we can go pretty large.
* ```borderw=10:bordercolor=white``` The default font color is black, but we want a nice retro white border around it.
* ```text='api.video'``` The text to output.
* ```x=(3840)/2-400:y=(2160)/2-400``` Finally the (x,y) coordinates to start the text.  This took some trial and error to get the locations in the right spot, but I'm pretty happy with the result:



## The full FFMPEG command

We've walked through all the steps used to create the command, but we cannot avoid the inevitable: we have to show the full FFMPEG command. It follows the basic premise above:

1. Input all the videos (with time = 9, start offsets and playback offsets)
2. filter all the videos
3. resize and or crop all the videos
4. overlay them all on a 4K background
5. add the text
6. output the video

```
ffmpeg  -ss 10 -t 9 -i  sorin.mp4  -ss 5 -t 9 -i olia.mp4	-ss 4 -t 9 -i maria.mp4  -ss 5 -t 9 -i josephine.mp4 -ss 2 -t 18 -i cedric.mov -ss 1 -t 9 -i yohann.mp4 -ss 6 -t 9 -i anne-sophie.mp4 -ss 2 -t 9 -i brady_erikka.mov -ss 0 -t 9  -i pierre.mov  -itsoffset 9 -ss 3 -t 9 -i   T3.mov -itsoffset 9 -ss 6 -t 9  -i fancy.mov -ss 0 -itsoffset 9 -t 9 -i romain.mov	-itsoffset 9 -ss 11 -t 9 -i jb.mov -itsoffset 9 -ss 0 -t 9 -i doug.mp4 -itsoffset 9 -ss 0 -t 9 -i stephanie.mov -ss 10 -itsoffset 9 -t 9 -i  sorin.mp4  -ss 5 -itsoffset 9 -t 9 -i olia.mp4 -filter_complex '[4]scale=3840:2160[main];[0:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[0];[1:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[1];[2:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[2];[3:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[3];[4:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[4];[5:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[5];[6:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[6];[7:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[7];[8:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[8];[9:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[9];[10:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[10];[11:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[11];[12:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[12];[13:v]crop=in_w:in_w*9/16:0:in_h*0.1,scale=1280:720[13];[14:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[14];[15:v]crop=in_w:in_w*9/16:0:in_h*0.2,scale=1280:720[15];[main][0]overlay=0:0[main0];[main0][1]overlay=1280:0[main1];[main1][2]overlay=2560:0[main2];[main2][3]overlay=0:720[main3];[main3][4]overlay=1280:720[main4];[main4][5]overlay=2560:720[main5];[main5][6]overlay=0:1440[main6];[main6][7]overlay=1280:1440[main7];[main7][8]overlay=2560:1440[main8];[main8][9]overlay=0:0[main9];[main9][10]overlay=1280:0[main10];[main10][11]overlay=2560:0[main11];[main11][12]overlay=0:720[main12];[main12][13]overlay=2560:720[main13];[main13][14]overlay=0:1440[main14];[main14][15]overlay=1280:1440[main15];[main15][16]overlay=2560:1440[finalnotext];[finalnotext]drawtext=enable:fontfile=BradBunR.ttf:fontsize=400:borderw=10:bordercolor=white:text='The':x=(3840)/2-200:y=(2160)/2-600,drawtext=enable:fontfile=BradBunR.ttf:fontsize=400:borderw=10:bordercolor=white:text='api.video':x=(3840)/2-400:y=(2160)/2-400,drawtext=enable:fontfile=BradBunR.ttf:fontsize=400:borderw=10:bordercolor=white:text='Bunch':x=(3840)/2-200:y=(2160)/2+200'   apibunch.mp4
```

Yeah, it's a long one.  But the results speak for themselves - a fun video showing off the api.video team!

## Conclusion

Install FFMPEG!  Get 9 friends to film a video - and create your own "The <insert name> Bunch."  We hope you'll upload it to api.video and share the results with us!

large_bbtitle2.jpeg

medium_bbtitle2.jpeg

small_bbtitle2.jpeg

thumbnail_bbtitle2.jpeg

bbtitle2.jpeg

A very Brady video: Using FFMPEG to recreate the Brady Bunch introduction

Overlaying two videos with FFMPEG

[FFMPEG](https://api.video/what-is/ffmpeg) is a free and open source software project that offers many tools for video and audio processing. It's designed to run on a command line interface, and has many different libraries and programs to manipulate and handle video files.

## The Question

### How can I best overlay 2 live camera feeds with FFMPEG to produce one live stream to broadcast at api.video? I’m using a raspberry PI, and having a lot of latency issues.

We then began working on the FFMPEG command that he was using to build the livestream.

For testing purposes, I replaced the two livestreams with videos stored locally on my computer.  My goal is to create a video that looks like this: Die Hard with an Indiana Jones (and the Last Crusade) overlay.


video overlay of Diehard and Indiana Jones

large_screenshot-2021-08-23-at-18-20-16.png

medium_screenshot-2021-08-23-at-18-20-16.png

small_screenshot-2021-08-23-at-18-20-16.png

thumbnail_screenshot-2021-08-23-at-18-20-16.png

screenshot-2021-08-23-at-18-20-16.png



My command began with:



```
ffmpeg  -i DieHard.mp4 -i Indianajones.mp4
```

The ```-i``` tells ffmpeg that these are input files, and then the filename.  If you were using a livestream, or a remote video you could use a url.

The end of the ffmpeg command looks like:



```
-f flv rtmp://broadcast.api.video/s/23df9302-5897-4fca-8b27-0eb39ea67ab2
```

This says to FFMPEG, save the video output as flv and send it to my livestream endpoint.  When video begins arriving at this endpoint, your api.video stream will begin broadcasting whatever video is being sent. At this point, it will just take the first input (DieHard.mp4) and stream that file - ignoring Indiana Jones. I need to do some work to add in the Indiana Jones video...

If you want to test locally, you can change the output to:


```
test.mp4
```

To save the video locally to your computer.

### Initial command

```
ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 test.mp4
```

This takes in the 2 videos, outputs just DieHard.mp4 and saves it into test.mp4.  We want to overlay the Indiana Jones video.

The first thing we’ll do is scale the Diehard video to 1080p (it is 1080p, but this will ensure that all future videos fit correctly), and we will give the output of this transformation the name ‘main’

```
 -filter_complex '[0:v]scale=-1:1080[main]’
```

This command says - take the video from input 1, and resize it to -1:1080. The 2nd value is pretty obvious - make the video 1080 high. The ‘-1’ on the horizontal axis says make the height 1080, and scale the width so that the video is not stretched in any way.  If the input video is 16:9,the output will be 1920x1080. If the input video is 4:3, the output will be 1440x1080.

Let’s resize Indiana Jones to 360p (again using the ‘-1’ to ensure that the video is not stretched), and name the resized video “overlay”.  We place a semicolon between the two commands:

```
 -filter_complex '[0:v]scale=-1:1080[main];[1:v]scale=-1:360[overlay]’
```

Now we have 2 correctly sized videos, let’s set the overlay on top of the main video.  In creating the overlay, we could hard code the coordinates, but we can also use variables. In this case, main_w and main_h are for the main video width and height, and overlay_w and overlay_h are assigned to the overlay video.

We'll place the overlay in the bottom right - 25 pixels from the right and 25 pixels from the bottom.  To do this, we’ll subtract 25 and the dimension of the overlay from the dimension of the main video. The command will look like:


```
[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)
```

The ```[main]``` video is used as the main video, and the ```[overlay]``` as the overlay, and then we use the variables to place the overlay video.

The full command now looks like:

```
ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 -filter_complex '[0:v]scale=-1:720[main];[1:v]scale=-1:240[overlay];[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)' test.mp4

```

And the video looks exactly how we’d like it to look! The audio, however, is from the main video: Die Hard.  We can remove the audio (making the video silent) by only exporting the video from the output video.  To do this, we can use the ```map``` parameter.  We’ll name the overlay filter [out] and then just map the first track (track 0) to the output:


```
ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 
-filter_complex '[0:v]scale=-1:720[main];[1:v]scale=-1:240[overlay];[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)[out]' -map '[out]'0:0  test.mp4
```

This results in a silent movie.  We can programmatically pick which audio track (track 1 in both cases) with a second ```-map``` command:

* ```-map 0:1``` selects the Die Hard audio (the original 0 index input, track 1)
* ```-map 1:1``` selects Indiana jones audio (the original 1 index input, track 1)



Here are the final 2 commands (with the stream parameters added in:

```
ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 -filter_complex '[0:v]scale=-1:720[main];[1:v]scale=-1:240[overlay];[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)[out]' -map '[out]'0:0 -map 0:1 -f flv rtmp://broadcast.api.video/s/23df9302-5897-4fca-8b27-0eb39ea67ab2

ffmpeg  -i DieHard.mp4 -i Indianajones.mp4 -filter_complex '[0:v]scale=-1:720[main];[1:v]scale=-1:240[overlay];[main][overlay]overlay=(main_w-overlay_w-25):(main_h-overlay_h-25)[out]' -map '[out]'0:0 -map 1:1 -f flv rtmp://broadcast.api.video/s/23df9302-5897-4fca-8b27-0eb39ea67ab2
```

Here’s what the final result looks like (Die Hard audio version):



In order to create his live stream, our developer simply changed the input from 2 movie trailers to his 2 live streams, and it all *just worked*.

## Conclusion

As Indiana Jones famously did not say: “What is FFMPEG? Fortune and Glory kid. Fortune and Glory.”

And we are pretty sure John McClain never said “Yippiee Ki Yay FFMPEG!”

But that is because neither of them ever got the chance to play with the tooling that allowed us to create the video above.

In this post, we’ve talked through how to scale the dimensions of multiple input videos, how to overlay them, and then how to apply an audio track to the videos.  Finally, the video created by ffmpeg is streamed to api.video for live streaming consumption by your viewers.


Use FFMPEG to resize and overlay two videos, and then live stream them

Featured article

# api.video blog

Blog

Follow the api.video Blog to explore tutorials, video use cases, and learn about on-demand and live-streaming video updates.