live - create

Video Get

Normal

doug.jpg

twitter-brands.svg

Doug Twitter

Doug Sillars

large_api-video-blog-tutorials.png

medium_api-video-blog-tutorials.png

small_api-video-blog-tutorials.png

thumbnail_api-video-blog-tutorials.png

api-video-blog-tutorials.png

Tutorials

JavaScript

We recently launched [record.a.video](https://record.a.video), a new web app that can record your camera and screen right from the browser!  When you are done recording, it uploads the video to [api.video](https://api.video) to create a link for easy sharing of your video.

In the process of building this app, I learned a lot about a few web APIs, so I thought I would write a bit more detail in how I used these APIs and how they work.

* In this post, we'll use getUserMedia API to record the user's camera and microphone.

* In [post 2](https://api.video/blog/tutorials/building-record-a-video-the-screencapture-api/), I discussed recording the screen, using the Screen Capture API. Using the video streams created in posts 1& 2, I draw the combined video on a canvas.

* In [post 3](https://api.video/blog/tutorials/building-record-a-video-the-mediarecorder-api/), I'll discuss the MediaRecorder API, where I create a video stream from the canvas to create the output video.  This output stream feeds into the video upload (for video on demand playback) or the live stream (for immediate and live playback).

* In [post 4](https://api.video/blog/tutorials/building-record-a-video-using-the-webspeech-api-for-live-captioning/), I'll discuss the Web Speech API. This API converts audio into text in near real time, allowing to create 'instant' captions for any video being created in the app. This is an experimental API that only works in Chrome, but was so neat, I included it in record.a.video anyway. 

You can also check out our [live stream API reference documentation](https://docs.api.video/reference/api/Live-Streams#list-all-live-streams).


## MediaDevices.getUserMedia()

Let's start at the beginning, With record.a.video, we grab the camera and microphone inputs from the device to record the video. This API allows the browser to interact with external media devices, and read their output.

In record.a.video, we enumerate the video and audio inputs to display them as options for the recording:




large_screenshot-2021-03-30-at-10-33-48.png

medium_screenshot-2021-03-30-at-10-33-48.png

small_screenshot-2021-03-30-at-10-33-48.png

thumbnail_screenshot-2021-03-30-at-10-33-48.png

screenshot-2021-03-30-at-10-33-48.png

Once the capture begins, we display the chosen camera, and record from the chosen microphone.

So, how does this work?

## Enumerating the devices

When you call the enumerateDevices, you get all of the details of the devices:

```
    navigator.mediaDevices.enumerateDevices()
    .then(function(devices) {
        devices.forEach(function(device) {
        console.log(device.kind + ": " + device.label +" id = " + device.deviceId);
        
            if(device.kind =="videoinput"){
                //add a select to the camera dropdown list
              //  var option = document.createElement("option");
              console.log(device);
                deviceIds[counter] = (device.deviceId);
                deviceNames[counter]= (device.label);
                counter++;
            }
    });
```

If we look at the console log, we get a list of all the input and outputs:


```
audioinput: Default - Blue Snowball (046d:0ab9) id = default
audioinput: Blue Snowball (046d:0ab9) id = 693957e7a6c63d00cf9068338ec0108bfdfad2f108182bc988f7ed79430d5024
audioinput: Logitech StreamCam (046d:0893) id = ec257829b4d910400bdff0fe8639e3e2c5b9bd761c1cd4454b8f272672e4d482
audioinput: MacBook Pro Microphone (Built-in) id = be26a24fb1ca054193d51c80a6c875a091a285153959ddae8ab24f55c939bcd5
audioinput: Iriun Webcam Audio (Virtual) id = 5e911d92f330fb9333fee989407c5c84c3b99af29046d8182e582545e772687f
videoinput: Logitech StreamCam (046d:0893) id = 8d59c8e7bc02076c4230ba70125c03491020950b008769bd593bdb13e33c1ce7
videoinput: FaceTime HD Camera (Built-in) (05ac:8514) id = e3fce20226c193876c5ff206407fd4815ad5b1e6329e67a8e82c9636d8e75c8d
audiooutput: Default - External Headphones (Built-in) id = default
audiooutput: U32J59x (DisplayPort) id = b342ee4661c78101936e50a2b5a3e5080d5ed748e031547e492c7ab0eeddd9df
audiooutput: External Headphones (Built-in) id = 4a3980d8579a418193b1e8ff46771f204e87b2997b90d0d4e7e60a4acaae1235
audiooutput: MacBook Pro Speakers (Built-in) id = b7045191ebec43d797348478004f25472c868bcae61ad04a7de73f70878e27b2
```

Here we learn that, yes, I have a lot of audio/video devices hooked into my computer.  (mental note: why is Iriun webcam only presenting as audio, and not as video?)

Each device has a:
* kind: audio or video; input or output
* label: A text description for human consumption
* deviceId: a random string that uniquely IDs the device.


We'll use the deviceId to decide which device to broadcast, but in record.a.video, use the labels in the form - since they are better descriptions for our users.

## Picking a broadcast

Once a user has chosen their video input, we can make a request to obtain this video. (The same approach works for audio as well, so I'm only describing video here):

```
            navigator.getUserMedia = (navigator.mediaDevices.getUserMedia ||
                            navigator.mediaDevices.mozGetUserMedia ||
                            navigator.mediaDevices.msGetUserMedia ||
                            navigator.mediaDevices.webkitGetUserMedia);
```

When requesting a camera, you can apply constraints to the request to enforce exactly what you would like, for example:

```
            cameraW=1280;
            cameraH=720;
            cameraFR=25;

            var camera1Options = {
                audio:false,
                video:{
                    deviceId: deviceIds[0],
                    width: { min: 100, ideal: cameraW, max: 1920 },
                    height: { min: 100, ideal: cameraH, max: 1080 },
                    frameRate: {ideal: cameraFR}
                }
            };
```

Here I set the ideal video as 1280x720, but allow for differences with the min & max parameters.  This means that if the camera cannot provide 1280x720, it will give me a similar size, but in the available sizes for the camera.

Note: I have  set the audio to false.  In this demo, I am applying the video onto the screen with a video tag. If the browser were to play audio from the camera, the mic would pick it up, and you'd get an awful feedback loop. I want to avoid this. I could just mute the video, but I thought it would be cool to take another Media Stream for the audio, and apply that track to my stream during recording (after it appears on the screen).

### Getting the stream and placing it on the page

I have a video element called 'video1'. I assign the getUserMedia to display there:

```
 video1= document.getElementById("video1");           navigator.mediaDevices.getUserMedia(camera1Options).then(function(stream1){
                video1.srcObject=stream1;
```


When this call is made - the browser will ask the user for permission to use the camera. If the user accepts the sharing request - the browser grabs the camera feed, and applies it into the video1 element:

small_screenshot-2021-03-30-at-13-34-37.png

thumbnail_screenshot-2021-03-30-at-13-34-37.png

screenshot-2021-03-30-at-13-34-37.png

thumbnail_screenshot-2021-03-30-at-13-37-29.png

screenshot-2021-03-30-at-13-37-29.png

That's all there is to it!

With the getUserdevices API, pulling the video feed from your camera to the browser just takes a few lines of code.

## If one camera is great, what about two?

While most desktops do not have multiple cameras, most smartphones do. I thought it would be very cool to extract a video feed from 2 cameras on one device. 

(Imagine a real estate walkthrough where you can see the surroundings, and the face of the person presenting.)  So I built a demo application. You can see this application in action at [record.a.video/index1.html](https://record.a.video/inex1.html).  

I basically repeat the camera1 and video1 code above with camera2 and video2, and use the first two cameras reported by the enumeration query.

When I test the page on desktop, I get both cameras to stream (which is super cool!) Unfortunately on Android & iOS, only one camera can stream at a time - resulting in the first camera grabbing a still image from the start or capture, and going moving to an inactive state as the second camera becomes active. (On my phone, the front camera has the still image, and the rear camera continues to show video).

I imagine that this is a battery/CPU saving setting, as running 2 cameras full time on a phone battery would quickly use up the battery.


## Conclusion

The MediaDevices.getUserMe4dia() API allows the browser to use the external camera and mics to display (and record) content.  [Build your own version](https://dashboard.api.video/register) of a video recording system today! If you have any questions or suggestions, please share them on our [community forum](https://community.api.video). Happy building! 

MediaDevices.getUserMedia() for camera and audio recording

large_piqsels-com-id-oslvc.jpg

medium_piqsels-com-id-oslvc.jpg

small_piqsels-com-id-oslvc.jpg

thumbnail_piqsels-com-id-oslvc.jpg

piqsels-com-id-oslvc.jpg

Building record.a.video part 1: MediaDevices.getUserMedia() for camera and audio recording

Video upload

NodeJS


With the explosive growth of remote working in the past year, we’ve all worked to find different ways of communicating. Slack and Zoom have become major tools for communication as we are all no longer in the same building or office.  

Apps like Clubhouse have become ‘places’ to go and hang out and chat with others.

One app that has featured explosive growth over the past year is Loom. Loom is an app that lets you record short video messages - sharing your screen or just using your camera. When you are done, the video is saved on a server, and you are given a link.  This link is easy to share: on social media, on Slack etc. It is a great way to communicate an idea, or to express your point - it’s everything that is great about video, packaged in an easy to use interface.  

It is no wonder that Loom is so successful.  My only issue is that I either have to use an installed app on my Mac (265 MB installed), or a Chrome extension (I did not go this approach, as Chrome eats too much memory already).

## Can I build a Loom-like app - in the browser?

I lead developer relations at api.video.  We offer a full service API for video transcoding, hosting and delivering video streams.  It is incredibly fast, and can be used to share videos in the same way Loom does.  So this raised 2 questions:

1. If I delegate the ‘hard’ bits of the video handling to api.video, can I build a recording app in the browser? 

2. Can I get this app running in all four major modern browsers?  Desktop & mobile?

TL;DR:  

Question 1: Yes - you can build a website to record and share video (screen sharing and camera video). It will even stream live video!

Question 2: Almost all browsers.  On the desktop, the full featured app works great in Chrome, Edge Firefox, and Safari 14.0.3 (there are some support issues that will fail below Safari 14).  

For Mobile: it works on Android (without screen sharing). Unfortunately, the APIs used are not yet in mobile Webkit, so no iOS support at this time.

Try it out! The application is live at [record.a.video](https://record.a.video). You can also check out our [live stream API reference documentation](https://docs.api.video/reference/api/Live-Streams#list-all-live-streams).

## How does it work?


large_screenshot-2021-03-26-at-11-13-30.png

medium_screenshot-2021-03-26-at-11-13-30.png

small_screenshot-2021-03-26-at-11-13-30.png

thumbnail_screenshot-2021-03-26-at-11-13-30.png

screenshot-2021-03-26-at-11-13-30.png

When you open [record.a.video](https://record.a.video), you'll get a menu as above. (You may not get the available cameras and microphones - press "start capture" and allow camera/microphone permissions, then reload the page).

* The top bar allows you to switch between Live streaming and recording a video.
* Choose your camera and mic (if you have more than one).
* Select the screen layout - do you want to show just the camera or screen, or do you want to overlay the camera on the screen?
* Live captions:  If you are using Chrome, we can add live captions to the video being created.  Pick top or bottom for the captions.
* "Show recording" if you leave this checked, you'll see the video being recorded in the browser.  Our CEO dislikes the "vortex effect" you see when sharing the screen, so we added this button for him to disable the view. (👋 Hi Cedric!)

Now that you've chosen all your options, you're ready to start capture!

### Recording

Once you've started recording, you'll see 3 new views appear:

* The screen being shared
* The camera
* The recording (showing screen, camera or both). (Unless you hid the recording).


small_screenshot-2021-03-26-at-11-22-05.png

thumbnail_screenshot-2021-03-26-at-11-22-05.png

screenshot-2021-03-26-at-11-22-05.png



## Record and stream right from your browser!

So it works!  Recording your screen and camera in the browser with live streaming or recording and uploading for asynchronous viewing.  

What completely blows my mind about this application is that leveraging web APIs, this site weighs in at just under 300KB.  There is a NodeJS backend to handle the streaming (we have to convert the format to RTMP for ingestion into api.video), but if you remove the live streaming component, this app runs with just JavaScript (you can even run it locally on your computer!)

## See it in action

##  Curious about the details

I'm really glad you asked!  We've written a series of posts describing the APIs used to collect video streams, how they were cpmbined, and how they are streamed:

* In [post 1](https://api.video/blog/tutorials/grabbing-camera-streams-with-mediadevices-getusermedia/), I talked about the getUserMedia API to record the user's camera and microphone.

* In [post 2](https://api.video/blog/tutorials/building-record-a-video-the-screencapture-api/), I discussed recording the screen, using the Screen Capture API. Using the video streams created in posts 1 & 2, I draw the combined video on a canvas.

* In [post 3](https://api.video/blog/tutorials/building-record-a-video-the-mediarecorder-api/), I'll discuss the MediaRecorder API, where I create a video stream from the canvas to create the output video.  This output stream feeds into the video upload (for video on demand playback) or the live stream (for immediate and live playback).

* In [post 4](https://api.video/blog/tutorials/building-record-a-video-using-the-webspeech-api-for-live-captioning/), I'll discuss the Web Speech API. This API converts audio into text in near real time, allowing to create 'instant' captions for any video being created in the app. This is an experimental API that only works in Chrome, but was so neat, I included it in record.a.video anyway. 



## Conclusion

Record.a.video lets you easily record and share video - live or asynchronously! 


Go give it a shot [record.a.video](https://record.a.video), and if you want to build your own - the code is on [Github](https://github.com/apivideo/recordavideo). In the coming days, we’ll add some technical posts describing how we built it.

If you have any comments or suggestions, please leave us a note in our [community forum](https://community.api.video). 


Building "Loom" in the browser: record a video with api.video

large_loom-app.png

medium_loom-app.png

small_loom-app.png

thumbnail_loom-app.png

loom-app.png

Player SDK

## Animated video preview


Traditionally, a video preview on the web is a poster image from the video, with a big play button in the middle.  The poster can be a unique image, but more typically is a frame from the actual video. The poster is an attribute of the video tag, so adding the image is very easy.

But animation draws your eye. When you see the page moving, you're going to look to see what is going on, and by seeing that there is a movie, can watch it.  So, how can you add an animated preview?

Many applications that use an animated preview use an animated GIF. We all love GIFs, and use them in our Slack conversations, add them on Twitter, often to great comedic effect. But - when you use a gif on Twitter - guess what?  It is actually a looping mp4!

So why do many video APIs fall back to animated GIFs?

Another image of the same street musician playing

large_streetmusicianplaying.jpg

medium_streetmusicianplaying.jpg

small_streetmusicianplaying.jpg

thumbnail_streetmusicianplaying.jpg

streetmusicianplaying.jpg



## GIF: Not intended as a platform for animation

If we enter our time machine back to 1989, we can read what the creators of GIF have to say about animated GIFs: it is right in the [specifications](https://www.w3.org/Graphics/GIF/spec-gif89a.txt):

>Animation - The Graphics Interchange Format is not intended as a platform for
animation, even though it can be done in a limited way.

The reason is that video uses time based compression - allowing for compression across frames of the video, while a 15 fps animated GIFs has 15 static GIF images per second. That means: really big files.  


## Using HTML5 video

With just a few lines of code, we can loop the actual video file without any editing of the file:

```
<video loop muted autoplay controls id="video">
	<source src="vi4blUQJFrYWbaG44NChkH27.mp4#t=5">
</video>

<script>
	var video = document.getElementById("video");
	var START_TIME = 5;
    var END_TIME=10;

	 video.addEventListener("timeupdate", () => {
		 console.log(video.currentTime);
		 if(video.currentTime >= END_TIME){
			 video.currentTime = START_TIME;
			video.play();
	      }
	  });
	
	
	</script>

```

In the video tag, we append the video url with #t=5, telling the video player to start the playback at 5 seconds. Aside: if we used #t=5,10, it will stop the video at t=10 seconds, but the video does not loop.

To ensure the video loops, we add a listener to 'timeupdate' (which fires every 250 ms).  When this exceeds the end time, we reset to start time, and begin the video playback again.

## Using api.video


We can do the exact same thing with api.video [player SDK](https://docs.api.video/sdks/player/apivideo-player-sdk):

```
    <iframe id="target" width="100%" height="100%"></iframe>
    <script>
        const START_TIME = 5;
        const ENDTIME = 10;

        // create the player in the #target element
        var player = new PlayerSdk("#target", {
            id: "vi5oDagRVJBSKHxSiPux5rYD",
            autoplay: true,
            muted: true,
            hideControls: true,
        });

        player.addEventListener("ready", () => {
            player.setCurrentTime(START_TIME);
        })

        setInterval(() => {
            player.getCurrentTime((time) => {
                if (time >= ENDTIME) {
                    player.setCurrentTime(START_TIME);
                }
            });
        }, 200);
    </script>
```

You can try it out in a [JSFiddle](https://jsfiddle.net/3f4gpnjd/1/)

## Using video

The great thing about the implementations above is that no additional files have to be generated: we keep using video to display video.  Do you want to change the interval of the loop?  All you need to do is change the start/stop variables in the JavaScript, and the change will be instantaneous. A GIF would require re-encoding.

If the user decides to watch the video, you already have a great deal of the files needed to start the video in your local cache - so playback will be super fast.

## Conclusion

We can create looping "GIF-like" experiences with videos uploaded to api.video with just a few lines of JavaScript.  There's no added cost to creating or changing your files - since it is all based on the video files already in the system. And, since we are just programmatically playing back a selected part of the video, making changes on the start, finish and duration of the video is easy.  Try it yourself.  Create an [api.video account](https://dashboard.api.video/register) today!

*Thanks to Olivier Lando from the Ecosystem team for creating the JSFiddle.*

Adding an "animated GIF" with api.video

large_output.jpg

medium_output.jpg

small_output.jpg

thumbnail_output.jpg

output.jpg

Sometimes your video content flows nicely from one video to another. If this is the case, having the player automatically load the 'next' video in a sequence would be convenient.

In this post, we'll use the [JavaScript Player SDK](https://docs.api.video/sdks/player/apivideo-player-sdk) to control the videos that appear in the player.  The demo application is available at [watch.a.video/playlist](https://watch.a.video/playlist), and the code is on [GitHub](https://github.com/dougsillars/videoPlaylist/blob/main/index.html).



## Video Playlist

Each video at api.video has a unique videoId. You can get this value from the video API, or you can extract it from the go.api.video URL for each video. I selected four videos from my account, and created an array with their videoIds:

```
var videos = [
            "vi7RSVfjmrCmQZgS12j7VW5X", //Edinburgh bagpipes
            "vi4uIigPjKOXiVnBGLjHDIMs", //clouds from a plane
            "vi3DZ8BOJBt3DxqenDSDXL1Z", //Zurich Christmas church
            "vi1rdVw3OxmAzpCM7mJmlz2c" //Munich surfing
        ];
var counter = 0;
```
 
This array will be our playlist.  We also create a counter that we will use to loop through the array.

## Creating a video player instance

The SDK makes it very easy to create a video on a page.  The first step is to include the [SDK script](https://github.com/apivideo/player-sdk/tree/master/dist) file:
```
<script src="sdk.js"></script>
```

In the above example, I simply have a div with the ID myVideo:

```
<div id="myVideo" ></div>
```

when the page loads, we call the createVideo function. This will use the player SDK to create a player instance in the myVideo div.  The videoId is set to the first video in the videos array, and we tell the player to autoplay - so the video starts as soon as it has loaded.

```
window.onload = createVideo(counter);

        function createVideo(counter) {
            console.log("video", counter +videos[counter]);
            window.player = new PlayerSdk("#myVideo", { 
               id: videos[counter], 
                autoplay: true
                // ... other optional options s
            });
            player.addEventListener('play', function() { 
                //console.log("playing");
                onPlay(counter);
            });
            player.addEventListener('ended',function() { 
                console.log("ended");
                counter ++;
                //if we hit the end of the array - start over again
                if (counter == videos.length){
                    counter = 0;
                }
                onEnd(counter);
            });
    
        }
```

There are 2 listeners attached to the player we created: 'play' and 'ended'

### Play Listener

The play listener fires when the video starts playing. Obviously, this happens at the start, but also when unpaused, and also whenever there is a 'seek' - the user jumps ahead or behind in the video.

the onPlay() function simply updates the 'videoInfo' div:

```
        function onPlay(counter) {
           // console.log("onPlay");
            console.log("counter" ,counter);
            document.getElementById('videoInfo').innerHTML = "video playing " + counter;
            console.log("video playing");
        }
```

### Ended Listener

When the video ends, this event fires.  We increment the counter (or if we are an the end of the array - we reset counter to 0), and call onEnd():

```
        function onEnd(counter){
            //console.log("onEnd");
            document.getElementById('videoInfo').innerHTML = "video over" ;
            //console.log("video over");
            player.destroy();
            //video is over - so start another one...
            createVideo(counter);
        }
```

onEnd destroys the current player, and then calls the createVideo function to create a new player with the new video in it.  

## Jump to the next/previous video

There are 2 arrow images to the left and to the right of the video. Clicking these will cause the video to skip to the next video (the right arrow) or the previous video (left arrow).

This is controlled with a clicklistener. Here's how we move to the "previous" video:

```
        document.getElementById("prev").onclick = function() {
            counter--;
            if( counter < 0){
                counter = videos.length -1;
            }
            player.destroy();
            createVideo(counter);
        }
```

It simply decreases the counter by 1 (and handles the case if we are at counter=0 by going to the last video), destroys the current player, and creates a new player with the new counter value. 

The "next" clickListener works the same way (and the code is in the repo.

## Playlist

onEnd calling createVideo creates a loop: after the video ends, start the next video.  This effectively creates a playlist that can be controlled by the array.


## Conclusion

Using the api.video Player SDK, we have created a way to play one video after another.  Give it a try, and let us know what you think on our [community page](https://community.api.video)!




Creating Video Playlists

small_playlist.png

thumbnail_playlist.png

playlist.png

Delegated Upload

Video Status

Video tagging

Video list

Video update

## Video Content Moderation

User generated content (UGC) is taking over the internet. From sites offering video product reviews to vlogging and online education - video is being created at the fastest pace yet (and shows no sign of stopping). Many companies are looking for ways to easily incorporate UGC video on their sites.  

large_screenshot-2020-10-23-at-12-17-19.png

medium_screenshot-2020-10-23-at-12-17-19.png

small_screenshot-2020-10-23-at-12-17-19.png

thumbnail_screenshot-2020-10-23-at-12-17-19.png

create-review.png

All of our customers who allow UGC to be posted on their site worry about a few bad actors working to ruin their brand by posting inappropriate content. In order to protect their brand and the content on their site - evaluating any user generated content before it is allowed to be placed on the website is an important step. Traditional content moderation requires human moderators (which takes time and is expensive).

In this post, I'll walk through an alternative: using machine learning to moderate videos. Before publishing, it is scanned using machine learning for content, tested against rules, and then either accepted or rejected based on those rules.  Super fast, works 24 hours a day, and there will be no human error in the categorisation of the video.

As a demonstration, we've built a site with [api.video](https://api.video) for video hosting, and using Hive AI to power the video moderation.

## The Basics

## api.video
api.video is a full service API-based video hosting solution.  You can use our APIs to upload videos, modify and serve streaming video.  Every video that is uploaded is transcoded into a video stream, and can be delivered in a custom video player. If you haven't created an api.video account yet, you can do that [here](https://dashboard.api.video/register). 

In this solution, we'll use the [delegated upload](https://docs.api.video/reference/api/Videos) (used like a public key) for uploading the videos.  We'll use the video [tagging](https://api.video/blog/tutorials/video-tagging-best-practices/) function to label each video based on the moderation results. We can search all of the tagged videos to deliver the videos (and custom player) to users based on their moderation state.

### Hive AI

Once the videos are uploaded, we need to run moderation before allowing them to appear on the website.  

Hive AI has several moderation suites. We've set up our API to use 2 endpoints. For short videos (under 25s) - we scan 1 frame every second, and for long videos, one frame every 5 seconds. (The demo above only works for 25s long videos).

The API is trained to identify several subjects that, depending on the context of your application, might be important to moderate. In this post, we will use a small subset of the trained models:

* Safe for work/Not Safe for Work
* Yes/No: Female nudity
* Yes/No: Male nudity
* Yes/No: Female swimwear
* Yes/No: Shirtless male
* Yes/No: guns
* Yes/No: smoking

After each frame is analyzed, a JSON file with data for each frame analyzed is returned for analysis.  

After analysis, the video is tagged with moderation value, ensuring that it only appears on pages appropriate for the video.

### Code

This application is available on [GitHub](https://github.com/apivideo/videoModeration). The general flow is:

1. On Upload, a video is tagged "needs moderation" (and will not appear on the site).
2. Hive AI performs frame analysis.
3. Based on the analysis, the video is tagged as "SFW" or "NSFW" (etc.).

moderation workflow, ai moderator, automated moderator

large_screenshot-2020-10-23-at-12-49-40.png

medium_screenshot-2020-10-23-at-12-49-40.png

small_screenshot-2020-10-23-at-12-49-40.png

thumbnail_screenshot-2020-10-23-at-12-49-40.png

moderation-workflow.png

large_screenshot-2020-10-22-at-15-17-28.png

medium_screenshot-2020-10-22-at-15-17-28.png

small_screenshot-2020-10-22-at-15-17-28.png

thumbnail_screenshot-2020-10-22-at-15-17-28.png

moderation-landing-page.png

On entering the site, users are presented with an upload form.  As we walk through this example, we'll be following the upload of the intro theme to the classic TV show Baywatch.

With api.video, we have created a delegated upload key - which allows us to place the code publicly on the webpage without exposing our API private key. The form takes the video, and uploads the video to api.video.  The uploader uses the [Blob API](https://api.video/blog/tutorials/uploading-large-files-with-javascript/) to break the videos into 50MB segments for uploading.

For the purposes of the demo, we show how many chunks are created, and update the progress of each chunk, in addition to the total video upload:


small_screenshot-2020-10-23-at-13-34-15.png

thumbnail_screenshot-2020-10-23-at-13-34-15.png

video-upload-status.png

The response from the upload provides the api.video ```videoId```, which is used to identify the video at api.video.  This is then sent to the NodeJS backend as a POST (along with the video's name).

On the Node server, we begin the process of moderating the video.  First, we call the [update video](https://docs.api.video/reference/api/Videos#update-a-video-object) endpoint to add the video's name and to tag the video "needsScreening" to indicate that it has entered the moderation queue.

### Transcoding

When the video is uploaded, api.video's servers begin the process of creating different size/bitrate videos to provide adaptive bitrate streaming.  We also create a mp4 version of the video. For content moderation, we need to submit the mp4 to Hive AI. The Node server pings api.video's [video status](https://docs.api.video/reference/api/Videos#retrieve-video-status-and-details) endpoint every 2 seconds to determine when the mp4 is ready. Initially, the API will indicate that the video is not playable (transcoding has not yet started). Once transcoding has started, the API lists the encoding status of every format being created, so we can monitor the encoding status of the mp4.

medium_screenshot-2020-10-23-at-13-36-55.jpg

small_screenshot-2020-10-23-at-13-36-55.jpg

thumbnail_screenshot-2020-10-23-at-13-36-55.jpg

server-video-upload-status.jpg

Once the mp4 is created, we can create our connection to Hive AI, and make the request for content moderation.

The request looks like this:
```JSON
{
  method: 'POST',
  url: 'https://api.thehive.ai/api/v2/task/sync',
  headers: {
    accept: 'application/json',
    authorization: 'token {API TOKEN}'
  },
  form: {
    image_url: 'https://cdn.api.video/vod/vi1iHWIy6Doy0LBJl3ajaED0/mp4/1080/source.mp4'
  }
}
```

and a few seconds later, a huge JSON response comes back - 39 categories * x frames analysed.  Let's look at a snip of one frame to see the sort of information we get (for brevity, I've only included the first 5 categories that are returned):

```JSON
	{
		"class": "general_not_nsfw_not_suggestive",
		"score": 0.00460230773187999
	},
	{
		"class": "general_nsfw",
		"score": 5.180850871024288e-06
	},
	{
		"class": "general_suggestive",
		"score": 0.995392511417249
	},
	{
		"class": "no_female_underwear",
		"score": 0.9998768576722025
	},
	{
		"class": "yes_female_underwear",
		"score": 0.00012314232779748514
	}

```

"Not NSFW" means "Not Not Safe for Work" which, removing the double negative is "safe for work." This score is combined with "not suggestive", and is scored at 0.004.  Since 0 is a low score, and 1 is a high score, this means that the API has determined that this frame is not considered appropriate for work.  Looking at the next 2 values, general NSFW is also very small, but the "general suggestive" is 99.9%. Since the yes:no scores will always add up to one, this means that the general suggestiveness is what makes this frame not safe for work.

video moderator, automated content moderator, ai moderation

large_screenshot-2020-10-24-at-19-46-39.png

medium_screenshot-2020-10-24-at-19-46-39.png

small_screenshot-2020-10-24-at-19-46-39.png

thumbnail_screenshot-2020-10-24-at-19-46-39.png

automated-content-moderator.png

### Compiling the scores

Hive AI gives you scores for each category on each frame, but it is up to us to define the pass/fail criteria for our videos.  

In order to do this, I take all the scores for each category, and place them in an array. With the data sequestered for each category, I can calculate the min, max, average and median of each score. I also calculate how many frames appear over 0.9 as a 90% certainty ("yes_smoking") and how many frames appear under 0.1 (90% certainty of "no_smoking.")  


medium_screenshot-2020-10-22-at-15-25-19.jpg

small_screenshot-2020-10-22-at-15-25-19.jpg

thumbnail_screenshot-2020-10-22-at-15-25-19.jpg

frame-analysis-code.jpg

In the above "SFW" array (really the 'not not safe for work' response, but that is a mouthful), of 22 frames, I finde:
* min score: 0 (not safe for work)
* max score: 1 (safe)
* average:  0.55 right in the middle!
* median: 0.75 
* count of frames > 0.9: 9
* count of frames <0.1: 7

As you can see, 9 frames are over 0.9 (safe!), but 7 are below 0.1 (not safe!). If the median falls below 0.9, than means at least 50% of the frames are not "certain" to be safe for work ( which I place at the 90% threshold).  Based on these numbers, my rudimentary pass/fail algorithm deems the Baywatch intro "NSFW."  

If you wanted to prevent videos with smoking from appearing to your audience, just one frame with a 90% certainty of smoking would be enough to cause the video to be categorised as "yes_smoking." I used this same threshold for guns, nudity, underwear and swimwear. 

>Interestigly, this does push the Schindler's List trailer into the "yes smoking" category.

Based on these metrics, it is not terribly surprising that the Baywatch video is flagged for "yes female swimwear" and "yes shirtless male."  Also unsurprisingly, the algorithm found an absence of smoking and guns.

large_screenshot-2020-10-22-at-15-25-26.jpg

medium_screenshot-2020-10-22-at-15-25-26.jpg

small_screenshot-2020-10-22-at-15-25-26.jpg

thumbnail_screenshot-2020-10-22-at-15-25-26.jpg

content-categories-code.jpg

Once we have the categories measured for each application, we can remove the tag ```needsScreening``` and add in the new tags from the moderation. This is done with the [Video Update](https://docs.api.video/reference/api/Videos#update-a-video-object) endpoint.  

And that ends the moderation process for the video.

### Video Categories

Now that each video has been categorised, it is easy to display each video category. Each video's categories have been added to the video as a tag and the [List videos](https://docs.api.video/reference/api/Videos#list-all-video-objects) endpoint allows us to search by tag - returning every video that does not have smoking, or is safe for work, etc.  Using Node, we can generate the list of videos (sorted newest first), and send them to the client:

```Javascript
app.get('/no_smoking', (req, res) => {
	//get list of no smoking videos
	client = new apiVideo.Client({ apiKey: apiVideoKey});
	let recordedList = client.videos.search({"tags":'no_smoking', "sortBy":"publishedAt","sortOrder":"desc"});
	recordedList.then(function(list) {
		console.log("list of tagged videos");
		console.log(list);
		return res.render('videos',{list});  
	}).catch((error) => {
	    console.log(error);
	});
		
});

```

The API returns the metadata for each video (including links for the video), so creating a page with each video iFrame is a pretty simple task.  On each page in the sample app, you can display (and watch) each of the videos assigned to a category.  In the demo, I use Pug for the rendering

```
 each video in list
    p #{video.title}
    iframe(type="text/html", src=video.assets.player, width = "960", height="540",frameborder="0", scrollling="no")
    p #{video.publishedAt}
   
```

For example, the "yes guns" page has 3 movie trailers (as I write this post): Indiana Jones and the Last Crusade, Die Hard and the latest James Bond,  

medium_screenshot-2020-10-24-at-19-51-28.jpg

small_screenshot-2020-10-24-at-19-51-28.jpg

thumbnail_screenshot-2020-10-24-at-19-51-28.jpg

list-of-videos.jpg

## Conclusion

There we have it - we have uploaded a video to api.video, and before it is displayed on the site, it is moderated by Hive AI for several categories of inappropriateness.  Based on the analysis for each frame, the video is categorised into buckets and displayed on the appropriate page on the website.


If you have questions about using content moderation with your videos at api.video, feel free to reach out on our [community forum](https://community.api.video) or comment on the [GitHub repo](https://github.com/apivideo/videoModeration).  We'd love to see how you are using moderation to sort and categorise your videos, and the rubrics you utilise to decide what categories a video might fall into. If you haven't created your api.video account yet, you can do that in just a fewmomnets by follwoing [this link](https://dashboard.api.video/register). Happy building! 

Video moderation with machine learning

video content, machine learning, moderator, AI moderator

large_screenshot-2020-10-23-at-12-04-32.png

medium_screenshot-2020-10-23-at-12-04-32.png

small_screenshot-2020-10-23-at-12-04-32.png

thumbnail_screenshot-2020-10-23-at-12-04-32.png

video-content-machine-learning-ai-moderator.png

api.video enables developers to build, scale and operate video in their own apps and platforms in minutes, with just a few lines of code. 
The service handles the end-to-end workflow, from video ingestion to worldwide video delivery. You can [test it for free](https://dashboard.api.video/register) right away and start building.

&nbsp;

**August 2021 update**
Since this post was written, we've published a library to simplify JavaScript upload of videos [read the blog post to learn more](https://api.video/blog/tutorials/new-video-uploader-javascript-library/).

&nbsp;

**17 February 2021 update**
The example code has been added to extract the filename, and then append it to the first upload. Without this step, your video title will be 'blob.'

&nbsp;


You can view the API reference documentation for the file upload endpoint here: [Upload a video](https://docs.api.video/reference/api/Videos#upload-a-video)

&nbsp;

***

As our presentations, PDFs, files, and videos get larger and larger, we are stretching remote servers’ ability to accept our files. With just a few lines of JavaScript, we can ensure that this error goes away, no matter what you are trying to upload. Keep reading to learn more.

&nbsp;

The most common error with large uploads is the server response: **HTTP 413:  Request Entity too Large**. Since the server is configured only to accept files to a specific size, it will reject any file larger than that limit.

&nbsp;

One possible resolution would be to edit your server settings to allow for larger uploads, but sometimes this is not possible for security or other reasons. (If the server limit gets raised to 2GB for videos, imagine the images that might get uploaded!)


Further, if a large file fails during upload, you may have to start the upload all over again.  How many times have you gotten an “upload failed” at 95% complete?  Utterly frustrating!

## Segments/Chunks

When you watch a streaming video from [api.video](https://api.video), Netflix or YouTube, the large video files are broken into smaller segments for transmission.  Then [the player](https://api.video/product/player/) on your device reassembles the video segments to play back in the correct order. 
What if we could do the same with our large file uploads? Break the large file into smaller segments and upload each one separately? We can, and even better, we can do it in a way that is seamless to our users!

&nbsp;

Baked into JavaScript are the File API and the Blob API, with full support across the browser landscape:

large_screenshot-2020-09-23-at-11-22-35.png

medium_screenshot-2020-09-23-at-11-22-35.png

small_screenshot-2020-09-23-at-11-22-35.png

thumbnail_screenshot-2020-09-23-at-11-22-35.png

blob-javascript-api-support.png

This API lets us accept a large file from our customer, and use the browser locally to  break it up into smaller segments, with our customers being none the wiser!

&nbsp;

Let’s walk through how you might use this to upload a large video to api.video.  
To follow along, the code is available on [Github](https://github.com/apivideo/blobUpload), so feel free to clone the repo and run it locally.

&nbsp;

To build your own uploader like this, you’ll need a free [api.video account](https://dashboard.api.video). Use this to create a [delegated upload token](https://docs.api.video/reference/api/Upload-Tokens). It takes just 3 steps to create using CURL and a terminal window. 

&nbsp;


> A delegated token is a public upload key, and anyone with this key can upload videos into your api.video account. We recommend that you place a [TTL (time to live) on your token](https://api.video/blog/product-updates/video-access-management-how-to-create-deliver-and-manage-private-videos-and-what/), so that it expires as soon as the video is uploaded.

&nbsp;

Now that you're back, we'll begin the process of uploading large files.

### Markup

The HTML for our page is basic (we could pretty it up with CSS, but it's a demo 😛):

There is an input field for a video file, and then there are 2 divs where we will output information as the video uploads.

&nbsp;

Next on the page is the ```<script>``` section - and here's where the heavy lifting will occur.  

We begin by creating some JavaScript variables:

&nbsp;

* input: the file input interface specified in the HTML.
* url:  the delegated upload url to api.video.  The token in the code above (and on Github) points to a sandbox instance, so videos will be watermarked and removed automatically after 24-72 hours. If you've created a delegated token, replace the url parameter 'to1R5LOYV0091XN3GQva27OS' with your token. 
* chunkCounter: Number of chunks that will be created.
* chunkSize: each chunk will be 6,000,000 bytes - just above the 5 MB minimum. For production, we can increase this to 100MB or similar.
* videoId: the delegated upload will assign a videoId on the api.video service. This is used on subsequent uploads to identify the segments, ensuring that the video is identified properly for reassembly at the server.
* playerUrl: Upon successful upload, this will output the playback url for the api.video player.

&nbsp;

Next, we create an EventListener on the input - when a file is added, split up the file and begin the upload process:

We name the file uploaded as 'file'. To determine the number of chunks to upload, we divide the file size by the chunk size. We round the number round up, as any 'remainder' less than 6M bytes will be the final chunk to be uploaded. This is then written onto the page for the user to see. (In a real product, your users probably do not care about this, but for a demo, it is fun to see).

## Slicing up the file

The function createChunk slices up the file.

Next, we begin to break the file into chunks.  Since the file is zero indexed, you might think that the last byte of the chunk we create should be ```chunkSize -1```, and you would be correct. However, we do not subtract one from the chunkSize.

> the first byte that will *not* be included in the new Blob (i.e. the byte exactly at this index is not included).

So, we must use ```chunkSize```, as it will be the first **byte** NOT included in the new Blob.

In the createChunk function, we determine which chunk we are uploading by incrementing the chunkCounter, and again calculate the end of the chunk (recall that the last chunk will be smaller than chunkSize, and only needs to go to the end of the file).

In the first chunk uploaded, we append in the filename to name the file (if we omit this, the file will be named 'blob.'

### The actual slice command

The ```file.slice``` breaks up the video into a 'chunk' for upload. We've begun the process of cutting up the file!

We then create a form to upload the video segment to the API. After the first segment is uploaded, the API returns a videoId that must be included in subsequent segments (so that the backend knows which video to add the segment to). 

On the first upload, the videoId has length zero, so this is ignored.  We add the chunk to the form, and then call the uploadChunk function to send this file to api.video. On subsequent uploads, the form will have both the videoId and the video segment.

## Uploading the chunk

Let's walk through the uploadChunk function:

We kick off the upload by creating a ```XMLHttpRequest``` to handle the upload. We add a listener so we can track the upload progress.  

### adding a byterange header

When doing a partial upload, you need to tell the server which 'bit' of the file you are sending - we use the byterange header to do this.

We add a header to this request with the byterange of the chunk being uploaded.
>Note that in this case, the end of the byterange should be the last byte of the segment, so this value is one byte smaller than the slice command we used to create the chunk.

The header will look something like this: 

### Upload progress updates

While the video chunk is uploading, we can update the upload progress on the page, so our user knows that everything is working properly.  We created the progress listener at the beginning of the uploadChunk function. Now we can define what it does:

First, we do a little bit of math to compute the progress. For each chunk we can calculate the percentage uploaded (```percentComplete```). Again, a fun value for the demo, but not useful for real users.

What our users want is the ```totalPercentComplete```, a sum of the existing chunks uploaded, but the amount currently being uploaded.

For the sake of this demo, all of these values are written to the 'chunk-information' div on the page.

large_screenshot-2021-01-28-at-10-31-53.png

medium_screenshot-2021-01-28-at-10-31-53.png

small_screenshot-2021-01-28-at-10-31-53.png

thumbnail_screenshot-2021-01-28-at-10-31-53.png

video-upload-progress-on-screen.png

### Chunk upload complete

Once a chunk is fully uploaded, we run the following code (in the onload event).

When the file segment is uploaded, the API returns a JSON response with the VideoId.  We add this to the videoId variable, so it can be included in subsequent uploads. 

To upload the next chunk, we increment the bytrange ```start``` variable by the chunkSize. If we have not reached the end of the file, we call the createChunk function with the videoId and the start. This will recursively upload each subsequent slice of the large file, continuing until we reach the end of the file.

### Upload complete

When ```start > file.size```, we know that the file has been completely uploaded to the server, and our work is complete! In this example, we know that the server can accept 5 MB files, so we break up the video into many smaller segments to fit uder the server size maximum.

When the last segment is uploaded, the api.video response contains the full video response (similar to the [get video endpoint](https://docs.api.video/reference/api/Videos#retrieve-a-video-object)). This response includes the player url that is used to watch the video. We add this value to the ```playerUrl``` variable, and add a link on the page so that the user can see their video. And with that, we've done it!

Page showing upload complete - with link to see the completed video

large_screenshot-2021-01-28-at-10-32-03.png

medium_screenshot-2021-01-28-at-10-32-03.png

small_screenshot-2021-01-28-at-10-32-03.png

thumbnail_screenshot-2021-01-28-at-10-32-03.png

video-upload-completed-with-link.png

## Conclusion

In this post, we use a form to accept a large file from our user. To prevent any 413: file too large upload errors, we use the ```file.slice``` API in the users' browser to break up the file locally. We can subsequently upload each segment until the entire file has been completely uploaded to the server.  All of this is done without any work from the end user. No more "file too large" error messages, improving the customer experience by abstracting a complex problem with an invisible solution!

When building a video uploading infrastructure, it us great to know that browser APIs can make your job building upload tools easy and painless for your users.

Are you using the File and Blob APIs in your upload service? Let us know how!
If you'd like to try it out, your can create a [free account](https://dashboard.api.video/register) and use the Sandbox environment for your tests.  
If this has helped you, leave a comment in our [community forum](https://community.api.video).



Uploading large files with JavaScript

small_JavaScript-LargeFiles.png

thumbnail_JavaScript-LargeFiles.png

JavaScript-LargeFiles.png

Uploading large files with JavaScript: File.slice() to the rescue!

Private Videos

Private Live

api.video offers a number of simple APIs to create and share video.  To protect videos from being shared, we offer the concept of private videos - where the ```public``` token for the video is set to ```false```.

These [private videos](https://api.video/blog/tutorials/tutorial-private-videos) have a URL token that can be used just once, and cannot be shared. To authenticate the token, we insert a custom header that must also be validated with the URL token.

If you use the api.video player - there is no development required - we do all the lifting of adding the authentication header for you.

If you are using a different player, you need to recieve this header from cdn.api.video and insert this header on each subsequent request.  To learn how, read on!

## Retrieving the header

For each unique view of a private video, you must generate a unique URL token.  It will look something like this:

>https://embed.api.video/vod/vitkytkieTnEvUjKE0VQpd2?token=c7cc5f68-1c98-49a0-a1c3-406e95d95228

Using Chrome DevTools, we can see what happens when we request this url:





large_picture-of-requests-in-chrome-devtools.png

medium_picture-of-requests-in-chrome-devtools.png

small_picture-of-requests-in-chrome-devtools.png

thumbnail_picture-of-requests-in-chrome-devtools.png

picture-of-requests-in-chrome-devtools.png

The 4th request is to cdn.api.video for the player.json.  The first request to this domain for a private video (either for the .mp4, the player.json, or the manifest.m3u8) will have a custom header in the response (```X-Token-Session```).


medium_x-token-session-in-headers.png

small_x-token-session-in-headers.png

thumbnail_x-token-session-in-headers.png

x-token-session-in-headers.png


This header and value must be sent on every subsequent request to cdn.api.video to ensure playback.

If you use the api.video player, each subsequent request has this header added  automatically.  For example, here is the request for the video thumbnail.jpg (in Safari, with a different session token):

request for the video thumbnail.jpg (in Safari)

large_request-for-the-video-thumbnail-jpg-in-safari.png

medium_request-for-the-video-thumbnail-jpg-in-safari.png

small_request-for-the-video-thumbnail-jpg-in-safari.png

thumbnail_request-for-the-video-thumbnail-jpg-in-safari.png

request-for-the-video-thumbnail-jpg-in-safari.png

At the bottom of the header list is the ```X-Token-Session```. 

## Using a different player

With a different video player, the code to insert this custom header must be added.  So lets walk through the steps to add a new request header in your code.

### Getting the session header

To retrieve the session header, simply request the first file from api.video (typically this is the m3u8 manifest file). It will look something like:

```
https://cdn.api.video/vod/vitkytkieTnEvUjKE0VQpd2/token/8e19097c-8310-4ce3-acba-206378119bc5/hls/manifest.m3u8
```

If you request this file from your player (or you can test just by loading this url in Safari), you'll receive the response with a lot of headers:

```
Timing-Allow-Origin: *
Cache-Control: no-store, no-store
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Content-Length: 590
Via: 1.1 PSfgblPAR2xf192:2 (W), 1.1 PSygldLON4zd14:8 (W), 1.1 PSygldLON2ew56:2 (W)
X-XSS-Protection: 1; mode=block, 1; mode=block
Date: Fri, 23 Oct 2020 10:02:57 GMT
Access-Control-Expose-Headers: Content-Type, Content-Length, Content-Range, X-Token-Session, Vary, Origin
X-Frame-Options: SAMEORIGIN, SAMEORIGIN
Access-Control-Allow-Credentials: true
Content-Type: application/vnd.apple.mpegurl
ETag: "5e8f75e8-24e"
X-Content-Type-Options: nosniff, nosniff
Last-Modified: Thu, 09 Apr 2020 19:22:16 GMT
x-proxy-cache: HIT
Server: PWS/8.3.1.0.8
x-ws-request-id: 5f92aa50_PSygldLON2ew56_14724-24181
x-token-session: 284a505f-d556-4de5-9498-c68260927551
x-px: ms PSygldLON2ew56LHR,ms PSygldLON4zd14LHR,ms PSfgblPAR2xf192CDG(origin)
```

Now we have the x-token-session for the playback of the video, and we must include this in each subsequent request.

### Minimal working example

The following piece of code shows how to retrieve the source file of a private video from a token value (error handling has been omitted for readability reasons).


## Adding the token to your header


Depending on the type of resource to retrieve, adding the X-Token-Session header may not be as straightforward as in the example above.

Below are some tips that you might find useful.

### Using MediaSource capabilities

If you manage the video tag yourself, you can use the [MediaSource interface](https://developer.mozilla.org/fr/docs/Web/API/MediaSource) to add headers to video files related requests.

### Using URL.createObjectURL() method

[URL.createObjectURL](https://developer.mozilla.org/fr/docs/Web/API/URL/createObjectURL) might be useful for some kinds of assets. For instance, you could display a video poster using an img tag with the following code:


### Using a service worker

[Service workers](https://developer.mozilla.org/fr/docs/Web/API/Service_Worker_API) can be used to intercept and modify any request made by your application and thus it gives you the ability to add headers. 


### Using video.js based players

If your player is based on [video.js player framework](https://videojs.com/), you can redefine the beforeRequest method on the xhr object of videojs.Hls . This will give you the ability to add the header on any video request. 


If you have any questions about this tutorial or would like to learn more about api.video please join our [community forum](https://community.api.video/). 

Video on demand

Live streaming

Analytics

Video infrastructure

Video player

See all features

Online learning

Short-form videos

Marketplace & E-commerce

Video messaging and recording

Quickstart guides

Documentation

Ecosystem catalog

Clients & SDKs

Community

Blog

Demos

Help center

JavaScript