api.video
SearchCtrl K

Features

Developers

Why and how api.video automated all their API clients

August 24, 2021 - Olivier Lando

Today, api.video automates all of its clients and general client documentation. It's fast and convenient for developers to get up and running with the language of their choice. But it wasn't always so easy.

The issue

By the start of 2021, we had 4 SDKs for the api.video API. We had one for each of the following languages:

  • Java
  • Node.js
  • Go
  • PHP

These SDKs had been built by different developers. They worked pretty well but there were some inconsistencies between them, for example:

  • How methods were named
  • How the client was instantiated
  • What features were included or updated

While we were trying to decide how to handle these inconsistencies, we also decided to meet the demands of our customers by developing two additional SDKs:

  • Python
  • C#

We also realized that we wanted to use the correct nomenclature for our clients. While SDK is a popular term, it’s usually misused. A true software development kit is a complete set of tools that facilitates the creation of applications. It’s typically a set of software components that allow you to handle more complex tasks in a particular way out-of-the-box. It can include helper tools for debugging, complete solutions for certain tasks, detailed documentation, different kinds of integration options and more. On the flipside, a client library creates reusable chunks of code you can use as building blocks to do more complex tasks - like the ones you might see in an SDK. We wanted to start using the term clients and made the decision to move to this term rather than SDK.

With that decision settled, we were still faced with maintenance for our growing list of clients. We realized that maintaining all 6 of the clients plus adding more of them and keeping them in sync with our API was going to be really painful. These considerations led us to question the possibility of generating these API clients automatically, instead of maintaining them manually.

The envisaged solution

An API client is a pretty simple thing. It’s mostly composed of models reflecting the API payloads, and methods that make the http calls. All this is pretty well described in the OpenAPI description of our API that we already had for documentation purposes. Therefore, we realized that it shouldn't be too strenuous to generate our API clients using the API description.

After a few investigations on how to achieve this, we chose to use the OpenAPI client generator. The reasons were the following:

  • OpenAPI generator is widely used, including in large projects
  • It supports a lot of target languages
  • It’s open source and highly customizable

Our requirements

Before we got started, we put together our requirements for the new clients. We wanted something that would work well for us and especially our customers. So we came up with this list:

  • Ease of use for our customers: Generated clients are often rough and don't handle the small particulars that an API may have. We didn't want the switch to generated clients to make our clients more complicated to use. That means things like authentication, results pagination, efficient upload of huge files using a chunk splitting mechanism, all must be handled seamlessly by the client.
  • Release efficiency: One of the goals of having a generated client is to be able to release new versions very quickly. In particular, when a simple change is made to our API (for example adding a new attribute in a payload), we wanted to be able to release a new version of all our API clients the same day.
  • Automation: We wanted to avoid manual actions as much as possible between the moment when the API description is updated and when new API clients are released.

The painful part: building the generator templates

For each target language, the OpenAPI generator uses a bundle of templates of the code files that will be generated. If you want to see what it looks like, you can take a look at the base templates for Typescript here: typescript templates. The templating system used is Mustache. Mustache syntax is really simple (sometimes too simple).

For each target language, we started from the base templates included in the OpenAPI generator and we updated them to make them match our needs. In particular, we had to modify them to:

  • Implement our authentication mechanism: the most common authentication mechanisms (OAuth2, basic password) are handled by most of the template bundle, but ours is based on an exchange of a private API key to an access token was not.
  • Handle our video upload mechanism: to improve reliability of huge video files transferred we use a mechanism based on file splitting into chunks. This is a particular use case that is of course not handled in the default templates.
  • Facilitate the use of the endpoints of our API that return paginated results (it must be easy to retrieve the results of the next page)
  • Make some client parameters modifiable by the user, such as: the size of the upload chunks, the API environment to use (sandbox or production), etc.

For some of these adaptations, modifying the templates was not enough, we had to go a little further by modifying the code of the generator itself. It’s pretty straightforward, each target language has a dedicated Java class responsible for generating the variables sent to the template engine from the API description. We simply had to write subclasses of these Java classes and put the needed changes in them.

The release process before automation

Once the template has been written, generating the code of a client is done by executing a single Maven command. In the weeks following the end of the development of our generator, the process of releasing new versions of the clients after an API change was very manual. It looked like this: 1. The API description was updated to reflect the change made in the API 2. Someone from the Ecosystem tech team launched the generator 6 times on their computer (1 execution per client) 3. For each generated client, the teammate ran the unit tests & integration tests 4. If everything went well, they committed the new code of each client to its dedicated GitHub repository (6 commits) 5. They created a GitHub release on each of the 6 repos 6. They manually released a new package on each package repository (npm.js for javascript, maven for java, pypi for python, etc.)

That was really painful and error-prone, so we decided to automate all of this.

The release process with automation

In total we have 7 GitHub repositories for our client : 1 for the generator itself, and one per generated client.

Thanks to GitHub Actions, we have implemented the following process:

When a pull request is made on the generator repository (this happens when the API description itself is updated, or when templates are changed), 6 GitHub actions are triggered on the generator repository: one per client. Each of them will:

  • clone the code of the client from its repository
  • launch the Maven command to generate a new version of the client
  • if something changed in the code of the client a branch is created on the client repository, a commit is made, and a PR is created

The GitHub action that creates the clients pull requests

After this step, is most cases, 6 pull requests have been created: one on each client repository, but it may be less (for instance in we updated the template of the Python client, only the code of the Python client will change and then a PR will only be created on the Python client repository).

For each PR created, a GitHub action will be triggered on the associated repository in order to run the unit tests and the integration tests.

If everything goes well (i.e. the tests are not failing), the pull request on the generator repository is merged by someone from the ecosystem team. That will trigger another GitHub action that will merge all the pull requests that have been created on the clients repository.

After that we manually create a release on each repository that will trigger the last actions responsible for deploying a new package on each package manager (Maven, npm.js, etc.)

Links

The client generator: https://github.com/apivideo/api-client-generator

The generator GitHub actions: https://github.com/apivideo/api-client-generator/tree/master/.github/workflows

The generated API clients: https://github.com/apivideo/php-api-client https://github.com/apivideo/go-api-client https://github.com/apivideo/python-api-client https://github.com/apivideo/java-api-client https://github.com/apivideo/nodejs-api-client https://github.com/apivideo/csharp-api-client

Conclusion

And that’s everything! We now have an automated solution that lets us keep up with all the new changes to our API while providing clients in the language of choice of our developers.

Olivier Lando

Head of Ecosystem

Connect your users with videos

Create a free account today