Speeding up builds with Remote Execution API

Building and testing software can be a daunting task at times. In the olden days the development and testing phase used to take place locally, on the developer’s machine.

That’s no longer the case - nowadays not only do we follow the practices of Continuous Integration and Continuous Delivery but we’re also trying to parallelize and distribute our builds as much as possible.

Currently available solutions, however, are not ideal. Over at Antmicro we already use various open source CI systems, such as GitLab CI or Jenkins. There are also multiple open source projects that are able to distribute builds across networked nodes, such as distcc.

But they all have one flaw in common - lack of flexibility and poor integration with local workflow. And that’s fine, they weren’t designed with that in mind.

Still, the need for those functionalities remained, so a new solution called Remote Execution API emerged to address it. Let’s take a closer look at it.

Remote Execution API: simple, yet powerful

On its own, the Remote Execution API is a specification created by Google, describing how a client and a server, together with its squad of workers, should talk to each other to harmoniously perform a build.

It has been successfully implemented by an array of open source projects (which we’ll look into further down the line).

Executing a single action involves sending the files required for the build, a command to be executed and a list of expected output files. This bundle is called an Action Message.

This makes for an easy to understand flow - upload necessary files, perform a command, and get a file back.

Caching

Apart from defining a clear and concise flow, the Remote Execution API also provides a mechanism for caching the files and results of Actions.

Now, provided that the codebase hasn’t changed, the build may be performed only once. Subsequent attempts to compile the same version of the code will result in downloading previously cached results.

This ensures that developers don’t experience unnecessary downtime resulting from having to compile a big piece of code.

Let’s talk implementations

Under the hood, REAPI uses gRPC as the application layer.

There are a couple of open source projects implementing the specification, notably BuildGrid, Bazel, recc and Antmicro’s custom open source client, distant-rec, which has been under heavy development for quite some time now and is available on GitHub.

The server implementation which we used and wholeheartedly recommend is BuildGrid. It’s written in Python and can be easily reused, which is what we did during the development of our universal client.

Embrace Remote Execution to streamline your workflow

While still being rough around the edges, Remote Execution API offers a promising solution for accelerating builds and giving the workflow of developers a more streamlined and integrated form.

Stay tuned for the next post on the topic, and while you do so, don’t forget to check out the awesome open source projects we’ve mentioned - including our custom client!

If we got you enthusiastic about REAPI and you would like to stay on the cutting edge of the distributed computing solutions, don’t hesitate to contact us at contact@antmicro.com to find out how we can help you develop your business.