Behind the Scenes: How We Built Our Developer CLI

When we started building our first version of the Zapier developer platform in 2012, we were unsure if engineers would devote much time to building apps on Zapier. At that time we had only around 10,000 users—so we focused on a browser interface that lets anyone build Zapier apps, regardless of technical proficiency. That interface, which we call the Web Builder, has supported most of the apps you see on Zapier today.

However, with that initial Web Builder, we quickly ran into the numerous rough edges in talking to APIs which required some pretty nasty workarounds. After about the 9th unique workaround we decided that instead of adding the 10th workaround, we should instead add the ability to execute custom JavaScript instead. This ended up being wildly popular, with about 90% of apps using some kind of scripting. The code in the Web Builder version of the platform was a big inspiration for us to build a modern CLI for the Zapier platform.

The final push for us to create the CLI was how unwieldy it was to execute custom JavaScript inside the Web Builder. Surprise surprise—engineers don't like coding in their browser! Add to that the inability to test, the lack of version control, and no team capabilities and you have a pretty miserable developer environment.

The CLI would solve all of that.

Building the CLI

We had a couple things that we wanted to accomplish with our CLI:

1. Provide a standard developer environment and tooling.
2. Use a standard language (we chose JavaScript).
3. Release the core code publicly.
4. Make it compatible with Zapier's 900+ apps.

We settled on a simple zapier command line utility written in JavaScript that would help users create, manage, and deploy standard Node.js packages to our production system. We broke apart the code into three distinct packages:

1. The CLI utility itself which contains all the helper commands for managing Zapier Node.js packages.
2. The JSON Schema for app definitions which validates and enforces the shape of Zapier apps.
3. The core library that contained all the glue code, HTTP helpers, logging, tools, and classic API client business logic.

Want to dive in? You can find the full code and document for each of these in the Zapier CLI Platform repository.

So how does it all work behind the scenes?

Creating, Building and Pushing an App

When you create an app via the zapier init . command, we're doing many of the same operations as npm init. We use a basic template similar to yeoman or cookiecutter to bootstrap the required index.js and package.json. This is purely local on your machine.

What happens when you run zapier register "My App" or zapier push is much more interesting!

The first thing the CLI must do is make sure you have an app registered with Zapier.com under your account. Once the remote app is created we also create a local .zapierapprc file to identify your app. Developers should check this into version control so all teammates using the CLI are working on the same app.

Behind the scenes, we have a REST API backed by Django that the zapier command will interact with, using whatever credentials you've already configured with a previous zapier login which stores a deploy key in ~/.zapierrc. This API is used by many commands as we'll illustrate below.

Next thing you'll do is a zapier build, which will take your local Zapier app and working directory, copy it to a tmp folder, do a fresh npm install, run some dependency analysis, and zip up all the needed .js files. The final zip file contains all the code and dependencies Zapier needs to run your package in the cloud.

Finally, you'd do a zapier upload. This takes the zip and uploads it to Zapier under the version you have defined in your package.json. This allows you to manage many live or legacy versions of your app, perfect for slowly deploying changes that might be risky.

If you'd like more information and examples, check out our CLI documentation which goes into more detail.

CLI Architecture

Once the zip of your app is uploaded to Zapier, behind the scenes we use AWS Lambda to run it in production. This involves Zapier creating an IAM role with no permissions in which to house the newly created function. Each version of a Node app gets an isolated function, meaning apps can have concurrent versions running at once.

After the function is created in AWS, Zapier can invoke it internally using the handy library boto3. The handler that the Lambda function exposes is one that is controlled by zapier-platform-core which allows us to intercept the invocation, perform any preparations, and call user code. For example, we might send an invoke call that looks something like this:

While an engineer would never interact directly with this JSON payload (our core wrapper would handle all this), it is fun to outline them anyways.

The command simply tells the handler what it wants to do. There are other options, including debug tools like "validate", but the majority of these are "execute".

The method tells the handler which user function or directive it wants to run this time. Typically this will be something like a function that returns an array of objects (if this is polling trigger, for example).

The bundle will contain the arguments commonly passed to the user function as defined by method.

The rpc_base and token (JWT) are endpoints that the handler can use to talk back to Zapier instead of just returning data from the function. For example, the z.stashFile utility requests a pre-signed Amazon S3 URL for uploading files, and the upcoming z.runWithLock acquires a Zap specific distributed lock.

And finally, the environment includes the Client ID code which is injected into process.env before user code is run. This is managed by the zapier env CLI command.

There are also some additional goodies we do behind the scenes. One of the more useful ones is the ability to use our built in z.request HTTP library to get automatic logging and authentication (depending on your app's configuration). Under the hood there we are using the now standard fetch built-in HTTP library.

Finally, there are some beta goodies like live app development which is made possible by calling a transparent RPC instead of AWS Lambda, in conjunction with using ngrok and zapier watch. While not officially launched or documented, you might give it a try and let us know how it works for you! :-D

Launching the CLI

In mid-March we finally launched the CLI after months of beta testing internally and externally. Almost all future apps and migrations Zapier develops internally will be performed on the CLI, ensuring that our external partners' experience is as great as our internal experience.

Overall, it has been a blast building the new platform and working with partners on their upcoming CLI applications. The CLI adds lots of missing features and brings the experience much closer to that of a classic development environment, and keeps us from building workarounds. But, if you are more comfortable with the Web Builder—don't worry, we're keeping that around too!

Now that you know about how it was built, try the Zapier CLI for yourself. Get started now!