Dogfooding Your API: How to Build New Features Faster 
by Consuming Your Code

Editor’s note: We publish occasional guest posts on the Zapier Engineering Blog, like this one from Jared Cheney, a senior software engineer at TSheets.

I believe that to build a great product, you’ve got to eat your own dog food—or, in other words, you have to use all the features and documentation of your product before you release them to your customers. You get the first taste of everything you build.

I learned this the hard way.

Pull Up a Bowl, It’s Chow Time

When we produced the first iteration of our API in 2008, we thought we had done a good job designing it for all of the use cases that were required. We thought it was cooler than a polar bear’s toenails, and we expected the integrations to start rolling in.

Eager to garner the fruit of our labor, we outsourced development of a desktop widget to a third-party developer. They worked on it for a while and we answered all sorts of questions about how to use our API to handle the requirements we’d given them. We even made some changes at their request. But when it finally came to using what they had developed, it didn’t work the way we wanted it to.

There were problems with how it managed the state of a clocked-in user, especially when that user switched between multiple methods for tracking their time throughout the day. As we dug into the details of how we might “fix” it, we realized the design and architecture of our API had forced them down the path they’d taken.

We finally concluded that our API as it was at that point wouldn’t work for the project the way we’d envisioned, and we ended up scrapping the desktop widget altogether. Had we done things differently with our API, it would have saved a lot of time and money—and we would have gained a usable product from our third-party developer.

From that point on, we decided on a few rules:

1. We would always be the first consumer of our API.
2. We would force ourselves to use our API for as many operations as we could, so we could implement learnings quickly and make adjustments as we built.
3. We would develop and document any new additions to our API in tandem with the features that required them.

Dogfooding Our Android App

At this point, we’d made several improvements to our API and it was getting the job done. But we were itching to get a do-over. We wanted to change some key architectural decisions we had made concerning authentication and our request/response flow.

Fast forward to 2012 and we were starting a new project to write a native TSheets Android application. Now we had the chance to rewrite our API and see if the rules we had set for ourselves would yield the benefits we envisioned. We set out with the firm goal to rely entirely on our API for our Android app to work with our service, and to develop everything with the intent that it would be made publicly available.

As we developed the new API, we would put ourselves in the shoes of an external developer and constantly ask ourselves, “How do I wish the API would behave?” We built out one component of our app at a time and as we learned about each piece we needed, we would develop the corresponding API endpoint to serve that need simultaneously. If we found we couldn’t consume the output efficiently or were missing information that we needed, we’d tweak things until it fit our needs exactly.

What’s Good for the Customer is Good for You

An early challenge we had to address came up as we were trying to make our application work in an offline mode—which required us to pull down a lot of data for our larger customers on their initial sync.

For example, we’d pull down the last four weeks’ worth of timesheets for the user and each timesheet could reference multiple fields, so we’d have to download the field definitions and the possible values for those fields and what field was actually chosen for each timesheet.

With our API following traditional REST patterns, we found ourselves making multiple calls to retrieve everything we needed just to display historical data, let alone everything needed to allow the user to clock in right away! The initial sync process was just taking too long. We were frustrated with the wait and feared our customers would be even less forgiving.

Our solution to this problem became something we include in every response from our API under the key supplemental_data. We decided that, along with the primary objects that were requested, we would also include an instance of every object that was referenced by any of the primary objects in the supplemental_data portion of the response.

Even though it required more overhead on the servers that generated the response, it was beautiful for the consumer of the API. Now we could retrieve 20 timesheets and have every bit of information necessary to “hydrate” those timesheets with the information that would make it usable for our end user. And we were able to accomplish it with a single API request.

Let the App’s Requirements Dictate the Endpoints

Because we were on both sides of the developer/API relationship during this whole process, progress was extremely fast. And we had advocates on both sides representing what was important, so compromises and re-engineering had to take place until both sides were satisfied.

Several patterns emerged on how we would format our responses and what they’d include, and we were able to apply these patterns to every endpoint that we developed. As we continued to develop the app, we let the requirements of the app dictate what endpoints we worked on next.

From our experience with a first, second, and third currently-in-progress iteration of our API, we’ve learned a few things:

1. Observe the YAGNI Rule.

"You Aren't Gonna Need It" goes by the acronym YAGNI, as described by Martin Fowler. Don’t build things out too far, or else you may end up looking back on a lot of wasted time and effort when you realize you either don’t ever use it or it needs to be changed to be useful.

2. Avoid “Hidden” or “Special” API Endpoints Only You Know About.

If you keep something hidden from external developers and reserved for yourself, you’re shielding yourself from the pain that they might feel. You want to put yourself in their position, so your sympathies will grow and motivate you to make your API the best for your needs. This will naturally benefit all of the external developers who are in the same boat with you.

Otherwise, you may begin to rely too heavily on some hidden features of your API to accomplish your needs. And if you have those needs, the chances are good that others who want to integrate with you will too.

If you absolutely must have an API capability that is restricted to your own applications, you should still document and design them with the idea that an external developer may use them at some point. Otherwise, the natural tendency is to get it “just good enough” for your own use, and technical debt is incurred that will have to be handled when you finally enable those features for external developers.

3. Make a List of Tips and Tricks as You Go.

And build them into your documentation! They should consist of the logic you used to accomplish flows that are common to the use of your API. For example, we have a recommended way to retrieve the assigned job codes for a user so you get both the assignments and the related job code objects in the same response (using the supplemental_data mentioned above).

We also show you how best to quickly determine what objects have changed since the last time you queried our API. Giving “recipes” like this to your consumers can help save them effort and help ensure that interaction with your API is efficient.

4. The End Result is a Lot Better Than Dog Food.

If you can be disciplined in developing and documenting new additions to your API in tandem with the features that require them, you’ll end up with an API that is full-featured, well-documented, and easy to use.

The Designer Should Write the Manual

As much as I’d like to lay claim to it, “dogfooding” is not a new concept. In 1989, Donald Knuth published a paper recounting the lessons he learned from the development of his TeX Typesetting software, in which he mentions the benefits that come from using a “dogfooding” approach:

“Thus, I came to the conclusion that the designer of a new system must not only be the implementer and the first large-scale user; the designer should also write the first user manual. The separation of any of these four components would have hurt TeX significantly. If I had not participated fully in all these activities, literally hundreds of improvements would never have been made, because I would never have thought of them or perceived why they were important.”

— Donald E. Knuth, The Errors of TeX, Software-Practice and Experience, VOL. 19(7), JULY 1989, pp 622

Amen, brother.

Because like they say in the Kibbles 'n Bits commercial, “Your dog’s ideal meal isn’t what he’s eating, it’s what you’re eating.” Me? I prefer the chicken flavor.