This post is the first in a series of blog posts on how we develop, deploy and operate an API as a product.
In this first post, we’ll discuss the goals we want to achieve with the proposed development and production setup. In follow up blog posts, we’ll discuss how we actually implemented the architecture, a development pipeline and production setup, and we'll discuss some of the tradeoffs and assumptions we made.
Let’s start with the product we would like to build: a startups API. Three years ago, our colleague Omar Mohout started to painstakingly collect data of the Belgian startup scene. At that time, nobody really had a good view on the startup scene in Belgium, let alone how it was evolving. Omar started gathering that data, and has since published many reports and insights on it. (You can find some of his analyses here, here or here.) As a result, Omar receives every so often requests from others to have access to that data, so they can do their own analyses. We also got the question to provide the community with a map and visual tools, all based on that data set. We thought it would be a good idea to expose that data through an official REST API, instead of sending excel files back and forth.
You can see the first app made on top of the API over at sirris.datascouts.eu.
The Requirements (at least some of them)
Since we are bringing the startups API to the market as a real product, not just as some internal thingie, the idea was to do that “by the book”. This means the API would need:
- An API portal where customers of our API can sign up for access to the API, manage their API keys, and where developers find information on how to use the API.
- An API management component that allows the API owner to define plans, rate limits, etc.
- A production setup that enforces the use of API keys, ensures that rate limits are respected, i.e. an api gateway.
- Some simple dashboards and analytics on API usage.
- A production setup that is robust, yet easy to maintain (did I mention that development and operations of the startup API would be done by 1 person, and that the API isn’t supposed to be a full time job?)
- Implement the necessary security practices to ensure that access to our startup data is protected at all times.
To add to these requirements above, we wanted to take advantage of the state of the art in (cloud) software engineering and DevOps and make sure that we aim for:
- developer parity: make the difference in setup between production and development as small as possible,making it easy to test, change and debug issues, should they arise.
- git push based continuous delivery: a development workflow that allows for fast, agile iterations, and that leverages test driven development and CI/CD. Ideally, a simple git push should trigger building, testing and deploying next iteration of the application.
- the whole setup should be fun and easy to work with.
Architectural view of your API
With these requirements in mind, and with our own capabilities and preferences in mind, we made the following technological choices:
- Implement the REST service in hapi (http://hapijs.com/), an open source webserver built in Node.js by Wallmart. Hapi comes out of the box support with for API’s and has excelent testing support through the lab library. We're used to developing in Node.js, so no learning curve here, hapi is well documented, which is a big plus.
- Mongodb as the data store: REST services produce mostly JSON these days, I therefor went with mongodb, since mongodb stores JSON (well technically BSON) directly, and uses JSON for querying. Another plus is that we have some inhouse experience with small scale mongo stuff.
- 3scale.net as an API management platform. 3scale offers developer portals right out of the box, provides an API gateway service (both as a service as well as self hosted) and has a free plan to start with. Alternatives could be mashape or apigee.
- Swagger for documenting the API. hapi has a swagger plugin that generates swagger documentation based upon the source code of the API itself, no extra work for me there (at least, that's the plan).
- For deployment, Docker containers will be used, scheduled, managed and orchestrated via Rancher. Using docker containers will make the app portable, so running the app locally or in production would more or less be the same.
- we’ll use Amazon Web Services as the infrastructure provider, but we decided not to go overboard with it and mainly use EC2 (virtual machines) and ELB.
- Let's Encrypt for everything ssl related. It's a free, open and automatic Certificate Authority that get's a lot of traction lately, what's not to like.
- Use an internal, Sirris hosted GitLab repository for our source code, linked to an Jenkins as a continuous integration and delivery server that builds, tests and deploys our code.
In this blog post we sketched the overall architecture we have in mind for the startups API. In the next instalment in this series, we’ll discuss the startups REST api itself, highlight some of our lessons learned when implementing the API on top of hapi and we’ll take our first steps in containerizing the application.