Return to Blog

Announcing Our v3 API Stack: Faster Data Using Varnish and Node.js

At Factual we’re pretty obsessed with data.  In particular we love building APIs that let you use data in new and interesting ways.  That was the motivation behind launching our two newest APIs, Resolve and Crosswalk.

Another thing we’re obsessed with is performance, which is why we’re excited to announce the release of our re-engineered API stack that makes calls to our API faster, more reliable, and more scalable.

In terms of speed, you will enjoy much faster query times when making calls to our APIs.  Our Places API which has over 30 million place entities is clocking in at ~100 ms.  Calls to Crossref and Crosswalk which contain over a billion connections are now returning results in just 50 ms.  Lastly, Factual Resolve, which is a more CPU intensive query and usually meant to be run offline, is down from 1200 ms to about 300 ms.

We rebuilt our API stack using a variety of open source tools including Varnish, HAProxy, Node.js, and Redis.  We’ve been long time fans of using Varnish as a HTTP accelerator.  Varnish shines when it comes to serving up relatively static content, but it’s not very good for handling user specific queries, authentication, and authorization.  Since our APIs rely extensively on the latter, we use Node.js to implement a thin business logic layer to handle the things Varnish isn’t so good at.

Factual API Stack using Node.js to handle authentication and authorization before hitting the Varnish cache.

We chose Node because of three F’s: it’s fast, flexible, and familiar. In particular, the flexibility is  what allowed us to use our Node layer to handle things like caching logic and load balancing, in addition to the aforementioned authentication and authorization. To make our Node layer scalable, we use multiple instances of Node tied together with Redis to keep things in sync.

Our new API stack is just one of many steps we’re taking to optimize and improve performance, but if you like faster data as much as we do, we think you’ll love this new release.  You can take our v3 APIs for a test drive by registering for a key here.