Comments Off on ProgrammableWeb : Why REST Keeps Me Up At Night

By daniel_jacobson, December 11, 2012 4:43 am

This post first appeared on ProgrammableWeb.com

With respect to Web APIs, the industry has clearly and emphatically landed on REST as the standard way to implement these services. And for good reason… REST, which is generally implemented as a one-size-fits-all solution, is an excellent choice for a most companies who wish to expose their content to third parties, mobile app developers, partners, internal teams, etc. There are many tomes about what REST is and how best to implement it, so I won’t go into detail here. But if I were to sum up the value proposition to these companies of the traditional REST solution, I would describe it as:

REST APIs are excellent at handling requests in a generic way, establishing a set of rules that allow a large number of known and unknown developers to easily consume the services that the API offers.

In this model, everyone knows how to behave and it can be incredibly powerful. The API providers establish a set of rules and the API consumers must adhere to those rules to get what they want from the API. It is perfect, right? In many cases, the answer is obviously yes. But in other cases, as our world scales and the number of ways for people to consume digital content and services continues to expand, this one-size-fits-all model is likely to fall short.

The potential shortcomings surface because this model assumes that a key goal of these APIs is to serve a large number of known and unknown developers. The more I talk to people about APIs, however, the clearer it is that public APIs are waning in popularity and business opportunity and that the internal use case is the wave of the future. There are books, articles and case studiescropping up almost daily supporting this view. And while my company, Netflix, may be an outlier because of the scale in which we operate, I believe that we are an interesting model of how things are evolving.

Netflix is currently available on over 800 different device types, including game consoles, mobile phones, TVs, Blu-ray players, tablets, computers, and almost any other device that can stream video. Our API alone handles more than two billion incoming requests on peak days, which translates into almost ten billion real-time outgoing requests from the API to internal dependency services. These numbers are up by about 70x from just two years ago. Most companies do not have that kind of scale, but it is clear that with the continued growth of the device market more companies are resetting their strategies to be less about the public API and more about internal consumption of their own APIs to support device proliferation. When this transition occurs, the API is no longer targeting “a large number of known and unknown developers.” Rather, the key audience is a small number of known developers.

The potential conflict between the internal and public use cases is in the design of the API itself. Keep in mind that the design implications will not be problematic in many scenarios. It becomes a potential problem if the breadth of devices becomes so wide that the variability of features across them becomes substantially harder to manage. It is the breadth of devices that creates a problem for the one-size-fits-all API solutions.

If your target is a small group of teams with whom you have close relationships, the dynamics around the API change. For Netflix, we persisted on the one-size-fits-all REST model for quite a while as more and more devices got added on top of the API. But given our scale, one thing has become increasingly obvious. Our REST API, while very capable of handling the requests from our devices in a generic way, is optimized for none of them. This is the case because our REST API focuses on resources that are meant to be granular representations of the data, from the perspective of the data. The granularity is exactly what allows the API to support a large number of known and unknown developers. Because it sets the rules for how to interface with the data, it also forces all of the developers to adhere to those rules. That means that each device potentially has to work a little harder (or sometimes a lot harder) to get the data needed to create great user experiences because devices are different from each other.

The differences across these devices can be varied and sometimes significant. Here are some examples of variances across devices that may be challenging for one-size-fits-all models:

• Different devices may have different memory capacity
• Some devices may require a unique or proprietary format or delivery method
• Some devices may perform better with a flatter or more hierarchical document model
• Different devices have different screen real estate sizes which may impact which data elements are needed
• Some devices may perform better having bits streamed across HTTP rather than delivered as a complete document
• Different devices allow for different user interaction models, which could influence the metadata fields, delivery method, interaction model, etc.

Just think about the differences between an iPhone and your TV and how they beg for different user experiences. Moreover, the XBox and the Wii, both of which project to the TV, are different in the way users interact with them as well as in the hardware constraints, both of which may require different APIs to support them. When considering more than 800 different device types, the variance across them becomes overwhelming. And as more manufacturers continue to innovate on these devices, the variance may only broaden.

To solve this issue, it is becoming increasingly common for companies (including Netflix) to think about the interaction model in a different way. Rather than having the API create a set of rather rigid rules and forcing the various devices to follow them, companies are now thinking about ways to let the UI have more control in dictating what is needed from a service in support of their needs. Some are creating custom REST-based APIs to support a specific device or category of devices. Others are thinking about greater granularity in REST resources with more batching of calls. Some are creating orchestration layers, such as ql.io, in their API system to customize the interaction. These are all smart and practical ways around the problem. But with the growing number of devices, the increasing urge for companies to be on as many of them as possible, and the desire for continued innovation across these devices, these various solutions are still somewhat restricted. They are still forcing the developers to adhere to server-side rules and non-optimized payloads in an effort to have a one-size-fits-all solution. These approaches are closer to the flexibility needed in that they are not as rigid as the typical REST-based solution, but when supporting as many devices as Netflix does, we believe they fall short for us.

For Netflix, our goal is to take advantage of the differences of these devices rather than treating them generically. As a result, we are handing over the creation of the rules to the developers who consume the API rather than forcing them to adhere to a generic set of rules applied by the API team. In other words, we have created a platform for API development.

Comments Off on ProgrammableWeb : 7 Ways to Make Your API More Successful

By daniel_jacobson, December 10, 2012 8:35 pm

I originally published this article to ProgrammableWeb.com on December 10, 2012

The purpose of a content API is to make the content available to its audience in the most useful and efficient way possible. To be a useful API, it needs to help the developers make their jobs easier. This could mean a wide range of things, including making it easier to dig into the API, allowing for greater flexibility in the responses, improved performance and efficiency for both the API and its consumer. Below are seven development techniques (all of which are part of the NPR API) that can help content providers improve the usefulness and efficiency of their APIs on both sides of the track. These techniques played a critical role in the success of the API which now delivers over 700 million stories per month to its users (more stats on the NPR API coming soon on our Inside NPR.org blog).

Be Flexible: Support Multiple Output Formats
Making the API as available and accessible as possible is very important in drawing developers to use it. So providing the content in a range of formats will increase the likelihood that the developer can rely on existing libraries and make as few changes to the code as possible.

The NPR API offers eight different output formats in an effort to improve efficiency for the developers. Above is a graph demonstrating the distribution of requests for each of the formats in July of 2009. As you can see, the majority of requests are to our proprietary XML markup (NPRML). That also means that almost 50% of the requests, or about 20M requests per month, use the other seven formats. In offering offering these other non-proprietary XML formats, the API is able to support developers that may have existing applications that pull in content in one of these standardized format, such as MediaRSS or Atom.

To make it even easier for people to use the API, NPR also launched with JavaScript and HTML “widgets”. The other six formats require more sophistication in order to put the content in an application or website. The widgets, however, are pre-designed feeds of NPR content (based on the developer’s selections) that can be easily dropped into a page.

Be Efficient: Handle Partial Response
This concept is now starting to get some more traction, now that Google announced partial response handling for some of their APIs. NPR’s API also makes extensive us of this feature because it really is tremendously valuable to the provider and the consumer of the API. For example, NPR stories contain a wide variety of fields and assets in the API. If the consumer is forced to handle the complete document, even if they only want a few fields, they have to endure all of the latency issues from the API itself as well as the additional processing power needed to handle the undesired fields.

As a result, NPR incorporated a “fields” parameter (the same parameter name used by Google) that can be used in the query string to limit the resulting document to only the fields of interest. This approach creates documents that are smaller and much more efficient. Overwhelmingly, more requests to the NPR API contain the fields parameter than those that do not (in fact, it isn’t even close).

Here are a few examples of how the same query to the NPR API, returning the same stories, delivers different documents based on the fields parameter (you will need to register for your own NPR API key to execute these queries):

http://api.npr.org/query?id=1001&apiKey=your_api_key

http://api.npr.org/query?id=1001&fields=title&apiKey=your_api_key

http://api.npr.org/query?id=1001&fields=title,teaser,text,image,audio&apiKey=your_api_key

An extension of partial response is to allow the developer to specify the number of items they would like in return. Some APIs return a fixed number of results, which can bloat the document just like the extra fields can. The NPR API, to counter this, allows the developer to pass in the number of results desired (with a fixed ceiling for any given request). To dig deeper in the results, we incorporated a “pagination” feature in the API. Here are some examples of how to control the number of stories:

http://api.npr.org/query?id=1001&numResults=5&apiKey=your_api_key

http://api.npr.org/query?id=1001&numResults=5&startNum=6&apiKey=your_api_key

Give Them Control: Allow for Customizable Output Markup (“Remapping Fields”)
As mentioned in the transform section, if the API can easily serve existing applications that expect specific markup, it potentially increases adoption and improves developer efficiency. To extend that functionality, the NPR API offers a function that we call “Remap” which essentially lets the developer modify the name of one or more XML elements or attributes in the output at request time. This is done in the query string and the API transforms the markup accordingly in real-time. Here are a few examples:

In this example, the remap parameter changes the story title to < specialTitle >:

http://api.npr.org/query?id=1001&remap=list.story.title:specialTitle&apiKey=your_api_key

In this example, the remap parameter changes the story title to < specialTitle > and it changes the image caption to < imageCaption >:

http://api.npr.org/query?id=1001&remap=list.story.title:specialTitle,list.story.image.caption:imageCaption&apiKey=your_api_key

In this example, the remap parameter changes the audio element’s id attribute to be named audioId:

http://api.npr.org/query?id=1001&remap=list.story.audio~id:audioId&apiKey=your_api_key

Another benefit to remap (which we have fortunately not had to use) is that it can be used to handle backward compatibility as the API grows and changes. NPR’s philosophy is to make sure that upgrades do not adversely affect existing functionality. That said, if an element or attribute does need to change, we could execute apache rewrites for all old API calls and have the remap function applied to have the output match that of the old markup. Alternatively, the developer could simply modify their API call instead of having to change their codebase to match the markup changes. (Although we do not intend to change existing markup, if we do, we would advise developers to upgrade their code accordingly. That said, rather than having the applications fail during the transition, remap could be used to temporarily handle requests until the full codebase can be upgraded).

Be Fast: Set Up a Comprehensive Caching Architecture
Performance is another critical aspect of APIs when it comes to enticing developers to use them. After all, if the API is sluggish, developers may not want to depend their application on it.

Smart caching of queries and results can really improve the speed of the system. NPR has implemented several layers of caching for the API, as follows:

• Base XML – Caching the full document for each item is important to prevent the system from executing disk I/O before doing any transform. We cache the Base XML first in memory and secondarily as XML files to eliminate the need to access our content database.
• Full Query Results – When compiling the list of items to be returned for any given story, it is important to cache the full list because popular applications that have many concurrent users (such as NPR Addict) are very likely to execute the same queries and expect the same results. The cached result is a single document containing the full list of all items and the full base XML for each.
• Transformed Query Results – The calling application, such as NPR Addict, expects the document to be transformed to fit the application’s needs. So, the results that get cached in Full Query Results may get transformed to MediaRSS while simultaneously removing extraneous fields. Caching the final results that get returned to the calling application enable fastest performance without compromising the system’s ability to use the other caching layers to produce different versions of the document.

Give Them Tools: Provide a Query UI with the Documentation
There are two truths about developers and documentation: the former always expects the latter, but seldom uses it. Of course, you cannot have an API without providing comprehensive documentation. That said, offering a simple user interface that helps developers get what they need from the API wil increase adoption and make life easier for them.

NPR’s API launched with a tool that we call the Query Generator. This tool exposes more than 6500 query-able IDs, methods for controlling the output format, fields to be returned, date and search restrictions, pagination, and more. Using the interface, the developer can select their options and have the tool create the query string for their API request. The developer can also see the results of that query inline before commiting it to their application. Almost exclusively, developers (including the NPR staff) use this tool to create queries, rather than reading the documentation.

Be Open: Eliminate Rate Limiting
Throttling or limiting access to APIs is an inherent disincentive for developers. Moreover, it is actually a detriment to the API provider. After all, the purpose of the API is to grant access to the content. If a given developer can only call the API 5000 times a day, and that developer creates a hugely popular application, the rate-limiting will inherently stifle the developer and the viral nature of the API.

Granted, most APIs use rate-limiting or tiered access levels to allow business people to control the graduation of API users. This seems counter-productive to me though. The better approach is to open access completely, identify those incredibly successful usages, then work with the developer accordingly on a mutually beneficial relationship. This way, applications are given full ability to grow and mature without arbitrary constraints.

Other APIs implement rate-limiting to protect the servers from unexpectedly high load. This is a legitimate risk which, if encountered, can adversely affect the performance of all users. That said, building complicated features into the system, such as rate-limiting, can be much more costly than configuring a scalable server architecture. Moreover, each request to the API will see slight latency increases as a result of the rate-limiting analysis. I know that latency is marginal, but why introduce any additional latency, especially when creating disincentives for developers?

Be Agile: Practice Iterative Development
Building your API over time has several benefits. First, it signals to the developer community that this API is meaningful to the provider and will continue to grow and get supported over time. This sounds trivial, but it is a very important part of the relationship with the community. If developers are not sure about your commitment to the API, are they likely to spend their own time building an application around it?

Another benefit of iterative development is that you do not have to get the API perfect the first time. I will qualify that by saying that, as a matter of principle, any release for an API should be done with the expectation that it will be supported for a long time. This is important because changes to existing API features will break the applications of those that use them. When I say the API doesn’t have to be perfect, I mean it does not have to be complete. New features can (and should) be added over time, extending its capability and making it more attractive for potential developers.

To put it another way, you will not have every detail of the API solved at the initial launch. It is much better to go live with the features that you know well while deferring those that you do not. Trying to cram in tenuous requirements will create headaches for you and for the community down the road. Spend the time necessary on figuring out the features, the supporting markup, the access and error methods, etc. before you commit to an API feature.

Comments Off on ProgrammableWeb : NPR API Architect Headed to Netflix

By daniel_jacobson, September 16, 2010 9:14 am

This article was originally published to ProgrammableWeb.com on September 16, 2010, upon the announcement that I was leaving NPR to join Netflix.

Daniel Jacobson, responsible for NPR’s trailblazing API, is leaving his post to join Netflix next month. Jacobson will become API Director of Engineering for the movie rental service, looking to support the company’s continued expansion to additional streaming devices. At NPR for over 10 years, Jacobson launched its API in 2008 and recently supported mobile devices that helped NPR’s traffic double in a year.

Jacobson joins Netflix at a time when the company is widely distributing its content to wherever media is played. All of these applications are supported by the Netflix API, which provides the meta-data, such as movie titles, and the ability to authenticate users to their own Netflix accounts. The new role is less about creating an API as it is expanding what’s already there. “More of the focal point will be continuing to evolve the APIs for the enterprise needs of the company,” Jacobson said.

While Netflix has been popular with developers, one major reason for an API is internal development, as Jacobson recently wrote in a guest post. “I think it’s a great fit because I think that’s exactly the model that NPR has taken,” Jacobson said. “It’s all about eating your own dog food.”

Before Jacobson launched NPR’s API, the organization had two outlets for its digital content: the website and what Jacobson called a “less-than-optimal mobile site.” Using the same APIs available to developers, NPR built apps for iPhone, Android and iPad, as well as a new mobile site. The result was 100% growth in NPR website traffic, mostly due to the apps. “As we launched apps, we saw additive pageviews. It wasn’t cannibalizing pageviews from the site,” Jacobson said.

NPR was among the first major media organizations to publish an API. When the we covered its launch, we noted it was the first talk radio API to provide access to the station’s content. Additionally, we compared it to the New York Times, which had announced but not released an API. The newspaper released its first API three months later.

Jacobson has been a frequent contributor to ProgrammableWeb as a guest author. For reference, here are all six of his blog posts so far:

• COPE: Create Once, Publish Everywhere
• Content Modularity: More Than Just Data Normalization
• Content Portability: Building an API is Not Enough
• 7 Ways to Make Your API More Successful
• How to Make Money With Your API
• Metrics for Content APIs: An NPR Case Study

His posts have provided a transparent view of how he ran the NPR API and we hope to continue to learn from his experience at Netflix.

Comments Off on ProgrammableWeb : Metrics for Content APIs: An NPR Case Study

By daniel_jacobson, September 10, 2010 1:27 pm

I originally posted this article to ProgrammableWeb.com on September 15, 2010.

This guest post comes from Daniel Jacobson, Director of Application Development for NPR. Daniel leads NPR’s content management solutions, is the creator of the NPR API and is a frequent contributor to the Inside NPR.org blog.

In my previous post, I discussed how companies can make money by using their content APIs to improve internal processes to enable rapid product development and to extend their reach. To successfully do this, however, this also requires a strong plan on how to capture appropriate metrics for the API.

At least from NPR’s perspective, the primary goal of the API is to get as many eyeballs on the content as possible. To achieve this goal, there are several ways to track the content as it travels through the API, each of which serve their own role. The following are the four key metric types that NPR is targeting:

• Request
• Response
• Impression
• Loyalty

Each of these are important in determining the true reach of the API, although their respective values to the overall equation are different. Moreover, each comes with its own challenges in capturing and parsing the data. Below is NPR’s definition of each of these metrics along with some basic data that NPR has (or doesn’t have) so you can see the relevance of each to our API strategy.

Request
The marketplace standard right now for tracking API metrics is based on API requests. While this metric is very useful and important, it is only a segment of the metrics needed to really determine the success of a content API. This is because requests do not translate into actual consumption – they merely create opportunties for consumption. To put it another way, tracking requests reveals information about how developers use the API even though the API itself is really just a means to get content in front of consumers. So, it is critical for producers of content APIs to be able to track how the content is consumed when distributed through the API.

The following chart details the growth of the NPR API over time in terms of API requests.

Response
Although the request metrics tell us what the developer asked for, they do not tell us what was delivered to the developer. Depending on the nature of the API, the response may include multiple items for each request and/or they could include warning and/or error codes and other information that gets returned to the user. A common example of this is an RSS feed which receives a single request but can deliver many stories. If the API captures only request metrics, it is missing the specifics around what was returned to the API developer.

The response data is critical as it tells you what content is potentially available to end-users.

Although NPR received more than 72 million requests to the API in August, it delivered over 1.3 billion stories over that same timeframe. This translates into roughly 18 stories per request. Clearly, by capturing only the request data, you are missing a very important part of the story.

Impression
Impressions are the first point in the metrics calculation where actual consumption is captured. By an impression, I mean a page view (or equivalent) where an end-user experiences the content that was delivered by the API. Generally, the way this metric gets captured in APIs is by putting an image beacon in a piece of the content. The beacon renders from the API’s server when it gets presented by the calling app, providing you with information about the content and its consumption every time it is viewed.

This is a very important metric because it is the impression that measures the number of eyeballs that see the content delivered by the API. For example, there are likely some requests that never get presented to a user if the calling application never presents them. Additionally, there are other requests that the calling application caches and gets presented multiple times to a user for that one request. Moreover, because a single request could return multiple items in a response, depending on how the requesting application handles it, there could be many impressions for that single request. As a result, the impression numbers could be substantially higher or lower than the request and/or response totals, depending on how the calling application interacts with the API. Because advertising revenues for many content APIs are dependent on actual consumption numbers (and not server traffic), the impression metric is much more important than the request or response totals.

The above image demonstrates how the NPR News iPhone App accesses the NPR API. In our app, a single API request is made to present the screen on the left. In that request, 25 stories get returned. Each of those stories contain the full story content, including images, audio and full text. The list view of all 25 stories garner a single page view. Clicking through one any of those stories results in the screen on the right, which is the full story page. The full story page garners another page view even though the iPhone app does not make another API request for it. In fact, if I launched the app, went to the Science page and looked at every story page from that list, it would result in 26 page views, all stemming from a single API request.

Loyalty
Once an impression is realized by the API, the next step is to create some relationships and loyalty around your content. After the user consumes a piece of content, did they carry on to another piece? Or do they have trackable sessions in the system already, perhaps from a different platform (whether delivered from the API or not)? There are several ways to try to make these relationships, but this is quite challenging and NPR is in the very early stages of trying to handle this. Our approach so far has been to use impression-related data mixed with query string parameters and session-related data (such as cookies).

A tangible example of this is if the content that gets delivered contains an audio or video asset. Generating an impression on the story is the first step. If the user then clicks on the audio, that click-through should also be attributed to the session attached to the API impression by passing tracking information to the audio URL so the audio piece can be related to the page view. By creating opportunities for the API content to create serendipitous experiences with other content, you are building a strong, more sellable content API.

As I mentioned before, capturing data for each of these metrics offers unique challenges. For example, to improve performance on our APIs, NPR uses a suite of caching layers. Moreover, the API has a lot of rights exclusion algorithms and transformations. As a result, it is increasingly difficult to ensure successful tracking of all of the metrics for all of the requests. Tracking impressions from APIs offers unique challenges since much of the content is getting distributed in XML, JSON or something comparable. How do you put a tracking beacon in the content? In which field should it go and how can you be sure that the calling application will consume that field? If you put it in multiple fields to ensure consumption, how do you prevent duplicate hits for a single page view? Finally, assuming that you are successful in accurately tracking metrics for each of the above, how do you convert them into a compelling story, one that offers value to the business?

I do not want to imply that NPR has solved all of these problems. Rather, we have built systems that do help us capture information about all phases of these metrics. But these systems are not bullet-proof. They do, however, give us more data about the content consumption from the API than merely request-based metrics, allowing us to learn more about how the API helps us achieve of greater goal: to increase the total number of eyeballs on our content.