Daniel Jacobson is the director of API engineering at Netflix. Prior to Netflix, Daniel was at NPR where he created the NPR API as well as the content management system that drives NPR.org, mobile platforms and all other digital presentations for NPR content. Daniel is also on the board of directors for OpenID.
This is Daniel Jacobson, Director of Engineering for the API here at Netflix. The Netflix API launched in 2008 with a focus on the public developer community. The expectation was that this community would build amazing and inspiring applications that would take Netflix to a new level in serving our members. While some fantastic things were built by this community, including sites like Instant Watcher, the transformational moment for the API was when we started to use it to deliver the streaming functionality to Netflix ready devices. In the last year, we have escalated this approach to deliver Netflix metadata through the API to hundreds of devices. All of this has culminated in tremendous growth of the API, as represented by the following chart:
The tremendous growth of the API over the last year is due to a combination of the increased number of users, more activity by our users, Netflix’s steady adoption of new devices over time, as well as chattier interfaces to the API.
Growing the API by about 37× in 13 months indicates a few things to us. First, it demonstrates the tremendous success of the API and the fact that it has become a critical system within Netflix. Moreover, it suggests that, because it is so critical, we have to get it right. When reviewing the founding assumptions of the API from 2008, it is now clear to us that the API needs to be redesigned to carry us into the future.
Establishing New Goals
In the two-and-a-half years that the API has been live, some major, fundamental changes have taken place, both with the API and with Netflix. I already mentioned the change in focus from an exclusively public API to one that also drives our device experiences. Additionally, at the time of the launch, Netflix was primarily focused on delivering DVDs. Today, while DVDs are still part of our identity, the growth of our business is streaming. Moreover, we are no longer US-only. In October, we launched in Canada with a pure streaming plan and we are exploring other international markets as well. Because of these fundamental changes, as well as others that have cropped up along the way, the goals of the API have changed. And because the goals have changed, the way the API needs to operate has as well.
Decreasing Total Requests
An example of where the current design is inefficient is in the way the API resources are modeled. Today, there are about 20 resources in the API. Some of these resources are very similar to each other, although they each have their own interfaces and responses. Because of the number of resources and the fact that we are adhering very closely to the REST conventions, our devices need to make a series of calls to the APIs to get all the content needed to render the user interface. The result is that there is a high degree of chattiness between the devices and the APIs. In fact, one of our device implementations accounts for about 50% of the total API calls. That same device, however, is responsible for significantly less streaming traffic. Why is this device so chatty? Can we design our API to reduce the number of calls needed to create the same experience? In essence, assuming everything remains static, could the 20+ billion requests that we handled in January 2011 have been 15 billion? Or 10 billion?
Decreasing Payload
If we reduce the number of requests to the API to achieve the same user experience, it implies that the payload of each request will need to be larger. While it is possible that this extra payload won’t noticeably impair performance, we still would like to reduce the total number of bits delivered. To do so, we will also be looking at ways to handle partial response through the API. Our goal in this approach will be to conceptualize the API as a database. A database can handle incredible variability in requests through SQL. We want the API to be able to answer questions with the same degree of variability that SQL can for a database. Other implementations, like YQL and OData, offer similar flexibility and we will research them as well. Chattiness and payload size (as well as their impact on the request/response model) are just two examples of the things we are researching in our upcoming API redesign. In the coming weeks, as we get deeper into this work, we will continue to post our thinking to this blog.
If these challenges seem exciting to you, we are hiring! Check out the jobs on the API team at our jobs site.
Last night, I had the pleasure of joining my old colleagues from NPR at the Online Journalism Association’s award ceremony in DC. First of all, it was great to see the gang again after three weeks as a non-NPR employee. It was also great to see NPR nominated for a wide range of awards (eight in total). The highlights for me, however, were that the NPR API won its first award (explicitly) and the fact that Kinsey Wilson won the Rich Jaroslovsky award (congratulations, Kinsey!).
The NPR API won the Gannett Foundation Award for Technical Innovation in the Service of Digital Journalism, which is a huge honor! I couldn’t be more proud of the NPR team, and more specifically the NPR Digital Media Tech Team, in claiming this. The work that the team did is amazing and it is great to see that work getting recognized in such a prestigious fashion. I am also very thankful that Kinsey invited me to accept the award on behalf of NPR. Representing such a tremendous team in this forum is a huge honor for me. While accepting the award, I explicitly thanked Zach Brand and Harold Neal as my “partners in crime” in getting the API live. While I stand by that statement, I do wish I explicitly thanked the rest of the Tech Team who has played a very important role in the evolution of the API since its launch in 2008.
Congratulations to Demian Perry, Jeremy Pennycook, Jennifer Oh and the other contributors to NPR Mobile as well, for winning the award for Outstanding Use of Emerging Technologies.
I look forward to seeing more great achievements from NPR going forward. I expect that there will be many…
Daniel Jacobson, responsible for NPR’s trailblazing API, is leaving his post to join Netflix next month. Jacobson will become API Director of Engineering for the movie rental service, looking to support the company’s continued expansion to additional streaming devices. At NPR for over 10 years, Jacobson launched its API in 2008 and recently supported mobile devices that helped NPR’s traffic double in a year.
Jacobson joins Netflix at a time when the company is widely distributing its content to wherever media is played. All of these applications are supported by the Netflix API, which provides the meta-data, such as movie titles, and the ability to authenticate users to their own Netflix accounts. The new role is less about creating an API as it is expanding what’s already there. “More of the focal point will be continuing to evolve the APIs for the enterprise needs of the company,” Jacobson said.
While Netflix has been popular with developers, one major reason for an API is internal development, as Jacobson recently wrote in a guest post. “I think it’s a great fit because I think that’s exactly the model that NPR has taken,” Jacobson said. “It’s all about eating your own dog food.”
Before Jacobson launched NPR’s API, the organization had two outlets for its digital content: the website and what Jacobson called a “less-than-optimal mobile site.” Using the same APIs available to developers, NPR built apps for iPhone, Android and iPad, as well as a new mobile site. The result was 100% growth in NPR website traffic, mostly due to the apps. “As we launched apps, we saw additive pageviews. It wasn’t cannibalizing pageviews from the site,” Jacobson said.
NPR was among the first major media organizations to publish an API. When the we covered its launch, we noted it was the first talk radio API to provide access to the station’s content. Additionally, we compared it to the New York Times, which had announced but not released an API. The newspaper released its first API three months later.
Jacobson has been a frequent contributor to ProgrammableWeb as a guest author. For reference, here are all six of his blog posts so far:
NPR’s director of application development, who set the vision for the APIs that have fueled a significant part of the network’s online growth, is leaving the nation’s capital to move to Silicon Valley where he will join Netflix in mid-October.
Daniel Jacobson will lead development efforts around the video rental giant’s APIs, including those that enable the transfer of content to set-top boxes and game consoles. like the Xbox, and mobile devices, like the iPhone and iPad.
Jacobson joined NPR 11 years ago and initially oversaw the development of the network’s content management system, which was launched in 2002 and which formed the underpinnings of the API system. The API system, which went live in 2008, allows NPR to easily share its content with publishers both inside and outside the network. The existence of the API has been credited with vastly accelerating NPR’s move into the mobile space, including its iPhone and iPad apps, as well as making possible the creation of its new Argo Network and enabling NPR to enjoy 100% growth in pageviews over the past year.
Comments Off on Interview : Daniel Jacobson Interviewed at Where 2.0 in 2011 
Themocracy