Comments Off on Presentation : The Structure of API Revolutions : Video and Slides Daniel Jacobson shares advice on dealing with evolving APIs based on his experience with Netflix APIs.
Comments Off on Interview : Insert Content Here Podcast, Episode 11: NPR’s COPE and Content APIs This podcast episode was published on Insert Content Here with Jeff Eaton on March 15, 2013
Daniel Jacobson recaps the history of NPR’s COPE, his work at Netflix, and the future of content APIs.
Host: Jeff Eaton
Guest: Daniel Jacobson, Netflix
Mentioned in this episode:
This episode’s transcript is in progress; the partial transcript below has been auto-generated by Otter.ai.
Jeff Eaton: Hi, I’m Jeff Eaton, senior architect at Lullabot. And your host for insert content here. today I’m here with Daniel Jacobson, the illustrious Daniel Jacobson. He’s not necessarily someone that you may have seen him in content circles or nerding out in the CMS world that much, but he’s actually one of the movers and shakers.
His role for quite a few years was Director of Application Development at NPR, where he oversaw the development of COPE – the Create Once Publish Everywhere infrastructure that made all kinds of waves in the content strategy and CMS world. He’s the author of O’Reilly and Associates’ APIs: A Strategy Guide. And now, he’s the director of engineering for the Netflix API. So, he is just all over the place when it comes to content, APIs, and reusable structured content. It’s a pleasure to have you here, Daniel, thank you for joining us.
Daniel Jacobson: Thanks for having me, Jeff. Thanks for that very nice intro.
Jeff Eaton: Well, you know, it’s funny, I was actually we reading your blog a couple of weeks ago. And, you talked about a presentation that you had done at a conference on the architecture and you said, “wow”. You know, a bunch of people in the room knew about it and you know, they were familiar with it.
And it seems like you were almost a little startled by that. Did you expect the work that you were doing on that project to become such a big discussion topic,
Daniel Jacobson: You know, I really didn’t. We had been working — this goes back to the time of NPR — we’d been working on these concepts and implementation since 2002.
I can give you the history on that if you’re interested, but we were just doing what we thought was right. You know, thinking about architecture, thinking about longevity of application development and knowing that there are going to be a host of things that we’re not going to be aware of down the road, so trying to prepare for nimbleness.
So then in 2009, that’s when I first published that blog post on programmable web about COPE, I was just taking this step of, “okay, we’ve done all this stuff in the spirit of sharing, let’s share it”. And it was interesting to see how some people latched onto it.
But what really has surprised me is we’re in 2013, people are still talking about it. It’s mind boggling for me. And if you look at the blog post, over half of the, the comments were from 2012. I don’t know, 70 or 80 comments. so what’s really kind of staggering for me is the fact that, we published it in 2009.
We’ve been thinking of it for a number of years and implementing it for that time. But it’s still, or it is now resonating. It’s very weird.
Jeff Eaton: Well, I’ll back up for a second for anybody who might not be familiar, what is COPE? You know, what’s the idea behind it.
Daniel Jacobson: Okay. I’ll give you a little bit of a history on how we arrived at COPE and some of the sensibilities that went into it. COPE stands for Create Once Publish Everywhere. And the idea there is, you want to maximize the leverage of content creation and minimize the effort to distribute. The way that the whole thing really started, it goes back a little deeper. It was actually in 2002 when we had a series of different ways of publishing content to NPR. We had HTML files. We had a Cold Fusion system with a SQL Server database that publish very thin radio pieces to the web. Then I had build this other system, a Java based system with an Informix database to capture a little bit richer content and offer an opportunity for co-branding with local stations. In 2002, we looked at this and said, well, this is kind of a mess. We’re spending all this energy publishing in three different ways and we have three different presentation layers. What we really need to do is to collapse them into a system that really leverages our editorial time and gets us an opportunity to distribute to all these different places.
So interestingly, part of the story is that I pitched the idea with my boss at the time Robert Holt, who’s now at Microsoft. We pitched the idea to actually spend a bunch of time collapsing these systems and building an awesome system that takes care of all of this. At the time the VP, her name was MJ Bear, she wanted us to do due diligence and explore all these different avenues, content management systems like TeamSite and vignette, which are all quite expensive. So we did all that due diligence but I wasn’t convinced that that was the right route. And in fact, I knew we could do a better job by building it in house. The inflection point was that she quit. She left the job and, I basically said, “Great, let’s go build it. There’s no impediment anymore before things settle, let’s get this built before the next VP.” And, so we did that. We hunkered down and that’s where we really started thinking about the philosophies of “separate content from display” and, and “content modularity” and things like that.
So this was back in 2002 and it was partially driven by the idea that everything we’re doing here of collapsing these data systems, we’re also doing a redesign of the presentation layer. If we’re doing that, it’s highly likely that in the future, we’re going to have another presentation layer, either a new one to replace the old one or an additional one. And it’s almost like this keeps happening. It’s cyclical.
We need something great. so let’s throw out the old, and so we said, “all right, this presentation layer is going to go away too. So we really need a decoupling. And that’s where a lot of these COPE philosophies started to soak in. And actually we launched that CMS in November, 2002 and it’s still the centerpiece of NPR today.
Jeff Eaton: Holy cow.
Daniel Jacobson: Yeah, the CMS has lasted 10 or 11 years and we see that as a kind of success. So we take a lot of pride in that.
Jeff Eaton: It seems like that that decoupling that you talked about from a pure software development and architecture standpoint, that feels like the right thing to do, but you know what you mentioned the idea of teasing apart the core business assets that are content that gets created and managed over time from the changing sort of ephemeral presentation, teasing those things apart. It feels like it makes good business sense too. It’s not just about architectural purity. It’s a way to make sure that you don’t have to dig up the foundation every time the house needs to get painted.
Daniel Jacobson: Absolutely. And I will say it’s not without some costs because there were certainly some cultural battles that went into those discussions. An example that I can offer is when we were doing this design, what we intentionally said is that the story is the atom of the system. That was another philosophy of what we were doing. And so the atom the center of the NPR system universe. And we basically said the story is super thin. And generally speaking in NPR, people think of the story as being a radio piece. And we said, “no, radio pieces are not a necessary component of a story, it’s an enhancing and enriching part.” But so are images. So is text, so are pull quotes. So is whatever else you want to imagine that you want to put in there, but they’re not necessarily. The only parts that are necessary are basically the title, a unique ID, a date. And I think we said a teaser, that’s it. That’s all you need for a story.
Jeff Eaton: Interesting.
Daniel Jacobson: And from there, there’s a hierarchical data model. Tell me if I’m getting too geeky.
Jeff Eaton: Oh, this is the kind of stuff I love.
Daniel Jacobson: So we had hierchical data model where we basically had a story own what we called a “resource” and a resource was any enhancing component of a story. And a resource is generic. And then the sub hierarchy underneath that were the particulars, which would be text, images, videos, external links, internal references…
Jeff Eaton: transcripts, I think just went live a little while ago.
Daniel Jacobson: Transcripts is another one for sure. Those were all enhancing enriching components, but not necessary to have a story on the site. And I know that you and others on this podcast have talked about blobs and chunks. So, we, we tried to be very surgical with this idea, we wanted to have everything be its own entity. We don’t want these blobs of data. In fact, every paragraph in the text are stored separately and very distinct fields. We really thought about how we can manage this stuff in modular ways so, as you were saying, we’re teasing it out so that our business can gain value down the road.
And the controversial part is, especially in 2002, NPR is a journalism company focused on broadcast. Saying that the radio or the audio piece is not essential was pretty controversial. And it takes a little, it took a little massaging inside to get people to understand the power of this model. And it took actually a number of years to really get everybody on board.
Jeff Eaton: I would imagine that some of the decisions that you made earlier, like the story is central, even if every single story has a radio component for the first several years, there’s a stake in the ground, just in the nomenclature that you’ve been able to get in there.
Daniel Jacobson: Sure. you know, we really talked about stories and lists. Those were the two core concepts and you start talking about it and eventually that becomes part of the vernacular and part of the culture. And really, every story can belong to any number of lists, any number of types of lists. Lists are really just aggregating mechanisms. And that cut against the culture as well. And NPR, because you think about morning edition for March 13th, you think there’s a rundown, there are 15 stories for that day. That’s really just another list. And it’s not really a program. It is, it happens to be that this list’s name is Morning Edition…
Jeff Eaton: It’s a particular branded aggregation of stories, not necessarily this thing that is the primary means that people approach and find content, right?
Daniel Jacobson: It’s not a radio program anymore. It’s a list, and you know, you can identify it by titles. But you gotta think about it more generically and that’s when we started introducing a broader set of topics, such as “News”, “Music”, and things like that. And it really created, tremendous opportunity for NPR actually.
Jeff Eaton: There’ve been a couple of things that have been written about the actual longterm impact of it and what it’s allowed NPR to do in terms of turning around new products that are targeted at emerging devices or platforms without having to go through the same sort of just profound, painful rearchitecting of everything that a lot of other companies are having to do. Do you think that it’s interesting, do you think that this set up NPR for taking advantage of mobile, because of all the work that had already been in place.
Daniel Jacobson: Unquestionably, yes. Following the story of the content management system creation, I’ve talked a bunch about not building a web publishing tool. So, we felt like we were building a content management system and we thought about it in terms of having the content management editing tool be just another their presentation layer. And you can have any number of presentation layers. And so that actually gave a lot of freedom. If the editing tool is just one presentation layer and the website is one, we can have any number of these layers. We can just tack on new presentation layers, some of which have write capabilities and some don’t.
So as that started to mature a little bit back in 2004, RSS started to emerge. It was easy just to spin up another PHP file and have it render a RSS feeds and you pass in parameters and it’s going to yield a topic or a program. And then shortly after that, podcasting started to emerge and we can just easily float an MP3 file in the RSS feeds.
And, it was about 2007 when the NPR music site launched. We started to see the fact that we had this single point of failure, an Oracle database, which we were growing out of. We actually need a different data redundancy model. We needed a cluster of databases to be able to scale it. And NPR was a nonprofit, so we couldn’t afford a million Oracle’s servers. So, there were a couple of things that had me thinking that we needed another level of abstraction. And that’s when we introduced the API. That gave us an opportunity to basically put the NPR music development on a different trajectory, as well.
Now that everything is, or at least was, we were moving in the direction of having our sites fueled from the API. We can more easily abstract away the database underneath the API and swap in a cluster of MySQL servers, multiple databases. And, so we started thinking of the API in those terms. And then sometime after that, we opened it up publicly.
After we opened it up, we realized, “wow, we should be using that for other presentation layers like mobile sites and iPhones and iPads” and feeding into the Livio radio, radio devices that are feeding NPR content. Now it’s going into cars. So, all of this strategy, which started early in 2002, I think was in retrospect, we were lucky with having that kind of architecture.
It put us in a great position to just tack on more presentation layers and allow them all to feed off of one central distribution channel, which was the API.
Jeff Eaton: No, it definitely makes sense. I mean, although I think it is interesting that the real rise of mobile web traffic, especially apps, this idea is a given.
Businesses that have content might need this cluster of different ways that people can get to their stuff. I think it feels like that’s broken and a lot of existing sites and a lot of existing workflows and a lot of existing platforms that companies have built out. And I think that sort of corresponds to where the huge spike in interest in the work that’s been done on COPE, really took off because suddenly everyone was feeling just such a tremendous amount of pain around that issue.
What you’re describing is interesting because it’s not the same kind of story your team had been working on this for a long time based on deeper needs than simply we need to go mobile or something like that.
Daniel Jacobson: Yeah, that’s right. I think we were lucky to have had a series of challenges, like financial challenges, not being able to afford vignette. We were lucky and that we were doing a series of things all at once. And we knew that we were redesigning and we were likely to have to redesign again. There’s a confluence of things that got us thinking early about it. We had no idea the iPhone was going to go nuts. I mean, if we had that kind of foresight, I should have been betting on the stock.
But we were very lucky and I think we actually made a series of decent decisions thereafter that put us in a good position. When I hear about folks who have been, who weren’t quite as lucky to have that confluence of events and they have to go back and retrofit. Well, now the space is much more complicated and you’re already embedded in your web publishing tool. So their rearchitecture at that stage is actually much more expensive and much more painful. It just seems like we did it at the right time. Actually, things were very lucky.
Jeff Eaton: You mentioned that the actual editorial tools, the things that the actual content creation people use is just another presentation layer in that sort of approach. How did that side of things evolve? Cause you know, the structured content approach that you’re describing isn’t necessarily a natural fit or a natural transition for people who aren’t used to saying data modeling is their weekend hobby.
Daniel Jacobson: Data modeling is really at the center of it all. I felt it was critically essential to start with a very clean data model. That that was the starting point. How do we, how do we imagine this content to be stored and thinking about it in those very teased out ways? So yeah. Text is very different than a teaser, which is very different than images.
And everything was really isolated. That’s where the content modularity part comes in. So we, that’s where we started. And if you have that well, that’s just your data repository. And then any number of apps can hit against that. And we had the website hitting against that, but we also had this suite of tools that could write to it. They can also access it, but then we started, over time experimenting with areas of the website, writing different user related things to the database. Basically, it starts with the database. That’s all you have. And then every, everything else is either reading or writing or both to that database. And we just thought about it in those terms.
Jeff Eaton: Was the user experience aspects of building out those tools, something that you were involved with or was that something that there was a different team of people trying to make the tools usable for this way of approaching things.
Daniel Jacobson: Okay. So I’m actually really glad you brought that up because I kind of glossed over that. First I want to be clear and maybe this kind of gives people some hope. When you say the team of people, the total team that I can think of that executed on this entire CMS project back in 2002, it was about four people.
Jeff Eaton: Oh, this is like learning how bacon gets made.
Daniel Jacobson: Yeah. I hope I don’t disappoint anybody.
Jeff Eaton: I think that’ll encourage a lot of people actually.
Daniel Jacobson: Yeah. I mean, it really started with this massive document that I put together that we used for due diligence to send out the vendors per the VPs request.
So we had a vision that we kind of pieced together and that vision, both encompassed what the data structure’s going to look like, what’s it going to look like in terms of conceptually the CMS or the content capture part as well as, directions for the website. We obviously then brought it in house.
We didn’t have a VP anymore. It was me and one other backend developer at the time. And there was one front end developer and there was a designer. And then my boss, so that was the team. We had a suite of editorial folks who contributed meaningfully to this. I don’t want to discount them, but in terms of the engineering, that was it.
So me and the other backend engineer, we wrote all the tools around the content management system. But before we did that, we were heavily informed by the designer. And I was pretty involved in this as well. We did a series of usability tests. We took data from both the usage patterns of our users online and the npr.org users. We took information about the three discreet ways that we publish. We had the Cold Fusion system, that Java system, the HTML files.
What are the kinds of things we’re building? We knew that actually, in the mode right now, at that moment too, we had a very limited sets of assets. It was very audio focused, but we knew that things coming down the road were going to be much more expansive in terms of the available assets: images were probably to come later, and full text. And we were hiring editors too on the online side specifically to build out those stories.
So we were informed by all of those things and did a whole series of mockups and clickable prototypes for the editors and sat down with them and said, “okay, well, what do you think of this? What do you think of that?” And then here’s the interesting part. We took all that data and we had to discount fair amount of it because we thought that they were still thinking about it too much like NPR, right?
Jeff Eaton: Yeah. They were thinking, “Oh, you’re building the new version of what we’re used to.”
Daniel Jacobson: Yes. Right. So, we took on all of that. There were a lot of really great ideas and fundamental things that drove the direction of where we’re going. But again, we had to think, we’re thinking bigger than this. We’re thinking there will be another design, there’s this and that.
So, we needed to discount a portion of our learnings. And, yeah, so all of that kind of boiled into the content management system. And I think you and Karen have talked about this fair amount, that you can’t just build tools as an engineer would build a tool, right? You’re building your website to have it be meaningfully useful to the website user.
You need the same mindset when you’re building the CMS. You want it to be infused with the sensibilities that will make them effective at what they’re doing. So we tried to take all those sensibilities and make something that they would succeed with, and evolve it over time.
Jeff Eaton: Especially with sites that have an existing infrastructure and stuff like that. It’s very easy to see those kinds of things. And imagine it’s just insurmountable, there’s definitely a lot of hurdles, especially, it feels like every year that passes there’s more weight that’s being put on a lot of the web properties, the different companies, but the idea is it doesn’t necessarily take an army to do this. It seems like it’s more of a wheel inside of the organization to take this kind of path.
Daniel Jacobson: Yeah. I have a couple of thoughts on that. First, generally I agree. I think the one area I would disagree is that the world is different now than 10 years ago. And I think, again, we were lucky to have gone down that route at the time that we did, because like you said, the weight is lighter in 2002 than in 2013. But in every other capacity, I agree. And you need to have commitment, and the commitment not only starts with the commitment to vision, but resourcing it appropriately. If you need these engineers to build this out, hire engineers, and I’m a big believer in the idea that excellent engineers are going to be, some people say, 10 times more effective than average engineers. So having really smart people who are able to execute on these things, you can really tease out what you’re going for. What’s the best way to execute on it? That’s going to pay way more dividends than just hiring a bunch of people and throwing money at it.
Jeff Eaton: What you were describing earlier was it was a very cross functional team, all working on these things. You were talking about working in close conjunction with the designers who are planning out what the new visual appearance of the site was going to be in how it was going to operate, working with the the content creators and the editorial team.
It wasn’t just a matter of wireframes that were thrown over the wall and you guys implemented it, which I think is one of the hardest scenarios for a lot of people who build an implement to find themselves in these days.
Daniel Jacobson: Yeah. It was kind of a scrum before scrum started taking off. I mean, that’s basically what it was. The only way it could have succeeded was very close collaboration with everybody on board with the message.
Jeff Eaton: So I guess coming back to that initial question, is it a little weird now to hear the stuff that you worked on turned into sort of the GoTo example slide in everyone’s presentations about structured content and reuse?
Daniel Jacobson: Yeah. It still freaks me out. It’s great. I love seeing it. I love talking to people about it and if there’s a way that I can help people, I love doing that. It is still a little surreal too, however, as I haven’t even been at NPR for the last two and a half years. And, so it’s interesting to still see it coming through the tweet stream once in a while.
Jeff Eaton: I can imagine. Well, you mentioned that you’re now at Netflix actually, and you’re the director of engineering for the Netflix API. How is that different? I mean, it seems like it’s an API and it deals with content, but it is really a big shift.
Daniel Jacobson: Yeah. It’s a huge shift in certain categories and it’s actually quite similar in others. The similarities are, we’re both media companies. But actually Netflix considers ourselves technology, company in media. I think NPR should be striving, if they are not already, for the same thing, as a technology company producing content for distribution, trying to get on multiple devices, reaching consumers with rich multimedia experiences.
Those kinds of things are similar. The scale. is fundamentally different. NPR, I think is a reasonably decent scale operation. but again, by the time I left it, my team was, including some contractors, around 20, I think. Here the engineering team is about 600.
The scale of the APIs: I think the NPR API is however many millions of requests a day. Maybe it’s a hundred, maybe it’s 50. I don’t remember the exact number. The API here does two and a half billion transactions a day. So what goes into those problems, you know, solving those problems? It’s a fundamentally different approach. And so contextually at NPR, it was even when I left, it was basically one team broken down into different groups, but focused on one pipeline and that pipeline was pretty interconnected. So, you have the content management system that publishes into a cluster of databases and the cluster of databases draw from an API, and the API distributes out to any number of destinations. The NPR engineering team was building pretty much all of that.
Jeff Eaton: Wow. 20 people or so!
Daniel Jacobson: Here it’s highly distributed. It’s an SOA model. lots of engineering teams focusing on specialized tasks. And my team does not really store any data. We don’t really have any editorial tools or anything like that. We’re basically a broker that takes data from other people’s systems and pass it across HTTP over to devices and people. So the core responsibilities for this team is making sure that we have a solid distribution pipe, scaling the system effectively with the growth of the company and growth of the system and ensuring resiliency. Those are the three key responsibilities I played out for the team. Whereas NPR, it’s building a lot of features and presentation layers, or managing a CMS.
So yeah, the scale I think is really at the core fundamentally different, and that drives a lot of the differences in other categories.
Jeff Eaton: Yeah. I think, at Netflix, you guys are responsible for what I think a double digit percentage of evening internet traffic or something, something like that.
Daniel Jacobson: Yeah. It’s 33%, right?
Jeff Eaton: That’s definitely a statistic. Not many people could claim.
Daniel Jacobson: I mean, there are all kinds of different ways that Netflix has massive scale. That’s one of them. The two and a half billion transactions a day, but we’re also an 800 different device types. It’s kind of mind boggling. When you think about some of these numbers.
Jeff Eaton: I am just blown away that there are that many kinds of devices to be on. I mean, I guess it makes sense, but it’s just, it is staggering.
When you think about that the device proliferation that we’re seeing, is it really difficult to keep up with it?
Daniel Jacobson: Yeah. And the scary part is it’s not done. Your fridge is going to have a screen on it. Why not have Netflix there? Right. Basically this is the beginning of it, actually, in my view.
Jeff Eaton: So how has it been, how has it been different? Just philosophically, like at NPR? I think a lot of the case for the API is about very public distribution of a lot of different information. But with Netflix, it’s different, it’s basically you’re API is part of a tool chain that allows you to provide a particular service to customers.
Are there ripple effect differences in how the two get approached?
Daniel Jacobson: I actually think you’ve mischaracterized NPR a little bit. Even by the time I left, I was starting to position it differently and I think that’s still the case. So, Javaun Moradi, he’s the PM there now for the API and the rest of the folks there. I think they focus intently on internal consumption. They still have the public API and still use it as a distribution mechanism to the member stations, among other things. But an overwhelming percentage of the traffic to the API is from NPR properties. It’s not from the general public domain. and then the next category would be the stations, and then the general public.
So I think in that regard, it’s very similar to Netflix. now that the percentages and the numbers are very different. So I think whatever their percentage is, 60, 70% at NPR. For Netflix, 99.9 plus percent of the traffic is from Netflix-ready devices. And we still have a public API, but that’s sees an incredibly small percentage of the traffic
Jeff Eaton: Then, I guess that is sort of a perception thing then. I mean, I’ve always heard a lot about the context of the public open API being there. And that’s actually a question that I think a lot of people have as well, “if my business doesn’t make sense for putting out a giant fire hose of all of my stuff to the world, how can I really leverage this stuff?” And I think that that’s part of it. You don’t have to think about it as just the all you can eat buffet of our stuff. It can be used for internal purposes too.
Daniel Jacobson: Yeah. I’d go a step further and I’d say it’s the majority. In fact, the overwhelming majority of value from API use will be the internal consumption. and this is from my background at NPR and at Netflix, but also in talking to a lot of people in the space.
I see other companies like Evernote and The Guardian and The New York Times. They all have similar pie charts where the overwhelming consumption is internal. So I actually think that we are seeing a shift in the marketplace towards internal consumption. People are looking at their businesses differently.
It’s like, how can we get on all these different devices? Let’s not worry as much about trying to piggyback on developers in their free time in their garage. And let’s dedicate our resources to building these things on our own. How do we get there? Let’s build an API so we can leverage the distribution rather than building one-offs all the time for either individuals, developers who are looking at dealing with the massive explosion of devices and channels.
Jeff Eaton: What would, what kind of advice would you have for them? What kind of major pitfalls would you tell them to steer clear of?
Daniel Jacobson: Well, definitely steer clear of lots of one-off development and steer clear of thinking, depending on who you are and what your business is, steer clear thinking of yourself as not being a software company.
I think that’s, to me, that’s the number one thing. If you can re-imagine whatever your business is and think of yourself as being a technology company, or at least partially a technology company, then you’re going to dedicate the resources to that. And if you’re dedicating resources to that, then you’re gonna have smart people who are thinking about these problems in the right way.
And I don’t think there’s a one size fits all approach for everybody, but I think if you have good people thinking about it, you’re going to end up with highly leverageable content management or distribution channels. You’re going to end up being much more nimble than you were otherwise.
So it’s probably sidestepping your question, but, I don’t know how else to say it because even at Netflix, we have very clearly stepped away from trying to be one size fits all for reaching all of the platforms we’re hitting.
Our REST API used to be a one size fits all model that we use for quite a while, and it felt like the right thing to do. But my view on that is that the API is a tool and when that tool runs its course, we need to move on to something that has greater pragmatic value. So I wouldn’t be beholden to any given technology. I’d be beholden to smart technologists who you trust to make good decisions.
Jeff Eaton: And it sounds like an important part of that is having a really clear coherent grasp of what it is that you’re trying to accomplish and what the longterm goals are too.
Daniel Jacobson: Absolutely. Yep. That goes right in with the commitment. If you’re committed to this, then think about how this is going to play out in five years and start planning for that.
Jeff Eaton: Well, I want to say thank you very much for joining us. it’s been a pleasure and, I hope that we’ll cross paths again in the future.
Daniel Jacobson: Absolutely. Thanks a lot, Jeff. I really appreciate it. And yeah, definitely.
Jeff Eaton: Thanks for listening to insert content here. If you’d like to catch up on our archives or keep up on our new episodes, visit us at dot com slash ideas slash podcasts slash insert content queue.
And you can also visit us directly at InsertContentHere.com.
Extra Story: Daniel Jacobson: Me and another NPR engineer, when we were doing a modeling exercise, we really got caught up on what to call the story, the atom of the content management system, in the database. And we debated about this for way too long. And her stance was let’s call it “page”, as in a web page or some page representation. And I wanted nothing to do with that because I thought that was too bound to a given presentation layer concept. I really wanted something way more generic. So I started throwing out ideas like “object” or something that really didn’t have a whole lot of meaning, but was abstract.
Jeff Eaton: Totally abstract.
Daniel Jacobson: Exactly. Yeah. And we went around this debate time and time again, and ultimately what we decided on was “thing”. So the central table in the system, and I think it still is the case today is called “thing”. We did that specifically so that we could not be worried about if a story going to end up on a mobile app or on an IP radio, which don’t really have a concept of “page”. It’s just, here’s this “thing”, we’re distributing it out and that’s it.
Comments Off on Presentation : API Revolutions – Netflix’s API Redesign Revolutions have a common pattern in technology and this is no different for the API space. This presentation discusses that pattern and goes through various API revolutions. It also uses Netflix as an example of how some revolutions evolved and where things may be headed.
Comments Off on Presentation : Set Your Content Free! : Case Studies from Netflix and NPR On February 8, 2013, I spoke at the Intelligent Content Conference 2013. When Scott Abel (aka The Content Wrangler) first contacted me to speak at the event, he asked me to speak about my content management and distribution experiences from both NPR and Netflix. The two experiences seemed to him to be an interesting blend for the conference. These are the slides from that presentation.
I have applied comments to every slide in this presentation to include the context that I otherwise provided verbally during the talk.
Comments Off on ProgrammableWeb : Why REST Keeps Me Up At Night This post first appeared on ProgrammableWeb.com
With respect to Web APIs, the industry has clearly and emphatically landed on REST as the standard way to implement these services. And for good reason… REST, which is generally implemented as a one-size-fits-all solution, is an excellent choice for a most companies who wish to expose their content to third parties, mobile app developers, partners, internal teams, etc. There are many tomes about what REST is and how best to implement it, so I won’t go into detail here. But if I were to sum up the value proposition to these companies of the traditional REST solution, I would describe it as:
REST APIs are excellent at handling requests in a generic way, establishing a set of rules that allow a large number of known and unknown developers to easily consume the services that the API offers.
In this model, everyone knows how to behave and it can be incredibly powerful. The API providers establish a set of rules and the API consumers must adhere to those rules to get what they want from the API. It is perfect, right? In many cases, the answer is obviously yes. But in other cases, as our world scales and the number of ways for people to consume digital content and services continues to expand, this one-size-fits-all model is likely to fall short.
The potential shortcomings surface because this model assumes that a key goal of these APIs is to serve a large number of known and unknown developers. The more I talk to people about APIs, however, the clearer it is that public APIs are waning in popularity and business opportunity and that the internal use case is the wave of the future. There are books, articles and case studiescropping up almost daily supporting this view. And while my company, Netflix, may be an outlier because of the scale in which we operate, I believe that we are an interesting model of how things are evolving.
Netflix is currently available on over 800 different device types, including game consoles, mobile phones, TVs, Blu-ray players, tablets, computers, and almost any other device that can stream video. Our API alone handles more than two billion incoming requests on peak days, which translates into almost ten billion real-time outgoing requests from the API to internal dependency services. These numbers are up by about 70x from just two years ago. Most companies do not have that kind of scale, but it is clear that with the continued growth of the device market more companies are resetting their strategies to be less about the public API and more about internal consumption of their own APIs to support device proliferation. When this transition occurs, the API is no longer targeting “a large number of known and unknown developers.” Rather, the key audience is a small number of known developers.
The potential conflict between the internal and public use cases is in the design of the API itself. Keep in mind that the design implications will not be problematic in many scenarios. It becomes a potential problem if the breadth of devices becomes so wide that the variability of features across them becomes substantially harder to manage. It is the breadth of devices that creates a problem for the one-size-fits-all API solutions.
If your target is a small group of teams with whom you have close relationships, the dynamics around the API change. For Netflix, we persisted on the one-size-fits-all REST model for quite a while as more and more devices got added on top of the API. But given our scale, one thing has become increasingly obvious. Our REST API, while very capable of handling the requests from our devices in a generic way, is optimized for none of them. This is the case because our REST API focuses on resources that are meant to be granular representations of the data, from the perspective of the data. The granularity is exactly what allows the API to support a large number of known and unknown developers. Because it sets the rules for how to interface with the data, it also forces all of the developers to adhere to those rules. That means that each device potentially has to work a little harder (or sometimes a lot harder) to get the data needed to create great user experiences because devices are different from each other.
The differences across these devices can be varied and sometimes significant. Here are some examples of variances across devices that may be challenging for one-size-fits-all models:
Just think about the differences between an iPhone and your TV and how they beg for different user experiences. Moreover, the XBox and the Wii, both of which project to the TV, are different in the way users interact with them as well as in the hardware constraints, both of which may require different APIs to support them. When considering more than 800 different device types, the variance across them becomes overwhelming. And as more manufacturers continue to innovate on these devices, the variance may only broaden.
If these ingredients are met, then you have the recipe for needing a new kind of API.
To solve this issue, it is becoming increasingly common for companies (including Netflix) to think about the interaction model in a different way. Rather than having the API create a set of rather rigid rules and forcing the various devices to follow them, companies are now thinking about ways to let the UI have more control in dictating what is needed from a service in support of their needs. Some are creating custom REST-based APIs to support a specific device or category of devices. Others are thinking about greater granularity in REST resources with more batching of calls. Some are creating orchestration layers, such as ql.io, in their API system to customize the interaction. These are all smart and practical ways around the problem. But with the growing number of devices, the increasing urge for companies to be on as many of them as possible, and the desire for continued innovation across these devices, these various solutions are still somewhat restricted. They are still forcing the developers to adhere to server-side rules and non-optimized payloads in an effort to have a one-size-fits-all solution. These approaches are closer to the flexibility needed in that they are not as rigid as the typical REST-based solution, but when supporting as many devices as Netflix does, we believe they fall short for us.
For Netflix, our goal is to take advantage of the differences of these devices rather than treating them generically. As a result, we are handing over the creation of the rules to the developers who consume the API rather than forcing them to adhere to a generic set of rules applied by the API team. In other words, we have created a platform for API development.
Panorama Theme by 
Themocracy
