Comments Off on Forbes : Why APIs Now Mean Business Transformation, Not Just Technology Infrastructure

By daniel_jacobson, March 23, 2016 12:12 pm

This article was originally written by Dan Woods and published on Forbes on March 23, 2016

Thirty-six years ago, my first consulting project was fixing an IBM 360 assembler language program that had broken because the behavior of one of the machine instructions had change subtly.  At that time, you could consider the definition of the IBM 360 assembler language an API to the hardware.

In the years that passed, the idea of the API has changed. We’ve moved from LPA libs, to JCL, to Microsoft DLLs, Java packages and Python libraries. The strict definition of the term “application programming interface” suggests that the API is a tool of abstraction and simplification. The application developer doesn’t want to want to have to worry about the details of how to send a text message from a mobile app or how to run an analytics routine in R or how to display a graphic using the D3 libraries.

Where is your API journey taking you?

Because of this legacy, APIs are most often seen as something deeply technical, and this impression lingers to this day. But as Google and Yahoo rose out of the wreckage of the dot com bust, as Facebook and Twitter were born, the technological essence of APIs is giving way to a more business-focused emphasis on enabling unfettered innovation.

The APIs of Google, Yahoo, Facebook, and Twitter in the mid and late 2000s were unusual in that they were public. These APIs did have a technological purpose. They abstracted the basic capabilities of publishers so that they became useful for developers. But the point and the value was unrestricted innovation. SAP had APIs decades before called BAPIs that allowed their massive ERP system to be controlled. The trick that the Internet darlings performed was to figure out that by making APIs public, by allowing self-service, a seething tribe of creative developers could put those APIs to use creating new ways to put the core services to work. John Musser, the founder of Programmable Web, was the prophet of this era.

For example, Google Maps was one of the first public APIs to be created, inspired in part because developers figured out how to use Google’s capabilities on their own in an unauthorized manner. Google quickly realized that this was a great idea, published APIs, and a flood of innovation and use of Google Maps followed. Yahoo, Twitter, Facebook, and Amazon followed the same pattern for many of their services.

So in this era, John Musser created a catalog of public APIs and Chet Kapoor of Apigee and other companies like Mashery, now part of Tibco, and others created products to allow everyone to get in the game of creating public APIs. Working with the CTO of Apigee, Greg Brail, and Daniel Jacobson, VP of Edge Engineering at Netflix, I co-authored APIs: A Strategy Guide (O’Reilly), a book about API strategy. (Disclosure: I have done work as an analyst and content marketer for Apigee, Intel, Tibco, Microsoft, and other companies who sell API technology. For a full list of my clients visit EvolvedMedia.com/clients.)

Forbes Daily: Join over 1 million Forbes Daily subscribers and get our best stories, exclusive reporting and essential analysis of the day’s news in your inbox every weekday.Sign Up

By signing up, you agree to receive this newsletter, other updates about Forbes and its affiliates’ offerings, our Terms of Service (including resolving disputes on an individual basis via arbitration), and you acknowledge our Privacy Statement. Forbes is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

The point of that book was to explain the notion of APIs from top to bottom, but also to explain how APIs had become far more than just technology infrastructure.

How APIs Have Become the Building Blocks of Business Transformation

Daniel Jacobson’s evolution of APIs at Netflix shows how APIs can become building blocks for business transformation. Jacobson is in charge of creating the API infrastructure to support product development. Netflix, like many other companies, had a set of “resource-based” APIs that exposed the core capabilities of the key operational systems. Using these APIs, you could get to the catalog information on Netflix and invoke services to play movies.

Jacobson and his team determined that the code they were writing to create Netflix clients for various families of devices such as iPhones, iPads, Android tablets, DVD players, and so on was far more complex than it needed to be. The problem was that resource-based APIs were built to expose the capabilities of key systems in a general way. That mean that the client code had to do all sorts of stuff like transform and combine data, connect information from a multitude of sources, and create data structures and navigation that matched the needs of the device.

Jacobson’s team realized that a new layer of APIs was needed, which they called experience-based APIs (see “How A Netflix Tech Innovation Can Unleash Creativity in Your Business”). This layer of APIs was created not by the teams in charge of the key operational systems, which created the general purpose resource-based APIs. Rather, the developers in charge of creating clients for all the devices used to access Netflix defined just the APIs needed for each family of devices. The experience-based APIs moved all the code for transforming and formatting data and adapting the resource-based APIs to the needs of the device into the experience-based API. Jacobson’s team found that this process could be accelerated by allowing the experience-based APIs to be created using a scripting language. Jacobson and his team packaged this idea as the Nicobar open source project.

APIs as a Unit of Digital Business Design

FinTech and other industries have taken the general pattern of experience-based APIs and adapted it to different circumstances. Consider these examples:

• Tradier has created a Brokerage-as-a-Service offering that is a combination of resource- and experience-based APIs. Tradier customers are able to embed the ability to research and trade stocks within their existing applications.
• Orchard Platforms has created a platform powered by APIs that normalizes access to lending marketplaces. Using this platform, large financial institutions can make loans on platforms like Prosper and Lending Club using automated underwriting.
• Xignite has created a set of APIs that normalizes access to a vast amount of financial data. Some of these APIs are resource-based, providing general access, and some are experience-based, focused on meeting a specific need.
• Bechtel has introduced huge amounts of efficiencies by creating a platform as well as resource- and experience-based APIs to enable the creation of mobile apps. On job sites around the world, instant access to information has dramatically reduced delays because workers don’t have to go to sheds to access computers.

The point to remember is that all of these APIs are both technical artifacts and units of digital business design. As technology infrastructure, both resource-based and experience-based APIs need to be supported by generic API gateway capabilities such as:

• Proxy support
• Authentication/Authorization
• SSL/TLS termination
• Encryption
• Logging
• Load balancing
• Routing
• Throttling
• Lightweight orchestration

But there is another business-focused process going on here as well: the process of developing the unit of business design. In other words, resource-based and especially experience-based APIs are not designed simply as technology artifacts, but as ways to enable a business purpose. The people involved in designing the desired digital business experience must be involved.

Enterprise-grade API management – the kind that can power digital transformation and drive a business – goes beyond an API gateway and supports an ecosystem of digital collaboration. Many types of software follow this path. The software starts out supporting specific, targeted functionality but ends up adding features because people want to collaborate and need supporting capabilities as the original function becomes more important.

When you are using APIs as a unit of digital business design, you need capabilities such as:

• Analytics about developers, operational performance, app performance, and business metrics
• A customized developer portal
• API monetization
• Multi-tenancy and support for high scale
• Global policy enforcement
• SDKs for all popular development environments to simplify the process of developing an API
• Support for adding application code and data to the API infrastructure to enable distribution of code and data
• A powerful transformation engine to speed the task of adapting resource-based APIs to experience-based APIs

It is perfectly possible to use APIs as a unit of business design without an enterprise-grade API management platform. The challenge then becomes creating enterprise capabilities when you need them and supporting them over time.

The API marketplace is evolving rapidly. Amazon recently released an API gateway product. Microsoft has one as well, and there are various types of open source toolkits and such. The functionality of many of these vendors’ products supports the notion that APIs are simply technology infrastructure.

But vendors like Apigee would argue that to really use APIs as a unit of digital business design, to accelerate the creation of business value that comes from apps that can be developed faster and that cost less to maintain because of practices like experience-based design, you need much more than an API gateway.

“APIs have gone well beyond just bits of technology – they are essentially the foundation for digital transformation,” said Apigee’s Kapoor. “We believe that businesses will need to use APIs and API management to support digital business initiatives or risk becoming increasingly irrelevant.”

The challenge in my view is the legacy that APIs have as a unit of technology infrastructure. The people who are buying solutions are often thinking small about APIs rather than thinking big. The challenge for anyone using APIs is to stop thinking of them only as technology and to start thinking about the results you want to achieve and how APIs can play a role in getting there faster.

Dan Woods is on a mission to help people find the technology they need to succeed. Users of technology should visit CITO Research, a publication where early adopters find technology that matters. Vendors should visit Evolved Media for advice about how to find the right buyers. See list of Dan’s clients on this page.

Comments Off on Forbes : How A Netflix Tech Innovation Can Unleash Creativity in Your Business

By daniel_jacobson, February 11, 2015 5:59 pm

This article was originally published by Dan Woods on Forbes on February 11, 2015

In the tech community, Netflix is a company that everyone learns from. But what is less widely celebrated are the broader business lessons of what Netflix has accomplished. I was bonked over the head by this point when I sat through a presentation about Experience-based APIs by Daniel Jacobson, VP of Edge Engineering for Netflix.

Jacobson and his team that builds and maintains the Netflix API platform have developed a way of working that solves a problem that is common in any company that is building its own technology: How do you best leverage the skills of everyone on a team so you can move as fast as possible? Jacobson and his team have figured this out for a general use case: Building many different applications based on a complex set of centralized resources. The structure of this solution, which relies on the idea of separation of concerns between different groups of developers, can help optimize the structure of many different types of teams.

For those who work on building software, Jacobson has embedded in the Nicobar open source project the technical capabilities needed to support for the separation of concerns and rapid innovation. I will describe briefly the Nicobar project, which was released yesterday, at the end of this article.

What are Experience-based APIs?

To understand the beauty of Experience-based APIs, we first must understand the challenge facing Netflix with respect to supporting many devices. Netflix client developers write software applications that allow Netflix to deliver its content over thousands of different devices types. Netflix has many teams of client developers, each of which specialize in creating applications for different types of devices. The goal of Jacobson’s team and the client development teams is to deliver a great user experience and make sure that Netflix works perfectly on any device you might use, from an iPad, to an Android phone, to a DVD player or smart TV.

In the Resource-based API platform the work gets done this way:

• The central API team creates APIs that make it easy to get to all of the data and resources of a company. These APIs are all about creating a general way for programs to get access to customer data, data about films, recommendations, and so on. The resources that a client program may need include data and services for:
  • User info
  • Movie metadata
  • Movie Ratings
  • Similar movies
  • My list
• The central team implements the APIs to deliver scalability and excellent performance.
• The client team takes these APIs and creates applications for a variety of devices. The problem is that the code for handing errors and adapting the data to the needs of the device all resides in the client application code.

The Resource-based API paradigm works just fine and is used all over the world. But Jacobson and his team wanted to find a way to move faster, create better software, and enable more optimized experiences for Netflix subscribers. They realized that different devices potentially needed a different set of data to support the user experience. The best way to make the client software simpler would be to have a new API that allowed as much as possible of the data translation, error handing, and other utility functions to happen on the server, not on the client.

But if you asked the central API team to create versions of APIs for all the different families of devices, the team would be overwhelmed. In addition, it would be hard for each of the client teams to actually ask for the right design for the APIs they would need. It would require trial and error and that would be hard to accomplish with many client teams competing for the attention of the central team.

Jacobson and his team realized they needed to create a new layer of API they called an Experience-based API. The idea is that the client team should design the perfect API to serve the needs of the device on which the client application will run.

For example, when writing the application code to create a page on the client using the resource-based approach, the developer may have to call a dozen APIs, handle errors from all of them, and then format the data for use on the page. All of this communication takes place over HTTP, which is less efficient than the networking methods used on the server.

But if the client developers could create their own API to support the needs of that page, and move all of the code for data access from the client to the server, lots of good things would happen such as:

• Client code would get simpler.
• More code could be shared across families of devices.
• New device families could be supported faster in a way that best fit the needs of the device.
• Performance could improve because network communication is optimized, reducing round trips from the device to the server.

So Jacobson and his team set out to create a new layer to enable client developers to create their own APIs. Here’s how it works:

• Client developers analyze the application that they want to build for a particular client and then decide on the APIs that they will need to support them.
• They define the endpoints of the APIs and protocols to handle the requests and responses that are optimized for their client
• The APIs are implemented by the client developers using the Groovy scripting language which is used to call Java APIs that will gather the data needed to handle the request. Groovy is easier for client developers to work with than Java but still is scalable enough.

This architecture, in the case of Netflix powered by Nicobar, allows the client teams to move faster and reduce the technical debt by making the client programs simpler. The code enabled by Nicobar is more shareable and easier to maintain.

“There are some significant engineering benefits to this structure, but the real win is increasing the velocity of innovation,” said Jacobson. “Because Experience-based APIs properly separate concerns across the two teams, the work gets done faster. The client teams control their own velocity and the server teams can focus on building the platform that supports them.”

Lessons for the Rest of Us

Jacobson, Greg Brail, Chief Architect of Apigee, and I were co-authors of “APIs: A Strategy Guide”. During the time we wrote the book in 2010, the world was focused on the impact that public APIs were having. Google, Facebook, Twitter, and Amazon all harnessed massive amounts of attention and energy as people put their public APIs to work.

At that time Jacobson argued that internal use of APIs would have the far larger impact. He saw that few companies, even Netflix, would be able to get a large amount of value from public innovation. Instead, the real victory would come as internal APIs increased efficiency, productivity, and the velocity of innovation. Experience-based APIs are one major proof point for Jacobson’s way of thinking.

I believe this two-tier structure that allows those closest to the problem to translate the Resource-based APIs to a more suitable form and create APIs that are adapted to their needs can be used in all sorts of companies who how have bottlenecks in their IT departments.

To make this work you need a front office IT staff, one that works with the business. These are the equivalent of the client teams at Netflix. The back office IT provides Resource-based APIs to the front office teams who then use simple tools, scripting languages, model driven development, content management systems, and so on to adapt the resources to the needs of the business. In large Wall Street Firms it is common to have highly skilled front office IT teams. I think that many companies could benefit from this model if they choose the right tools and create the right separation of concerns.

Nicobar 

For companies who create -multiple applications and are committed to the Java ecosystem, Nicobar is definitely worth a look. This library provides the core functionality to implement the Experience-based API program. Right now the Groovy language is the way that the adapter code to assemble the data needed for the Experience-based is written, but in the future other JVM languages may be supported.

The key capabilities of Nicobar are:

• Rapid application delivery via dynamic updates of components
• Ability to add support for multiple JVM languages

To learn more go to Nicobar: Dynamic Scripting Library for Java.

And, for the truly nerdy, here’s what the project wiki says about why the project was called Nicobar:

“Nicobar refers to a remote, archipelagic island chain in the Eastern Indian Ocean. It is aUNESCO World Biosphere Reserve, and like many other remote volcanic islands, it has rich biodiversity, and various endemic species are found here. Taken together with the surrounding Andaman group of islands, six different human tribes occupy these islands, each with its own spoken language. We felt that these isolated islands, evolving independently are a good metaphor for the polyglot, modular runtime framework we were setting out to build.”

Dan Woods is on a mission to help people find the technology they need to succeed. Users of technology should visit CITO Research, a publication where early adopters find technology that matters. Vendors should visit Evolved Media for advice about how to find the right buyers. See list of Dan’s clients on this page.

Comments Off on Forbes : Explaining the API Revolution to your CEO

By daniel_jacobson, February 12, 2012 12:01 pm

This article first appeared on Forbes on February 12, 2012

APIs: A Strategy Guide

For most of the past year, I have worked with two brilliant experts on APIs, Daniel Jacobson, at Netflix and Greg Brail, CTO of Apigee, to create a book that clearly explains the value of APIs. In researching the book, APIs: A Strategy Guide, we talked to dozens of other smart people who had led the creation of APIs for both internal and external use.

One of the most striking findings was how often API programs were started in secret, nurtured by the true believers in a clandestine way, slipped into production, and then brought to the awareness of senior management after the API was shown to be a success.

This pattern is understandable but ridiculous. Our book is an attempt to obviate this pattern by providing a top-to-bottom reference for people who want to understand the business value of APIs. But we are not naive. We know that the most likely people to read our book are also true believers and innovators who are already open to new ideas. The trick to accelerating the adoption of APIs and reaping the massive value they can create, is to convince skeptics. At some point, that means convincing the CEO, who, if sold on the idea, can bring all the resources of the organization to bear. APIs have been proven over and over to be a transformative force. It is time for technology leaders to force the issue. So, get yourself a meeting with your CEO and have a conversation along the following lines.

The Business Basics of APIs

When a CEO looks at the calendar entry for your meeting, he or she will likely think, “My god, this seems a bit technical for me. Shouldn’t the CIO or CTO be handling this?” So your first job is to make sure that the CEO understands that APIs are huge channels for business. Start the conversation by pointing out these facts:

APIs are not experimental: More than half of all the traffic to major companies like Twitter and eBay come through APIs

• APIs are channel to new customers and markets: APIs used externally unlock the power of partners to use business assets to extend the reach of your products or services to customers and markets you may not easily reach.
• APIs can be private: Much of the talk about APIs emphasizes their public use. Internal APIs should be part of every companies IT strategy.
• APIs promote innovation: Through an API, people who are passionate about a problem can solve it on their own.
• APIs are a better way to organize IT: APIs used internally can accelerate innovation by allowing everyone in a company to use each others assets without having to wait around for permission.
• APIs are not only for huge companies: The technology is standardized and able to be used by companies of all sizes.
• APIs create a path to lots of Apps: Apps are powered by APIs. When developers are motivated, they can use APIs and combinations of APIs to create new experiences for end users.
• APIs to create lots of apps that can lead to lots of customers: Apps are going to be a crucial channel in the next 10 years. There will be trillions of apps in the next decade vs.a billion web browsers in the last.

Then point the CTO to the post by Google engineer and Amazon alumnus Steve Yegge, that points out that APIs are so important to Amazon CEO Jeff Bezos that he threatened to fire anyone who didn’t expose their assets through APIs.

APIs from Twitter, Amazon, Google, and Facebook have been used to create thousands of applications. These victories are being followed by APIs from AT&T, Sears, E*Trade, Alcatel-Lucent, Accuweather, and hundreds of other companies. Point the CEO to the research from John Musser, founder of ProgrammableWeb.com, a site that tracks the growth of APIs.

At this point if your CEO isn’t interested, then there’s not much you can do.

However, the next question your CEO will likely have is, “What is an API?”

Drawing this simple graphic is a good way to start:

API Value Chain

Explain that APIs are a simple way to provide access to some type of business assets. The business asset can be information itself, information about a product or service, or direct access to the product or service.

Point out that to make an API successful, everyone in the value chain must benefit. Make sure you convey you are not suggesting that if you build an API, the world will come rushing to use it.

The value chain takes many forms. The organization that owns the business asset may or may not be the same as the organization that builds the APIs. Different people or organizations may build, distribute, and market the applications. At the end of the chain are end users who gets the benefit of the business asset. Often, many APIs are used to create a new experience for end users.

Finding your API Value Proposition

If you are getting anywhere with your conversation, your CEO will likely get impatient with the technology and want to know the payoff for your company. He or she may say, “Great, APIs are a big deal, but what does that have to do with us?”

Here’s where you need to come armed with some sort of experiment that you want to get support for. One of the best ways to start is with an internal use of APIs. A key point of our book, one tirelessly championed by Daniel Jacobson, is that internal use of APIs is going to have the largest impact for most businesses. Remember, Amazon’s Jeff Bezos was going to fire people if they didn’t create internal APIs.

So, what are the business assets? Usually information that would benefit the company if more people had it in their hands. Look at the backlog of requests to IT. Is there an API that could take several items off the list? Have this in mind.

It is much better to build skills internally and master the design, development, testing, and operational processes before putting an API in public. That said, with careful planning and a clear value proposition, it is possible to have your first API be a public one, especially if you are following a well-established method of using APIs.

If you do start with an external public API, make sure you have a long testing cycle and flesh out all business and legal concerns before launching. I recently talked about API design and development with Byron Sebastion of Heroku, the Ruby platform for development that was purchased by Salesforce.com in 2010. He said that all of their APIs are first used extensively internally before going through a long public beta test. “When you launch an API, you want to make sure you are confident it is right, because people will rely on it,” Sebastian said.

At this point, I hope you are successful, because then you will need to answer questions about business models, design, engineering, operations, and marketing, all of which are covered in our book.

This article covers one way to go about educating a CEO about APIs. If you have other approaches, please send them along.

Also, we are going to be expanding the book several times this year. If you have a good example of how APIs have been successful for your organization, please reach out to me.

Dan Woods is CTO and editor of CITO Research, a firm focused on advancing the craft of technology leadership. He consults for many of the companies he writes about. For more stories about how CIOs and CTOs can grow visit CITOResearch.com.