Open APIs: An indispensable link to customers
When Neil Fantom, a manager at the World Bank, sat down with the organization's technology team in 2010 to talk about opening up the bank's data to the world at large, he encountered a bit of unfamiliar terminology. "At that time I didn't even know what 'API' meant," says Fantom.
As head of the World Bank's open data initiative, which was announced in April 2010, Fantom was in charge of taking the organization's vast trove of information, which previously had been available only by subscription, and making it available to anyone who wanted it. The method of doing that, he would learn, would be an application programming interface, or API.
The API would put thousands of economic indicators, including rainfall amounts, education levels and birth rates -- with some metrics going back 50 years -- at the disposal of developers to mix and match and present in any way that made sense to them. The hope was that this would advance the bank's mission of fighting poverty on a global scale by tapping the creativity of others. "There are many people outside the bank who can do things with the data set we never thought about," says Fantom.
One developer, for instance, created an app that married the bank's rainfall data to Google Maps to estimate how much rainwater could be collected on rooftops and subsequently used to water crops in different parts of the world. Another app provides facts about energy consumption and shows individuals what they can do to fight climate change.
Fantom and the World Bank aren't alone in such pursuits. A decade ago, open APIs were a novelty, but in the past few years they've been put to use at an accelerating rate. ProgrammableWeb, a website that tracks public APIs, listed more than 8,800 in early April. According to the site, it took eight years, from 2000 to 2008, for the number of APIs to reach 1,000, and then just another 18 months to hit 2,000. The jump from 7,000 to 8,000 took just three months.
The APIs cover a wide range of categories, including business, shopping, messaging, mapping, telephone, social, financial and government, according to ProgrammableWeb. They're becoming as necessary to an organization as a website. "In business today, an open API is more or less table stakes. It's something you have to have," says Stephen O'Grady, an analyst at RedMonk, an analysis firm that focuses on developers. "Increasingly, your traction is going to be driven by how open and how programmatically manipulable your product is."
An Evolving Model
When Best Buy first launched its API, BBYOpen, in 2009, it gave developers access only to the chain's products catalog, with descriptions and prices for all the items it had on sale, in the hopes that doing so would bring in more customers. That was part of a deliberate strategy to start slowly, says Steve Bendt, director of emerging platforms at Best Buy. "We had to prove these things over time," he says. "We started to prove out that this is a very vibrant and viable area to pursue."
But external developers wanted more, so the company added the ability to access reviews and ratings for products, find nearby stores, check whether certain products were available at particular stores, and purchase items through the website via mobile app, perhaps with a single click if the user had linked a credit card to the app.
It's been a hit. The mobile apps ShopSavvy, RedLaser and Milo all use BBYOpen. The makers of the app get a commission on sales through Best Buy's affiliate program. Shoppers can search for an item, or scan a bar code, and get information on pricing from various sellers.
Of course, that could mean that a customer using the app might wind up buying from a Best Buy competitor, but Bendt says that since websites and mobile apps have changed how people shop, what's important for Best Buy is to be in the mix. "If we're not in the consideration set, that's a missed opportunity." And the fact that the API makes it possible for people to find out if products they've purchased are available for pickup at nearby stores helps give Best Buy a competitive edge over online-only retailers, he says. "Now you can search for, buy and pick up within a matter or 20 to 40 minutes," says Bendt.
Tips for Creating Open APIs
Here's what you need to know about creating open APIs to your data:
Make it easy.Outside developers -- those at your customers' shops -- may have great ideas for how to use the data you make available, but the API itself needs to be understandable and easy to work with. Clear documentation and helpful tools are must-haves.
Make sure your licensing terms are clear and fair.Successful APIs tend to have MIT-style open-source software licenses.
Use REST unless you absolutely need SOAP.About three quarters of all APIs are REST-based, according to ProgrammableWeb, with SOAP a distant second.
Be prepared for cultural resistance.Some of the data "owners" may be reluctant to share the jewels. You might explain how the World Bank, Best Buy, Bloomberg and others have used the technique to reach customers in new ways and/or further their organization's mission.
- Neil Savage
The idea of an in-store pickup option actually came from external developers, Bendt says, and it took the chain some effort to adapt its legacy system to make inventory data available through the API; the data needed to be reformatted to be compatible. "The systems were built at a time before Web services and APIs were in active use," he explains. "It wasn't built in a way to expose it externally to the developer."
The specifics of how the team did that varied depending on the data source, but generally they tried to expose some snapshot of the data, updated as frequently as possible. If the data proved useful, they found ways to make it available in closer to real time.
Getting existing systems to work with the new API was also a challenge at the World Bank, says Malarvizhi Veerappan, the bank's open data systems lead. Her group originally struggled with latency issues because their 8,000 economic indicators were not all directly linked to each another. It was important, she says, to create a structure that could incorporate all that historical data and grow as new information accumulated.
We didn't want the API to be a separate application. We wanted it to be part of everything else we did with the data. Malarvizhi Veerappan, open data systems lead, World Bank
"We didn't want the API to be a separate application. We wanted it to be part of everything else we did with the data," she says. "We needed to connect it back to our data system. It did require our improving our internal data system."
As the API grew, the team added performance monitoring and instituted policies to ensure good traffic flow. The organization also increased server capacity and added server redundancy to ensure availability of the API.
When financial information provider Bloomberg LP launched its Open Market Data Initiative in February 2012, the new open API -- BLPAPI -- was actually Version 3 of the software development kit the company had already been using internally, says Bloomberg CTO Shawn Edwards. In the old days, Bloomberg customers were given a dedicated terminal that connected them to the company's mainframe, which delivered market data, news and analysis.
Bloomberg's project has since evolved into a software package that customers install on their own systems. Even before making it open, the company used the API to develop specific applications that allow customers to manipulate Bloomberg data on their own desktops.
We're not in the business of selling software. We're going to win their business by providing the best services and the best data. Shawn Edwards, CTO, Bloomberg
With the launch of its open API, the company is now allowing customers to create their own apps, such as watch lists for selected securities or their own trading systems. It also allows outside developers to create apps that draw on other data sources besides Bloomberg's. "We're not giving away market data. What this allows people to do is integrate with other services," Edwards says. "The API is a piece of software that connects to the Bloomberg cloud."
It makes sense to let others do the app development, he explains. "We're not in the business of selling software," he says. "We're going to win their business by providing the best services and the best data."
When Bloomberg put out the open API, it decided to remove some of the features that the previous versions supported. There was discussion as to whether the API should be backward-compatible. "We said no," Edwards says. That meant some customers wound up with features that no longer worked, but Edwards says it makes the API less cluttered with obsolete functions.
Like most open APIs, the BLPAPI supports a variety of languages, so a developer can choose the best one for his app. Someone running an overnight batch process might choose Perl, or the recently released Python version. An electronic trading system would probably run on C or C++. Quantitative analysts, or quants, generally use the data in Matlab. The API also supports Java, .Net and C#, and Edwards says some developers are using an R wrapper as well.
One key to making an API successful lies in making it easy to use. Back in 2000, RedMonk's O'Grady says, APIs often used Web services protocols, but those proved too complex. Now about three-quarters of all APIs are REST-based, according to ProgrammableWeb, with SOAP a distant second. "Because developers overwhelmingly preferred this, it's now the dominant protocol for API systems," O'Grady says.
The Importance of Clarity
Another important requirement is having extensive, clear documentation, and tools to help developers do their jobs. Bloomberg's initial documentation was aimed more at the financial experts who are its customers, but it had to be reworked to tell developers what they needed to know.
Bloomberg will soon attempt to make BLPAPI easier for developers to use by providing a replay tool that will allow them to perform trial runs of their apps. Best Buy's BBYOpen also gives developers a set of tools, including a test console to run apps and an automatic widget generator. The World Bank offers a query builder that lets developers select options.
Tools and ideas for APIs don't all flow outward from the organizations; external developers often provide information and frameworks to help one another out. BBYOpen, for instance, offers libraries created by developers in Java, .Net, PHP and other languages. At the World Bank, there's a discussion forum where developers can ask questions and get answers from their peers.
"They don't wait for us to respond to questions in the forum," says Veerappan, who is working to add features to the forum and convert it into a knowledge base. "It's kind of interesting to see the knowledge that other developers have gained in the API."
Successful APIs tend to have MIT-style open-source software licenses; the World Bank, for example, uses an open source attribution license. O'Grady says one key to success is being very clear about the terms of service, and not having an overly restrictive license that discourages use.
For instance, he says Stack Overflow, a collaboratively edited question-and-answer site for programmers, has a very nice API, but the terms of using it are difficult to navigate. And he notes that Twitter irritated some developers by being too demanding about issues such as how the time stamp was formatted, or insisting that the word tweet be capitalized. While developers are unlikely to shun a widely used service such as Twitter for being difficult to work with, O'Grady says, "if your product isn't that popular [it's possible that] people will abandon it."
Another nontechnological challenge to creating an open API is getting other people in your organization to cede some control, because they're likely accustomed to dealing with proprietary information and maintaining authority over their brand. "I had to do a lot of convincing," Bloomberg's Edwards says. "It's a different way of thinking, when you've been controlling your product." But he says it was important to distinguish between the market data Bloomberg sells and things like the symbology and software that the company doesn't need to control. "The time for all these proprietary interfaces is gone," he says. "It doesn't add value anymore."
You've got to release the right kind of data with the right documentation. Really, it comes down to what customer problems are you going to solve by doing what you do. Steve Bendt, director of emerging platforms, Best Buy
Best Buy's Bendt faced similar concerns. "It was tough when we first started talking about an API platform," he says, noting that colleagues wondered, "What are they going to build? What if they create a bad experience?" The company addressed that with rules about how developers could use the data: They must attribute it to Best Buy, for instance, and they can't appropriate it for other purposes. Best Buy doesn't preapprove apps, but it does regular audits to make sure apps comply with the terms of service.
At the World Bank, there was concern that giving away data would mean giving up the revenue that paid for curation of the data. Fantom says the bank decided that a free model would be better for its main objective of fighting poverty. "By making these data available for free and using these tools, we've seen a massive increase in the use of our data," he says. "Once you start getting into this, it's pretty clear that this is the right thing to do."
All of these organizations say they are continually developing their APIs, adding new functionality, responding to feedback from developers and customers, and figuring out what data to make available. "You've got to release the right kind of data with the right documentation. Really, it comes down to what customer problems are you going to solve by doing what you do," Bendt says. "It's not a launch-it-and-leave-it type of capability. It's constant learning and constant improvement."
Savage is a freelance science and technology writer in Lowell, Mass. He can be reached at email@example.com.
This version of this story was originally published in Computerworld's print edition. It was adapted from an article that appeared earlier on Computerworld.com.
Read more about applications in Computerworld's Applications Topic Center.