Q&A: Visa dips a toe into the Hadoop pool

As cloud computing edges its way into the enterprise, the open-source Apache Hadoop project may well prove to be the poster child of the movement. Hadoop effectively gives enterprises the power of Google or Yahoo Web indexing for free, or for the cost of a CloudEra subscription if you want to involve Hadoop's core developers in your rollout. Credit card giant Visa is an early corporate adopter of Hadoop, and points to a bright future for the open-source project.

I caught up with Visa's Joe Cunningham, head of the technology strategy and innovation group, to talk about the company's adoption of Hadoop.

Q: What got you interested in Hadoop initially and how long have you been using Hadoop?
Joe Cunningham: It's early days for us here at VISA for Hadoop. It's still very much classified as a research and development activity.

My role is the head of technology strategy and research and development for the company. Our task is to look outside the company for interesting technologies on the landscape and identify potential opportunities for those technologies to add value to either the VISA business of VISA technology and then bring them in and play with them in our lab research environment until they are ready for mainstream or commercial activity.

Hadoop is one of those technologies we've been looking at for about a year and we think it offers certain value as an augmentation to existing systems and capabilities VISA has.

Q: How do you use Hadoop at VISA? What made you think it could be the best solution for what you're trying to accomplish?
Cunningham: The most important thing to remember is VISA obviously has a heritage of offerings--very large, very scalable, very reliable, and very secure services to the payments industry. And we're continuously trying to innovate and make those services more valuable to our clients and ultimately to cardholders.

We have a data challenge we attempt to meet every day in terms of the number of transactions we handle and therefore we think there's an opportunity to look at the skills VISA already has in the data analytics space with the power of Hadoop to handle very, very, very large volumes of data.

To put that in context, we handle approximately 200 million transactions a day at VISA. That works out to be about 8,000 transactions a second, and with that comes huge volumes of data and Hadoop offers the potential to harness some of that along with some of our existing capabilities to extract more value from those transactions.

Q: Are there particular directions in which you'd like to see Hadoop evolve?
Cunningham: I think we're interested in looking at Hadoop and looking at its evolution over time. We're certainly interested in how the Hadoop community continues to operate in this open-source environment.

My specific interest is how can Hadoop evolve from the alpha beta environment in which it is today to the mainstream and how can we continue to integrate it as a mainstream technology with all the existing platforms we have here at VISA.

I'll give you two examples. The operations management space is very important to us: how we guarantee the reliability and security of our systems and how Hadoop can be merged or integrated into that environment. And secondly, and I guess this is a common question, but how can we enable SQL-like access to some of the data via the Hadoop file system or via the Hadoop engine?

Q: Given that it's still early days for Hadoop at Visa, it's interesting that you're speaking at the upcoming Hadoop World conference, along with JP Morgan Chase, China Mobile, and other Hadoop users that may be further along the adoption curve. What are you going to be talking about?
Cunningham: I plan to talk about the application stream, so I'll be taking a business-focused view of how we see Hadoop offering value to Visa. If there are tech junkies in the room, they are probably not going to be as interested in what I talk about.

I plan to spend a little bit of time showcasing Visa's technology today to set the scene. I will talk a little bit about our research and development function and how it works with the rest of Visa. Then I'll spend some time expanding on what I call our information products business.

The information products business for Visa offers services to our clients that are, obviously, information-based. So, some of the use cases where we see Hadoop potentially offering value in the future are in the areas of transaction analysis (particularly for risk products and the modeling of risk scenarios), fraud analysis (assisting our clients in potentially managing fraud more carefully), and in the loyalty space where Visa offers services on behalf of our clients to cardholders.

There's an opportunity for us to combine the power of Hadoop with data analytics capabilities that Visa has to augment those services and products on behalf of our clients.

In fact, that's an area that I'm hoping to learn a lot at the event. In some industries, Hadoop is very much mainstream but for some others, it's still emerging and I'm trying to understand whereabouts on that hockey stick or [Gartner] Hype Cycle Hadoop is, or whether Hadoop is already mainstream and it's just a matter of us catching up.

It's always good to gauge and plot that evolution. I think you need to get to these events and talk to other companies and key leaders in the community to really understand where we fit and what we should be doing next at Visa.

For those interested in attending Hadoop World in New York, the organizers are giving Open Road readers a 25 percent discount if you register by September 24.