QCon San Francisco 2007 – Jinesh Varia – Amazon and Hadoop

This is not a summary of Jinesh Varia’s talk but some of the thoughts it triggered.

I am still struggling to make sense of all the possibilities that what he presented open. Essentially, amazon rents virtual servers by the hour at a very affordable price.

Which opens very heavy computing to any small company or individual. You can even build custom images for your servers. Some services actually offer to run you whole infrastructure on rented virtual servers.

In terms of business it could really break the price barrier to a lot of markets where you occasionally need heavy computing power (did I say financial industry) as it removes the need to build an under used data center. I fully realize that this must be a temporary solution (because your exposure to a single service provider would be too high) and that some questions need to be adressed like data confidentiality and integrity before this becomes a reality.

Which lead me to my next thought. How do you make it so that it is possible to use any of the companies that may provide the same kind of service ? Say that yahoo, google, amazon, sun, IBM offer that service (I believe some if not all of them do), how do you arbitrate which one you are using (based on pricing, power,…) in a way that is transparent to your organization. And when there is contention on these on demand resources, how do you reserve the capacity and ultimately exchange it.

In other words, can we organize a market and a derivative market for rented processing power ? I wonder if it already exists.

One Response to “QCon San Francisco 2007 – Jinesh Varia – Amazon and Hadoop”

  1. Hering Cheng Says:

    Here is an article on what Sun and Archipelago (now part of the NYSE) was trying to do in 2005: operating an exchange (like a stock exchange) trading CPU power. (http://www.itjungle.com/tug/tug021005-story03.html) This obviously went nowhere as Archipelago at the time became busy acquiring Pacific Exchange (where I used to work) and itself was bought by the NYSE a few months later. Just like the bond exchange that the NYSE is (still unsuccessfully) trying to operate, there has to be enough of breadth and depth in the marketplace for something like this to work.