An Emerging Industry in Big Data

In a previous blog post, I looked at data mining social media, specifically twitter, in order to gain information about people and locations.  A company called WiBiData is collecting and organizing data about online users on a much larger scale.
 

An emerging industry today involves big data mining.  WiBiData, which is funded by Google, uses a technique that “combines big-data aggregation and analysis tools like Hadoop and Hbase and Apache’s Avro.”  This technique allows WiBiData to locate data across several databases and compile and analyze it in a single location.  The major reason for doing this is to sell this data to companies who want to know more about their consumers.  Providing data as a service is becoming a big business as these companies look to gain an edge in sales.  WiBiData reportedly has an interface that companies can use to quickly sort and locate the data they need.  In the article, this method is related to Netflix and Amazon in the way that they provide recommendations for users.


A great example used in the article relates to something we learned in class about association rules.  In class we talked about how beer and diapers were commonly bought in pairs.  In the article, the example involves online shopping for a hammer.  In a hardware store, hammers and nails are generally located far from each other.  On Amazon.com, if a person adds a hammer to his or her cart, nails are recommended to the user by analyzing data in milliseconds. 


This video shows some interesting insight on understanding the current state of data analysis and how it relates to the user.  It also explains some of the new software and techniques used to overcome some of the challenges of making sense of big data.  The speaker is the co-founder and chief technology officer of WiBiData, Aaron Kimball.

Watch live streaming video from gigaombigdata at livestream.com

Resource: http://gigaom.com/2012/03/22/wibidata-structure-data-2012/