The New Shape of Big Data

The New Shape of Big Data

Companies are now seeking better tools than traditional statistical methods to analyze the vast amounts of data readily available. Researchers are increasingly scouring scientific papers and esoteric branches of mathematics like topology to make sense of complex data sets. A ton of data is now available in such vast scope and minute detail that it is sometimes difficult to just look at it in a two dimensional format of rows and columns is what software analysts proclaim. It is best to fully utilize this advent of data availability and target areas which benefit towards the goals of the project. Data is so complex that using old methods will not benefit an organization. A good application of esoteric mathematics would be to look at data arranged in shapes using topology.

Topology is a form of geometry that relies on the way humans perceive shapes. Topology helps researchers look at a set of data and think about its similarities, even when some of the underlying details may be different. It is just one of the new methods being explored. Many analysts expect to see a renaissance in advanced mathematics and algorithms as companies increasingly realize how valuable data is and how cheap it is available for storage using various platforms. Another initiative to build this is developed by IBM known as graph theory. This is a tool similar to topology. It maps the interactions of people on social networks. Based on the communications traffic, each person is a node and the communication is a link. Graph theory algorithms help discover the shortest path between nodes and reveal social cliques. Cliques are a group of 2- 12 people who interact with each other on a more regular basis. Cliques are more tightly interconnected than the community around them. It is easier to analyze these groups than targeting a large community of people at once. Automates search programs can determine what each cliques members are talking about and the company can use this data to create recommendation and advertising systems.

An example of another algorithm developed by Tellagence is one which will predict how information will travel as it moves through social networks, but assumes that the network will change constantly and what is most important is the context in which the data appears. This technique helped the company track down the source of some influential ideas being discussed online about the kind of integrated circuits known as field programmable gate arrays. Using this algorithm Tellagence identified a group of more than 100 people involved in an online discussion (similar to the class blog) and devised a strategy to approach the 2 or 3 people whose traffic patterns indicated that they were at the center of the discussion.

Source: The Wall Street Journal