With all the talk about prediction models, I decided to look at a model to try and predict something important: a way to make money. Big data and the stock market seem like a perfect union of a technology and application. The stock market has hundreds of years of historical performance data and no shortage of real-time performance data. One would assume that someone would have already created a model using Big Data techniques to accurately predict how the stock market will move and have become wealthy as a result. Indeed, many investment companies have swung the trend toward hiring statistics wizards to create algorithms to predict the stock market. A short description of a potential algorithm-based trading scenario can be found at this link under the heading, “One Possible Scenario”: http://www.futuristspeaker.com/2012/12/will-big-data-destroy-the-stock-market/. Just as a side note I do not agree with the slant of this article saying the use of algorithms will eventually destroy the stock market, although that scenario did occur in a book I read by Tom Clancy called Debt of Honor.
However, are these algorithms really the best way to predict the stock market anyway, or is there a better way. According to this article, http://www.dailymail.co.uk/sciencetech/article-2120416/Twitter-predicts-stock-prices-accurately-investment-tactic-say-scientists.html, researchers at Cal-Berkeley have developed a model based solely on Twitter data in following the stock market, and do not look at any real financial patterns like most stock market algorithms do. During a four-month 2010 test trial, the Twitter-based model performed better than the Dow Jones average and out-paced conventional baseline models by 1.4-11%. To make this model, the researchers did not use sentiment analysis (positive or negative reaction to a stock) but more looked at pure volume of tweets about a particular company as well as how those tweets linked to other tweets or users.
To be fair, I am not sure I would trust that Twitter data can accurately predict the stock market. Perhaps there is some connection between the amount of buzz around a company (maybe buzz about Apple releasing a new product) and its future price but beyond that I am skeptical about this relationship. The researchers from the article linked above also expressed their caution that this model was tested in a declining market, and they did not know how the performance would translate to a market with overall rising prices. Personally, I would rather stick with traditional analysis about stock prices until it can be proven why Twitter or any other social media engine would be more accurate than detailed company financial indicators.