Big Data in the Enterprise

                Times have changed over the last 40 to 50 years when it comes to R&D work in the U.S. During the 60’s and 70’s the government was leading the way on research and development with supercomputers,  creation of the Internet, and all of the ideas and patents from NASA. However, the private sector and free enterprise are now the bread winners when it comes to pushing the technology envelope. Now companies are always seeking out the new gadget or toy that their competitors don’t have. Apple has been known for leading the cell phone industry in touch screen technology. But as today’s enterprise grows and is becoming more data-driven, this is forcing the government to get involved in pushing “Big Data” down the road, albeit a little behind the curve. The government is having the same problems as private industry with collecting enormous volumes of data. Often times this data is of poor quality and needs “cleansing” and opened from their organizational servers (silos).  For almost 55 percent of global respondents, their upper management and senior leaders still don’t recognize the importance and need for Big Data management.  Merely collecting the data does not add value to the bottom line of the company. You still need the analysis, interpretation and inference into the data.

                The U.S. government has recognized where the future lies in respect to Big Data and last March the Obama Administration launched a $200 million dollar Big Data initiative. They want to help transform our ability to use Big Data for scientific discovery, environmental and biomedical research, education, and national security, said Dr. John P. Holdren, Assistant to the President and Director of the White House Office of Science and Technology Policy. Within this initiative, they hope to expand the workforce needed to develop and use Big Data technologies. The National Science Foundation is also encouraging research universities to develop interdisciplinary graduate programs to prepare the next generation of data scientists and engineers. The U.S. Geological Survey – Big Data for Earth System Science have issued grants for projects that will improve our understanding of issues such as species response to climate change, earthquake recurrence rates, and the next generation of ecological indicators. This Big Data initiative is one of the largest public technology investments in recent history. 

                The Army is already taking advantage of Big Data. They deployed a Hadoop cluster in Afghanistan known as a network connection point. This cluster can be dropped into a war zone in a 20-foot shipping container. This real time analysis of intelligence reports from around the world will allow the troops on the ground and commanders to better understand how to thwart the enemy. Other Government agencies are starting to invest in the future of big data with talent acquisition and growth from the inside. They are providing on-the-job training to develop expert skills like “open source analysis” of Web sites and social media networks to detect the recruiting strategies and information pipeline of extremist organizations. 

                Some of the vital traits and qualities of a Big Data analyst include understanding semantics, statistics, algorithms and analytics. Finding one individual with all of those traits is hard to find. The private sector is also competing with the government to acquire these rare individuals. Businesses can partner with universities and colleges to ensure that the next generation of scientist has an understanding of Big Data technologies. 

                In today’s “data-driven” world where everything creates data, one of the biggest problems for older companies is when their Big Data grows too big for their standard commercial computer systems.  They must make a decision to scale-up to a high performance computing system or scale-out by just adding nodes to a distributed computing network? With new companies in the ingest phase, a complete Big Data solution should empower the enterprise to plan for analysis and archiving of these enormous data sets.  They should be able to digest massive volumes of data at a high velocity. This data could include videos, RFID and sensor signals, to massive volumes of Tweets from Twitter (which includes approximately 400 plus million Tweets per day). With the technology available now, this can be done either in parallel into a cluster or into a scale-up in memory system that is capable of ingesting 4TB of data – roughly the equivalent of the Library of Congress – in 3 to 4 seconds. 

                As the need for Big Data analyst increase and as technology increase, we will see a large step forward within the next five years. Businesses and industry will transform just like they did leaving the analog days into the digital era. The potential effects on the health-care industry, when it comes to re-admissions and survival rates are endless.