IDC HPC Forum
9/6/16 – 9/8/16
Data is finance’s new currency, healthcare’s latest wonder drug, and the energy sector’s new oil.
Another day, another Big Data analogy.
All of the hype doesn’t change the fact that businesses across nearly every industry are gaining competitive advantage by extracting value from large datasets.
Econometrics is an area that has been cautious about Big Data. The field is built on a strong foundation of theory and methodology, and relies on a variety of approaches that differ significantly from those of Big Data analytics. For example, econometrics typically starts with a theory and then uses data analysis to prove or disprove it, while Big Data and machine learning work in reverse. Econometricians have also expressed concerns regarding the context, reliability and representativeness of such vast datasets.
However, it’s becoming clear that Big Data has the potential to be disruptive to traditional econometrics. Data collection over social sources has produced unprecedentedly large and complex datasets about human behavior and interaction, and this unstructured data has proven itself to be a goldmine of economic information.
Econometricians are certainly not strangers to data analysis; however the growing volume of economic data from diverse sources is driving the need to adopt new computational approaches and develop better data manipulation tools.
Econometricians entering the field today also face a bit of a learning curve, and find they require a combination of skills in both economics and computer science to deal with the increasing volume, variety, and velocity of data. Hal Varian, Chief Economist at Google offers this word of advice to current students of econometrics: “Go to the computer science department and take a class in machine learning.”
While econometricians might still be working out the “kinks” in their Big Data approaches, the analysis of large datasets is already driving a number of advancements across the field:
Machine learning by its very definition has the potential to rapidly alter the field of econometrics. The ability of computers to develop pattern recognition, and then learn from and make predictions based on data is a familiar task for econometricians, who on a daily basis analyze tremendously large volumes of economic data in order to form theories.
As Big Data continues to penetrate the methods of econometrics, the field will need to adopt new computational tools and approaches in order to extract insight from these increasingly large and complex economic datasets. The granularity offered by Big Data will enable econometricians to adopt new data-driven styles of analysis and investigation to help them resolve their biggest economic questions.