Big Data is less about the data itself and more about what you do withÂ the data. The application of statistics and statistical principles on the data helps you extract the information it contains. According to Wikipedia, statisticsÂ is the study of the collection, analysis, interpretation, presentation, and organization of data. The American Statistical Association defines statistics as “the science of learning from data, and of measuring, controlling, and communicating uncertainty.”
Statistics is considered to be one of the three primary pillars of the field of data science (the other two are content domain knowledge and computer science skills). While content domain expertise provides the context throughÂ which you identify the relevantÂ questions to ask, computer science skills help you get access to the relevant data and prepare them for analysis, statistics helps you interrogate thatÂ data to provide answers to yourÂ questions.
The Rise ofÂ Statistics
We have a lot of data and are generating a lot more of it. IDC saysÂ that we created 2.8 zettabytes in 2012. They estimate that number willÂ grow toÂ 40 zettabytes by 2020. It’s not surprising that Hal Varian, chief economist at Google, in 2009,Â said that “the sexy job in the next 10 years will be statisticians.” Statistics, after all, helps make sense of and get insight from data.Â The importance of statistics and statistical thinking in our datafied world can also be found in this excellent slideshareÂ by Diego Kuonen, a statistician.
Statistical skills are receiving increasingÂ attention in the world of business and education. LinkedIn found thatÂ statistical analysis and data mining was the hottest skill in 2014Â (see Figure 1).
Many companies are pursuing statistics-savvy people to help them make sense of their quickly-expanding, ever-growing, complex data.Â Job postings onÂ IndeedÂ show that the number of data science jobs continue to grow (see Figure 2).
University students are flocking to the field of statistics. Of the STEM Professions, statistics has been the fastest growing undergraduate degree over the past four years (see Figure 3).
The Fall of Statistics
The value of statistics is evident by the increase in number of statistics degrees and the Big Data jobs requiring statistical skills. These are encouraging headlines, no doubt, as more businesses are adopting what scientists have been using to solve problems for decades. But here are a few troubling trends that need to be considered in our world of Big Data.
McKinseyÂ estimates that the USÂ faces a shortage of up to 190,000 people with analytics expertise to fill theseÂ data science jobs as well as a shortage of 1.5 million people to fill managerial and analyst jobs who canÂ understand and make decisions based on the data. Where will we find theseÂ statistics-savvy people to fill the jobs of tomorrow? We may have to look outside the US.
In a worldwide study onÂ 15-year-old students’ reading, mathematics, and science literacy (the Program for International Student Assessment;Â PISA), researchers found that US teenagers, compared to children of other countries, ranked 27th (out of 34 countries) in math literacy (see Figure 4), many countries having significantly higher scores than the US. According to the NY Times, while 13% of industrialized nations reachedÂ the top two levels of proficiency in math, Â just 9% of US students did. In comparison,Â 55% of students from Shanghai reached that level of proficiency. In Singapore, 40% did.
Even the general US public is showing a decreased interest in statistics. Using Google Trends, I looked at the popularity of the term, statistics, among the general US public, comparing it with “analytics” and “big data.” While the number of searches for “big data” and “analytics” has increased, the number of searches of “statistics” has decreased steadily since 2004.
Summary and Major Trends
Statistics is the science of learning from data. Statistics and statistical thinking helps people understand the importance of data collection, analysis, interpretation and reporting of results.
In our Big Data world, statistical skills are becoming increasinglyÂ important for businesses. Companies are creating analytics-intensive jobs for statistics-savvy people, and universities are churning out more graduates with statistics degrees. On the other hand, there is expected to be a huge talent gap in the analytics industry. Additionally, the math literacy of US students is very low compared to the rest of the world. Finally,Â the USÂ general public’s interest in statistics has been decreasing steadily for about a decade.
Knowledge of Statistics Needed for Both Analysts and Consumers
Statistics and statistical knowledge areÂ not just for people who analyze data. They are also for people who consume, interpret and make decisions based the analysis of those data. Think of the data from wearable devices, home monitoring systems and health records and how they are turned into reports for fitness buffs, homeowners and patients. Think of CRM systems, customer surveys, social media posts and review sites and how dashboards are created to help front-line employees make better decisions to improveÂ the customer experience.
The better the grasp of statistics peopleÂ have, the more insight/value/use they will get from the data.Â In a recent study, I found that customer experience professionals had difficulty estimating size of customer segments based on customer survey metrics. Even though these customer experience professionals commonly use customer survey metrics to understand their customers, theyÂ showed extreme bias whenÂ solving thisÂ relatively simple problem. I assert that they would likely benefit (make fewer errors) if they understood statistics.
To get value from the data, you need to make sense of it, do something with it. How you do that is through statistics and applying statistical thinking to your data. Statistics is a way of helping peopleÂ get value from their data.Â As the number of things that get quantified (e.g., data) continues to grow, so will the value of statistics.
The Most Important Thing People Need to KnowÂ about Statistics
Statistics is the language of data. Like knowledge of your native language helps you maneuver in the world of words, statistics will help you maneuver in the world ofÂ data. As the world around us becomes more quantified, statistical skills will become more and more essential in our daily lives. If you want to make sense of our data-intensive world, you will need to understand statistics.
Iâm not saying that everyoneÂ needs an in-depth knowledge of statistics, but I do believe that everybody would benefit from knowing basic statistical concepts and principles. What is the most important thing you think people need to know about statistics and why? I would love to hear your answers in the comments section. Here is my take on this question.