Follow Datanami:
May 30, 2017

Overcoming the Quant Crunch


Think it’s hard to find trained data scientists in the job market now? According to a new report from IBM dubbed “The Quant Crunch,” the shortage will grow so big in the years to come that it could derail the delivery of big data benefits that we all expect.

IBM worked with the Business-Higher Education Forum and Burning Glass Technologies to gauge the current state of the job market for “quants,” those number-loving researchers who turn hectares of big data wheat into kernels of informational gold. What they found is the market is shifting in ways that were not entirely expected.

The new research starts with McKinsey’s seminal 2011 report, which predicted there would be 2.8 million workers in the United States with deep analytical talent or data-savvy skillsets by 2018. However, according to Burning Glass – which maintains a database of 130 million current and historical job listings — there are already about 2.5 million open job postings for people with data science and analytics (DSA) skills in the U.S, a number that’s expected to surpass 2.7 million by 2020.

The catch, of course, is that those are all open positions; it doesn’t factor in all those folks currently working in the DSA field. The people holding those positions would need to quit their jobs and re-enroll in the job market for McKinsey’s prediction to become true. McKinsey, it appears, vastly underestimated the need for quants.

“There is growing concern that the supply of DSA workers is lagging dangerously behind demand,” write the report authors, Steven Miller of IBM Analytics and Debbie Hughes of Business-Higher Education Forum. If action is not taken, “the DSA skills gap is in danger of widening, which would continue to undercut the promise of big data.”

Can We Close the Gap?

(excerpt from “The Quant Crunch”)

Is it possible to close the gap between supply of DSA workers and supply? Because we live in a market-based system, there is no overarching authority that can implement a quick fix. But over time, the market should eventually self-correct and find a balance between supply and demand.

In the meantime, each of the constituent parties – the universities who will train the people, the trained employees who do the DSA work, and the companies that pay the DSA workers’ salaries and benefit from the work – need to step up to address the shortage.

“To meet this explosive demand, businesses need to rethink hiring, training, and partnerships,” the authors write. “Higher education needs to be nimble and responsive, and its bachelor’s, graduate, certificate, and executive-level programs have to be responsive to workforce needs.”

Universities have been adding data science curriculum to address the need. This fall, for example, Syracuse University will debut a new master’s of data science engineering program.

Jeff Saltz, an associate professor who heads up the program, tells Datanami there’s a clear need to train a new generation of data scientists.

Summary of six core DSA fields, according to “The Quant Crunch” report

“For a number of years, it was ‘Let’s collect all this data. OK, now I have this data. What does it mean?’ That’s where data science comes in,” he says. “The question really comes down to, what do you do with all this data?”

Currently, DSA jobs are among the hardest to fill in the entire market. The average DSA job is open for 45 days, which is 5 days longer than average, the “Quant” report authors write. And folks in the DSA field are well-compensated, earning an average annual salary of $80,265, which is nearly $9,000 more than average compared to all bachelor’s and master’s-level jobs.

But another problem is semantics. There are no standard terms for many of the job titles involved here. For example, the Bureau of Labor Statistics doesn’t even recognize “data scientist” or “data engineer,” two of the most widely used positions for people in the DSA field, according to the report’s authors. (The term DSA itself is not widely used, either, but we’ll give them that.)

The 300 (Analytical Skills)

With the vast Burning Glass database at their fingertips, Miller and Hughes identified 300 core analytical skills that factor heavily in DSA jobs. The list covers everything from general analytical competencies and database architecture to using R and Hadoop.

Next, the authors broke the DSA field down into six major functional areas. They included:

  • Data Scientists and Advanced Analysts;
  • Data Analysts;
  • Data Systems Developers;
  • Analytics Managers;
  • Functional Analysts;

    Excerpt from “The Quant Crunch”

  • and Data-Driven Decision Makers.

Obviously, not everybody who does DSA work will have a business card that says “data scientist” or “data engineer” on it. Those two jobs do exist, however, and they require the holders of those titles to have the most advanced skill sets for things like machine learning and model development. In terms of analytical rigor, these positions rank 100 out of 100. They also tend to earn more money than their brethren on the DSA spectrum.

But in many cases, the DSA jobs of the future won’t require rigorous analytical credentials. Instead, a position such as credit analyst, GIS specialist, or marketing analytics manager may only rank 70 or 80 out of the 100-point analytical rigor scale.

Those 300 analytical skills will be spread widely across the spectrum. A budget analyst may benefit from having skills in SQL and SAS, but they won’t necessarily need to know MapReduce or MongoDB, whereas a marketing analyst may need ETL and pivot-table skills to do his job, but won’t necessarily need to know Tableau or Apache Pig.

Excerpt from “The Quant Crunch”

Complicating things is the fact that many DSA jobs will be hybrid positions that require a variety of technical computing and domain-specific skills, the authors find. Analytics Managers, for instance, must not only be experts in fields like human resources or marketing, but also much possess skills in analytics, project management, financial planning, and budgeting.

“Preparing workers for these roles is problematic, since these skills cut across a diverse mix of functional areas,” the authors write.

Want ultimate job security? Become a Data-Driven Decision Maker, such as a data-enabled marketing manager. According to the report, this class of worker will comprise one-third of the data savvy professional job market by 2020. Between now and then, there will be 110,000 new Data-Driven Decision Maker positions added to the job rolls across the country.

There are some people who still think data science is a fad. “At some point I’m sure [the hype] will die down, but I haven’t seen it at all yet,” Syracuse’s Saltz says.

To download “The Quant Crunch,” see

Related Items:

Universities Roll Out New Data Science Programs

Data Analytics in Higher Education: A Mixed Bag

Six Big Name Schools with Big Data Programs