Black Belts Take Heart: Big Demand for Statisticians

In several other posts (Career opportunities from Big Data, The Era of Big Data) we’ve seen the importance for both individuals and organizations to develop the skills to glean insight from ever-increasingly mounds of data. For years Black Belts and others who work in the world of applied statistics have done so with the challenge that few people understand what it is they do for a living. Afterall, statistics and analytics are rarely the topic of cocktail parties. But in addition to Hollywood A-listers portraying data nerds (Brad Pitt & Moneyball; Moneyball press conference) there is also the trend of increased needs in all areas of the economy for people who can sort and sift through Big Data.

Back in the fall of 2009 the New York Times’ Steve Lohr wrote:

At Harvard, Carrie Grimes majored in anthropology and archaeology and ventured to places like Honduras, where she studied Mayan settlement patterns by mapping where artifacts were found. But she was drawn to what she calls “all the computer and math stuff” that was part of the job. “People think of field archaeology as Indiana Jones, but much of what you really do is data analysis,” she said. Now Ms. Grimes does a different kind of digging. She works at Google, where she uses statistical analysis of mounds of data to come up with ways to improve its search engine. Ms. Grimes is an Internet-age statistician, one of many who are changing the image of the profession as a place for dronish number nerds. They are finding themselves increasingly in demand — and even cool.

The rising stature of statisticians is a byproduct of the recent explosion of digital data. In field after field, computing and the Web are creating new realms of data to explore — sensor signals, surveillance tapes, social network chatter, public records and more. Yet data is merely the raw material of knowledge. “We’re rapidly entering a world where everything can be monitored and measured,” said Erik Brynjolfsson, an economist and director of the Massachusetts Institute of Technology’s Center for Digital Business. “But the big problem is going to be the ability of humans to use, analyze and make sense of the data.”

The new breed of statisticians tackle that problem. They use powerful computers and sophisticated mathematical models to hunt for meaningful patterns and insights in vast troves of data. The applications are as diverse as improving Internet search and online advertising, culling gene sequencing information for cancer research and analyzing sensor and location data to optimize the handling of food shipments.

Though at the fore, statisticians are only a small part of an army of experts using modern statistical techniques for data analysis. Computing and numerical skills, experts say, matter far more than degrees. So the new data sleuths come from backgrounds like economics, computer science and mathematics.

I.B.M., seeing an opportunity in data-hunting services, created a Business Analytics and Optimization Services group in April. The unit will tap the expertise of the more than 200 mathematicians, statisticians and other data analysts in its research labs — but that number is not enough. I.B.M. plans to retrain or hire 4,000 more analysts across the company.

In another sign of the growing interest in the field, an estimated 6,400 people are attending the statistics profession’s annual conference in Washington this week, up from around 5,400 in recent years, according to the American Statistical Association. The attendees, men and women, young and graying, looked much like any other crowd of tourists in the nation’s capital. But their rapt exchanges were filled with talk of randomization, parameters, regressions and data clusters.

The reference to IBM’s Business Analytics and Optimization Services group (IBM BAO) underscores the growing need for organizations to muscle-build these capabilities in their organizations, especially those organizations wanting to excel at things like consumer marketing, risk assessment and complex supply chain management.

Indeed, the very skills that made it difficult for Black Belts to explain to their friends and relatives what they did for a living are also making them more marketable. But it also highlights the increasing penalty organizations will pay if they are lacking quality data and the talent to know what to do with that data (The Good, the Bad and the Ugly).