Career opportunities from Big Data

Quantity of global data in exabytes (1 exabyte = 1 billion gigabytes)

One of the major stories of the past year and one that will only grow in importance with each coming year, is the so-called Big Data issue and opportunity. Simply put, the cost of storing data is becoming so cheap, and the amount of data from a range of sensors, social media, and smart phones is growing so immense, that the shear extent of the data sets and the ability to cross-connect them in real-time will provide opportunities for breakthroughs in marketing, product design, product and service delivery, order generation and payments, process improvement and so on.

The Economist, in their 2012 outlook, wrote:

This deluge has long been a topic among geeks; 2012 will be the year in which the Big Data trend gets noticed beyond their ranks. Many more firms will start to analyse huge piles of data to optimise ­everything from their supply chains to their customer relationships.

In the past, because data storage was expensive, lots had to be thrown away. Information was locked up in computer systems and could not be combined with other sources. Even if firms were able to aggregate their data in one place (called a “data warehouse”) and sift through the information, it was often already out of date.

All this is changing—rapidly. The price for storage is plunging; by 2020 storing a petabyte will cost a mere $4, predicts Forrester, a market-research firm. Software to handle such huge amounts of data is improving, too. Hadoop, a new type of database, can be used to sift through big data streams in real time—and not just orderly numbers, but also “unstructured” data, for instance any kind of text.

The implications for performance improvement professionals is that there will be a growing need to add the skills and tools of analyzing big data sets to the traditional process improvement tools. Perhaps not every Black Belt in every situation will have the need or opportunity, but in a growing number of sectors, and in certain areas of these businesses, breakthroughs in supply chains, marketing, and real-time process management will require performance improvement leaders to have the skills to exploit big data.

There is a large talent shortage of these skills, says The Economist:

In a world oozing data, sensors measure everything from speed to smell. Smartphone applications generate vast quantities of “data exhaust”: information that is produced when users engage in, say, skiing, and, thanks to a motion sensor, the phone can tell whether somebody has just had a bad fall.

Social media are adding to the flood: the number of messages on Twitter will exceed 500m per day by the end of 2012. By opening their data vaults, some ­national and local governments are doing the same.

Most important, firms will discover that they can extract value from the data. A study by the McKinsey Global Institute, a think-tank, found that analysing health-care data could yield $300 billion-worth of savings in America alone. One of the pioneers in this field is Britain’s National Institute for Health and Clinical Excellence, which uses large datasets to investigate the cost and benefit of new drugs and existing expensive treatments.

Firms will also use data for new business models. Rolls-Royce no longer needs to sell all its jet engines; it can charge for their use. By continuously assessing their performance, it can predict when engines are more likely to fail, so that customers can schedule engine changes.

Already, the number of data-driven start-ups is growing—particularly when it comes to social media. Lexalytics, for instance, analyses the sentiment of utterings on Twitter, Facebook and other such services. Klout measures the influence of social-media users (and some firms give people with a high Klout score preferential treatment).

For Big Data to become huge, however, there are still hurdles to leap. For one thing, the tools to analyse data are not yet good enough. And people with the skills to analyse data are scarce and will become scarcer. By 2018 there will be a “talent gap” of between 140,000 and 190,000 people, says the Mc­Kinsey Global Institute. The main problem may be privacy. It is unlikely that people will want to live in a “yotta world” (1,000 trillion gigabytes) in which their every move is instantly digitised and added to the flood of public data.