Big Data is the New Engine of the Internet
Mobile devices and expanding networks of sensors are generating huge amounts of data that are increasingly searchable and can be shared to discern patterns and, potentially, to solve problems, according to an annual report on the state of the Internet.
The “Internet Trends 2014” survey by Mary Meeker, an analyst with venture capital firm Kleiner, Perkins Caufield, Byers, also asserts that the “mining [of a] rising amount of data has the potential to yield patterns that help solve basic, previously unsolvable problems but create new challenges related to individual rights.”
Among big data trends identified in the Internet report were:
- The amount of “uploadable, findable, sharable” real-time data is soaring.
- Sensor usage driven by emerging trends like the “Internet of Things” is rising sharply.
- Data processing costs are plummeting thanks to cloud computing and storage.
- Better user interfaces “aided by data-generating consumers” are helping to make data more useful.
- Data mining and other analytics tools are improving, helping to identify previously unseen patterns.
- “Pattern-driven problem solving” is emerging.
Driving the data flood are photos shared on social media sites like Facebook and Instagram. The survey estimates that 1.8 billion photos are posted each day on those sites plus Flickr, Snapchat, and WhatsApp.
Indeed, the New York Times reported in early June that the National Security Agency collects “millions of images per day” that include facial recognition quality images.
Meeker estimated that consumer data is growing at a rate of more than 50 percent annually to more than 4 zetabytes. She forecasts that the total will grow to 13 zetabytes by 2016.
Meanwhile, sensor data growth is being driven by the proliferation of networked devices. For example, the Samsung Galaxy S5 smartphone alone contains up to 10 sensors. Accelerometers, proximity sensors, and compasses are packaged in devices called MEMS, for micro-electromechanical systems. The Internet survey estimates that global shipments of MEMS devices in consumer electronics products reached nearly 8 billion units in 2013.
As sensors proliferate, the cost of processing and storing all the data they generate is plummeting. Thanks to the cloud, computing costs have dropped by one-third annually between 1990 and 2013. That drop also reflects the Moore’s Law dictum that computing performance tends to double on average every two years.
Moreover, Meeker’s annual survey found that storage costs have dropped 38 percent annually since 1992. Add to this, bandwidth costs have been trending down at an annual rate of 27 percent since 1999.
In turn, the data analytics tools used to sift through the zetabytes of consumer data are steadily improving, Meeker said, adding that the fastest growing segment of “valuable,” or tagged data, is being generated by the emerging “Internet of Things.” These networks of devices and sensors are “capturing and sending data, increasingly in real time,” the Internet survey stressed.
The explosion of data is being leverage to begin solving problems. Early applications include energy conservation devices for the home and connected cars.
The survey also points to health care applications as one of the most promising ways to leverage big data to solve real-world problems. For example, data analytics have helped drive down the cost and time need to sequence genomes to as little as $1,000 and 24 hours.
Moreover, the annual survey notes that these data trends are just emerging and will continue to evolve rapidly.