The Big Data Inside Amazon’s New Fire Phone

The new Fire phone that Amazon launched this week looks like your ordinary black smartphone, with a bright 4.7-inch screen, sharp 1080p camera, and stereo speakers. But this little device does some stuff that Google and Apple haven’t tried yet in the area of text, sound, and image recognition on a massive scale.

From a big data point of view, the most interesting bit about Fire is the Firefly function. To use this function, users point their camera at some object in the real world, snap a picture of it by pressing the Firefly button, and voila! The data (including images, audio, GPS location data, and a timestamp) is automatically uploaded to Amazon’s cloud, where algorithms automatically identify the object or the sound.

Amazon says Firefly currently can identify more than 100 million items, including household products, songs, movies, TV shows, phone numbers, Web and email addresses, and QR codes. Once an item is identified, Firefly lets users take action, such as making a purchase on Amazon’s e-commerce website.

Amazon is attempting to identify all the objects users encounter with its Firefly image- and sound-recognition service.

That’s the most obvious tie-in to buying a phone from the world’s largest e-tailer, but there are other possible actions include saving names and contact information from a business card, automatically opening websites or sending email, or posting something about it on Facebook and other social media sites.

Now, this isn’t the first such app for a smartphone. Amazon already built similar functionality into its Flow app for iPhones and Android, and Google and Microsoft offer similar product-recognition systems as well. Ditto to the song, movie, and TV show recognition systems, which aren’t new either.

What makes Firefly unique from a big data perspective is the scale that Amazon is bringing to its image and sound recognition systems. Whenever a user takes a photo or a video, it’s uploaded to Amazon’s Firefly storage service. The phone sports 32GB of Flash storage, but Amazon is giving customers an unlimited amount of cloud storage from its massive data centers.

The more photos and videos that Fire users take, the more photos and images get stored in the Firefly repository, and the better Amazon’s algorithms get at identifying stuff in the world all around. Amazon does give users the option of deleting photos and videos from the Firefly repository. But the default setting is to store them in Firefly, and until users remove the files, Amazon will be using them to train its machine learning models on the world according Firefly users. Amazon will also be offering third-party apps access to the Firefly identification service through an SDK.

This relationship between the Firefly function and Amazon’s customers is what led VentureBeat‘s John Koetsier to declare the phone “the biggest privacy invasion ever.”

“By storing all the photos you’ll ever take with Firefly, along with GPS location data, ambient audio, and more metadata than you can shake a stick at in Amazon Web Services, Amazon will get unprecedented insight into who you are, what you own, where you go, what you do, who’s important in your life, what you like, and, probably, what you might be most likely to buy,” Koetsier writes.

“Babies in your pictures? Sell that dame diapers,” he continues. “Lots of old-school hot rods? See if you can sell Billy Bob some NASCAR shwag, or maybe beef jerky. Outdoorsy, are you, with your pictures of remote mountaintops and idyllic forest meadows? Clearly you need hiking boots and granola. Looking at a business card? Perhaps things she likes will be things you’ll like, too.”

Amazon is already using shopping information to profile its customers. Every major Web property–from Google to Microsoft to Apple to Facebook—is collecting user data and watching what its users do to come up with targeted sales pitches, if not for itself then for its biggest advertisers. That’s what the big data game is all about, and probably the biggest workload for first-gen Hadoop clusters.

As we enter the next phase of the data explosion, the proliferation of connected sensors will give us new sources of data to exploit. As the world’s biggest data center operator, Amazon is well positioned to turn all of this unstructured data into selling opportunities. Amazon may need to clarify some of its privacy policies surrounding Firefly–particularly if it begins offering a facial recognition or voice recognition filter with its Firefly SDK.

But if users find the service useful, we’ll undoubtedly see other big firms try to duplicate this attempt to collect and correlate consumer smartphone data on a massive scale. It’s the future. Get used to it.

Related Items:

Microsoft Spins Up a Machine Learning Cloud

Deep Neural Networks Power Big Gains in Speech Recognition

The Big Potential of that Little Data Miner in Your Pocket