Applications » Data Mining


How Spark Helps ClearStory Achieve Data Harmony

Jul 28, 2014 |

When the folks at ClearStory Data set out to build a big data harmonization tool that could automatically converge diverse data streams into a single coherent view for analysis, they figured they’d have to develop much of the underlying technology themselves. Then they heard of the Apache Spark project at Cal Berkeley’s AMPlab, and the rest, as they say, is history. ClearStory Data was founded several years ago by two high-tech veterans, Sharmila Mulligan and Vaibhav Nivargi, with the idea Read more…

FDA Mines Billing Data for Drug Interaction Insight

Jul 22, 2014 |

The Food and Drug Administration is five years into a pilot program aimed at identifying hazardous drug interactions by mining the medical billing records of millions of Americans. The program, dubbed Mini-Sentinel, is a creative application of big data technologies that has the potential to improve people’s lives. The FDA started funding the Mini-Sentinel project in 2009 with the goal of coming up a better way to monitor for unintended side effects of prescription drugs. It’s impossible to eliminate all Read more…

Slicing and Dicing Music Data for Fun and Profit

Jul 21, 2014 |

The advent of big data analytics promises to have a profound impact on many aspects of human life, including how we work and play. Big data is even influencing the arts, where the field of music data science is rearranging our relationship with music. We’re in the midst of a boom in music data science that can be traced back to 1999, when two important events occurred. First, Shawn Fanning unleashed Napster to the world, thereby giving people the power Read more…

Can You Trust Your Algorithms?

Jul 15, 2014 |

Algorithms are critical to how we interact with data. And as the volume and variety of data increases, so does our reliance on algorithms to give us the answers we seek. But how much faith should you put into those algorithms, and how can you be sure they’re not misleading you? They’re not simple questions, but through the use of algorithmic differentiation techniques, data scientists can get more precise answers. Algorithmic differentiation, sometimes called automatic differentiation, is a technique used Read more…

Where Does Spark Go From Here?

Jul 11, 2014 |

The excitement behind Apache Spark reached an apex last week during the 2014 Spark Summit put on by Databricks, the company behind the in-memory analytics phenomenon. With a large community of users and growing support from software vendors, the future for Spark certainly appears bright. But there’s a large amount of work ahead to fulfill the promise of Spark, including hardening various components. Providing an easier-to-use alternative to MapReduce is the first use case for Spark, which is said to Read more…

News In Brief

DARPA Seeks to Leapfrog Big Data with ‘Big Mechanisms’

Jul 31, 2014 |

A Defense Department research program seeks to “leapfrog” advanced big data analytics by developing automated search technologies that could help explain the causes and effects that drive complex systems. The Defense Advanced Research Projects Agency (DARPA), which is credited with developing the Internet’s basic architecture, launched its “big mechanism” initiative earlier this year that aims to develop automated tools that could uncover causal models hidden in big data. The classic example of a big mechanism is the 1854 map of Read more…

UK Regulators Seek to Balance Big Data, Privacy

Jul 29, 2014 |

As concerns grow over how companies collect and use personal data, a UK regulator has issued guidelines for how companies leveraging big data must operate under Britain’s data protection law. The UK Information Commissioner’s Office recently released a report on July 28 that seeks a balance between operating within Britain’s Data Protection Act of 1998 without slowing data innovation. The report acknowledges that big data analytics frequently focus on areas other than the collection and use of personal data. It Read more…

NFL’s 49ers Launch Data Drive to Boost Fan Base

Jul 28, 2014 |

The San Francisco 49ers are now brought to you by the $1.2 billion Levi’s Stadium in Santa Clara. The NFL team’s “fan engagement program,” dubbed “Faithful 49″ is presented by team sponsor Esurance and a couple of data collectors you may have heard of: Google and Facebook. As the Niners prepare to open their spanking new 68,500-seat stadium in Silicon Valley, the team is jumping into the data collection business by offering fans incentives like flying in the team charter Read more…

Commerce Department to Hire Data Czar

Jul 23, 2014 |

In an effort to make greater use of government data, the U.S. Commerce Department is hiring its first chief data officer and convening an industry panel to make agency data more accessible. Several agencies within the Commerce Department like the National Weather Service produce and disseminate huge volumes of meteorological data that are used by private enterprises like the Weather Channel and AccuWeather. “The Department of Commerce is working to unleash more of its data to strengthen the nation’s economic Read more…

Auto Data Being Culled to Understand Driver Preferences

Jul 21, 2014 |

Automotive electronics that have enabled the installation of what amount to networks in vehicles are delivering a raft of new telematics and other features for drivers. These systems are also generating huge volumes of data – 25 gigabytes an hour, according to one estimate – that auto makers are struggling to harness as a way to better understand customer trends and preferences. Call centers, sales and marketing were the traditional methods used to gather customer data. Emerging big data platforms Read more…

This Just In

ProfitBricks Accelerating Big Data Projects with SDN and Infiniband

Apr 2, 2014 |

ProfitBricks, the price/performance leader in Cloud Computing IaaS, is now helping big data systems integrators like Altoros bring the power of the cloud to the high demand of its customers. Thanks to second generation technologies like software-defined networking (SDN) and InfiniBand that are at the heart of the ProfitBricks cloud, Altoros customers can easily deploy Hadoop clusters for both temporary projects and continuous big data analysis programs.

Dell Acquires StatSoft

Mar 24, 2014 |

Dell today announced the acquisition of StatSoft, a leading provider of advanced analytics solutions that deliver a wide range of data mining, predictive analytics and data visualization capabilities. StatSoft combines comprehensive statistical analysis with advanced analytics to help organizations better understand their businesses, predict change, increase agility and control critical systems.

TDWI Checklist Report Helps Enterprises Explore Big Data Analytics

Mar 13, 2014 |

TDWI Research has released its newest Checklist Report, Utilizing Big Data Analytics with Hadoop. The report examines how enterprises can gain competitive advantage using advanced analytics and how several technologies are coming together to form the fabric of an analytics ecosystem.

UC San Diego Offering Big Data Seminar

Mar 6, 2014 |

The significance and real-world impact of Big Data — the popular term used to describe the exponential growth and availability of data, both structured and unstructured — will be explored at a specia lUniversity of California San Diego Extension seminar titled “Big Data at Work: A Conversation with the Experts.”

Glassbeam and Hitachi Data Systems Expand Partnership

Feb 19, 2014 |

Glassbeam, Inc., the machine data analytics company, today announced a global OEM agreement with Hitachi Data Systems Corporation (HDS). Glassbeam’s flexible any data, any device, any cloud platform delivers advanced machine data analytics that provides OEM partners like HDS proactive information to manage storage, servers, networks and other IT assets for their enterprise customers.