Follow Datanami:

Tag: stream processing

Fueled by Kafka, Stream Processing Poised for Growth

Once a niche technique used only by the largest organizations, stream processing is emerging as legitimate technique for dealing with massive amounts of data generated every day. While it's not needed for every data chal Read more…

Managing Streaming Flink Apps Is About To Get Easier

Apache Flink has emerged as a powerful platform for building real-time stream processing applications. However, not every organization has the resources to go all in on Flink the way Netflix, Uber, and Alibaba have. That Read more…

A Peek Inside Kafka’s New ‘Exactly Once’ Feature

Here's some great news for Apache Kafka users: The open source software will support exactly once semantics for stream processing with the upcoming version 0.11 release, thereby eliminating the need for application devel Read more…

Yahoo’s Massive Hadoop Scale on Display at Dataworks Summit

Yahoo put its massive Hadoop investment on display this week at Dataworks Summit, the semi-annual big data conference that it co-hosts with Hortonworks. While Hadoop is no longer the conference headliner that it once Read more…

Hortonworks Shifts Focus to Streaming Analytics

Hortonworks started life providing a Hadoop distribution that allowed customers to process big data at rest. But these days, the company has shifted its much of its attention and resources to streaming analytics, or proc Read more…

Sparse Fourier Transform Gives Stream Processing a Lifeline from the Coming Data Deluge

When James Cooley and John Tukey introduced the Fast Fourier transform in 1965, it revolutionized signal processing and set us on course to an array of technological breakthroughs. But today's overwhelming data sets requ Read more…

How Pandora Uses Kafka

As a big Hadoop user, Pandora Media is no stranger to distributed processing technologies. But when the music streaming service decided to transition its ad tracking system from a batch-oriented system into a real-time o Read more…

Google/ASF Tackle Big Computing Trade-Offs with Apache Beam 2.0

Trade-offs are a part of life, in personal matters as well as in computers. You typically cannot have something built quickly, built inexpensively, and built well. Pick two, as your grandfather would tell you. But appare Read more…

The Real-Time Future of ETL

We're on the cusp of a huge uptick in data generation thanks to the IoT, but most of that data will never be landed in a central repository or stored for any length of time. To get a handle on this morass of data, enterp Read more…

Kafka ‘Massively Simplifies’ Data Infrastructure, Report Says

What's behind the rapid rise in Apache Kafka? According to a new survey of Kafka users by Confluent, the commercial venture behind Kafka, the data pipeline's capability to "massively simplify" the IT infrastructure of en Read more…

Flink Aims to Simplify Stream Processing

Apache Flink has emerged as a powerful framework for building real-time stream processing applications that has gained traction by some of the most progressive tech companies in the world, including at Netflix, Uber, and Read more…

How Kafka Redefined Data Processing for the Streaming Age

The Apache Kafka phenomenon reached a new high today when Confluent announced a $50 million investment from the venture capital firm Sequoia. The investment signals renewed confidence that Kafka is fast becoming a new an Read more…

Exactly Once: Why It’s Such a Big Deal for Apache Kafka

Organizations building real-time stream processing systems on Apache Kafka will be able to trust the platform to deliver each messages exactly once when they adopt new Kafka technology planned to be unveiled this spring, Read more…

Kafka Gets Server, API Upgrades

The latest feature release from the Apache Kafka stream processing community includes server support for time-based search along with more than 200 bug fixes and other improvements. Confluent Inc., founded by the team Read more…

Acquisition Validates Concord’s Event-Based Approach

We suggested during the summer that companies looking for a stream-processing engine for fast data applications featuring high-throughput and low-latency may want to check out startup Concord Systems Inc. Someone did: Read more…

MemSQL Delivers an ‘Exactly Once’ Real-Time Pipeline

MemSQL today unveiled a new release of its in-memory relational database that can process a real-time flow of messages from Apache Kafka using "exactly once" semantics. The NewSQL database accomplished the feat by creati Read more…

Flink Distro Now Available from data Artisans

The company behind Apache Flink, data Artisans, today launched the first commercial distribution of the upstart stream processing engine. The new software product, called the dA Platform, is identical to the open source Read more…

Apache Flink Gears Up for Emerging Stream Processing Paradigm

We're close to the next release of Apache Flink, the stream processing engine developed by the Apache Software Foundation. Flink version 1.1.0 will bring new SQL interface for working with streaming data, while bigger ch Read more…

Concord Claims 10x Performance Edge on Spark Streaming

Organizations that are looking for a stream processing engine upon which to build fast data applications featuring high-throughput and low-latency may want to check out Concord, a new framework that emerged from the ad-t Read more…

Merging Batch and Stream Processing in a Post Lambda World

It wasn't long ago that developers looked to the Lamba architecture for hints on how to design big data applications that needed elements of both batch and streaming data. But already, the Lamba architecture is falling o Read more…

Datanami