
Tag: data pipeline
Data Integration and Observability Provider Crux Nabs $50m in Funding
Crux, a cloud-based provider of data integration and observability tools that claims to have more than 250 data connectors, today announced that it raised $50 million in a Series B round of venture capital. The San Franc Read more…
AWS Seeks an End to ETL
Extract, transform, and load. It’s a simple and ubiquitous thing in IT. And yet everybody seems to hate it. The latest company to pile on to ETL is AWS, which declared an effort to end ETL yesterday at re:Invent. Ad Read more…
How Snowplow Breaks Down Data Barriers
If you have suspicions about your data, you’re not alone. The AI and data analytics dreams of many a company have been broken by poor data management and defective ETL pipelines. But by enforcing data schema at the poi Read more…
A Data Platform for Chatbot Development
One of the most compelling use cases for AI at the moment is developing chatbots and conversational agents. While the AI part of the equation works reasonably well, getting the training data organized to build and train Read more…
CI/CD Pipeline: 7 Advantages To A Continuous Integration Approach to Data Pipelines
When it comes to modern software development, it’s not surprising that companies have a need for speed. But if you develop software too quickly, it can mean sacrificing quality, security and compliance. DevOps and con Read more…
Exploring the Top Options for Real-Time ELT
Competitive advantage in today’s world rests on a company’s ability to innovate and adapt to a rapidly changing environment. To do that, organizations must adopt real-time thinking in the way they approach the design Read more…
Airflow Available as a New Managed Service Called Astro
Companies can now get an Apache Airflow data orchestration environment up and running in less than an hour via Astro, a new managed service launched today by Astronomer, the commercial entity behind the popular open-sour Read more…
Building Continuous Data Observability at the Infrastructure Layer
Data is the lifeblood of business today, but getting it where it needs to go is hard, especially as data volumes grow. Data pipelines become the repeatable method for moving this digital crude, but monitoring the flows f Read more…
Monte Carlo Hits the Circuit Breaker on Bad Data
Data pipelines are critical conduits of information for data-driven companies. But what happens when the data in the pipeline becomes corrupted? In some situations, you want to immediately stop the flow of data, which is Read more…
Databricks Ships New ETL Data Pipeline Solution
Databricks today announced the general availability (GA) of Delta Live Tables (DLT), a new offering designed to simplify the building and maintenance of data pipelines for extract, transform, and load (ETL) processes usi Read more…
ETL Tool Apache Hop Graduates Incubator
Apache Hop, a metadata-driven data orchestration tool used to design and build pipelines, today emerged from incubator status and was named a Top-Level Project at the Apache Software Foundation, clearing the way for more Read more…
Inside AutoTrader UK’s Data Observability Pipeline
In the course of shifting its analytics estate to the cloud, AutoTrader UK has adopted many new tools and technologies, including BigQuery, Looker, and dbt, which have helped to democratize data access among users. Along Read more…
Bad Data Pipelines Costing Companies Big, Fivetran Finds
Stop us if you’ve heard this one before: Overworked data engineer builds faulty data pipeline, which leads to bad data, which leads to a bad outcome. It may be the same old song, but it’s also the current state of af Read more…
Hands-Off: Manual Data Integration Tasks Plummeting, Gartner Says
While the need to integrate data has never been greater, the addition of machine learning and other forms of automation is driving a large reduction in the amount of manual data management tasks that human workers are re Read more…
The Data Mesh Emerges In Pursuit of Data Harmony
The data mesh is a new concept that’s emerging in big data circles. Similar in some respects to data fabrics, the data mesh provides a way to reconcile and hopefully overcome the challenges posed by previous data archi Read more…
Data Pros Are Maxed Out: Survey
The demands of the data-driven life are wearing on data professionals, according to survey results unveiled this week by data pipeline automation solution developer Ascend.io, which found that 96% of data professionals a Read more…
50 Years Of ETL: Can SQL For ETL Be Replaced?
It’s hard to imagine data warehousing without ETL (extract, transformation, and load). For decades, analysts and engineers have embraced no-code ETL solutions for increased maintainability. Does this mean that Struct Read more…
Future Proofing Data Pipelines
Data pipelines are critical structures for moving data from its source to a destination. For decades, companies have used data pipelines to move information, such as from transactional with analytic systems. However, as Read more…
Prophecy Spins Up Low-Code Data Pipeline Tool
In recent years, the shortage of data engineers has at times exceeded the shortage of data scientists. To help close the gap, a Silicon Valley startup called Prophecy today unveiled a low-code data engineering tool that Read more…