Follow Datanami:

Tag: data transformation

J.P. Morgan Launches ‘Containerized Data’ Solution in the Cloud

Getting access to consistent, high-quality data ranks as one of the toughest challenges in big data, advanced analytics, and AI. It’s a challenge that is being taken up by Fusion by J.P. Morgan with its new Containeriz Read more…

Matillion Bringing AI to Data Pipelines

Data engineers historically have toiled away in the virtual basement, doing the dirty work of spinning raw data into something usable by data scientists and analysts. The advent of generative AI is changing the nature of Read more…

DataForge Sets New Standard for the Future of Data Platforms

Data engineering often requires the utilization of SQL scripting for data transformation within the database. However, this can result in lengthy scripts, recurring copy-paste patterns, the need for schema changes across Read more…

Data Quality Getting Worse, Report Says

For as long as “big data” has been a thing, data quality has been a big question mark. Working with data to make it suitable for analysis was the task that data professionals spent the bulk of their time doing 15 yea Read more…

Tristan Handy’s Audacious Vision of the Future of Data Engineering

Tristan Handy is a lot of things: co-creator of dbt, founder and CEO of dbt Labs, and self-described “startup person.” But besides leading dbt Labs to a $4 billion valuation, he is one more thing: An audacious dreame Read more…

dbt Labs Tackles Data Project Complexity with Mesh at Coalesce

Adoption of dbt has skyrocketed since it was launched just seven years ago, and today, more than 30,000 organizations use the open source software in production. The complexity of data projects has also increased, with 5 Read more…

Starburst Brings Dataframes Into Trino Platform

Starburst customers who prefer to manipulate data using dataframes as opposed to regular SQL will be happy with a pair of announcements made today. That includes the introduction of PyStarburst, which provides a PySpark- Read more…

In Search of Data Model Repeatability

Everybody wants to be data-driven--that much is clear. But that desire doesn’t necessarily translate into real business results, especially in competitive industries like ecommerce. Data quality has long been a burr Read more…

Semantic Layer Belongs in Middleware, and dbt Wants to Deliver It

The folks at dbt Labs think the data transformation tool is the proper place to implement and manage a semantic data layer, as opposed to the BI tool or the data warehouse, where it has traditionally resided. And later t Read more…

Data Integration and Observability Provider Crux Nabs $50m in Funding

Crux, a cloud-based provider of data integration and observability tools that claims to have more than 250 data connectors, today announced that it raised $50 million in a Series B round of venture capital. The San Franc Read more…

Inside AutoTrader UK’s Data Observability Pipeline

In the course of shifting its analytics estate to the cloud, AutoTrader UK has adopted many new tools and technologies, including BigQuery, Looker, and dbt, which have helped to democratize data access among users. Along Read more…

8 Key Considerations for Embarking on a Data Integrity Journey

Modern enterprises are reliant on data, and as the volume of it increases, making that data useful is absolutely critical. However, more data also means the likelihood of incomplete, inconsistent data sets is on the rise Read more…

The ‘Rage Design’ Behind Flatfile’s Onboarding Success

David Boskovic was excited to join a company called Envoy back in 2016. He had worked with B2B startups since he was 18, and was looking forward to helping another tech startup scale an idea. But that excitement turned t Read more…

AWS Tackles Real-Time Data Transformation with S3 Object Lambda

Cloud object stores like S3 have become the default storage repository for many companies. But operational challenges arise when one tries to use a single object store as a universal repository for multiple applications, Read more…

Cloud Is the New Center of Gravity for Data Warehousing

The great migration of data into the cloud didn’t start in 2020, but it certainly accelerated throughout the year. And according to a new survey from IDG, the overwhelming majority of companies are planning to expand t Read more…

Informatica Likes Its Chances in the Cloud

Quick: Name a company that made its name in the 1990s and 2000s by providing data integration tools for enterprise analytics running in on-prem data centers, but has since pivoted the cloud and was even named Snowflake� Read more…

Why You Need Data Transformation in Machine Learning

Thanks to machine learning and the advancements in software and technology, enterprises can now process and understand their data much faster using modern tools with established algorithms. This effectively allows them t Read more…

How ML Helps Solve the Big Data Transform/Mastering Problem

Despite the astounding technological progress in big data analytics, we largely have yet to move past manual techniques for important tasks, such as data transformation and master data management. As data volumes grow, t Read more…

The Seven Sins of Data Prep

Data preparation is often considered a necessary precursor to the “real” work found in visualizing or analyzing data, but this framing sells data prep short. The ways in which we cleanse and shape data for downstream Read more…

Data Engineers in Hot Demand

The big data community has been dealing with the data scientist shortage ever since big data became a thing. Now we're learning that there's possibly an even bigger shortage of another type of data professional: the data Read more…

Datanami