DataForge Open-Sources First Functional Data Programming Framework

CHICAGO, May 8, 2024 — DataForge, a leading data integration provider, has open-sourced a new framework for developing and managing data transformation code: DataForge Core. Tailor-made for high-growth companies building rapidly evolving data products, DataForge Core has introduced modern computer science principles to data engineering and defined the future of data platform development and transformation code management.

Until now, the only option for data teams was to build monolithic data platforms using coding patterns dating back to the 1970s, resulting in data platforms that grow exponentially in complexity and cost as demands for data and analytics increase. Enter DataForge Core. By using functional structures and other modern software engineering concepts, DataForge Core breaks up the monolith and accelerates data product development. By changing the foundational paradigms of the code, DataForge Core allows for easy expansion and growth of data platforms to keep up with the accelerating demand for data, reporting, and AI within modern enterprises.

“This groundbreaking shift eliminates the most tedious data plumbing chores and makes data engineering exciting again,” explained Vadim Orlov, DataForge co-founder and CTO. “It enables Data Engineers to direct their complete attention towards generating business value from data.”

DataForge Core makes developing data products easier than ever:

• Faster development cycles: With functional programming, it is easier to convert business logic into code and add to existing code as requirements change. Accelerate your data team’s cycles by eliminating tedious and manual tasks such as orchestration and data backfill, increasing the ability to quickly meet new demands for data sets, insights, and advanced analytics.
• Enable AI and ML: With simple and easy to follow patterns, Data Scientists can write high-quality data pipelines for feature engineering and batch inference. With native integration with Spark SQL and Databricks, DataForge Core provides the structure to allow Data Scientists to focus on model development and avoid creating a quagmire of data preparation scripts.
• Governance and Auditability: DataForge Core includes a metadata repository that stores a compiled copy of the code in database tables. This means that even with thousands of lines, it is easy to quickly find a single code snippet by writing a SQL query, allowing teams in high-compliance environments to perform code audits and lineage analysis.

“By bringing DataForge Core to the open-source community, we are reaffirming our belief that innovation happens through collaboration, not isolation,” said Matt Kosovec, co-founder and CEO of DataForge. “We have just scratched the surface of what is possible by thinking differently and believe we will need the help of both data engineering and computer science communities to evolve DataForge quickly enough to keep up with the demand for data and AI products.”

About DataForge

DataForge‘s mission is to make data management, integration, and analysis faster and easier than ever. DataForge, the Declarative Data Management platform, automates data transformation, orchestration, and observability. By bringing functional programming to data engineering, DataForge introduces a new paradigm for building data solutions. Avoid the pitfalls of procedural scripting and take advantage of modern software engineering principles to automate orchestration, promote code reuse, and maximize observability. Experience a new era of data engineering with DataForge, where functional programming and automation pave the way for scalable data platforms.

Source: DataForge