People to Watch 2023 – Karthik Ramasamy
Every year, real time data processing is predicted to go mainstream, but so far it hasn’t broken out of its niche status. Will 2023 be different, and if so, why?
At Databricks, we predict 2023 is going to be yet another great year for real time data processing. Streaming workloads on our platform have been growing at 140-150% YoY (as presented in Data + AI summit 2022) and we are running more than 7 million of them. The launch of Delta Live Tables (DLT) makes streaming extremely simple, using declarative language like SQL and automated operations. It is definitely going mainstream.
What will be the biggest impediments to success with stream data processing in 2023? What are the biggest technical or business hurdles?
One of the biggest challenges will be around new APIs and languages to learn. It’s difficult to enable existing data teams when they’re so familiar with the languages and tools they already know. Another challenge is the need to build the complex operational tooling required to deploy and maintain streaming data pipelines that run reliably in customers’ production environments. Finally, real time and historical data often live in separate systems, and incompatible governance models can limit the ability to control access for the right users and groups.
Databricks wants to be the one-stop-shop for data analytics, machine learning, and stream processing. Why will it succeed?
The lakehouse architecture is key to success because all the data is stored in a common format. Databricks provides tightly integrated solutions for different types of data processing with a well-known compute engine that is based on open source Apache Spark. In the context of data streaming, Databricks’ Lakehouse offers a single platform for streaming and batch data so data teams can eliminate silos and centralize their security and governance models.
Databricks enables data engineers, data scientists and analysts to easily build streaming data workloads with the languages and tools they already know and with the APIs they already use. We simplify development and operations by leveraging out-of-the-box capabilities that automate much of the production aspects associated with building and maintaining real-time data pipelines.
Outside of the professional sphere, what can you share about yourself that your colleagues might be surprised to learn – any unique hobbies or stories?
My favorite hobby is photography. I took a class while in grad school about how to compose what goes in a photo and how to get the correct settings. I mainly shoot photographs of natural scenic beauties. I started with a Nikon SLR film camera and graduated to using slides and then moved to digital SLR. Now phone cameras are so advanced that I just carry my iPhone.
March 28, 2024
- MineOS Unveils AI Asset Discovery
- Cloudera Survey Reveals 90% of IT Leaders Believe that Unifying the Data Lifecycle on a Single Platform is Critical for Analytics and AI
- Snowflake Enhances Secure, Cross-Cloud Collaboration for High Value Business Outcomes with Snowflake Data Clean Rooms
- Domo Announces Winners of the 2024 Community Ovation Awards
March 27, 2024
- New MLPerf Inference Benchmark Results Highlight the Rapid Growth of Generative AI Models
- Qlik Advances Real-time Data Analytics with Solace PubSub+ Platform Integration
- Samsung Unveils Expanded CXL Memory Module Portfolio at Memcon 2024, Enhancing AI and HPC
- Celestial AI Closes $175M Series C Funding Round Led by US Innovative Technology Fund
- Databricks Launches DBRX: A New Standard for Efficient Open Source Models
- Astronomer Unveils New Capabilities in Astro to Streamline Enterprise Data Orchestration
- DataVisor Introduces Enhanced Anti-Money Laundering Solution to Support Financial Institutions
March 26, 2024
- Dremio Announces General Availability on Microsoft Azure
- Elastic Study Highlights Soaring Optimism in Generative AI Investment Despite Data and Security Challenges
- Dataiku and PwC Bring Practical AI Solutions to Regulated Industries
- HPE Leverages GenAI to Enhance AIOps Capabilities of HPE Aruba Networking Central Platform
- DataStax and Microsoft Collaborate to Make it Easier to Build Enterprise Generative AI and RAG Applications with Legacy Data
- SQream and Qantm AI Partner to Solve Enterprise Data and AI Challenges
- Neo4j Announces Collaboration with Microsoft to Advance GenAI and Data Solutions
- InterSystems Expands IRIS Data Platform with Vector Search to Support Next-Gen AI Applications
March 25, 2024
Most Read Features
Sorry. No data so far.
Most Read News In Brief
Sorry. No data so far.
Most Read This Just In
Sorry. No data so far.
Sponsored Partner Content
-
Supercharge Your Data Lake with Spark 3.3
-
Learn How to Build a Custom Chatbot Using a RAG Workflow in Minutes [Hands-on Demo]
-
Overcome ETL Bottlenecks with Metadata-driven Integration for the AI Era [Free Guide]
-
Gartner® Hype Cycle™ for Analytics and Business Intelligence 2023
-
The Art of Mastering Data Quality for AI and Analytics
Sponsored Whitepapers
Contributors
Featured Events
-
Data Universe
April 10 - April 11New York United States -
Call & Contact Center Expo
April 24 - April 25Las Vegas NV United States -
AI & Big Data Expo North America 2024
June 5 - June 6Santa Clara CA United States -
AI Hardware & Edge AI Summit 2024
September 10 - September 12San Jose CA United States