

via Shutterstock
The shipping industry is the lifeblood of the world economy, moving 90% of merchandise valued at over $19 trillion in 2014, according to the World Trade Organization. However, the forecasts relied on by many shipping companies leave something to be desired. Now, with big data analytics settling into the wheelhouse, the industry is set to make a sizable leap in efficiency.
It’s easy to forget how quickly the field of big data analytics has arisen, and what steps a given organization must take to be in a position to take advantage of it. For industries like financial services that digitized their process long ago, the move up to using predictive analytics was not a traumatic one. In healthcare, the passage of the American Recovery and Reinvestment Act in 2009 carried the mandate to shift to electronic medical records by 2014, which greased the wheels for the adoption of big data analytics in medicine.
But some industries are far behind the digital curve. The shipping industry, in particular, has lagged behind other businesses in moving beyond paper-based processes and using centralized computers to automated key processes. To be sure, there are big, customized ERP systems in place at many of the biggest shippers. However, many of the smaller ones have only dipped their toes into the digital waters. The challenge is formidable, to be sure, but that makes the pay-off potentially much bigger.
Industry leaders are stepping up to confront the challenges and opportunities that digitization and big data present to shipping. The advent of GPS navigation, real-time weather data feeds, and RFID-equipped smart containers promise to transform the shipping business, optimizing the delivery of goods and the distribution of containers, according to Dieter Berg, the head of business development for Munich Re Global Marine Partnership.
“I’m convinced that the entire field of traffic and port logistics as well as just-in-time shipping will change,” Berg writes in a story on the impact of big data and the Internet of Things (IoT) on his company’s website.
Container Optimization

World shipping traffic (Copyright B.S Halpern/Wikimedia Commons)
Multi-stop trip routing in is a notoriously difficult problem to solve, and it’s no different in the shipping business. There are more than 100,000 ships in the maritime shipping fleet, ferrying merchandise-filled containers and tanks full of bulk cargo to ports thousands of miles across the ocean. In 2012, it was estimated there were 20.5 million standard containers, around the world, accounting for 60% of the world’s seaborne trade.
One company that’s aiming to optimize the flow of containers around the world is the Silicon Valley startup ClearMetal. Founded two years ago by data scientists and engineers from Stanford University and Google, ClearMetal takes a math-based approach to predicting what containers should be where, and when.
“We’re a data science and artificial intelligence company,” says Adam Compain, ClearMetal CEO and co-founder. “We predict all events in the logistics cycle with precision. By doing that, we’re able to provide different parties in the supply chain – whether it’s a shipper, retailers, or freight forwarders or terminals – with actionable insights into how they can better manage their operations.”
Forecast: Not So Excel-lent
Among the various parties in the global supply chain, much of the planning for container placement is done in off-the-shelf forecasting tools, Excel, or worse–between the ears of an experienced manager.

The largest “megaship” can hold 18,000 containers (Alex Kolokythas Photography/Shutterstock)
“Excel and intuition are not that sophisticated in managing the kinds of variability there are in all the elements that go into operations,” Compain says. “It’s simply not really a sharp knife. They’re top-down approaches and static models. They look at overall averages, pump data through a static model, and come out with a forecast.”
Those forecasts give the average shipper less-than-stunning results when it comes to positioning a fleet of 100,000 to a million shipping containers in the optimal position. The shippers need to decide whether to keep a container sitting in a certain location, in hopes that a retailer will fill it up and move it, or to re-position it to a location where it’s more likely to be utilized.
Getting the right mix of containers in space and time is not easy, especially considering the vagaries of global commerce. “There’s massive complexity and uncertainty around how customers are going to behave, how markets are going to move, and how the operations are going to perform.”
Machine Learning to the Rescue
Rather than take an aggregate approach to predicting container demand, ClearMetal uses machine learning and data science techniques to forecast demand at a much more granular level, right down to the individual shipping container.
It all starts, as every data science project does, with the data. ClearMetal feeds its random forest models with data around booking information, vessel voyage information, and container movement data. The company uses these models to power Monte Carlo simulations that try to predict exactly where and when containers will be needed at each leg of their journeys. The models, which run on Amazon’s AWS cloud, provide up to an eight-week window into the future.

ClearMetal data sources
“Our system is actually looking at every piece of the shipping cycle in a way that’s tailored to the shipment industry to come up with much more accurate and dynamic real-time model and forecast of what will happen,” Compain says.
ClearMetal seeks to provide specific answers to specific questions. “As a carrier we’d provide them an application that helps them underrated exactly where they have containers, where they need them, and what their imbalance is,” he continues. “For freight-forwarders, it’s an aggregation helping them understand, of the bookings they get from shipper customers, what will actually dispatch and ship and precisely where. And for a terminal, it might be predictions and applications around exactly when a vessel will arrive and berth, or how containers might flow in and out of their gate.”
In proof of concepts, ClearMetal has demonstrated that it can deliver container demand predictions that are 30% to 60% more accurate than the predictions shippers are currently using. For a large shipping company with hundreds of thousands of containers, that can translate into a savings well into the millions of dollars. All told, the company estimates more than $2 billion could be saved by optimizing the shipping supply chain.
ClearMetal is tackling just one aspect of the massive shipping industry. But as more sensors are installed and companies become more comfortable using the Internet of Things (IoT), there are a number of other applications that are possible to boost the efficiency of logistics operations, especially in the areas of security, safety, just-in-time shipping, and food spoilage.
Related Items:
How GPU-Powered Analytics Improves Mail Delivery for USPS
Exploring New Arenas for Geospatial Data
July 3, 2025
- FutureHouse Launches AI Platform to Accelerate Scientific Discovery
- KIOXIA AiSAQ Software Advances AI RAG with New Version of Vector Search Library
- NIH Highlights AI and Advanced Computing in New Data Science Strategic Plan
- UChicago Data Science Alum Transforms Baseball Passion into Career with Seattle Mariners
July 2, 2025
- Bright Data Launches AI Suite to Power Real-Time Web Access for Autonomous Agents
- Gartner Finds 45% of Organizations with High AI Maturity Sustain AI Projects for at Least 3 Years
- UF Highlights Role of Academic Data in Overcoming AI’s Looming Data Shortage
July 1, 2025
- Nexdata Presents Real-World Scalable AI Training Data Solutions at CVPR 2025
- IBM and DBmaestro Expand Partnership to Deliver Enterprise-Grade Database DevOps and Observability
- John Snow Labs Debuts Martlet.ai to Advance Compliance and Efficiency in HCC Coding
- HighByte Releases Industrial MCP Server for Agentic AI
- Qlik Releases Trust Score for AI in Qlik Talend Cloud
- Dresner Advisory Publishes 2025 Wisdom of Crowds Enterprise Performance Management Market Study
- Precisely Accelerates Location-Aware AI with Model Context Protocol
- MongoDB Announces Commitment to Achieve FedRAMP High and Impact Level 5 Authorizations
June 30, 2025
- Campfire Raises $35 Million Series A Led by Accel to Build the Next-Generation AI-Driven ERP
- Intel Xeon 6 Slashes Power Consumption for Nokia Core Network Customers
- Equal Opportunity Ventures Leads Investment in Manta AI to Redefine the Future of Data Science
- Tracer Protect for ChatGPT to Combat Rising Enterprise Brand Threats from AI Chatbots
June 27, 2025
- Inside the Chargeback System That Made Harvard’s Storage Sustainable
- What Are Reasoning Models and Why You Should Care
- Databricks Takes Top Spot in Gartner DSML Platform Report
- LinkedIn Introduces Northguard, Its Replacement for Kafka
- Change to Apache Iceberg Could Streamline Queries, Open Data
- Agentic AI Orchestration Layer Should be Independent, Dataiku CEO Says
- Why Snowflake Bought Crunchy Data
- Fine-Tuning LLM Performance: How Knowledge Graphs Can Help Avoid Missteps
- Top-Down or Bottom-Up Data Model Design: Which is Best?
- The Evolution of Time-Series Models: AI Leading a New Forecasting Era
- More Features…
- Mathematica Helps Crack Zodiac Killer’s Code
- ‘The Relational Model Always Wins,’ RelationalAI CEO Says
- Confluent Says ‘Au Revoir’ to Zookeeper with Launch of Confluent Platform 8.0
- DuckLake Makes a Splash in the Lakehouse Stack – But Can It Break Through?
- Solidigm Celebrates World’s Largest SSD with ‘122 Day’
- The Top Five Data Labeling Firms According to Everest Group
- Supabase’s $200M Raise Signals Big Ambitions
- Toloka Expands Data Labeling Service
- With $17M in Funding, DataBahn Pushes AI Agents to Reinvent the Enterprise Data Pipeline
- Databricks Is Making a Long-Term Play to Fix AI’s Biggest Constraint
- More News In Brief…
- Astronomer Unveils New Capabilities in Astro to Streamline Enterprise Data Orchestration
- Databricks Unveils Databricks One: A New Way to Bring AI to Every Corner of the Business
- Seagate Unveils IronWolf Pro 24TB Hard Drive for SMBs and Enterprises
- Gartner Predicts 40% of Generative AI Solutions Will Be Multimodal By 2027
- BigBear.ai And Palantir Announce Strategic Partnership
- Astronomer Introduces Astro Observe to Provide Unified Full-Stack Data Orchestration and Observability
- Databricks Donates Declarative Pipelines to Apache Spark Open Source Project
- Deloitte Survey Finds AI Use and Tech Investments Top Priorities for Private Companies in 2024
- Code.org, in Partnership with Amazon, Launches New AI Curriculum for Grades 8-12
- Databricks Announces Data Intelligence Platform for Communications
- More This Just In…