Follow Datanami:
June 15, 2016

The Weather Company Absorbs ‘Deep Thunder’ to Bolster Modeling

(Fesus Robert/Shutterstock)

IBM is combining two of its weather forecasting resources—including The Weather Company’s regional and global models and local and hyperlocal forecasts from “Deep Thunder”—to create a single combined forecasting model that executives say will be more accurate than what came before.

Deep Thunder has been an IBM Research project for 20 years, and has been used to produce extremely accurate forecasts for specific events or specific places, such as Brazilian monsoons in 2010, Hurricane Irene’s landfall in New York in 2011, and airflow in Chinese windfarms. Deep Thunder can predict what the weather will be with a 0.2 mile resolution in 10-minute intervals going out three or four days. By comparison, the National Weather Service’s forecast model looks at the country across a grid of 12km squares in one-hour increments going out 10 days.

Deep Thunder is generally more accurate than what The Weather Company provides today, but that’s largely due to TWC’s focus, which is providing global and regional weather forecasts. From a global view, what happens in one valley compared to the next is not critically important.

But now that Deep Thunder lives under the care of TWC, the two teams will share notes, with the goal of improving the organization’s weather forecasts, says Mary Glackin, head of forecasting for TWC.

“When I look at IBM Research and the efforts they put in, while they use the same physics package that we use in TWC, they’ve really been focused on local and hyper-local scales,” Glackin says. “We had different areas of emphasis. We had two complementary efforts here. And when I think about it, the whole is going to be greater than the sum of its parts.”

Three elements of Deep Thunder really stand out to Glackin, an accomplished leader in meteorology who spent 30 years at NOAA before joining TWC in 2015. First is the attention that IBM researchers paid to tuning the models that deliver such fine-grained accuracy.


Deep Thunder’s forecast model provides better vertical resolution than what The Weather Channel previously offered

Glackin also likes how the Deep Thunder team assimilates data into models. “They’ve done not only some of the assimilation we do at TWC, and done it better, but they also are assimilating data sets we hadn’t been assimilating,” she tells Datanami. “That does make a difference ultimately in the accuracy of the forecast.”

Lastly, the way the Deep Thunder team couples their forecasts with machine learning models to answer real-world questions has impressed Glackin. This helps in areas such as predicting power output from a wind farm based on short-term weather forecasts.

“They’ve done a really nice job coupling those supply and demand energy models,” Glackin says. “When we bring those capabilities into the The Weather Channel, we’ll be able to scale to various industry sectors in a really effective manner.”

From a practical standpoint, the supercomputers that run Deep Thunder and The Weather Company’s model (called RPM) will remain separate, for the time being, Glackin says. Deep Thunder will continue to run on a supercomputer in IBM’s lab in Poughkeepsie, New York, while RPM will continue to run on IBM high performance computing (HPC) systems in Andover, Massachusetts. (The models that generate 20 billion or so weather forecasts for consumers are hosted on the Amazon Web Services cloud.)

The TWC and Deep Thunder teams will share and blend forecast models to serve clients. IBM has done a lot of work on building models that incorporate data from soil-moisture sensors and from bodies of water. The existence of water, whether in the ground or on the ground, can impact weather in small but noticeable ways.

For example, if the goal is to better predict air pollution around Lake George (a large lake in the Adirondack Mountains of Upstate New York), then Deep Thunder’s superior hydrologic models will be used to get a better handle on the impact that all that water has on weather and the creation of potential “dead zones” that can trap air pollution, Glackin says.

In the Lake George example, TWC may want to use the entire 1/3 of a kilometer resolution that Deep Thunder is capable of delivering, while other times, that resolution may not be necessary. “You have some dials on this, and you can dial in what you need,” Glackin says. “[But] you don’t want to overkill it with resolution that doesn’t actually help.”

The real payoff is going to be in business to business applications in various sectors, Glackin says, such as predicting pop-up thunderstorms around major airports and in major air corridors in the Northeast.

shutterstock_airline_Cardinal arrows

Weather data collected in real-time from airliners is playing an increasingly important role in forecast models (Cardinal Arrows/Shutterstock)

“It is going to be noticeable running these models at finer resolution, both in time and in space to support critical decision-making,” she says. “We know from looking at use cases, when you’re really focused on an area of the world and you really doing a really good job managing your data going in, you get a better answer than a large regional solution or a global solution. We know that will make a difference.”

Deep Thunder will also gain access to a treasure trove of observational data that the TWC has access to through about 300,000 personal weather stations connected through the Weather Underground. This on-the-ground data is proving to be very valuable to forecasting weather in some regions of the globe that may not have sophisticated government-backed observation networks, such as in Asia and Africa.

And TWC is working on getting even more data to feed the newly combined Deep Thunder-RPM model. TWC is already pulling real-time air turbulence data from aircraft through partnerships with airlines. But in the future, the models could be assimilated with data from IoT-type use cases, such as by pulling barometric pressure data observations that are taken from people’s smartphones, or detecting when windshield wipers are running on cars. (Doppler radar can see rain in the clouds, but it isn’t very good at detecting rain actually hitting the ground).

“We’re looking at other creative ways to extract environmental data from other sources out there,” Gacklin says. “I think this is one of the existing things going forward. Leveraging those rich data sets will help, particularly in the short term.”

Related Items:

Big Data Heads to Orbit with Weather Sats

Data ‘Assimilation’ Seen as Weak Link in U.S. Weather Forecasts

Farmers Plant for Hyper-Local Forecasts with IBM’s ‘Deep Thunder’