In-Memory Analytics to Boost Flight Ops for Major US Airline
A major US airline is about to go live on a new in-memory analytics system that will help it better anticipate and respond to disruptions in flight operations, potentially saving it tens of millions of dollars per year and improving the experience of its passengers, according to its developer.
Flying a modern airliner is an expensive affair. According to airlines.org, direct operating costs in items such as fuel, crew, and maintenance runs about $81 per minute, or about $20,000 for a four-hour flight. When flights are delayed due to storms, mechanical breakdowns, or other events, the costs add up quickly. In fact, when the passenger’s time (calculated at $45 per hour) is added in, delays cost $9 billion in 2014, the industry group says.
Some delays, such as those from bad weather, cannot be avoided. But there is a lot of room for improvement in how airlines deal with the delays that will inevitably crop up during day-to-day flight operations.
One company at the forefront of implementing real-time analytics in the airline industry is Gigaspaces. Adi Paz, the general manager of the IMC business unit at Israel-based company, recently briefed Datanami on the impact its in-memory technology will have on the flight operations of a major US airline.
The airline–which doesn’t want its name being used in print and thus will remain anonymous in this story–has adopted Gigaspaces in-memory data grid (IMDG) technology, called XAP, as the core foundational element for a group of new applications that it’s implementing at its US headquarters.
The new applications are replacing existing system that were designed over a decade ago to work on relational technology. While the relational technology has served the airline well, it hasn’t been able to deliver the speed to insight that the airline would like to have. With the maturation of IMDGs like Gigaspaces’ XAP, the technology is now available to tackle some of the intractable real-time computational challenges that have remained outside its grasp.
The new apps are being built atop a three-tiered stream processing technology stack that was designed to continually ingest data from many different sources and provide batch and real-time analysis upon a sliding 40-day window of data. The effective utilization of in-memory computing provides the underlying breakthrough that enabled the airline to move beyond the batch-based analytics.
The airline’s stack consists of XAP as the core data grid or “pre-processing” tier; Spark Streaming and Spark SQL in the “computational” tier; and visualization tools from Tableau and Apache Zeppelin in the upper level “perspectives” tier. IBM InfoSphere Streams is used as the messaging layer. The tiers are separated to enable the airline to scale each system independently to handle increased reads and writes.
The airport 360 system, in particular, has the potential to significantly improve flight operations and minimize the impact of delays, Paz says. The current system struggles to keep up with ever-changing conditions of its operations, which span the globe and involves thousands of flights and hundreds of thousands of passengers each day.
“You have data from different domains–flights, passengers, crew, cargo, etc.” he says. “If the ETA for one flight changes, that’s going to impact the next flight that this aircraft is going to be on, it’s going to impact passengers connecting from somewhere else on this flight, and it’s going to impact crews that are using this flight as well. You have this multi-dimensional graph that you need to do this delay propagation against, and all that needs to happen in sub-second instances.”
The airline’s existing Airport 360 system was first implemented in 2005 atop a relational data warehouse. The airline has since moved it to Oracle Exadata hardware, but the system still couldn’t handle all the variables the airline was throwing at it. “They tried to speed it up as much as they can,” he says, “but they couldn’t break the 10-minute refresh barrier. With us, we get that to below 200 milliseconds.”
The new in-memory system will also improve the airline’s modeling and simulation capability. The airline will have a group of data scientists and analysts exploring the data through Tableau, Spark, and Zeppelin to identify ways it can be more efficient. The simulations will also drive systems, such as the customer delay index, which displays the likelihood of a passenger mis-connects occurring at any given point in time.
The simulations will give the airline’s flight managers the insight they need to better handle delays. Variables such as fuel costs can impact decisions, such as whether to delay a flight at the point of origin or at the destination, while industry regulations (such as restriction on how many hours a pilot can work) can impact how flight crews are re-scheduled.
The use of data modeling to inform decision making is a big deal. Previously, the airline made these sorts of decisions by assembling data from disparate systems and trying to analyze them on an ad-hoc basis. Effectively, the airline had no capability to model its operations. In some respects, they were flying by the seat of their pants.
“These are high-value questions they couldn’t answer before,” Paz says. “When you intersect fuel costs, cost of delays, scheduling, and regulations on flight crew, [the airline] was incurring about a $9 million loss per month.”
While the airline won’t be able to recoup all of those losses, it should be able to cut into it significantly. “You can’t prevent a storm,” Paz points out. “But at least you can be more intelligent about re-accommodating customers.”
Gigaspaces has worked with many large companies, including retailers and banks, to implement IMDG technology to quickly derive insight from fast-moving data. This is one of the first engagements the company has had in the airline business, and so far the technology has been promising. “It’s kind of their secret sauce against low-cost airlines who are damaging their revenue,” Paz says.