Preventing the Next 9/11 Goal of NORAD’s New Streaming Data Warehouse

(mr.-teerapon-tiuekhom/Shutterstock)

The skies above North America are constantly scanned with a variety of radar technologies. But with upwards of 20,000 planes and billions of birds aloft at any given time (and who knows what else), spotting potential threats before they can strike is not an easy task. But thanks to the contract the US Government awarded to Kinetica earlier this year, we should soon have a system capable of doing just that.

In February, Kinetica was granted a five-year deal contract to build Pathfinder, a new system composed of Kinetica’s GPU-powered relational database that will occupy a couple of hundred nodes in the AWS GovCloud. The system, which is worth upwards of $100 million, is currently in development.

Awarded by the Defense Innovation Unit (DIU), North American Aerospace Defense Command (NORAD) and United States Northern Command (USNORTHCOM), the program is set to improve the situational awareness for the US and Canadian militaries, says Amit Vij, Kinetica’s president and co-founder.

“You often hear ‘Is it a bird? Is it a plane? Is it Superman?’” Vij tells Datanami. “We’re kind of doing that for them, leveraging massive big data, streaming data feeds from a multitude of different sensors and doing anomaly detection on the fly.”

Kinetica has done this kind of work before. Back when it was known as GIS Federal, the company helped the United States Army Intelligence and Security Command (INSCOM) at Fort Belvoir, bring GPU-accelerated computing resources to fast-moving geographic and time-series data in support of the U.S. military’s terrorist-hunting efforts.

Kinetica’s SQL database forms the heart of its real-time analytics offering (Image source Kinetica)

“We started in the same vein,” Vij says. “We were incubated by the Army and NSA in taking structured and unstructured data feeds from their variety of different data sensors to track terrorists in real time, and really primarily focused in on Centcom (U.S. Central Command). Now seven to 10 years later, we’re at it again. We’re very more mature in our product capability, but we’re trying to prevent the next 9/11 from happening.”

The numbers behind the Pathfinder program are mind-boggling. The program’s primary mission is entity classification, or identifying the nature of the objects occupying U.S. airspace. The primary data source for Pathfinder is unclassified radar data from the Federal Aviation Administration, with some additional data feeds, such as data from aircraft equipped with transponders.

According to Vij, there are around 1,000 sources of data that are constantly feeding data into Pathfinder via Apache NiFi. Upwards of five billion new objects per day are ingested into Kinetica’s distributed relational database, which currently has about one trillion rows, Vij says. The combination of this deep historical record of activity, plus the real-time feed of current activity on top, gives Pathfinder a unique capability to bring machine learning and real-time analytics to bear on the objects occupying our airspace.

Vij uses an analogy to explain what Pathfinder is doing.

“Before COVID, a person would go to work, they have their favorite bars or restaurants they would go to Monday, Wednesday, and Friday. ‘I’m going to this place Friday, Saturday I’m going to this friend’s house,’” he says. “As you gather that information over the course of months and years, people follow certain patterns right and they don’t really move too much outside their comfort [zone]. And if they do, that becomes an anomaly. That’s something that I haven’t seen before, and it’s a change in what historically has happened.

“We’re doing that with planes,” he continues. “This Southwest flight goes from LA to San Francisco five times a day on this schedule. It’s off by a few minutes here and there. But this is it’s pattern of life. Now, if that plane veers off left or right, or in the case of 9/11, so many things deviated from that normal pattern of life. Our database has all this information, it’s gathering it constantly. We’re assessing vulnerability in real time with a multi-faceted approach, leveraging machine learning and AI to see what’s historically happened in the past to predict and provide fields of deviation for what is in the norm of certain classes of aircraft.”

Every object that takes to the sky above the United States and Canada is tracked by this system. From large jetliners lumbering across the continent to flocks of migratory geese, it all ends up in the Kinetica database.

“A Boeing 777 has a very distinct signature [compared] to an F-16 or F-35 versus a cruise missile, versus any other kind of aircraft, based on kinematic profiles, based on landing,” Vij says. “[There are] multiple different areas we are inspecting and providing analysis on.”

Nearly 20 years after the fateful events of September 11, 2001, the United States Government has taken many steps to improve security, both on the ground and in the air. In the unlikely event that a terrorist was able to bypass those new security measures and successfully hijack a large aircraft full of jet fuel, thanks to Pathfinder, military and government decision-makers will know about it much more quickly.

Former postal worker Doug Hughes evaded air traffic control to land a gyrocopyter near the Capitol on April 15, 2015.

But the threat isn’t confined to large jets. Because the volume of radar data far exceeds what even the most capable radar operators can process, there exists the potential for smaller aircraft to slip by undetected. That is exactly what happened on April 15, 2015, when a disgruntled former postal worker landed a gyrocopter on a lawn near the Capitol. The folks watching the airspace above Washington, D.C. thought it was a passing weather system, Vij says.

“There’s only so much data a human can actually understand at one time,” he says. “There’s 10,000 to 20,000 aircraft flying at any given time, and each one of those are producing signatures at multiple times a second. Compounding that with weather data and all these other kinds of factors, it’s hard for human beings to comprehend and discern what this is.”

When it goes live, the Pathfinder program should lighten the load on human opeators, thereby enabling them to focus their time and energy on the most pressing needs.

The goal is “to have this fast layer on top of a data lake that can quickly ingest analyze [and] run models and visualize this to prevent those kinds of scenarios from happening,” Vij says. “Where you can take 70-80% of that problem and apply machine learning and analytics to do this and make this automated, then we can make our warfighters, our soldiers more lethal.”

Related Items:

Kinetica Gets $50M for Converged GPU Analytics

How GPU-Powered Analytics Improves Mail Delivery for USPS

GPU-Powered Terrorist Hunter Eyes Commercial Big Data Role