‘Lifelong’ Neural Net Aims to Slash Training Time
Among the consequences of big data is a wealth of relevant minutiae that can be used to train machine learning and other models. That often translates into processing-intensive steps required to train models to perform a specific task.
In response, technology startups and government-funded university researchers are promoting a new machine intelligence approach called “lifelong” machine learning as a way to accelerate model training. The latest entrant is Neurala, which this week took the wraps off its lifelong deep neural network technology.
The Boston-based startup claims its AI approach can slash training time from upwards of 15 hours to as little as 20 seconds by utilizing “incremental learning.” The approach is said to eliminate the need to completely retrain a model every time new piece of information is added.
“Our technology allows for a massive reduction in the time it takes to train a neural network and all but eliminates the time it takes to add new information,” said Anatoli Gorchetchnikov, Neurala’s CTO.
The lifelong learning framework also is gaining traction among researchers seeking to overcome current machine learning limitations such as inability to adapt to new situations beyond the narrow tasks models were trained to perform.
For example, the Defense Advanced Research Project Agency is funding a lifelong machine learning effort that includes early work by Columbia University researchers toward building and training a “self-replicating” neural network. The goal is to develop a system that can adapt “by using knowledge of its own structure,” DARPA said.
Neurala said Wednesday (May 9) its lifelong deep neural network software is initially designed to boost the autonomy of robots, drones and other “smart” products. The framework utilizes a deep neural network pretrained on the ImageNet database of images organized by keyword. It also uses specific data sets required for specific tasks.
The downside to the conventional approach is that a newly trained deep neural network is fixed, meaning retraining is required when adding new data on all objects. Neurala’s lifelong approach is touted as enabling incremental learning—mimicking the way human and animal brain circuitry works to adapt to new information “on the fly.”
The incremental learning approach that uses software to learn and understand “objects on the edge” also is said to eliminate the need for cloud-based servers to handle huge training data sets, the company notes. Training time is reduced further by reducing the number of server instances required during training. Neurala further claims that inference, memory and model precision “are not significantly affected” using its approach.
“We can envision this technology slashing compute powers in server farms and enabling networks to be assembled on the fly [based] on custom data,” Gorchetchnikov added.
By mimicking how the brain learns and analyzes its environment, the company’s software is aimed at products ranging from industrial drones to consumer electronics.
Neurala initially developed deep learning neural network software for NASA planetary missions where processing power, battery life and communications are limited. “Instead of designing deep learning neural networks for supercomputers, the Neurala Brain was designed to work where the decisions need to be made on [a] device,” the company noted.
Neurala’s spinoff technology may complement similar government research efforts designed to advance the frontiers of lifelong machine learning. Along with U.S.-funded researcher at Columbia University, DARPA last week announced expansion of its lifelong machine learning effort to include a “dual-memory architecture” project along with adaptive learning and a “memory reconsolidation” effort. The latter is designed to understand the biological process of memory formation and storage and how memories could then be used to solve specific problems.