IBM Extends Jupyter Notebooks for AI Development
IBM has released a new open source toolkit with AI extensions to the popular Jupyter Notebooks data science development platform.
The Elyra AI Toolkit extends the industry standard JupyterLab user interface with the goal of simplifying development of AI and other data science models. IBM said this week the initial release includes a visual editor for building AI pipelines along with the ability to run interactive notebooks as batch jobs. Other features include Python script execution and a “hybrid runtime” capability based on Jupyter Notebooks’ enterprise gateway.
The gateway is designed to ease the scaling of enterprise workloads. IBM said Elyra (pronounced, el-EYE-rah) would ease workload development. Elyra “aims to help data scientists, machine learning engineers and AI developers through the model development lifecycle complexities,” the company added in a blog post announcing the open source release.
The visual editor for building AI pipelines is designed to ease the conversion of multiple notebooks into simpler workflows. The user interface also allows the submission of single notebook as a batch job.
Among the Jupyter extensions is the ability to “decompose” different AI development tasks into different notebooks, allowing each to use different frameworks such as TensorFlow for deep learning models. The editor can then be used to build notebook-based AI pipelines.
Multiple notebooks can then be converted into batch jobs or workflows.
The stable release in March of a related Kubernetes-based automation tool called Kubeflow includes a Jupyter Notebooks controller. Elyra currently supports the Kubeflow Pipelines runtime.
Along with Kubeflow and Kubeflow Pipelines, IBM said it is also contributing to PyTorch, Spark, TensorFlow and other machine learning model development tools.
Jupyter users welcomed the open source notebook extensions. “As much as AI tools rely on vast amounts of data and computational resources, the human in the loop remains the critical element for both asking the right questions and making decisions responsibly,” said Fernando Pérez, co-founder and co-director of Project Jupyter.