Follow Datanami:

Tag: file system

AWS Delivers ‘Lightning’ Fast LLM Checkpointing for PyTorch

AWS customers who are training large language models (LLMs) will be able to complete their model checkpoints up to 40% faster thanks to improvements AWS has made with its Amazon S3 PyTorch Lightning Connector. The compan Read more…

Getting the Upper Hand on the Unstructured Data Problem

Unstructured data accounts for the vast majority of data stored in the world today, and it’s growing at a geometric rate. Organizations today may have petabytes of the stuff spread around various object stores and file Read more…

The Journey to Effective Data Management in HPC

Imagine a simple interface for data search across an organization’s local and cloud storage. The search would return relevant data types, their location, and automatically extracted metadata. From there, advanced analy Read more…

Blurred Storage Lines: Clouds That Appear Like On-Prem

Cloud storage is growing fast as companies look to take advantage of low cost and flexible storage options for terabytes and petabytes of data. But for all the convenience of cloud storage, sometimes it's just better whe Read more…

Quobyte Plugs TensorFlow Into Scale-Out File System

Quobyte designed its file system with scalability in mind. It also designed it to handle multiple types of files and workload without burdening human administrators with extra complexity. Both of those characteristics ar Read more…

Amazon S3 Dominance Spawns Storage Challengers

When it comes to storing data, Amazon's Simple Storage Service (S3) is a runaway success. With trillions of files under management spanning many exabytes worth of data, S3 is the largest digital storage system in the kno Read more…

Quantum Adds Global Smarts to StorNext File System

Companies that use Quantum's StorNext platform to store massive amounts of data this week got a glimpse of new storage capabilities that should make it easier to access their data horde from anywhere in the world. St Read more…

Meet Alluxio, the Distributed File System Formerly Known as Tachyon

Today marks the availability of Alluxio 1.0, the first major release of the memory-centric, virtual distributed file system formerly known as Tachyon. The graduate of UC Berkeley's AMPLab is already being used to manage Read more…

Quantcast Opens Exabyte-Ready File System

When ingesting data volumes that tip the multi-exabyte per year scales, even the most notorious, robust systems for managing scale can start to crumble under their own weight. When the cost and complexity of this volume mount, what’s a growing company to do but look beyond the buzz for its own solutions?...We speak with Quantcast CEO and VP of R&D about the.... Read more…

SC11 Video Feature: Garth Gibson on RAID, Roots and Reliability

In this video feature, we sit down with Garth Gibson, co-founder and CTO of high performance storage company, Panasas. During our chat we touch on the roots of his company, explore the way file systems have evolved to meet new demands, and wrap up with a section on big data and the storage demands behind it. Read more…

Datanami