In simple English
This tool stores and serves large sets of data for artificial intelligence training. It connects to Amazon, Google, and Azure storage to fetch your files quickly. You can run it on a single Linux machine or across a cluster of servers. It helps your training scripts read data without waiting on slow network drives. You need some system administration skills to set it up and run it.
What it is
AIStore is an elastic distributed storage stack designed for high-throughput AI applications. It features linear scalability, integrated caching, and native support for common deep learning frameworks.
What it is good for
Solo builders and researchers who need fast data loaders for PyTorch or TensorFlow without paying for managed cloud storage accelerators.
- Compatible with the Amazon S3 (Simple Storage Service, a web service that offers storage through web APIs) API so your existing code works without changes.
- Connects directly to multiple cloud providers while caching files locally for fast access.
- Includes dedicated PyTorch (a machine learning library for Python) integration for efficient data loading during training.
- Requires a dedicated Linux environment and multi-node planning for full distributed features.
- High configuration complexity with many cluster nodes and storage disks to manage.
- Demands careful monitoring of memory and CPU (Central Processing Unit, the main chip that does calculations in a computer) limits under heavy loads.
MinIO or standard network file shares
PyTorch
- dbeaver/dbeaver52k stars
A free desktop program that connects to more than a hundred kinds of database and lets you browse and edit them.
- dimitri/pgloader6.5k stars
Loads data into PostgreSQL from other databases and from files, keeping going when individual rows fail.
./scripts/install_from_binaries.sh