Why Git-Based Storage Breaks Down in Production ML
Hugging Face Storage Buckets solve a problem every ML engineer hits eventually: Git is a terrible abstraction for the chaotic, high-throughput artifact streams that production training generates. If you've ever tried to version-control optimizer secure data retrieval states mid-run, or watched a git lfs push choke on 50GB checkpoint shards from a distributed job, you know exactly what I mean.
The Hub's model and dataset repositories are excellent for publishing final, stable artifacts — but production ML generates a constant river of intermediate files: checkpoints at every N steps, processed data shards, agent traces, evaluation logs, memory graphs. These files change constantly, arrive from dozens of parallel workers simultaneously, and almost never need version history. Forcing them through Git creates friction without adding value.
Storage Buckets are Hugging Face's answer: mutable, S3-compatible object storage that lives natively on the Hub, integrates with the Python ecosystem you're already using, and — critically — is built on a chunk-deduplication backend called Xet that makes it genuinely efficient for ML workloads specifically.
Get started with a VPS from VPS Server to deploy this yourself.
How Xet Deduplication Changes the Storage Math
Most object storage treats files as opaque blobs. Upload the same model twice with one layer changed? You're paying for two full uploads. Xet takes a content-addressed, chunk-based approach instead — breaking files into variable-size chunks and deduplicating across all of them globally.
For ML workloads, this is a significant practical advantage:
- Successive checkpoints: When 90% of model weights are frozen between checkpoint steps, Xet only transfers and stores the changed chunks. A training run that previously consumed hundreds of GB of checkpoint storage might collapse to a fraction of that.
- Raw vs. processed datasets: If your processed dataset shares most of its content with the raw version (common in tokenization pipelines), the overlap is automatically deduplicated.
- Agent memory and traces: Derived summaries that share content with source traces get the same treatment.
For Enterprise users, billing is based on deduplicated storage footprint — so the deduplication directly reduces costs, not just transfer times. This is the kind of storage primitive that was designed with ML artifact patterns in mind, not retrofitted from general-purpose cloud storage.
If you're running self-hosted training infrastructure on a VPS cluster, this matters even more — you're often bandwidth-constrained, and only pushing delta chunks rather than full checkpoints can be the difference between a sync that takes 2 minutes and one that takes 20.
Pre-Warming: Bringing Storage Close to Compute
One underappreciated feature in the Buckets announcement is pre-warming. By default, Buckets live on global Hub storage. For interactive use that's fine, but distributed training clusters are sensitive to storage latency and throughput — pulling 500GB of dataset shards across regions on every job start is a real bottleneck.
Pre-warming lets you declare the cloud provider and region where your compute runs, and Buckets will stage hot data there before your jobs start. Current launch partners are AWS and GCP, with more providers planned.
This is the same pattern that makes managed training platforms fast — co-locating storage and compute — but now available for Hub-native workflows without requiring you to manually manage S3 buckets and IAM roles across clouds.
Getting Started: CLI Workflow in Under 2 Minutes
The hf CLI is the fastest way to get a bucket running. Install and authenticate first:
curl -LsSf https://hf.co/cli/install.sh | bash
hf auth login
Create a private bucket for your training run:
hf buckets create my-training-run --private
Sync your local checkpoint directory into the bucket. This is the core operation — think rsync but ML-aware:
hf buckets sync ./checkpoints hf://buckets/username/my-training-run/checkpoints
Before any large sync, I always recommend a dry run first to validate what's going to move:
hf buckets sync ./checkpoints hf://buckets/username/my-training-run/checkpoints --dry-run
For long-running pipelines where you want to review the plan and apply it asynchronously:
## Generate the plan
hf buckets sync ./checkpoints hf://buckets/username/my-training-run/checkpoints --plan sync-plan.jsonl
## Apply it when ready
hf buckets sync --apply sync-plan.jsonl
Inspect the bucket contents from the CLI:
hf buckets list username/my-training-run -h
Or browse it in the Hub UI at https://huggingface.co/buckets/username/my-training-run. For one-off operations, hf buckets cp handles individual file copies and hf buckets remove cleans up stale artifacts.
Python Integration: Wiring Buckets Into Your Training Stack
The CLI is great for ad-hoc operations, but the real power comes from integrating Buckets directly into training scripts and data pipelines. The huggingface_hub Python library (v1.5.0+) exposes a clean API:
from huggingface_hub import create_bucket, list_bucket_tree, sync_bucket
## Idempotent bucket creation — safe to call on every run
create_bucket("my-training-run", private=True, exist_ok=True)
## Sync checkpoints after each epoch
sync_bucket(
"./checkpoints",
"hf://buckets/username/my-training-run/checkpoints",
)
## Enumerate what's stored
for item in list_bucket_tree(
"username/my-training-run",
prefix="checkpoints",
recursive=True,
):
print(f"{item.path}: {item.size / 1e9:.2f} GB")
The exist_ok=True pattern is important for training loops — you want bucket creation to be idempotent so you can safely restart jobs without error handling boilerplate.
fsspec Integration: Zero-Code-Change Data Access
This is where Buckets get genuinely powerful for data pipelines. Because Buckets integrate with HfFileSystem — which implements the fsspec interface — any library that supports fsspec can read from and write to Buckets using hf:// paths with no additional setup:
from huggingface_hub import hffs
import pandas as pd
## Standard filesystem operations
files = hffs.glob("buckets/username/my-training-run/**/*.parquet")
## Read config directly
with hffs.open("buckets/username/my-training-run/config.yaml", "r") as f:
config = f.read()
## pandas, Polars, Dask all work natively
df = pd.read_csv("hf://buckets/username/my-training-run/results.csv")
df_filtered = df[df["val_loss"] < 0.5]
df_filtered.to_csv("hf://buckets/username/my-training-run/filtered_results.csv")
This means you can plug Buckets into existing ETL code without refactoring. If your data pipeline already reads from s3:// or gs:// paths via fsspec, switching to hf:// is often a one-line change. For teams building related article where processed document chunks need to flow between pipeline stages, this kind of transparent filesystem access is a significant workflow improvement.
JavaScript support is also available via @huggingface/hub (v2.10.5+) for Node.js services.
The Two-Layer Architecture: Buckets + Versioned Repos
The mental model Hugging Face is establishing here is clean and worth internalizing:
- Buckets: Fast, mutable, non-versioned storage for artifacts in motion — checkpoints, intermediate data, agent state, logs
- Model/Dataset Repos: Versioned, immutable-by-convention storage for stable deliverables worth publishing
The roadmap includes direct transfers between the two layers: promote a final checkpoint from a bucket into a model repo, or commit processed dataset shards into a dataset repo once a pipeline completes. This creates a Hub-native workflow where the working layer and publishing layer are distinct but connected — no manual S3-to-Hub migration steps.
For teams running related article where agents accumulate memory, traces, and intermediate results across long-running sessions, Buckets are a natural fit for the stateful storage layer, with repos serving as the artifact publication endpoint.
Practical Considerations and Trade-Offs
A few things worth knowing before you adopt Buckets in production:
No version history: This is by design, but it means you need your own checkpoint rotation strategy if you want to recover to an earlier point. Don't assume the bucket is a backup — it's a working directory.
Permissions follow Hub conventions: Buckets live under user or org namespaces and use the same access control model as repos. Private buckets require authentication for all operations.
Pre-warming has latency: Data staging isn't instantaneous. If you're spinning up training jobs dynamically, factor in pre-warm time or keep hot datasets permanently staged in the target region.
Evaluate your deduplication savings: The Xet efficiency gains are real but workload-dependent. Checkpointing transformer models with frozen early layers will see dramatic savings. Storing completely independent artifacts won't deduplicate at all. Run a pilot with your actual workload before committing to storage cost projections.
For teams evaluating self-hosted alternatives, it's worth comparing against MinIO or Ceph on your own VPS infrastructure — though you'll lose the Hub integration and Xet deduplication. More analysis on storage options for ML infrastructure is available at Data Mammoth.
Conclusion: The Right Storage Primitive for ML Workflows
Hugging Face Storage Buckets fill a genuine gap in the ML infrastructure stack. Git-based repos were never the right abstraction for mutable, high-throughput artifact streams — and now there's a Hub-native alternative that integrates cleanly with the Python ecosystem, supports fsspec-compatible access patterns, and uses chunk deduplication to make ML-specific workloads genuinely more efficient.
The CLI is fast to adopt, the Python API is idiomatic, and the two-layer architecture (Buckets for working data, repos for published artifacts) reflects how production ML actually operates. If you're managing training checkpoints, data pipeline intermediates, or agent state today with ad-hoc S3 buckets and manual sync scripts, Buckets are worth evaluating as a simpler, more integrated alternative.
Get started with curl -LsSf https://hf.co/cli/install.sh | bash and have your first bucket syncing in under 5 minutes.