#1
git-annex
Command-line tool that manages large files with Git by keeping file contents out of commits, providing sync, backup, encryption and many storage backends.
git-annex lets Git track and manage large files without storing their contents inside the Git repository. It records file identity, location, and metadata while keeping actual file content in annexed object stores, enabling sync, backup and archival workflows across local drives and remote backends.
Key Features
- Keeps file contents outside commits while storing pointers and metadata in Git, enabling versioning of metadata without bloating repositories
- Supports many "special remotes" and backends (S3, WebDAV, directory, rsync, bup, Tahoe, Glacier, bittorrent and more) for flexible storage and transfer
- Optional encryption and signing of content (GnuPG/gcrypt) and multiple checksum/key-value backends for integrity and detectability
- git-annex assistant and webapp provide an easier UI for folder synchronisation and managing queued transfers; core functionality remains CLI-first
- FUSE-backed presentation of annexed files as work-tree symlinks to enable transparent access without full local copies
- Location tracking and preferred-content policies let you locate which drive or remote holds specific files and automate movement (cron-friendly)
- Scales to very large collections and offline/archival workflows, with features for deduplication, clustering, and selective retrieval
Use Cases
- Long-term archiving: track datasets and media across multiple offline archival drives while preserving Git history of organization
- Nomadic/offline workflows: selectively fetch or drop large files on laptops, USB drives, and servers to save space while keeping metadata in Git
- Distributed sharing of large assets: exchange big files between collaborators or servers using multiple special-remote backends and sync policies
Limitations and Considerations
- Many convenience features (webapp, assistant, some backend support) require optional build-time libraries; packaged builds from distributions may omit features
- The webapp listens on localhost by default and must be configured for remote access; some platform ports (Windows/Android) are marked beta
- There is a learning curve for matching workflows and configuring special remotes; operations are primarily command-line oriented
git-annex is a mature, integrity-focused tool for bringing Git-style tracking to large files while avoiding repository bloat. It is best suited to users who need flexible backend support, offline/archival workflows, and programmable control over file placement and retrieval.