# registry-backup
A command line utility for downloading all .crate files hosted by a Cargo registry server.
Use cases:
- **Backup:** retrieve a registry server's files for backup storage
- **Export:** pull the files so you can host them at another registry server
## Example Usage:
Specify the registry index either as a local path (`--index-path`)...
```console
$ git clone https://github.com/rust-lang/crates.io-index.git
$ registry-backup \
--index-path crates.io-index \
--output-path crates.io-crate-files \
--requests-per-second 10
```
...or as an `--index-url` instead:
```console
$ registry-backup \
--index-url ssh://git@ssh.shipyard.rs/shipyard-rs/crate-index.git \
--output-path shipyard-rs-crate-files \
--auth-token ${AUTH_TOKEN} # for private registry, need auth
```
## Install
```console
$ cargo install registry-backup --git https://git.shipyard.rs/jstrong/registry-backup.git
```
## Runtime Options
```console
$ ./target/release/registry-backup --help
registry-backup 0.3.0
Jonathan Strong < jstrong @ shipyard . rs >
Download all .crate files from a registry server
USAGE:
registry-backup [OPTIONS]
OPTIONS:
--index-url < URL >
URL of the registry index we are downloading .crate files from. The program expects that
it will be able to clone the index to a local temporary directory; the user must handle
authentication if needed
--index-path < PATH >
instead of an index url, just point to a local path where the index is already cloned
-a, --auth-token < TOKEN >
If registry requires authorization (i.e. "auth-required" key is set to `true` in the
`config.json` file), the token to include using the Authorization HTTP header
-o, --output-path < PATH >
Directory where downloaded .crate files will be saved to [default: output]
--overwrite-existing
Download files when if .crate file already exists in output dir for a given crate
version, and overwrite the existing file with the new one. Default behavior is to skip
downloading if .crate file already exists
-U, --user-agent < USER_AGENT >
Value of user-agent HTTP header [default: registry-backup/v0.3.0]
-R, --requests-per-second < INT >
Requests to registry server will not exceed this rate [default: 100]
-M, --max-concurrent-requests < INT >
Independent of the requests per second rate limit, no more than
`max_concurrent_requests` will be in flight at any given moment [default: 50]
-c, --config-file < PATH >
Specify configuration values using the provided TOML file, instead of via command line
flags. The values in the config file will override any values passed as command line
flags. See config.toml.sample for syntax of the config file
--filter-crates < REGEX >
Only crates with names that match --filter-crate regex will be downloaded
--dry-run
Don't actually download the .crate files, just list files which would be downloaded.
Note: --requests-per-second and --max-concurrent-requests are still enforced even in
--dry-mode!
-h, --help
Print help information
-V, --version
Print version information
```
## Configuration File
A toml configuration file may be used instead of command line flags. A sample file (`config.toml.sample`) is included. From the example file:
```toml
dry-run = false
filter-crates = "^."
[registry]
index-url = "ssh://git@ssh.shipyard.rs/shipyard-rs-public/crate-index.git"
# alternatively, specify a local dir
# index-path = "/path/to/cloned/index"
auth-token = "xxx"
[http]
user-agent = "registry-backup/v0.1.0"
requests-per-second = 100
max-concurrent-requests = 50
[output]
path = "output"
overwrite-existing = false
```
## Build From Source
```console
$ git clone https://git.shipyard.rs/jstrong/registry-backup.git
$ cd registry-backup
$ just release-build # alternatively, cargo build --bin registry-backup --release
# ./target/release/registry-backup --help
# cp target/release/registry-backup ~/.cargo/bin/
```
## Running Tests
```console
$ just test # alternatively, cargo test
```
## Justfile
The repository includes a `justfile` with functionality for building, testing, etc.
Included commands:
```console
$ just --list
Available recipes:
cargo +args='' # cargo wrapper; executes a cargo command using the settings in justfile (RUSTFLAGS, etc.)
check +args='' # cargo check wrapper
debug-build +args='' # cargo build wrapper - builds registry-backup in debug mode
generate-readme # generate updated README.md
get-crate-version
install # cargo install registry-backup via git dep
pre-release # check, run tests, check non-error output for clippy, run rustfmt
release # release version (regenerate docs, git tag v0.0.0)
release-build +args='' # cargo build --release wrapper - builds registry-backup in release mode
release-prep # get everything all ready for release
show-build-env # diagnostic command for viewing value of build variables at runtime
test +args='' # cargo test wrapper
update-readme # re-generate README.md and overwrite existing file with output
update-readme-and-commit # re-generate, overwrite, stage, and commit
update-readme-and-stage # re-generate, overwrite, and stage changes
verify-clean-git # verify no uncommitted changes
```
The commands that mirror cargo commands (e.g. `just test` ) are included for the purpose of convenience, so that various options (e.g. `RUSTFLAGS='-C target-cpu=native` ) can be included without typing them out each time.
## Generating `README.md`
This file is generated using a template (`doc/README.tera.md`) rendered using updated outputs of the CLI menu, config sample, and other values.
This version of `README.md` was generated at `Fri, 09 Sep 2022 05:50:07 +0000` based on git commit `b7621e78` .
To (re-)generate the `README.md` file, use the justfile command:
```console
$ just generate-readme
```
