# registry-backup A command line utility for downloading all .crate files hosted by a Cargo registry server. Use cases: - **Backup:** retrieve a registry server's files for backup storage - **Export:** pull the files so you can host them at another registry server ## Example Usage: Specify the registry index either as a local path (`--index-path`)... ```console $ git clone https://github.com/rust-lang/crates.io-index.git $ RUST_LOG=info registry-backup \ --index-path crates.io-index \ --output-path crates.io-crate-files \ --requests-per-second 10 ``` ...or as an `--index-url` instead: ```console $ RUST_LOG=info registry-backup \ --index-url ssh://git@ssh.shipyard.rs/shipyard-rs/crate-index.git \ --output-path shipyard-rs-crate-files \ --auth-token ${AUTH_TOKEN} # for private registry, need auth ``` ## Install ```console $ cargo install registry-backup --git https://git.shipyard.rs/jstrong/registry-backup.git ``` ## Runtime Options ```console $ ./target/release/registry-backup --help registry-backup 0.4.1 Jonathan Strong Download all .crate files from a registry server USAGE: registry-backup [OPTIONS] OPTIONS: --index-url URL of the registry index we are downloading .crate files from. The program expects that it will be able to clone the index to a local temporary directory; the user must handle authentication if needed --index-path instead of an index url, just point to a local path where the index is already cloned -a, --auth-token If registry requires authorization (i.e. "auth-required" key is set to `true` in the `config.json` file), the token to include using the Authorization HTTP header -o, --output-path Directory where downloaded .crate files will be saved to [default: output] --overwrite-existing Download files when if .crate file already exists in output dir for a given crate version, and overwrite the existing file with the new one. Default behavior is to skip downloading if .crate file already exists --output-format What format to use for the output filenames. Works the same as Cargo's registry syntax for the "dl" key in the `config.json` file in a reigstry index. See [Cargo docs](https://doc.rust-lang.org/cargo/reference/registries.html#index-format) for additional details. Not specifying this field is equivalent to specifying "{crate}/{version}/download", the default. The resulting path specified by the format should be relative; it will be joined with the --output-path. (i.e. it should not start with "/".) -U, --user-agent Value of user-agent HTTP header [default: registry-backup/v0.4.1] -R, --requests-per-second Requests to registry server will not exceed this rate [default: 100] -M, --max-concurrent-requests Independent of the requests per second rate limit, no more than `max_concurrent_requests` will be in flight at any given moment [default: 50] -c, --config-file Specify configuration values using the provided TOML file, instead of via command line flags. The values in the config file will override any values passed as command line flags. See config.toml.sample for syntax of the config file --filter-crates Only crates with names that match --filter-crate regex will be downloaded --dry-run Don't actually download the .crate files, just list files which would be downloaded. Note: --requests-per-second and --max-concurrent-requests are still enforced even in --dry-mode! -h, --help Print help information -V, --version Print version information ``` ## Configuration File A toml configuration file may be used instead of command line flags. A sample file (`config.toml.sample`) is included. From the example file: ```toml dry-run = false filter-crates = "^." [registry] index-url = "ssh://git@ssh.shipyard.rs/shipyard-rs-public/crate-index.git" # alternatively, specify a local dir # index-path = "/path/to/cloned/index" auth-token = "xxx" [http] user-agent = "registry-backup/v0.1.0" requests-per-second = 100 max-concurrent-requests = 50 [output] path = "output" overwrite-existing = false format = "{crate}/{version}/download" ``` ## Build From Source ```console $ git clone https://git.shipyard.rs/jstrong/registry-backup.git $ cd registry-backup $ just release-build # alternatively, cargo build --bin registry-backup --release # ./target/release/registry-backup --help # cp target/release/registry-backup ~/.cargo/bin/ ``` ## Running Tests ```console $ just test # alternatively, cargo test ``` ## Justfile The repository includes a `justfile` with functionality for building, testing, etc. Included commands: ```console $ just --list Available recipes: cargo +args='' # cargo wrapper; executes a cargo command using the settings in justfile (RUSTFLAGS, etc.) check +args='' # cargo check wrapper debug-build +args='' # cargo build wrapper - builds registry-backup in debug mode generate-readme # generate updated README.md get-crate-version install # cargo install registry-backup via git dep pre-release # check, run tests, check non-error output for clippy, run rustfmt release # release version (regenerate docs, git tag v0.0.0) release-build +args='' # cargo build --release wrapper - builds registry-backup in release mode release-prep # get everything all ready for release show-build-env # diagnostic command for viewing value of build variables at runtime test +args='' # cargo test wrapper update-readme # re-generate README.md and overwrite existing file with output update-readme-and-commit # re-generate, overwrite, stage, and commit update-readme-and-stage # re-generate, overwrite, and stage changes verify-clean-git # verify no uncommitted changes ``` The commands that mirror cargo commands (e.g. `just test`) are included for the purpose of convenience, so that various options (e.g. `RUSTFLAGS='-C target-cpu=native`) can be included without typing them out each time. ## Generating `README.md` This file is generated using a template (`doc/README.tera.md`) rendered using updated outputs of the CLI menu, config sample, and other values. This version of `README.md` was generated at `Thu, 08 Dec 2022 02:23:17 +0000` based on git commit `3241e207`. To (re-)generate the `README.md` file, use the justfile command: ```console $ just generate-readme ```