registry-backup

A command line utility for downloading all .crate files hosted by a Cargo registry server.

Use cases:

• Backup: retrieve a registry server's files for backup storage
• Export: pull the files so you can host them at another registry server

Example Usage:

Specify the registry index either as a local path (--index-path)...

$ git clone https://github.com/rust-lang/crates.io-index.git
$ registry-backup \
    --index-path crates.io-index \
    --output-path crates.io-crate-files \
    --requests-per-second 10

...or as an --index-url instead:

$ registry-backup \
    --index-url ssh://git@ssh.shipyard.rs/shipyard-rs/crate-index.git \
    --output-path shipyard-rs-crate-files \
    --auth-token ${AUTH_TOKEN} # for private registry, need auth

Install

$ cargo install registry-backup --git https://git.shipyard.rs/jstrong/registry-backup.git

Runtime Options

$ ./target/release/registry-backup --help

registry-backup 0.2.4
Jonathan Strong <jstrong@shipyard.rs>
Download all .crate files from a registry server

USAGE:
    registry-backup [OPTIONS]

OPTIONS:
        --index-url <INDEX_URL>
            URL of the registry index we are downloading .crate files from. The program expects that
            it will be able to clone the index to a local temporary directory; the user must handle
            authentication if needed

        --index-path <INDEX_PATH>
            instead of an index url, just point to a local path where the index is already cloned

    -p, --output-path <PATH>
            Directory where downloaded .crate files will be saved to [default: output]

    -u, --user-agent <USER_AGENT>
            Value of user-agent HTTP header [default: registry-backup/v0.2.4]

        --requests-per-second <REQUESTS_PER_SECOND>
            Requests to registry server will not exceed this rate [default: 25]

        --max-concurrent-requests <MAX_CONCURRENT_REQUESTS>
            Independent of the requests per second rate limit, no more than
            `max_concurrent_requests` will be in flight at any given moment [default: 10]

    -a, --auth-token <AUTH_TOKEN>
            If registry requires authorization (i.e. "auth-required" key is set to `true` in the
            `config.json` file), the token to include using the Authorization HTTP header

    -c, --config-file <CONFIG_FILE>
            Specify configuration values using the provided TOML file, instead of via command line
            flags. The values in the config file will override any values passed as command line
            flags. See config.toml.sample for syntax of the config file

    -h, --help
            Print help information

    -V, --version
            Print version information

Configuration File

A toml configuration file may be used instead of command line flags. A sample file (config.toml.sample) is included. From the example file:

# index repo url
index-url = "ssh://git@ssh.shipyard.rs/shipyard-rs-public/crate-index.git"

# alternatively, specify a local dir
# index-path = "/path/to/cloned/index"

[http]
user-agent = "registry-backup/v0.1.0"
requests-per-second = 40
max-concurrent-requests = 20

[output]
path = "output"
format = "/{crate}/{version}/download"

# auth-token = "xxx"

Build From Source

$ git clone https://git.shipyard.rs/jstrong/registry-backup.git
$ cd registry-backup
$ just release-build # alternatively, cargo build --bin registry-backup --release
# ./target/release/registry-backup --help
# cp target/release/registry-backup ~/.cargo/bin/

Running Tests

$ just test # alternatively, cargo test

Justfile

The repository includes a justfile with functionality for building, testing, etc.

Included commands:

$ just --list

Available recipes:
    cargo +args=''           # cargo wrapper; executes a cargo command using the settings in justfile (RUSTFLAGS, etc.)
    check +args=''           # cargo check wrapper
    debug-build +args=''     # cargo build wrapper - builds registry-backup in debug mode
    generate-readme          # generate updated README.md
    get-crate-version
    pre-release              # check, run tests, check non-error output for clippy, run rustfmt
    release                  # release version (regenerate docs, git tag v0.0.0)
    release-build +args=''   # cargo build --release wrapper - builds registry-backup in release mode
    release-prep             # get everything all ready for release
    show-build-env           # diagnostic command for viewing value of build variables at runtime
    test +args=''            # cargo test wrapper
    update-readme            # re-generate README.md and overwrite existing file with output
    update-readme-and-commit # re-generate, overwrite, stage, and commit
    update-readme-and-stage  # re-generate, overwrite, and stage changes
    verify-clean-git         # verify no uncommitted changes

The commands that mirror cargo commands (e.g. just test) are included for the purpose of convenience, so that various options (e.g. RUSTFLAGS='-C target-cpu=native) can be included without typing them out each time.

Generating README.md

This file is generated using a template (doc/README.tera.md) rendered using updated outputs of the CLI menu, config sample, and other values.

This version of README.md was generated at Fri, 09 Sep 2022 03:23:10 +0000 based on git commit 73f5fee4.

To (re-)generate the README.md file, use the justfile command:

$ just generate-readme