registry-backup

A command line utility for downloading all .crate files hosted by a Cargo registry server.

Use cases:

• Backup: retrieve a registry server's files for backup storage
• Export: pull the files so you can host them at another registry server

Example Usage:

Specify the registry index either as a local path (--index-path)...

$ git clone https://github.com/rust-lang/crates.io-index.git
$ RUST_LOG=info registry-backup \
    --index-path crates.io-index \
    --output-path crates.io-crate-files \
    --requests-per-second 10

...or as an --index-url instead:

$ RUST_LOG=info registry-backup \
    --index-url ssh://git@ssh.shipyard.rs/shipyard-rs/crate-index.git \
    --output-path shipyard-rs-crate-files \
    --auth-token ${AUTH_TOKEN} # for private registry, need auth

Install

$ cargo install registry-backup --git https://git.shipyard.rs/jstrong/registry-backup.git

Runtime Options

$ ./target/release/registry-backup --help

registry-backup 0.4.1
Jonathan Strong <jstrong@shipyard.rs>
Download all .crate files from a registry server

USAGE:
    registry-backup [OPTIONS]

OPTIONS:
        --index-url <URL>
            URL of the registry index we are downloading .crate files from. The program expects that
            it will be able to clone the index to a local temporary directory; the user must handle
            authentication if needed

        --index-path <PATH>
            instead of an index url, just point to a local path where the index is already cloned

    -a, --auth-token <TOKEN>
            If registry requires authorization (i.e. "auth-required" key is set to `true` in the
            `config.json` file), the token to include using the Authorization HTTP header

    -o, --output-path <PATH>
            Directory where downloaded .crate files will be saved to
            
            [default: output]

        --overwrite-existing
            Download files when if .crate file already exists in output dir for a given crate
            version, and overwrite the existing file with the new one. Default behavior is to skip
            downloading if .crate file already exists

        --output-format <FORMAT>
            What format to use for the output filenames. Works the same as Cargo's registry syntax
            for the "dl" key in the `config.json` file in a reigstry index. See [Cargo
            docs](https://doc.rust-lang.org/cargo/reference/registries.html#index-format) for
            additional details. Not specifying this field is equivalent to specifying
            "{crate}/{version}/download", the default.
            
            The resulting path specified by the format should be relative; it will be joined with
            the --output-path. (i.e. it should not start with "/".)

    -U, --user-agent <USER_AGENT>
            Value of user-agent HTTP header
            
            [default: registry-backup/v0.4.1]

    -R, --requests-per-second <INT>
            Requests to registry server will not exceed this rate
            
            [default: 100]

    -M, --max-concurrent-requests <INT>
            Independent of the requests per second rate limit, no more than
            `max_concurrent_requests` will be in flight at any given moment
            
            [default: 50]

    -c, --config-file <PATH>
            Specify configuration values using the provided TOML file, instead of via command line
            flags. The values in the config file will override any values passed as command line
            flags. See config.toml.sample for syntax of the config file

        --filter-crates <REGEX>
            Only crates with names that match --filter-crate regex will be downloaded

        --dry-run
            Don't actually download the .crate files, just list files which would be downloaded.
            Note: --requests-per-second and --max-concurrent-requests are still enforced even in
            --dry-mode!

    -h, --help
            Print help information

    -V, --version
            Print version information

Configuration File

A toml configuration file may be used instead of command line flags. A sample file (config.toml.sample) is included. From the example file:

dry-run = false
filter-crates = "^."

[registry]
index-url = "ssh://git@ssh.shipyard.rs/shipyard-rs-public/crate-index.git"
# alternatively, specify a local dir
# index-path = "/path/to/cloned/index"
auth-token = "xxx"

[http]
user-agent = "registry-backup/v0.1.0"
requests-per-second = 100
max-concurrent-requests = 50

[output]
path = "output"
overwrite-existing = false
format = "{crate}/{version}/download"

Build From Source

$ git clone https://git.shipyard.rs/jstrong/registry-backup.git
$ cd registry-backup
$ just release-build # alternatively, cargo build --bin registry-backup --release
# ./target/release/registry-backup --help
# cp target/release/registry-backup ~/.cargo/bin/

Running Tests

$ just test # alternatively, cargo test

Justfile

The repository includes a justfile with functionality for building, testing, etc.

Included commands:

$ just --list

Available recipes:
    cargo +args=''           # cargo wrapper; executes a cargo command using the settings in justfile (RUSTFLAGS, etc.)
    check +args=''           # cargo check wrapper
    debug-build +args=''     # cargo build wrapper - builds registry-backup in debug mode
    generate-readme          # generate updated README.md
    get-crate-version
    install                  # cargo install registry-backup via git dep
    pre-release              # check, run tests, check non-error output for clippy, run rustfmt
    release                  # release version (regenerate docs, git tag v0.0.0)
    release-build +args=''   # cargo build --release wrapper - builds registry-backup in release mode
    release-prep             # get everything all ready for release
    show-build-env           # diagnostic command for viewing value of build variables at runtime
    test +args=''            # cargo test wrapper
    update-readme            # re-generate README.md and overwrite existing file with output
    update-readme-and-commit # re-generate, overwrite, stage, and commit
    update-readme-and-stage  # re-generate, overwrite, and stage changes
    verify-clean-git         # verify no uncommitted changes

The commands that mirror cargo commands (e.g. just test) are included for the purpose of convenience, so that various options (e.g. RUSTFLAGS='-C target-cpu=native) can be included without typing them out each time.

Generating README.md

This file is generated using a template (doc/README.tera.md) rendered using updated outputs of the CLI menu, config sample, and other values.

This version of README.md was generated at Thu, 08 Dec 2022 02:23:17 +0000 based on git commit 3241e207.

To (re-)generate the README.md file, use the justfile command:

$ just generate-readme