rust-fil-proofs

# Filecoin Proving Subsystem The **Filecoin Proving Subsystem** (or FPS) provides the storage proofs required by the Filecoin protocol. It is implemented entirely in Rust, as a series of partially inter-dependent crates – some of which export C bindings to the supported API. There are currently several different crates: - [**Storage Proofs (`storage-proofs`)**](./storage-proofs) A library for constructing storage proofs – including non-circuit proofs, corresponding SNARK circuits, and a method of combining them. - [**Storage Proofs Core (`storage-proofs-core`)**](./storage-proofs/core) A set of common primitives used throughout the other storage-proofs sub-crates, including crypto, merkle tree, hashing and gadget interfaces. - [**Storage Proofs PoRep (`storage-proofs-porep`)**](./storage-proofs/porep) `storage-proofs-porep` is intended to serve as a reference implementation for _**Proof-of-Replication**_ (**PoRep**), while also performing the heavy lifting for `filecoin-proofs`. Primary Components: - **PoR** (**_Proof-of-Retrievability_**: Merkle inclusion proof) - **DrgPoRep** (_Depth Robust Graph_ **_Proof-of-Replication_**) - **StackedDrgPoRep** - [**Storage Proofs PoSt (`storage-proofs-post`)**](./storage-proofs/post) `storage-proofs-post` is intended to serve as a reference implementation for _**Proof-of-Space-time**_ (**PoSt**), for `filecoin-proofs`. Primary Components: - **PoSt** (Proof-of-Spacetime) - [**Filecoin Proofs (`filecoin-proofs`)**](./filecoin-proofs) A wrapper around `storage-proofs`, providing an FFI-exported API callable from C (and in practice called by [lotus](https://github.com/filecoin-project/lotus) via cgo). Filecoin-specific values of setup parameters are included here. ## Security Audits The `rust-fil-proofs` proofs code and the [Filecoin Spec](https://bafybeidxw5vxjdwsun2zc2illagf43v6w5r5w63vg455h7vjesbyqssg64.ipfs.dweb.link/algorithms/sdr/) has undergone a [proofs security audit](audits/Sigma-Prime-Protocol-Labs-Filecoin-Proofs-Security-Review-v2.1.pdf) performed by [Sigma Prime](https://sigmaprime.io/) and been deemed free of *critical* or *major* security issues. In addition to the security review, the document provides the summary of findings, vulnerability classifications, and recommended resolutions. All known issues have been resolved to date in both the code and the specification. `rust-fil-proofs` has also undergone a [SNARK proofs security audit performed by Dr. Jean-Philippe Aumasson and Antony Vennard](audits/protocolai-audit-20200728.pdf) and been deemed free of *critical* or *major* security issues. In addition to the security analysis, the document provides the audit goals, methodology, functionality descriptions and finally observations on what could be improved. All known issues have been resolved to date. ## Design Notes Earlier in the design process, we considered implementing what has become the **FPS** in Go – as a wrapper around potentially multiple SNARK circuit libraries. We eventually decided to use [bellman](https://github.com/zkcrypto/bellman) – a library developed by Zcash, which supports efficient pedersen hashing inside of SNARKs. Having made that decision, it was natural and efficient to implement the entire subsystem in Rust. We considered the benefits (self-contained codebase, ability to rely on static typing across layers) and costs (developer ramp-up, sometimes unwieldiness of borrow-checker) as part of that larger decision and determined that the overall project benefits (in particular ability to build on Zcash’s work) outweighed the costs. We also considered whether the **FPS** should be implemented as a standalone binary accessed from Filecoin nodes either as a single-invocation CLI or as a long-running daemon process. Bundling the **FPS** as an FFI dependency was chosen for both the simplicity of having a Filecoin node deliverable as a single monolithic binary, and for the (perceived) relative development simplicity of the API implementation. If at any point it were to become clear that the FFI approach is irredeemably problematic, the option of moving to a standalone **FPS** remains. However, the majority of technical problems associated with calling from Go into Rust are now solved, even while allowing for a high degree of runtime configurability. Therefore, continuing down the same path we have already invested in, and have begun to reap rewards from, seems likely. ## Install and configure Rust **NOTE:** If you have installed `rust-fil-proofs` incidentally, as a submodule of `lotus`, then you may already have installed Rust. The instructions below assume you have independently installed `rust-fil-proofs` in order to test, develop, or experiment with it. [Install Rust using rustup.](https://www.rust-lang.org/en-US/install.html) ## Build **NOTE:** `rust-fil-proofs` can only be built for and run on 64-bit platforms; building will panic if the target architecture is not 64-bits. Before building you will need OpenCL to be installed. On Ubuntu, this can be achieved with `apt install ocl-icd-opencl-dev`. Other system dependencies such as 'gcc/clang', 'wall' and 'cmake' are also required. You will also need to install the hwloc library. On Ubuntu, this can be achieved with `apt install hwloc libhwloc-dev`. For other platforms, please see the [hwloc-rs Prerequisites section](https://github.com/daschl/hwloc-rs). ``` > cargo build --release --all ``` ## Test ``` > cargo test --all ``` ## Benchmarks The main benchmarking tool is called `benchy`. `benchy` has several subcommands, including `merkleproofs`, `prodbench`, `winning_post` and `window_post`. You can run them with various configuration options, but some examples are below: ``` > cargo run --release --bin benchy -- merkleproofs --size 2 > cargo run --release --bin benchy -- winning-post --size 2 > cargo run --release --bin benchy -- window-post --size 2 > cargo run --release --bin benchy -- prodbench ``` ### Window PoSt Bench usages The Window PoSt bench can be used a number of ways, some of which are detailed here. First, you can run the benchmark and preserve the working directory like this: ``` cargo run --release --bin benchy -- window-post --size 2KiB --cache window-post-2KiB-dir --preserve-cache ``` Then if you want to run the benchmark again to test commit-phase2, you can quickly run it like this: ``` cargo run --release --bin benchy -- window-post --size 2KiB --skip-precommit-phase1 --skip-precommit-phase2 --skip-commit-phase1 --cache window-post-2KiB-dir ``` Alternatively, if you want to test just GPU tree building, you can run it like this: ``` cargo run --release --bin benchy -- window-post --size 2KiB --skip-precommit-phase1 --skip-commit-phase1 --skip-commit-phase2 --cache window-post-2KiB-dir ``` Note that some combinations of arguments will cause destructive changes to your cached directory. For larger benchmark sector sizes, it is recommended that once you create an initial cache, that it be saved to an alternate location in the case that it is corrupted by a test run. For example, the following run sequence will be guaranteed to corrupt your cache: ``` # Do NOT run this sequence. For illustrative purposes only: # Generate clean cache cargo run --release --bin benchy -- window-post --size 2KiB --cache window-post-2KiB-dir --preserve-cache # Skip all stages except the first cargo run --release --bin benchy -- window-post --size 2KiB --skip-precommit-phase2 --skip-commit-phase1 --skip-commit-phase2 --cache broken-cache-dir ``` The reason this fails is because new random piece data is generated (rather than loaded from disk from a previous run) in the first step, and then we attempt to use it in later sealing steps using data from previously preserved run. This cannot work. There is also a bench called `gpu-cpu-test`: ``` > cargo run --release --bin gpu-cpu-test ``` Some results are displayed at the comm