Deuxfleurs/garage
42
41
Fork 57
Code Issues 79 Pull requests 9 Releases 29 Wiki Activity

Compile Garage with Nix #117

Merged
quentin merged 1 commit from feature/static into main 2021-10-19 15:43:10 +00:00
Conversation 2 Commits 1 Files changed 23 +4007 -136
quentin commented 2021-10-04 16:30:14 +00:00
Owner
Copy link

This PR provides 2 things:

  • a development environment through nix-shell
    • it embeds some publishing scripts
    • it packages and imports kaniko, a tool to build docker images inside containers
  • a building tool through nix-build
    • it supports cross compilation
    • it supports binary caching

It also implies some modifications to Drone:

  • we need a pre-configured host volume and a runner with a nix label see more here

To release a binary, pick a commit on Drone and promote it.
To release an extra binary, publish a tag and promote the Drone build.
To release a stable binary, publish a tag matching this regex v[0-9]+\.[0-9]+\.[0-9]+, and promote the Drone build.

Dev notes Nix could be an elegant way to track our dependencies and cross-compile Garage. This PR tracks my tests and must not be merged yet.

Goals

  • Easily get a shell with the same environment than the CI
  • Provides an easy way to share a common environment
  • Quickly install your development environment
  • Provides simple cross compilation / static compilation
  • Replace different tools (shell scripts, dockerfiles and Makefiles) with a single unified solution
  • Minimal disruption of developers' workflow

Features

Working:

  • Build are working and defined from nix expressions
  • Built dependencies are cached
  • We can build statically compiled binaries (except for armv7l)
  • Patch cargo2nix for target_arch - https://github.com/superboum/cargo2nix
  • Propose a development environment with shell.nix
  • Use a cache backed by S3
  • Drone integration
  • Deactivate clippy borrow check and open an issue
  • Add a promote field to Drone
  • Publish artifacts
  • Use DRONE_TAG when available
  • Generate Docker containers
  • Set up a weekly cron
  • Patch cargo2nix to support armv6
  • Maybe add a cache for releases too (especially i686)
  • Build a version listing/index
  • Update documentation
  • Automatically deploy listing/index
  • Test generated binaries on my old CPUs without AES accel
  • Clean Makefiles, scripts and Dockerfiles
  • Find a workaround for git version
  • Create a Docker org for Deuxfleurs

Remaining to do for this PR:

  • Rebase

Ideally:

  • Contribute an armv7l + musl target
  • publish multi arch containers
  • Tweaking features (not sure it is a good idea)

Not tested yet:

  • build on a tag

Targets

I also add to the command line --max-jobs auto --cores $(nproc).

[OK] Linux, x86_64, Static:

nix-build --arg crossSystem '{ config = "x86_64-unknown-linux-musl"; }'

[OK] Linux, aarch64, Static:

nix-build --arg crossSystem '{ config = "aarch64-unknown-linux-musl"; }'

[OK] Linux, i686, Static

nix-build --arg crossSystem '{ config = "i686-unknown-linux-musl"; }'

[OK] Linux, armv7l, Dynamic

nix-build --arg crossSystem '{ config = "armv7l-unknown-linux-gnueabihf"; }'

[OK] Linux, armv6lhf, Static

nix-build --arg crossSystem '{ config = "armv6l-unknown-linux-musleabihf"; }'

[KO] Windows, x86_64 - x86_64-pc-windows-gnu

not considered but could be easy:

  • linux power pc (musl-power) powerpc64le-unknown-linux-musl OR linux power pc 64bits (ppc64-musl) powerpc64-unknown-linux-musl
  • netbsd x86_64 (x86_64-netbsd) x86_64-unknown-netbsd

From the nix documentation:

armv7 is backwards-compatible with armv6 and armv5tel;

So I suggest we compile for:

  • x86_64
  • i686
  • aarch64
  • armv6

Build, Sign and Cache

Signing keys are generated with:

nix-store --generate-binary-cache-key nix.web.deuxfleurs.fr cache-priv-key.pem cache-pub-key.pem

We copy the secret key in our nix folder:

cp cache-priv-key.pem /etc/nix/signing-key.sec

Manually sign

We can sign the whole store with:

nix sign-paths --all -k /etc/nix/signing-key.sec

Or simply the current package and its dependencies with:

nix sign-paths --recursive -k /etc/nix/signing-key.sec

Setting a key in nix.conf will do the signature at build time automatically without additional commands, edit the nix.conf of your builder:

secret-key-files = /etc/nix/signing-key.sec
max-jobs = auto
cores = 8

Now you are ready to build your packages:

cat > $HOME/.awsrc <<EOF
export AWS_ACCESS_KEY_ID="xxx"
export AWS_SECRET_ACCESS_KEY="xxx"
EOF

# source each time you want to send on the cache
source ~/.awsrc

# copy garage build dependencies (and not only the output)
nix-build
nix-store -qR --include-outputs $(nix-instantiate default.nix) 
  | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'
  
# copy shell dependencies
nix-build shell.nix -A inputDerivation
nix-copy-closure result/ --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'

More example of nix-copy

# nix-build produces a result/ symlink
nix copy result/ --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'

# alternative ways to use nix copy
nix copy nixpkgs.garage --to ...
nix copy /nix/store/3rbb9qsc2w6xl5xccz5ncfhy33nzv3dp-crate-garage-0.3.0 --to ...

Clear the cache:

mc rm --recursive --force garage/nix/

A desirable nix.conf for a consumer:

substituters = https://cache.nixos.org https://nix.web.deuxfleurs.fr
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nix.web.deuxfleurs.fr:eTGL6kvaQn6cDR/F9lDYUIP9nCVR/kkshYfLDJf1yKs=

And now, whenever you run a command like:

nix-shell
nix-build

Our cache will be checked.

Legacy, if you do not want to modify nix.conf

Now, to fetch, running the following command as root worked:

source ~/.awsrc
nix-build 
  --option extra-binary-caches 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' 
  --option require-sigs false

We have many problems:

  1. We have no signature yet, and verification must be disabled from command line or nix.conf with require-sigs = false
  2. We must run the command as root as we are not a trusted user, we must add our username to nix.conf eg. trustedUsers = ["quentin", "root"]
  3. We do not want to set the cache on command line, can be set with "binary-caches = s3://xxxx https://xxxx"
  4. We might want to make this cache public

Test strategy

Goals:

  • Test fast, Fail fast
  • Tests must be run in conditions as close to the production as possible

Pseudo script for testing:

# Inject aws variables
# Inject signing key
# Configure the nix daemon

# Checks and Tests
nix-shell cargo fmt -- --check
nix-shell cargo clippy -- --deny warnings

nix-build --argstr target x86_64-unknown-linux-musl

nix-build --argstr target x86_64-unknown-linux-musl --argstr compileMode test --out-link test
./test*/bin/garage_api*
./test*/bin/garage_model*
./test*/bin/garage_rpc*
./test*/bin/garage_table*
./test*/bin/garage_util*
./test*/bin/garage_web*
./test*/bin/garage* # -- this will contain our integration tests later I think. Will be run in a nix-shell.


# need to update $PATH in the script, will be integrated later
nix-shell --run script/test-smoke.sh

# push the cache
nix-build --no-out-link shell.nix -A inputDerivation 
  | nix-copy-closure --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' 
  
nix-instantiate --argstr target x86_64-unknown-linux-musl 
  | xargs nix-store -qR --include-outputs 
  | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'

Pseudo script for promotion:

# set aws variable
# configure the nix daemon
# @TODO 

# amd64
nix-build --argstr target x86_64-unknown-linux-musl

# we might not want to keep a cache for our promotion
nix-instantiate --argstr target x86_64-unknown-linux-musl 
  | xargs nix-store -qR --include-outputs 
  | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'
  
# etc.
nix-build --argstr arch i686
nix-build --argstr arch arm 
nix-build --argstr arch aarch64
# find a way to push binaries to Garage
# find a way to build docker images
# find a way to push docker images

Drawbacks

  • Must run cargo2nix -f each time we add/change a dependency to generate Cargo.nix. Dependencies in this output file are not sorted and thus diff is not exploitable yet (there is an issue on the upstream project)
  • Build is not parallelized by default, you must use --max-jobs (build multiple derivations at once) and --cores (use multiple core to build a derivation). auto set the value to the number of cores you have.
  • If you use submodules in your crate (like crdt and replication in garage_table), you must list them in default.nix
  • Windows target does not work but might be solved. might be solvable through overrides. Indeed, we pass x86_64-pc-windows-gnu but mingw need x86_64-w64-mingw32
  • We must use my patched version of cargo2nix to enable i686 and armv7l compilation. We might need to contribute to cargo2nix in the future. At least, the package is relatively small and the amount of logic implied could be worse, and I have already read large parts of it
  • Nix has no armv7 + musl toolchains
  • git_version does not work

Drone's Drawbacks

  • Host volume does not work on Drone
  • Parallelism seems to work only on the same runner which is not interesting for us
  • Unable to use the official Docker plugin because
    • it requires to mount docker socket in the container (but there is no doc about it)
    • you cant set the platform when building
  • Unable to use buildah because it needs CLONE_USERNS capability
  • Unable to use the kaniko plugin for Drone as we can't set the target platform

Ref

This PR provides 2 things: - a development environment through `nix-shell` - it embeds some publishing scripts - it packages and imports kaniko, a tool to build docker images inside containers - a building tool through `nix-build` - it supports cross compilation - it supports binary caching It also implies some modifications to Drone: - we need a pre-configured host volume and a runner with a nix label [see more here](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/branch/main/app/drone-ci/integration) To release a binary, pick a commit on Drone and promote it. To release an *extra* binary, publish a tag and promote the Drone build. To release a *stable* binary, publish a tag matching this regex `v[0-9]+\.[0-9]+\.[0-9]+`, and promote the Drone build. <details><summary>Dev notes</summary> Nix could be an elegant way to track our dependencies and cross-compile Garage. This PR tracks my tests and must not be merged yet. ## Goals - Easily get a shell with the same environment than the CI - Provides an easy way to share a common environment - Quickly install your development environment - Provides simple cross compilation / static compilation - Replace different tools (shell scripts, dockerfiles and Makefiles) with a single unified solution - Minimal disruption of developers' workflow ## Features Working: - Build are working and defined from nix expressions - Built dependencies are cached - We can build statically compiled binaries (except for `armv7l`) - Patch cargo2nix for target_arch - https://github.com/superboum/cargo2nix - Propose a development environment with `shell.nix` - Use a cache backed by S3 - Drone integration - Deactivate clippy borrow check and open an issue - Add a promote field to Drone - Publish artifacts - Use DRONE_TAG when available - Generate Docker containers - Set up a weekly cron - Patch cargo2nix to support armv6 - Maybe add a cache for releases too (especially i686) - Build a version listing/index - Update documentation - Automatically deploy listing/index - Test generated binaries on my old CPUs without AES accel - Clean Makefiles, scripts and Dockerfiles - Find a workaround for git version - Create a Docker org for Deuxfleurs Remaining to do for this PR: - Rebase Ideally: - Contribute an armv7l + musl target - publish multi arch containers - Tweaking features (not sure it is a good idea) Not tested yet: - build on a tag ## Targets I also add to the command line `--max-jobs auto --cores $(nproc)`. [OK] **Linux, x86_64, Static**: ```bash nix-build --arg crossSystem '{ config = "x86_64-unknown-linux-musl"; }' ``` [OK] **Linux, aarch64, Static**: ``` nix-build --arg crossSystem '{ config = "aarch64-unknown-linux-musl"; }' ``` [OK] **Linux, i686, Static** ``` nix-build --arg crossSystem '{ config = "i686-unknown-linux-musl"; }' ``` [OK] **Linux, armv7l, Dynamic** ``` nix-build --arg crossSystem '{ config = "armv7l-unknown-linux-gnueabihf"; }' ``` [OK] **Linux, armv6lhf, Static** ``` nix-build --arg crossSystem '{ config = "armv6l-unknown-linux-musleabihf"; }' ``` [KO] Windows, x86_64 - `x86_64-pc-windows-gnu` not considered but could be easy: - linux power pc (musl-power) `powerpc64le-unknown-linux-musl` OR linux power pc 64bits (ppc64-musl) `powerpc64-unknown-linux-musl` - netbsd x86_64 (x86_64-netbsd) `x86_64-unknown-netbsd` From the nix documentation: > armv7 is backwards-compatible with armv6 and armv5tel; So I suggest we compile for: - x86_64 - i686 - aarch64 - armv6 ## Build, Sign and Cache Signing keys are generated with: ``` nix-store --generate-binary-cache-key nix.web.deuxfleurs.fr cache-priv-key.pem cache-pub-key.pem ``` We copy the secret key in our nix folder: ``` cp cache-priv-key.pem /etc/nix/signing-key.sec ``` Manually sign We can sign the whole store with: ``` nix sign-paths --all -k /etc/nix/signing-key.sec ``` Or simply the current package and its dependencies with: ``` nix sign-paths --recursive -k /etc/nix/signing-key.sec ``` Setting a key in `nix.conf` will do the signature at build time automatically without additional commands, edit the `nix.conf` of your builder: ```toml secret-key-files = /etc/nix/signing-key.sec max-jobs = auto cores = 8 ``` Now you are ready to build your packages: ```bash cat > $HOME/.awsrc <<EOF export AWS_ACCESS_KEY_ID="xxx" export AWS_SECRET_ACCESS_KEY="xxx" EOF # source each time you want to send on the cache source ~/.awsrc # copy garage build dependencies (and not only the output) nix-build nix-store -qR --include-outputs $(nix-instantiate default.nix) | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' # copy shell dependencies nix-build shell.nix -A inputDerivation nix-copy-closure result/ --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' ``` More example of nix-copy ``` # nix-build produces a result/ symlink nix copy result/ --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' # alternative ways to use nix copy nix copy nixpkgs.garage --to ... nix copy /nix/store/3rbb9qsc2w6xl5xccz5ncfhy33nzv3dp-crate-garage-0.3.0 --to ... ``` Clear the cache: ```bash mc rm --recursive --force garage/nix/ ``` --- A desirable `nix.conf` for a consumer: ```toml substituters = https://cache.nixos.org https://nix.web.deuxfleurs.fr trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nix.web.deuxfleurs.fr:eTGL6kvaQn6cDR/F9lDYUIP9nCVR/kkshYfLDJf1yKs= ``` And now, whenever you run a command like: ``` nix-shell nix-build ``` Our cache will be checked. Legacy, if you do not want to modify `nix.conf` Now, to fetch, running the following command as root worked: ```bash source ~/.awsrc nix-build --option extra-binary-caches 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' --option require-sigs false ``` We have many problems: 1. We have no signature yet, and verification must be disabled from command line or `nix.conf` with `require-sigs = false` 2. We must run the command as root as we are not a trusted user, we must add our username to `nix.conf` eg. `trustedUsers = ["quentin", "root"]` 3. We do not want to set the cache on command line, can be set with "binary-caches = s3://xxxx https://xxxx" 4. We might want to make this cache public ## Test strategy Goals: - Test fast, Fail fast - Tests must be run in conditions as close to the production as possible Pseudo script for testing: ```bash # Inject aws variables # Inject signing key # Configure the nix daemon # Checks and Tests nix-shell cargo fmt -- --check nix-shell cargo clippy -- --deny warnings nix-build --argstr target x86_64-unknown-linux-musl nix-build --argstr target x86_64-unknown-linux-musl --argstr compileMode test --out-link test ./test*/bin/garage_api* ./test*/bin/garage_model* ./test*/bin/garage_rpc* ./test*/bin/garage_table* ./test*/bin/garage_util* ./test*/bin/garage_web* ./test*/bin/garage* # -- this will contain our integration tests later I think. Will be run in a nix-shell. # need to update $PATH in the script, will be integrated later nix-shell --run script/test-smoke.sh # push the cache nix-build --no-out-link shell.nix -A inputDerivation | nix-copy-closure --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' nix-instantiate --argstr target x86_64-unknown-linux-musl | xargs nix-store -qR --include-outputs | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' ``` Pseudo script for promotion: ```bash # set aws variable # configure the nix daemon # @TODO # amd64 nix-build --argstr target x86_64-unknown-linux-musl # we might not want to keep a cache for our promotion nix-instantiate --argstr target x86_64-unknown-linux-musl | xargs nix-store -qR --include-outputs | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' # etc. nix-build --argstr arch i686 nix-build --argstr arch arm nix-build --argstr arch aarch64 # find a way to push binaries to Garage # find a way to build docker images # find a way to push docker images ``` ## Drawbacks * Must run `cargo2nix -f` each time we add/change a dependency to generate `Cargo.nix`. Dependencies in this output file are not sorted and thus `diff` is not exploitable yet (there is an issue on the upstream project) * Build is not parallelized by default, you must use `--max-jobs` (build multiple derivations at once) and `--cores` (use multiple core to build a derivation). `auto` set the value to the number of cores you have. * If you use submodules in your crate (like `crdt` and `replication` in `garage_table`), you must list them in `default.nix` * Windows target does not work but might be solved. might be solvable through [overrides](https://github.com/cargo2nix/cargo2nix/blob/master/overlay/overrides.nix). Indeed, we pass `x86_64-pc-windows-gnu` but mingw need `x86_64-w64-mingw32` * We must use my patched version of cargo2nix to enable i686 and armv7l compilation. We might need to contribute to cargo2nix in the future. At least, the package is relatively small and the amount of logic implied could be worse, and I have already read large parts of it * Nix has no armv7 + musl toolchains * `git_version` does not work ## Drone's Drawbacks - Host volume does not work on Drone - Parallelism seems to work only on the same runner which is not interesting for us - Unable to use the official Docker plugin because - it requires to mount docker socket in the container (but there is no doc about it) - you cant set the platform when building - Unable to use buildah because it needs CLONE_USERNS capability - Unable to use the kaniko plugin for Drone as we can't set the target platform ## Ref - https://doc.rust-lang.org/nightly/rustc/platform-support.html - https://nix.dev/tutorials/cross-compilation - https://nixos.org/manual/nix/unstable/package-management/s3-substituter.html - https://fzakaria.com/2020/09/28/nix-copy-closure-your-nix-shell.html - http://www.lpenz.org/articles/nixchannel/index.html </details>
quentin force-pushed feature/static from 95e5036bc3 to e1747b96d2 2021-10-18 08:05:52 +00:00 Compare
quentin force-pushed feature/static from 27e1997530 to d31dd7987f 2021-10-19 11:11:00 +00:00 Compare
quentin changed title from WIP: Compile Garage with Nix to Compile Garage with Nix 2021-10-19 11:11:34 +00:00
quentin force-pushed feature/static from d31dd7987f to 46ad81ddc0 2021-10-19 11:37:46 +00:00 Compare
lx approved these changes 2021-10-19 11:44:18 +00:00
lx left a comment
Owner
Copy link

I'm very happy with this PR!

I agree we should merge this ASAP, as long as it is not in a broken state (I trust Quentin for this). Most (all?) of the remarks here can be kept for other PRs we will open later.

  • Question: do we really need Cargo.nix to be in the repository? If it is deterministcally generated from Cargo.lock, can't we generate it dynamically when it is needed?

  • About the patched cargo2nix, my worry is that we will have to maintain the patched version forever. We should therefore try to either 1/ upstream our patch, OR 2/ find a workaround to do the same thing with the non-patched version.

  • Why exactly does git_version! not work when building using Nix? Is there a solution to this? Has anyone else had this problem when building with Nix, and what solution have they found?

  • For the CI it's good we have a pipeline that is as fast as possible for all commits, and that does the minimum. This means for example that we want to parallelize steps as much as possible, we don't want to lose time rebuilding caches (we can do this asynchronously with a cron), etc. I think we're already gaining a lot of speed with the Nix docker volume, can we go further and split the pipeline that does build+test from the one that does fmt+clippy? (so that they can run in parallel)

I'm very happy with this PR! I agree we should merge this ASAP, as long as it is not in a broken state (I trust Quentin for this). Most (all?) of the remarks here can be kept for other PRs we will open later. - Question: do we really need `Cargo.nix` to be in the repository? If it is deterministcally generated from `Cargo.lock`, can't we generate it dynamically when it is needed? - About the patched `cargo2nix`, my worry is that we will have to maintain the patched version forever. We should therefore try to either 1/ upstream our patch, OR 2/ find a workaround to do the same thing with the non-patched version. - Why exactly does `git_version!` not work when building using Nix? Is there a solution to this? Has anyone else had this problem when building with Nix, and what solution have they found? - For the CI it's good we have a pipeline that is as fast as possible for all commits, and that does the minimum. This means for example that we want to parallelize steps as much as possible, we don't want to lose time rebuilding caches (we can do this asynchronously with a cron), etc. I think we're already gaining a lot of speed with the Nix docker volume, can we go further and split the pipeline that does build+test from the one that does fmt+clippy? (so that they can run in parallel)
.drone.yml
@ -12,1 +15,4 @@
environment:
HOME: /drone/garage
steps:
lx commented 2021-10-19 10:19:49 +00:00
Owner
Copy link

To speed up the CI, I was thinking we could do these steps in parallel pipelines:

  • pipeline 1 = build + test-smoke
  • pipeline 2 = fmt + clippy
To speed up the CI, I was thinking we could do these steps in parallel pipelines: - pipeline 1 = build + test-smoke - pipeline 2 = fmt + clippy
quentin commented 2021-10-19 14:23:39 +00:00
Author
Owner
Copy link

I think we should study that in a separate PR

I think we should study that in a separate PR
quentin marked this conversation as resolved
.drone.yml Outdated
@ -106,0 +107,4 @@
- |
nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' \
$(nix-store -qR --include-outputs \
$(nix-instantiate --argstr target x86_64-unknown-linux-musl --arg release false))
lx commented 2021-10-19 10:21:33 +00:00
Owner
Copy link

Do we really want to update the cache with all intermediary builds? Its taking time in the CI and it will make the bucket grow uncontrolably. Maybe we can do this not at all commits but on a nightly trigger, OR rely on the nix docker volume exclusively for these artifacts.

Do we really want to update the cache with all intermediary builds? Its taking time in the CI and it will make the bucket grow uncontrolably. Maybe we can do this not at all commits but on a nightly trigger, OR rely on the nix docker volume exclusively for these artifacts.
quentin commented 2021-10-19 14:24:33 +00:00
Author
Owner
Copy link

it should be done only through a cron now.

it should be done only through a cron now.
quentin marked this conversation as resolved
.drone.yml Outdated
@ -140,0 +175,4 @@
temp: {}
environment:
TARGET: x86_64-unknown-linux-musl
lx commented 2021-10-19 10:23:23 +00:00
Owner
Copy link

Can't we factorize these many pipelines into only one? In application of "don't repeat yourself", it limits the number of changes to make when we want to make a change, and it limits the chance of making a mistake.

If the only reason is performance (with many pipelines builds run in parllel), maybe we can say for now it is not important, and we will have it again when we template the .yml file?

Can't we factorize these many pipelines into only one? In application of "don't repeat yourself", it limits the number of changes to make when we want to make a change, and it limits the chance of making a mistake. If the only reason is performance (with many pipelines builds run in parllel), maybe we can say for now it is not important, and we will have it again when we template the .yml file?
quentin commented 2021-10-19 14:25:48 +00:00
Author
Owner
Copy link

I think we should put that in another PR, where we use a different CI or we use jsonnet or starlark

I think we should put that in another PR, where we use a different CI or we use jsonnet or starlark
quentin marked this conversation as resolved
.drone.yml Outdated
@ -140,0 +207,4 @@
- name: nix_config
path: /etc/nix
commands:
- nix-shell --run ./script/test-smoke.sh || (cat /tmp/garage.log; false)
lx commented 2021-10-19 10:24:29 +00:00
Owner
Copy link

Do we really want to run this here? It was already ran for this commit in the non-promoted pipelines.

Do we really want to run this here? It was already ran for this commit in the non-promoted pipelines.
quentin commented 2021-10-19 14:13:22 +00:00
Author
Owner
Copy link

I am convinced we want to run our tests on the release binary we will publish and not suppose that the same source code built earlier in debug will generate this time a working release build.

(Let's suppose we have 2 runners and 1 is buggy in a very strange way. We build on debug on the 1st one and we release on the 2nd one a non-tested binary that is broken)

I am convinced we want to run our tests on the release binary we will publish and not suppose that the same source code built earlier in debug will generate this time a working release build. (Let's suppose we have 2 runners and 1 is buggy in a very strange way. We build on debug on the 1st one and we release on the 2nd one a non-tested binary that is broken)
quentin marked this conversation as resolved
Makefile Outdated
@ -4,4 +0,0 @@
clear; cargo build
doc:
cd doc/book; mdbook build
lx commented 2021-10-19 10:26:00 +00:00
Owner
Copy link

Can we please keep at least the all and doc targets?

A makefile acts like a kind of documentation: many devs will just try typing "make" to see what happens, and it can bootstrap them quickly if they don't know the cargo or mdbook commands.

Can we please keep at least the `all` and `doc` targets? A makefile acts like a kind of documentation: many devs will just try typing "make" to see what happens, and it can bootstrap them quickly if they don't know the cargo or mdbook commands.
quentin commented 2021-10-19 14:26:09 +00:00
Author
Owner
Copy link

Done

Done
quentin marked this conversation as resolved
doc/book/src/development/devenv.md Outdated
@ -18,0 +50,4 @@
```
nix-build
```
lx commented 2021-10-19 11:28:26 +00:00
Owner
Copy link

Mention here the following variants:

  • nix-build -j <n> for faster build
  • nix-build --arg release true for optimized build
Mention here the following variants: - `nix-build -j <n>` for faster build - `nix-build --arg release true` for optimized build
quentin commented 2021-10-19 14:14:58 +00:00
Author
Owner
Copy link

If you copied our nix.conf file, you should already have -j xx preconfigured. Will try to explain that better

If you copied our nix.conf file, you should already have `-j xx` preconfigured. Will try to explain that better
quentin commented 2021-10-19 14:37:20 +00:00
Author
Owner
Copy link

Modified with many examples

Modified with many examples
quentin marked this conversation as resolved
doc/book/src/development/devenv.md Outdated
@ -18,0 +60,4 @@
cargo2nix -f
```
Many tools like rclone, mc (minio-client), or awscli will be available in your environment and will be useful to test Garage.
lx commented 2021-10-19 11:33:05 +00:00
Owner
Copy link

the mention of awscli is misleading, the tool is actually called aws on the command line (aws s3 cp ...)

the mention of `awscli` is misleading, the tool is actually called `aws` on the command line (`aws s3 cp ...`)
quentin commented 2021-10-19 13:27:03 +00:00
Author
Owner
Copy link

aws is the name of the command, awscli is the name of the package.

Some references:

But I think I will put the command first and the package name between parenthesis.

`aws` is the name of the command, `awscli` is the name of the package. Some references: - https://src.fedoraproject.org/rpms/awscli - https://docs.aws.amazon.com/en_us/cli/latest/userguide/install-cliv2-linux.html#cliv2-linux-install - https://aur.archlinux.org/packages/aws-cli-v2/ - https://pypi.org/project/awscliv2/ But I think I will put the command first and the package name between parenthesis.
quentin marked this conversation as resolved
doc/book/src/development/release_process.md Outdated
@ -0,0 +163,4 @@
drone info
```
The CLI tool is very self-discoverable, just happen `--help` to each subcommands.
lx commented 2021-10-19 11:33:46 +00:00
Owner
Copy link

append*

append*
quentin marked this conversation as resolved
lx commented 2021-10-19 11:45:50 +00:00
Owner
Copy link

Forgot this comment:

  • the .drone.yml is long and I don't like the fact that we are repeating the same thing three or four times. If we could merge these four sections into only one (at the cost of not running them in parallel) that would be nice for this PR.
Forgot this comment: - the `.drone.yml` is long and I don't like the fact that we are repeating the same thing three or four times. If we could merge these four sections into only one (at the cost of not running them in parallel) that would be nice for this PR.
quentin commented 2021-10-19 13:18:10 +00:00
Author
Owner
Copy link

Question: do we really need Cargo.nix to be in the repository? If it is deterministcally generated from Cargo.lock, can't we generate it dynamically when it is needed?

The output .nix file is not deterministic (the tool has some unsorted arrays that are not always output in the same order) but the generated derivation should be deterministic.

Integrating in our Nix build may not be that simple as currently this file is a dependency of our build.

We can still keep in mind this for a future PR.

About the patched cargo2nix, my worry is that we will have to maintain the patched version forever. We should therefore try to either 1/ upstream our patch, OR 2/ find a workaround to do the same thing with the non-patched version.

I have already opened a PR here: https://github.com/cargo2nix/cargo2nix/pull/201.
There is no possible workaround, the only option would be to drop cargo2nix.

Why exactly does git_version! not work when building using Nix? Is there a solution to this? Has anyone else had this problem when building with Nix, and what solution have they found?

Because git_version calls git and expect a .git folder. In Nix, we copy only the files of each crate in a clean environment, we do not have access to the .git folder. In many cases, it is even considered an undesirable side effect, here we have a sub-crate (src/garage) that is dependant of the main project (.git) and this information is not shipped when the program is installed via cargo bin or through a tarball.

I do not have a precise example but, when programs have undesired side effects at compile time, often nix packagers just patch them. A similar example is when a program try to put its build date in the final binary.

I think having an environment variable at build time is the most packaging friendly way to track fron which source the program has been compiled, and it can be adapted to many packaging approaches.

For the CI it's good we have a pipeline that is as fast as possible for all commits, and that does the minimum. This means for example that we want to parallelize steps as much as possible, we don't want to lose time rebuilding caches (we can do this asynchronously with a cron), etc. I think we're already gaining a lot of speed with the Nix docker volume, can we go further and split the pipeline that does build+test from the one that does fmt+clippy? (so that they can run in parallel)

We might do that but we could also improve cargo2nix to cache clippy intermediate steps. But it is blocked by this issue: https://github.com/cargo2nix/cargo2nix/issues/107

> Question: do we really need Cargo.nix to be in the repository? If it is deterministcally generated from Cargo.lock, can't we generate it dynamically when it is needed? The output .nix file is not deterministic (the tool has some unsorted arrays that are not always output in the same order) but the generated derivation should be deterministic. Integrating in our Nix build may not be that simple as currently this file is a dependency of our build. We can still keep in mind this for a future PR. > About the patched cargo2nix, my worry is that we will have to maintain the patched version forever. We should therefore try to either 1/ upstream our patch, OR 2/ find a workaround to do the same thing with the non-patched version. I have already opened a PR here: https://github.com/cargo2nix/cargo2nix/pull/201. There is no possible workaround, the only option would be to drop cargo2nix. > Why exactly does git_version! not work when building using Nix? Is there a solution to this? Has anyone else had this problem when building with Nix, and what solution have they found? Because git_version calls git and expect a `.git` folder. In Nix, we copy only the files of each crate in a clean environment, we do not have access to the `.git` folder. In many cases, it is even considered an undesirable side effect, here we have a sub-crate (src/garage) that is dependant of the main project (.git) and this information is not shipped when the program is installed via `cargo bin` or through a tarball. I do not have a precise example but, when programs have undesired side effects at compile time, often nix packagers just patch them. A similar example is when a program try to put its build date in the final binary. I think having an environment variable at build time is the most packaging friendly way to track fron which source the program has been compiled, and it can be adapted to many packaging approaches. > For the CI it's good we have a pipeline that is as fast as possible for all commits, and that does the minimum. This means for example that we want to parallelize steps as much as possible, we don't want to lose time rebuilding caches (we can do this asynchronously with a cron), etc. I think we're already gaining a lot of speed with the Nix docker volume, can we go further and split the pipeline that does build+test from the one that does fmt+clippy? (so that they can run in parallel) We might do that but we could also improve cargo2nix to cache clippy intermediate steps. But it is blocked by this issue: https://github.com/cargo2nix/cargo2nix/issues/107
quentin force-pushed feature/static from 46ad81ddc0 to dc017a0cab 2021-10-19 14:56:21 +00:00 Compare
quentin merged commit dc017a0cab into main 2021-10-19 15:43:10 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
lx
Labels
Clear labels   
AdminAPI

Issues concerning the administration API

  
Bug

   
Check AWS

The behaviour should be checked against AWS S3

  
CI

Issues with Nix, Drone, Rust Compiler, etc.

  
Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  
Critical

   
Documentation

   
Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
Improvement

   
Low priority

For issues that we don't want to spend time on until made absolutely necessary

  
Newcomer

A bug that should be newcomer friendly.

  
Performance

   
S3 Compatibility

   
Testing

   
Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No labels
AdminAPI
Bug
Check AWS
CI
Correctness
Critical
Documentation
Ideas
Improvement
Low priority
Newcomer
Performance
S3 Compatibility
Testing
Usability
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#117
No description provided.
Delete branch "feature/static"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?