trinity-1686a/garage
2
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests 1 Projects Releases Wiki Activity
2083 commits 16 branches 0 tags 42 MiB
Commit graph

30 commits

Author SHA1 Message Date
Alex Auvolat
390a5d97fe nix, ci: build with Crane 
This removes our dependency on cargo2nix, which was causing us some
issues. Whereas cargo2nix creates one Nix derivation per crate, Crane
uses only two derivations:

1. Build dependencies only
2. Build the final binary

This means that during the second step, no caching can be done. For
instance, if we do a change in garage_model, we need to recompile all of
the Garage crates including those that do not depend on garage_model.
On the upside, this allows all of the Garage crates to be built at once
using cargo build logic, which is optimized for high parallelism and
better pipelining between all of the steps of the build. All in all,
this makes most builds faster than cargo2nix.

A few other changes have been made to the build scripts and CI:

- Unit tests are now run within a Nix derivation. In fact, we have
  different derivations to run the tests using LMDB and Sqlite as
  metadata db engines.

- For debug builds, most CI steps now run in parallel (with the notable
  exception of the smoke test that runs after the build, which is
  inevitable).

- We no longer pass the GIT_VERSION argument when building debug builds
  and running the tests. This means that dev binaries and test
  binaries don't know the exact version of Garage they are from. That
  shouldn't be an issue in most cases.

- The not-dynamic.sh scripts has been fixed to fail if the file does not
  exist.
 2025-02-03 16:39:50 +01:00
Alex Auvolat
83887a8519 nix build: remove clippy build env that doesn't work 2025-01-12 17:51:33 +01:00
Florian Klink
a6e4b96ca9 shell.nix: expose devShellFull 
This allows accessing devShells.full from shell.nix.
 2024-04-23 11:59:37 +03:00
Alex Auvolat
f7ae966ed3
[nix-improvements] special case for Docker's "386" architecture 2024-02-09 12:49:17 +01:00
Alex Auvolat
561fad0b44
[nix-improvements] get rid of Drone 2024-02-09 12:19:16 +01:00
Alex Auvolat
d35d4599de
[nix-improvements] use kaniko and manifest-tools from nixpkgs, simplify 2024-02-09 11:15:52 +01:00
Alex Auvolat
9900368380
[nix-improvements] modernize Nix infrastructure 2024-02-09 11:10:13 +01:00
Alex Auvolat
8b6a44a53d
[woodpecker] convert drone pipelines to woodpecker 2024-02-08 18:24:52 +01:00
Alex Auvolat
d50fa2a562
nix: add clang to flake.nix and shell.nix 2023-10-26 11:19:22 +02:00
Alex Auvolat
d3fffd30dc use mold linker when invoking cargo manually (not in nix build scripts) 2023-10-10 13:56:48 +02:00
Alex Auvolat
f251b4721f Apply nixfmt to all .nix files; fix devshell and add it to cache 2023-01-26 12:25:48 +01:00
Alex Auvolat
60c26fbc62
Inject last modified date as git_version; flake cache uploading 2022-11-16 23:47:10 +01:00
Alex Auvolat
e76dba9561
Make repository into a Nix flake 2022-11-16 23:25:34 +01:00
Alex Auvolat
a54a63c491
Add function to upload a build and its dependencies to the cache 
to faster bootstrap new runner nodes
 2022-10-18 14:19:19 +02:00
Alex Auvolat
8fc93abc79
Some things are now in result-bin 2022-10-18 13:39:21 +02:00
Alex Auvolat
7c0c229934
move refresh_toolchain 2022-10-18 12:15:31 +02:00
Alex Auvolat
d104ae8711
Add step to generate multi-arch Docker container in CI 2022-09-26 19:09:55 +02:00
Quentin Dufour
fcb04843f7
Run clippy in nix, leveraging nix caching ability 2022-07-26 18:27:52 +02:00
Quentin Dufour
76cb34a0ae
Fail if compiled binary is dynamic 2022-07-26 18:27:46 +02:00
trinity-1686a
64c193e3db Add a K2V client library and CLI (#303) 
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#303
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-05-18 22:24:09 +02:00
Alex Auvolat
f0d0cd9a20
Remove strum crate dependency; add protobuf nix dependency 2022-03-14 10:53:00 +01:00
Max Audron
9d44127245
add support for kubernetes service discovery 
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
 2022-03-12 13:05:52 +01:00
Quentin Dufour
2a3afcaf65 Test WinSCP 2022-03-03 14:29:10 +01:00
Quentin Dufour
5d19f3d2d7
Add integration tests to Drone 2022-02-10 17:55:50 +01:00
Quentin Dufour
084dcdbd3a
Upgrade cargo2nix 2022-02-10 17:55:50 +01:00
Quentin Dufour
6f7ef11537 Generate and upload a JSON result 2022-02-05 22:09:43 +01:00
Quentin
b4592a00fe Implement ListMultipartUploads (#171) 
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
 2022-01-12 19:04:55 +01:00
Quentin Dufour
93f8d59e4c
Extract toolchain build from the CI 2021-10-29 11:34:01 +02:00
Quentin Dufour
cc1caa87fb
Use Rust binaries from Nix instead of rustup 2021-10-29 11:34:01 +02:00
Quentin Dufour
dc017a0cab
Build Garage with Nix 2021-10-19 16:56:07 +02:00