Commit graph

159 commits

Author SHA1 Message Date
e645bbd3ce smoke test: add multipart upload test with part re-upload 2023-06-09 16:23:37 +02:00
58563ed700 Add multipart upload using aws s3api 2023-06-09 16:23:37 +02:00
ea3bfd2ab1 Minio tests for multipart upload behaviour:
- upload part renumbering test
- part skipping test
2023-06-09 16:23:37 +02:00
351d734e6c Merge branch 'main' into next 2023-05-09 12:40:08 +02:00
4ea7983093
Helm: Increment patch version 2023-05-08 08:03:21 +02:00
d5e39d11eb
Helm: Include newer config parameters as values
Add all missing parameters from the reference manual.
Primarily to enable the use of the new lmdb engine
2023-05-08 07:47:31 +02:00
fa78d806e3 Merge branch 'main' into next 2023-04-25 12:34:26 +02:00
66ed0bdd91
Update Helm chart versions (app + chart)
* chart version: 0.4.0
* app version: v0.8.2
2023-03-23 20:20:46 +01:00
50bce43f25
refactor(helm): use stable as image tag for init container 2023-01-27 00:08:33 +01:00
ac6751f509
doc(helm): removed extra line 2023-01-27 00:08:33 +01:00
b999bb36af
feat(helm): ability to monitor garage via prometheus 2023-01-27 00:08:33 +01:00
d20e8c9256
feat(helm): allow to override the init container image 2023-01-27 00:08:32 +01:00
fd03b184b3
fix(helm): file permission issues when running as non-root user
Specify the user group for the garage (and init) process and ensure
that the persistent storage is mounted with the correct file system
group
2023-01-27 00:08:32 +01:00
da6f7b0dda
feat(helm): ensure that config changes trigger a pod rollout 2023-01-27 00:08:32 +01:00
e17970773a
refactor(helm): removed metadataDir and dataDir config variable
The variables were only templated into the configuration file and
did not change the pod mountpaths, so the variables were not necessary
2023-01-27 00:08:32 +01:00
88b66c69a5
feat(helm): allow to override the default configuration file
Signed-off-by: Patrick Jahns <kontakt@patrickjahns.de>
2023-01-27 00:08:31 +01:00
80e4abb98d Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' () from jpds/garage:key-create-standardize into next
Reviewed-on: 
2023-01-04 10:35:49 +00:00
570e5e5bbb
Merge branch 'main' into next 2023-01-04 11:34:43 +01:00
Jonathan Davies
8be862aa19 Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time. 2023-01-03 11:11:12 +00:00
559e924cc2 Bump the helm chart version 2022-12-25 13:33:44 +01:00
e9b0068079 Set hostPath type for volumes 2022-12-25 13:30:14 +01:00
49a138b670 Fix volume handling and persistence flag 2022-12-25 13:30:14 +01:00
e94d6f78d7 Enable daemonset deployment using the helm chart
DaemonSet is a k8s resource that schedules one instance per node,
which is useful for some garage deployment use cases, including
managing garage nodes using k8s node labels
2022-12-25 13:30:14 +01:00
7a0014b6f7 chore(helm): bump chart number 2022-12-11 23:11:56 +00:00
edb0b9c1ee feat(helm): allow to add custom labels to created ingress resources 2022-12-11 23:11:56 +00:00
f58a813a36 refactor(helm): disable the ingress per default
The default values forces people to create an ingress resources,
where per default an ingress is not necessary to start garage.

If someone wants to utilize an ingress, he would need to define
the values for the ingress either way, so enabling the ingress
explicitly makes more sense, then requiring it to be disabled per default
2022-12-11 23:11:56 +00:00
2c2e65ad8b
Merge commit 'ec12d6c' into next 2022-12-11 18:41:15 +01:00
a7af0c8af9
Add best practices and doc of monitoring (fix ) 2022-11-16 13:27:24 +01:00
fc2729cd81
Fix integration test 2022-11-08 15:19:46 +01:00
d6c77ea327
Fix helm chart with correct configuration syntax 2022-10-18 22:30:05 +02:00
8fc93abc79
Some things are now in result-bin 2022-10-18 13:39:21 +02:00
db0c8b3980 Updates values.yml with some opinionated and untested defaults 2022-09-30 18:46:57 +02:00
6dba7dadf4 Add missing ClusterRole and bindings for CRDs 2022-09-30 18:46:57 +02:00
b71fa2ddf4 Generate random RPC secret if not provided 2022-09-30 18:46:57 +02:00
37a73d7d37 Move documentation to book 2022-09-30 18:46:57 +02:00
d0f08c254e Add secret to overrides 2022-09-30 18:46:57 +02:00
fa52558ca1 Add configuration instructions to README 2022-09-30 18:46:57 +02:00
131cc2532b Cleanup values.yaml 2022-09-30 18:46:57 +02:00
a93dcce841 Add helm chart 2022-09-30 18:46:57 +02:00
e935861854
Factor out node request order selection logic & use in manager 2022-07-29 12:25:03 +02:00
76cb34a0ae
Fail if compiled binary is dynamic 2022-07-26 18:27:46 +02:00
62f0715abe Add/Fix OpenTelemetry 2022-04-07 16:12:35 +02:00
7e1ac51b58 Add files to quickly test k8s 2022-04-07 16:12:35 +02:00
bfb4353df5
Update Grafana dashboard 2022-03-14 10:55:30 +01:00
a19341b188
Add Grafana dashboard for Garage 2022-03-14 10:54:25 +01:00
dc8d0496cc
Refactoring: rename config files, make modifications less invasive 2022-03-14 10:53:51 +01:00
b6561f6e1b
Add docker-compose for traces & metrics 2022-03-14 10:51:52 +01:00
mricher
e349af13a7
Update dependencies and add admin module with metrics
- Global dependencies updated in Cargo.lock
- New module created in src/admin to host:
  - the (future) admin REST API
  - the metric collection
- add configuration block

No metrics implemented yet
2022-03-14 10:51:12 +01:00
c00b2c9948 Functional tests for admin commands 2022-03-07 17:32:07 +01:00
8df1e186de Functional tests for website endpoints 2022-03-07 17:32:07 +01:00
2ef60b8417 Functional test for multipart endpoints 2022-03-07 17:32:07 +01:00
1e639ec67c Functional test for ListMultipartUploads 2022-03-07 17:32:07 +01:00
0f4e0e8bb9 Move ListObjects tests to Rust 2022-03-07 17:05:10 +01:00
2a3afcaf65 Test WinSCP 2022-03-03 14:29:10 +01:00
935670690f
Probably fix test-smoke 2022-02-02 17:34:19 +01:00
f67029ce2a Improve testing conf + test CORS 2022-02-01 17:55:14 +01:00
c99f55c420
Add restriction on part ordering in CompleteMultipartUpload 2022-01-25 12:45:00 +01:00
94f0e7c135 Test ListParts endpoint with awscli 2022-01-21 10:42:35 +01:00
7c049f1c94 Fix extreme value to be less extreme so that integration test works on 32bits 2022-01-17 12:56:29 +01:00
b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD () ()
Closes .

Reviewed-on: 
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 10:55:31 +01:00
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
b4592a00fe Implement ListMultipartUploads ()
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: 
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
1eb972b1ac Add compression using zstd ()
fix 

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: 
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 11:26:43 +01:00
8811bb08e6
In ListBuckets, hide entry if no perms 2021-11-22 12:10:28 +01:00
85b2e4ca29
Start socat only once
Fixes 
2021-11-17 10:59:32 +01:00
c94406f428
Improve how node roles are assigned in Garage
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
2021-11-16 16:05:53 +01:00
de4276202a
Improve CLI, adapt tests, update documentation 2021-10-25 14:21:48 +02:00
dc017a0cab
Build Garage with Nix 2021-10-19 16:56:07 +02:00
ddb2b29bfd
Rename datacenters into zones (doc not yet updated) 2021-05-28 14:07:36 +02:00
b490ebc7f6
Many improvements on ring/replication and its configuration:
- Explicit "replication_mode" configuration parameters that takes
  either "none", "2" or "3" as values, instead of letting user configure
  replication factor themselves. These are presets whose corresponding
  replication/quorum values can be found in replication/mode.rs

- Explicit support for single-node and two-node deployments
  (number of nodes must be at least "replication_mode", with "none"
  we can have only one node)

- Ring is now stored much more compactly with 256*8 + n*32 bytes,
  instead of 256*32 bytes

- Support for gateway-only nodes that do not store data
  (these nodes still need a metadata_directory to store the list
  of bucket and keys since those are stored on all nodes; it also
  technically needs a data_directory to start but it will stay
  empty unless we have bugs)
2021-05-28 14:07:36 +02:00
c8aa1eb481
Add preliminary support for Duck 2021-05-15 10:24:20 +02:00
88925ebe22
add clippy to the CI 2021-05-03 22:11:37 +02:00
631c36b3ff S3 API: support ListBuckets 2021-05-03 21:55:30 +02:00
ef4d6e782a
Add minio & rclone to our functional tests
It is now possible to configure which clients
you do not want to test with the env variable SKIP_XXX=1,
XXX being the client name. eg. SKIP_S3CMD=1 ./script/test-smoke.sh
2021-05-02 14:59:58 +02:00
e01f74e763
Introduce test case that demonstrates (the & problem) 2021-04-27 23:09:26 +02:00
640e3921d8
Use pre-prepared Docker image in CI to speed things up 2021-04-27 16:37:11 +02:00
5ab33fddac Refactor CLI and prettify CLI outpu 2021-03-12 18:12:31 +01:00
6a3dcf3974 Rename n_tokens into capacity 2021-03-10 14:52:03 +01:00
5e33c3cfc9 Use smaller capacities for nodes 2021-03-05 16:58:34 +01:00
d7e005251d Not fully tested: new multi-dc MagLev 2021-03-05 16:22:29 +01:00
49c25a1509 Simulate stuff moving around 2021-02-25 10:53:33 +01:00
1abbca37c4 Add adapted version of maglev for multi-dc 2021-02-21 19:14:28 +01:00
24f924afdb Maglev simulation 2021-02-21 18:32:13 +01:00
e59322041a Evaluate hash functions 2021-02-21 15:11:15 +01:00
217b0dfd68 Add script to simulate different kinds of rings 2021-02-21 14:30:26 +01:00
3b023c0c3b try to fix smoke test
dev cluster: don't ipv6 (fixes smoke test in container?)
2021-02-17 22:13:11 +01:00
36814be447 Fix S3 ListObjects result and replace println!s by debug!s 2021-01-16 16:05:54 +01:00
086e5be290 Update testing script 2020-12-17 21:04:59 +01:00
1119d466e7 Fix S3 command 2020-12-10 20:19:22 +01:00
e8c12072ce Merge branch 'master' into feature/website 2020-12-10 20:12:56 +01:00
e13fd09543 Reduce garage.1.rnd size to store it inline 2020-12-06 13:33:08 +01:00
a92868504f Indentation & comments 2020-12-06 10:23:14 +01:00
d2d1fc676d Test awscli/s3cmd interactions 2020-12-06 10:19:01 +01:00
a12930075d Test garage list & delete commands 2020-12-06 10:04:17 +01:00
28055b708f Improve README, add more tests 2020-12-06 09:13:47 +01:00
132c54b807 wip smoke test 2020-12-05 19:26:10 +01:00
8df0b322ab Fix merge error 2020-11-29 17:40:36 +01:00
a512db342e Merge branch 'master' into feature/website 2020-11-29 17:40:03 +01:00
2f11191f60 Use ipv6 localhost for dev cluster and different port numbers 2020-11-29 17:31:58 +01:00
54c3a023f0 Use aws cli version 2 2020-11-29 17:27:49 +01:00