165 commits

Author	SHA1	Message	Date
Florian Klink	cb359b4434	doc: update endpoint_url documentation ... Since `awscli` `>=1.29.0` or `>=2.13.0` it is now possible to use the `AWS_ENDPOINT_URL` environment variable, or the `endpoint_url` config key to override the endpoint URL. This means, the aws bash function to wrap with --endpoint-url is not necessary anymore. Update invocations to reflect that. https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html https://github.com/aws/aws-cli/issues/4454#issuecomment-1626116607	2023-10-02 17:16:11 +03:00
Alex Auvolat	9ac1d5be0e	add upgrade test for garage 0.8 -> 0.9	2023-09-27 14:57:37 +02:00
Alex Auvolat	ad6b1cc0be	Merge branch 'main' into next	2023-09-11 13:14:18 +02:00
Alex Auvolat	51eac97260	update version to 0.8.4	2023-09-05 23:28:12 +02:00
Alex Auvolat	2e90e1c124	Merge branch 'main' into next	2023-08-29 11:32:42 +02:00
Alex Auvolat	cece1be1bb	bump version to 0.8.3	2023-08-28 13:17:26 +02:00
Alex Auvolat	e645bbd3ce	smoke test: add multipart upload test with part re-upload	2023-06-09 16:23:37 +02:00
Alex Auvolat	58563ed700	Add multipart upload using aws s3api	2023-06-09 16:23:37 +02:00
Alex Auvolat	ea3bfd2ab1	Minio tests for multipart upload behaviour: ... - upload part renumbering test - part skipping test	2023-06-09 16:23:37 +02:00
Alex Auvolat	351d734e6c	Merge branch 'main' into next	2023-05-09 12:40:08 +02:00
Jonatan Steuernagel	4ea7983093	Helm: Increment patch version	2023-05-08 08:03:21 +02:00
Jonatan Steuernagel	d5e39d11eb	Helm: Include newer config parameters as values ... Add all missing parameters from the reference manual. Primarily to enable the use of the new lmdb engine	2023-05-08 07:47:31 +02:00
Alex Auvolat	fa78d806e3	Merge branch 'main' into next	2023-04-25 12:34:26 +02:00
Kamil Banach	66ed0bdd91	Update Helm chart versions (app + chart) ... * chart version: 0.4.0 * app version: v0.8.2	2023-03-23 20:20:46 +01:00
Patrick Jahns	50bce43f25	refactor(helm): use stable as image tag for init container	2023-01-27 00:08:33 +01:00
Patrick Jahns	ac6751f509	doc(helm): removed extra line	2023-01-27 00:08:33 +01:00
Patrick Jahns	b999bb36af	feat(helm): ability to monitor garage via prometheus	2023-01-27 00:08:33 +01:00
Patrick Jahns	d20e8c9256	feat(helm): allow to override the init container image	2023-01-27 00:08:32 +01:00
Patrick Jahns	fd03b184b3	fix(helm): file permission issues when running as non-root user ... Specify the user group for the garage (and init) process and ensure that the persistent storage is mounted with the correct file system group	2023-01-27 00:08:32 +01:00
Patrick Jahns	da6f7b0dda	feat(helm): ensure that config changes trigger a pod rollout	2023-01-27 00:08:32 +01:00
Patrick Jahns	e17970773a	refactor(helm): removed metadataDir and dataDir config variable ... The variables were only templated into the configuration file and did not change the pod mountpaths, so the variables were not necessary	2023-01-27 00:08:32 +01:00
Patrick Jahns	88b66c69a5	feat(helm): allow to override the default configuration file ... Signed-off-by: Patrick Jahns <kontakt@patrickjahns.de>	2023-01-27 00:08:31 +01:00
Alex	80e4abb98d	Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next ... Reviewed-on: #459	2023-01-04 10:35:49 +00:00
Alex Auvolat	570e5e5bbb	Merge branch 'main' into next	2023-01-04 11:34:43 +01:00
Jonathan Davies	8be862aa19	Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time.	2023-01-03 11:11:12 +00:00
kaiyou	559e924cc2	Bump the helm chart version	2022-12-25 13:33:44 +01:00
kaiyou	e9b0068079	Set hostPath type for volumes	2022-12-25 13:30:14 +01:00
kaiyou	49a138b670	Fix volume handling and persistence flag	2022-12-25 13:30:14 +01:00
kaiyou	e94d6f78d7	Enable daemonset deployment using the helm chart ... DaemonSet is a k8s resource that schedules one instance per node, which is useful for some garage deployment use cases, including managing garage nodes using k8s node labels	2022-12-25 13:30:14 +01:00
Patrick Jahns	7a0014b6f7	chore(helm): bump chart number	2022-12-11 23:11:56 +00:00
Patrick Jahns	edb0b9c1ee	feat(helm): allow to add custom labels to created ingress resources	2022-12-11 23:11:56 +00:00
Patrick Jahns	f58a813a36	refactor(helm): disable the ingress per default ... The default values forces people to create an ingress resources, where per default an ingress is not necessary to start garage. If someone wants to utilize an ingress, he would need to define the values for the ingress either way, so enabling the ingress explicitly makes more sense, then requiring it to be disabled per default	2022-12-11 23:11:56 +00:00
Alex Auvolat	2c2e65ad8b	Merge commit 'ec12d6c' into next	2022-12-11 18:41:15 +01:00
Alex Auvolat	a7af0c8af9	Add best practices and doc of monitoring (fix #419)	2022-11-16 13:27:24 +01:00
Alex Auvolat	fc2729cd81	Fix integration test	2022-11-08 15:19:46 +01:00
Alex Auvolat	d6c77ea327	Fix helm chart with correct configuration syntax	2022-10-18 22:30:05 +02:00
Alex Auvolat	8fc93abc79	Some things are now in result-bin	2022-10-18 13:39:21 +02:00
Maximilien R	db0c8b3980	Updates values.yml with some opinionated and untested defaults	2022-09-30 18:46:57 +02:00
Maximilien R	6dba7dadf4	Add missing ClusterRole and bindings for CRDs	2022-09-30 18:46:57 +02:00
chemicstry	b71fa2ddf4	Generate random RPC secret if not provided	2022-09-30 18:46:57 +02:00
chemicstry	37a73d7d37	Move documentation to book	2022-09-30 18:46:57 +02:00
chemicstry	d0f08c254e	Add secret to overrides	2022-09-30 18:46:57 +02:00
chemicstry	fa52558ca1	Add configuration instructions to README	2022-09-30 18:46:57 +02:00
chemicstry	131cc2532b	Cleanup values.yaml	2022-09-30 18:46:57 +02:00
chemicstry	a93dcce841	Add helm chart	2022-09-30 18:46:57 +02:00
Alex Auvolat	e935861854	Factor out node request order selection logic & use in manager	2022-07-29 12:25:03 +02:00
Quentin Dufour	76cb34a0ae	Fail if compiled binary is dynamic	2022-07-26 18:27:46 +02:00
Quentin Dufour	62f0715abe	Add/Fix OpenTelemetry	2022-04-07 16:12:35 +02:00
Quentin Dufour	7e1ac51b58	Add files to quickly test k8s	2022-04-07 16:12:35 +02:00
Alex Auvolat	bfb4353df5	Update Grafana dashboard	2022-03-14 10:55:30 +01:00
Alex Auvolat	a19341b188	Add Grafana dashboard for Garage	2022-03-14 10:54:25 +01:00
Alex Auvolat	dc8d0496cc	Refactoring: rename config files, make modifications less invasive	2022-03-14 10:53:51 +01:00
Maximilien	b6561f6e1b	Add docker-compose for traces & metrics	2022-03-14 10:51:52 +01:00
mricher	e349af13a7	Update dependencies and add admin module with metrics ... - Global dependencies updated in Cargo.lock - New module created in src/admin to host: - the (future) admin REST API - the metric collection - add configuration block No metrics implemented yet	2022-03-14 10:51:12 +01:00
Quentin Dufour	c00b2c9948	Functional tests for admin commands	2022-03-07 17:32:07 +01:00
Quentin Dufour	8df1e186de	Functional tests for website endpoints	2022-03-07 17:32:07 +01:00
Quentin Dufour	2ef60b8417	Functional test for multipart endpoints	2022-03-07 17:32:07 +01:00
Quentin Dufour	1e639ec67c	Functional test for ListMultipartUploads	2022-03-07 17:32:07 +01:00
Quentin Dufour	0f4e0e8bb9	Move ListObjects tests to Rust	2022-03-07 17:05:10 +01:00
Quentin Dufour	2a3afcaf65	Test WinSCP	2022-03-03 14:29:10 +01:00
Alex Auvolat	935670690f	Probably fix test-smoke	2022-02-02 17:34:19 +01:00
Quentin Dufour	f67029ce2a	Improve testing conf + test CORS	2022-02-01 17:55:14 +01:00
Alex Auvolat	c99f55c420	Add restriction on part ordering in CompleteMultipartUpload	2022-01-25 12:45:00 +01:00
Quentin Dufour	94f0e7c135	Test ListParts endpoint with awscli	2022-01-21 10:42:35 +01:00
Alex Auvolat	7c049f1c94	Fix extreme value to be less extreme so that integration test works on 32bits	2022-01-17 12:56:29 +01:00
Jill	b45dcc1925	Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156) ... Closes #64. Reviewed-on: #156 Co-authored-by: Jill <kokakiwi@deuxfleurs.fr> Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>	2022-01-17 10:55:31 +01:00
Alex Auvolat	1ee8f596ee	Testing for UploadPartCopies and bugfixes in AWS signatures	2022-01-13 14:03:30 +01:00
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
trinity-1686a	1eb972b1ac	Add compression using zstd (#173) ... fix #27 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #173 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 11:26:43 +01:00
Quentin Dufour	8811bb08e6	In ListBuckets, hide entry if no perms	2021-11-22 12:10:28 +01:00
Quentin Dufour	85b2e4ca29	Start socat only once ... Fixes #124	2021-11-17 10:59:32 +01:00
Alex Auvolat	c94406f428	Improve how node roles are assigned in Garage ... - change the terminology: the network configuration becomes the role table, the configuration of a nodes becomes a node's role - the modification of the role table takes place in two steps: first, changes are staged in a CRDT data structure. Then, once the user is happy with the changes, they can commit them all at once (or revert them). - update documentation - fix tests - implement smarter partition assignation algorithm This patch breaks the format of the network configuration: when migrating, the cluster will be in a state where no roles are assigned. All roles must be re-assigned and commited at once. This migration should not pose an issue.	2021-11-16 16:05:53 +01:00
Alex Auvolat	de4276202a	Improve CLI, adapt tests, update documentation	2021-10-25 14:21:48 +02:00
Quentin Dufour	dc017a0cab	Build Garage with Nix	2021-10-19 16:56:07 +02:00
Alex Auvolat	ddb2b29bfd	Rename datacenters into zones (doc not yet updated)	2021-05-28 14:07:36 +02:00
Alex Auvolat	b490ebc7f6	Many improvements on ring/replication and its configuration: ... - Explicit "replication_mode" configuration parameters that takes either "none", "2" or "3" as values, instead of letting user configure replication factor themselves. These are presets whose corresponding replication/quorum values can be found in replication/mode.rs - Explicit support for single-node and two-node deployments (number of nodes must be at least "replication_mode", with "none" we can have only one node) - Ring is now stored much more compactly with 256*8 + n*32 bytes, instead of 256*32 bytes - Support for gateway-only nodes that do not store data (these nodes still need a metadata_directory to store the list of bucket and keys since those are stored on all nodes; it also technically needs a data_directory to start but it will stay empty unless we have bugs)	2021-05-28 14:07:36 +02:00
Quentin Dufour	c8aa1eb481	Add preliminary support for Duck	2021-05-15 10:24:20 +02:00
Trinity Pointard	88925ebe22	add clippy to the CI	2021-05-03 22:11:37 +02:00
Quentin Dufour	631c36b3ff	S3 API: support ListBuckets	2021-05-03 21:55:30 +02:00
Quentin Dufour	ef4d6e782a	Add minio & rclone to our functional tests ... It is now possible to configure which clients you do not want to test with the env variable SKIP_XXX=1, XXX being the client name. eg. SKIP_S3CMD=1 ./script/test-smoke.sh	2021-05-02 14:59:58 +02:00
Quentin Dufour	e01f74e763	Introduce test case that demonstrates #59 (the & problem)	2021-04-27 23:09:26 +02:00
Alex Auvolat	640e3921d8	Use pre-prepared Docker image in CI to speed things up	2021-04-27 16:37:11 +02:00
Alex Auvolat	5ab33fddac	Refactor CLI and prettify CLI outpu	2021-03-12 18:12:31 +01:00
Alex Auvolat	6a3dcf3974	Rename n_tokens into capacity	2021-03-10 14:52:03 +01:00
Alex Auvolat	5e33c3cfc9	Use smaller capacities for nodes	2021-03-05 16:58:34 +01:00
Alex Auvolat	d7e005251d	Not fully tested: new multi-dc MagLev	2021-03-05 16:22:29 +01:00
Alex Auvolat	49c25a1509	Simulate stuff moving around	2021-02-25 10:53:33 +01:00
Alex Auvolat	1abbca37c4	Add adapted version of maglev for multi-dc	2021-02-21 19:14:28 +01:00
Alex Auvolat	24f924afdb	Maglev simulation	2021-02-21 18:32:13 +01:00
Alex Auvolat	e59322041a	Evaluate hash functions	2021-02-21 15:11:15 +01:00
Alex Auvolat	217b0dfd68	Add script to simulate different kinds of rings	2021-02-21 14:30:26 +01:00
Alex Auvolat	3b023c0c3b	try to fix smoke test ... dev cluster: don't ipv6 (fixes smoke test in container?)	2021-02-17 22:13:11 +01:00
Alex Auvolat	36814be447	Fix S3 ListObjects result and replace println!s by debug!s	2021-01-16 16:05:54 +01:00
Quentin Dufour	086e5be290	Update testing script	2020-12-17 21:04:59 +01:00
Quentin	1119d466e7	Fix S3 command	2020-12-10 20:19:22 +01:00
Quentin	e8c12072ce	Merge branch 'master' into feature/website	2020-12-10 20:12:56 +01:00
Quentin	e13fd09543	Reduce garage.1.rnd size to store it inline	2020-12-06 13:33:08 +01:00
Quentin	a92868504f	Indentation & comments	2020-12-06 10:23:14 +01:00
Quentin	d2d1fc676d	Test awscli/s3cmd interactions	2020-12-06 10:19:01 +01:00
Quentin	a12930075d	Test garage list & delete commands	2020-12-06 10:04:17 +01:00