537 commits

Author	SHA1	Message	Date
withinboredom	a4f9f19ac3	remove size limitation in UploadPartCopy (#280) ... This removes the >1mb s3_copy restriction. This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248). Co-authored-by: Rob Landers <landers.robert@gmail.com> Reviewed-on: #280 Co-authored-by: withinboredom <landers.robert@gmail.com> Co-committed-by: withinboredom <landers.robert@gmail.com>	2022-04-19 12:49:43 +02:00
Alex Auvolat	5d404dcd54	Add missing opentelemetry features	2022-04-08 14:21:04 +02:00
Alex Auvolat	94f1e48fff	Update to netapp 0.4.2 (a tiny fix)	2022-04-07 11:50:03 +02:00
Alex Auvolat	cb5836d53c	Bring maximum exponential backoff time down from 16h to 1h	2022-04-07 11:49:29 +02:00
Alex Auvolat	0091002ef2	New replication modes and their documentation	2022-03-28 16:26:04 +02:00
Alex Auvolat	8f9cf3a5d1	fix a clippy lint	2022-03-28 15:48:55 +02:00
Alex Auvolat	913f7754bb	Add blocks in errored state to garage stats	2022-03-28 15:47:23 +02:00
Alex Auvolat	42dde54126	Log admin GET requests at debug level instead of info ... to reduce noise in logs	2022-03-28 15:46:52 +02:00
Alex Auvolat	9d0ed78887	Add feature flag for Kubernetes discovery	2022-03-24 16:57:43 +01:00
Alex Auvolat	509d256c58	Make layout optimization work in relative terms	2022-03-24 15:27:14 +01:00
Alex Auvolat	2814d41842	Allow garage layout assign to assign to several nodes at once	2022-03-24 15:27:13 +01:00
Alex Auvolat	7e0e2ffda2	Slight change and add comment to layout assignation algo	2022-03-24 15:27:13 +01:00
Alex Auvolat	413ab0eaed	Small change to partition assignation algorithm ... This change helps ensure that nodes for each partition are spread over all datacenters, a property that wasn't ensured previously when going from a 2 DC deployment to a 3 DC deployment	2022-03-24 15:27:10 +01:00
Alex Auvolat	3dc9214172	Add lots of comments on how the resync queue works ... (I don't really want to change/refactor that code though)	2022-03-23 10:25:39 +01:00
Alex Auvolat	077dd1cde9	Clippy	2022-03-23 10:25:39 +01:00
Alex Auvolat	e480aaf338	Make background tranquility a configurable parameter	2022-03-23 10:25:19 +01:00
Alex Auvolat	8fd6745745	Move block RC code to separate rc.rs	2022-03-23 10:25:19 +01:00
Alex Auvolat	c3982a90b6	Move DataBlock out of manager.rs	2022-03-23 10:25:19 +01:00
Alex Auvolat	c1d9854d2c	Move block manager to separate module	2022-03-23 10:25:15 +01:00
trinity-1686a	8565f7dc31	cleanup	2022-03-23 10:22:37 +01:00
trinity-1686a	8db6b84559	add test for create bucket and put website with streaming signature	2022-03-23 10:22:37 +01:00
trinity-1686a	1eb7fdb08f	add test framework for arbitraty S3 requests ... and implement some basic test with it	2022-03-23 10:22:36 +01:00
KokaKiwi	e934934f14	garage_api: Update streaming payload stream unit tests	2022-03-23 10:22:36 +01:00
KokaKiwi	98545a16dd	garage_api: Handle streaming payload early in request handling	2022-03-23 10:22:36 +01:00
Alex Auvolat	db46cdef79	Update netapp to v0.4.1	2022-03-15 17:09:57 +01:00
Alex Auvolat	ba6b56ae68	Fix some new clippy lints	2022-03-14 12:27:49 +01:00
Alex Auvolat	0af314b295	Add comment for fsync	2022-03-14 11:54:00 +01:00
Alex Auvolat	d78bf379fb	Fix resync queue to not drop items	2022-03-14 11:51:37 +01:00
Alex Auvolat	f7e6f4616f	Spawn a single resync worker	2022-03-14 11:51:37 +01:00
Alex Auvolat	dc5ec4ecf9	Add appropriate fsync() calls in write_block ... to ensure that data is persisted properly	2022-03-14 11:51:32 +01:00
Alex Auvolat	fe62d01b7e	Implement exponential backoff for resync retries	2022-03-14 11:41:20 +01:00
Alex Auvolat	9b2b531f4d	Make admin server optional	2022-03-14 10:54:25 +01:00
Alex Auvolat	2377a92f6b	Add wrapper over sled tree to count items (used for big queues)	2022-03-14 10:54:25 +01:00
Alex Auvolat	203e8d2c34	Bump version to 0.7 because of incompatible Netapp	2022-03-14 10:54:24 +01:00
Alex Auvolat	f869ca625d	Add spans to table calls, change span names in RPC	2022-03-14 10:54:12 +01:00
Alex Auvolat	0cc31ee169	add missing netapp telemetry feature	2022-03-14 10:54:11 +01:00
Alex Auvolat	dc8d0496cc	Refactoring: rename config files, make modifications less invasive	2022-03-14 10:53:51 +01:00
Alex Auvolat	d9a35359bf	Add metrics to web endpoint	2022-03-14 10:53:50 +01:00
Alex Auvolat	2a5609b292	Add metrics to API endpoint	2022-03-14 10:53:36 +01:00
Alex Auvolat	818daa5c78	Refactor how durations are measured	2022-03-14 10:53:35 +01:00
Alex Auvolat	f0d0cd9a20	Remove strum crate dependency; add protobuf nix dependency	2022-03-14 10:53:00 +01:00
Alex Auvolat	55d4471599	Remove ... at end of hex IDs	2022-03-14 10:52:31 +01:00
Alex Auvolat	bb04d94fa9	Update to Netapp 0.4 which supports distributed tracing	2022-03-14 10:52:30 +01:00
Alex Auvolat	8c2fb0c066	Add tracing integration with opentelemetry	2022-03-14 10:52:13 +01:00
Alex Auvolat	2cab84b1fe	Add many metrics in table/ and rpc/	2022-03-14 10:51:50 +01:00
Maximilien R	1e2cf26373	Implement basic metrics in table	2022-03-14 10:51:17 +01:00
mricher	e349af13a7	Update dependencies and add admin module with metrics ... - Global dependencies updated in Cargo.lock - New module created in src/admin to host: - the (future) admin REST API - the metric collection - add configuration block No metrics implemented yet	2022-03-14 10:51:12 +01:00
Max Audron	9d44127245	add support for kubernetes service discovery ... This commit adds support to discover garage instances running in kubernetes. Once enabled by setting `kubernetes_namespace` and `kubernetes_service_name` garage will create a Custom Resources `garagenodes.deuxfleurs.fr` with nodes public key as the resource name. and IP and Port information as spec in the namespace configured by `kubernetes_namespace`. For discovering nodes the resources are filtered with the optionally set `kubernetes_service_name` which sets a label `garage.deuxfleurs.fr/service` on the resources. This allows to separate multiple garage deployments in a single namespace. the `kubernetes_skip_crd` variable allows to disable the creation of the CRD by garage itself. The user must deploy this manually.	2022-03-12 13:05:52 +01:00
Quentin Dufour	c00b2c9948	Functional tests for admin commands	2022-03-07 17:32:07 +01:00
Quentin Dufour	8df1e186de	Functional tests for website endpoints	2022-03-07 17:32:07 +01:00
Quentin Dufour	2ef60b8417	Functional test for multipart endpoints	2022-03-07 17:32:07 +01:00
Quentin Dufour	1e639ec67c	Functional test for ListMultipartUploads	2022-03-07 17:32:07 +01:00
Quentin Dufour	cfea1e0315	Functional tests for bucket endpoints	2022-03-07 17:32:02 +01:00
Quentin Dufour	05eb79929e	Functional tests for object operations	2022-03-07 17:05:10 +01:00
Quentin Dufour	0f4e0e8bb9	Move ListObjects tests to Rust	2022-03-07 17:05:10 +01:00
Alex Auvolat	8a5bbc3b0b	More permissive OPTIONS on S3 API	2022-03-01 11:15:16 +01:00
Alex Auvolat	97f245f218	Add tracing output to signature calculation	2022-02-28 12:22:39 +01:00
Alex Auvolat	8129a98291	Process CORS earlier in pipeline	2022-02-28 12:22:39 +01:00
trinity-1686a	f6f8b7f1ad	Support for PostObject (#222) ... Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html) - [x] routing PostObject properly - [x] parsing multipart body - [x] validating signature - [x] validating policy - [x] validating content length - [x] actually saving data Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr> Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #222 Reviewed-by: Alex <alex@adnab.me> Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2022-02-21 23:02:30 +01:00
Jill	84613e66a2	garage(tests): Remove RNG stuff	2022-02-11 10:50:55 +01:00
Jill	c8b30ebc79	garage(tests): Remove superfluous test	2022-02-10 17:55:51 +01:00
Jill	d7decda3f4	garage(tests): Add random suffix to created buckets.	2022-02-10 17:55:50 +01:00
Jill	cd13ea461b	garage(tests): Add some unsafe-usage doc and tweaks	2022-02-10 17:55:50 +01:00
Quentin Dufour	5d19f3d2d7	Add integration tests to Drone	2022-02-10 17:55:50 +01:00
Jill	3baa841d6f	tests: Fix garage integration test	2022-02-10 17:55:49 +01:00
Jill	dd407e7041	tests: Add garage integration tests (base)	2022-02-10 17:55:49 +01:00
Alex Auvolat	af261e1789	Fix a bug when a migration is followed by a rebalance ... Nodes would stabilize on different encoding formats for the values, some having the pre-migration format and some having the post-migration format. This would be reflected in the Merkle trees never converging and thus having an infinite resync loop.	2022-02-10 17:38:27 +01:00
Alex Auvolat	ae2f32baf1	Hide deleted key in bucket info (fix #211)	2022-02-02 17:12:48 +01:00
Alex Auvolat	c99f55c420	Add restriction on part ordering in CompleteMultipartUpload	2022-01-25 12:45:00 +01:00
Alex Auvolat	acdf893362	Fix partnumber	2022-01-25 12:25:23 +01:00
Alex Auvolat	338b1b83ee	Implement part_number for GetObject	2022-01-24 21:04:42 +01:00
Alex Auvolat	6dab836f3a	Multipart improvements ... - support part_number for HeadObject - add checks in complete_multipart_upload	2022-01-24 21:04:40 +01:00
Alex Auvolat	513a6b15f9	Handle OPTIONS on website endpoint	2022-01-24 12:32:28 +01:00
Alex Auvolat	ea7fb901eb	Implement {Put,Get,Delete}BucketCors and CORS in general ... - OPTIONS request against API endpoint - Returning corresponding CORS headers on API calls - Returning corresponding CORS headers on website GET's	2022-01-24 11:58:00 +01:00
Trinity Pointard	820924534a	use clamp instead of min(max())	2022-01-24 11:56:59 +01:00
Quentin Dufour	440374524b	Implement ListParts	2022-01-21 10:42:30 +01:00
Quentin Dufour	fe003d6fbc	Add ListPartsResult structure	2022-01-20 16:38:55 +01:00
trinity-1686a	e55fa38c99	Add date verification to presigned urls (#196) ... fix #96 fix #162 by returning Forbidden instead Bad Request Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #196 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2022-01-18 12:22:31 +01:00
trinity-1686a	178e35f868	refactor s3_router and api_server to make unused Endpoint parameters more obvious	2022-01-17 15:50:24 +01:00
Jill	fdcddbe168	Fix Multipart Upload with WinSCP (#164) (#193) ... Closes #164. Reviewed-on: #193 Co-authored-by: Jill <kokakiwi@deuxfleurs.fr> Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>	2022-01-17 11:18:40 +01:00
Jill	b45dcc1925	Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156) ... Closes #64. Reviewed-on: #156 Co-authored-by: Jill <kokakiwi@deuxfleurs.fr> Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>	2022-01-17 10:55:31 +01:00
Alex Auvolat	d4dd2e2640	Make use of website config, return error document on error	2022-01-13 14:25:19 +01:00
Alex Auvolat	9eb211948e	Allow setting index document and error document on the CLI	2022-01-13 14:25:19 +01:00
Alex Auvolat	3ea8ca1b9e	Implement GetBucketWebsite	2022-01-13 14:23:52 +01:00
Alex Auvolat	f7349f4005	Add quotes in returned etags	2022-01-13 14:03:33 +01:00
Alex Auvolat	1ee8f596ee	Testing for UploadPartCopies and bugfixes in AWS signatures	2022-01-13 14:03:30 +01:00
Alex Auvolat	6617a72220	Implement UploadPartCopy	2022-01-13 13:58:47 +01:00
Alex Auvolat	3770a34e3d	Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187)	2022-01-13 13:56:55 +01:00
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
Alex Auvolat	3586c7257c	Clarify error message	2022-01-10 14:10:04 +01:00
Alex Auvolat	17ea28a438	Fix trivial bug in CLI	2022-01-10 12:38:33 +01:00
Alex Auvolat	7ee11f0eb6	Fix unit tests	2022-01-05 17:34:48 +01:00
Alex Auvolat	168a90dfb5	Fix some error codes	2022-01-05 17:07:36 +01:00
Alex Auvolat	fb1e31add0	Small CLI changes	2022-01-05 16:28:46 +01:00
Alex Auvolat	135858d067	Implement DeleteBucket	2022-01-05 16:28:19 +01:00
Alex Auvolat	8395030e48	Implement CreateBucket	2022-01-05 15:56:48 +01:00
Alex Auvolat	9431090b1e	Implement key allow|deny --create-bucket	2022-01-05 15:12:59 +01:00
Alex Auvolat	677ab60cc1	Small changes in key model and refactoring	2022-01-04 18:59:17 +01:00
Alex Auvolat	df35feba18	New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates	2022-01-04 12:53:14 +01:00
Alex Auvolat	1bcd6fabbd	New buckets for 0.6.0: small changes ... - Fix bucket delete - fix merge of bucket creation date - Replace deletable with option in aliases Rationale: if two aliases point to conflicting bucket, resolving by making an arbitrary choice risks making data accessible when it shouldn't be. We'd rather resolve to deleting the alias until someone puts it back.	2022-01-04 12:52:47 +01:00