723 commits

Author	SHA1	Message	Date
Alex Auvolat	077dd1cde9	Clippy	2022-03-23 10:25:39 +01:00
Alex Auvolat	e480aaf338	Make background tranquility a configurable parameter	2022-03-23 10:25:19 +01:00
Alex Auvolat	8fd6745745	Move block RC code to separate rc.rs	2022-03-23 10:25:19 +01:00
Alex Auvolat	c3982a90b6	Move DataBlock out of manager.rs	2022-03-23 10:25:19 +01:00
Alex Auvolat	c1d9854d2c	Move block manager to separate module	2022-03-23 10:25:15 +01:00
trinity-1686a	8565f7dc31	cleanup	2022-03-23 10:22:37 +01:00
trinity-1686a	8db6b84559	add test for create bucket and put website with streaming signature	2022-03-23 10:22:37 +01:00
trinity-1686a	1eb7fdb08f	add test framework for arbitraty S3 requests ... and implement some basic test with it	2022-03-23 10:22:36 +01:00
KokaKiwi	e934934f14	garage_api: Update streaming payload stream unit tests	2022-03-23 10:22:36 +01:00
KokaKiwi	98545a16dd	garage_api: Handle streaming payload early in request handling	2022-03-23 10:22:36 +01:00
Alex Auvolat	db46cdef79	Update netapp to v0.4.1	2022-03-15 17:09:57 +01:00
Alex Auvolat	ba6b56ae68	Fix some new clippy lints	2022-03-14 12:27:49 +01:00
Alex Auvolat	0af314b295	Add comment for fsync	2022-03-14 11:54:00 +01:00
Alex Auvolat	d78bf379fb	Fix resync queue to not drop items	2022-03-14 11:51:37 +01:00
Alex Auvolat	f7e6f4616f	Spawn a single resync worker	2022-03-14 11:51:37 +01:00
Alex Auvolat	dc5ec4ecf9	Add appropriate fsync() calls in write_block ... to ensure that data is persisted properly	2022-03-14 11:51:32 +01:00
Alex Auvolat	fe62d01b7e	Implement exponential backoff for resync retries	2022-03-14 11:41:20 +01:00
Alex Auvolat	9b2b531f4d	Make admin server optional	2022-03-14 10:54:25 +01:00
Alex Auvolat	2377a92f6b	Add wrapper over sled tree to count items (used for big queues)	2022-03-14 10:54:25 +01:00
Alex Auvolat	203e8d2c34	Bump version to 0.7 because of incompatible Netapp	2022-03-14 10:54:24 +01:00
Alex Auvolat	f869ca625d	Add spans to table calls, change span names in RPC	2022-03-14 10:54:12 +01:00
Alex Auvolat	0cc31ee169	add missing netapp telemetry feature	2022-03-14 10:54:11 +01:00
Alex Auvolat	dc8d0496cc	Refactoring: rename config files, make modifications less invasive	2022-03-14 10:53:51 +01:00
Alex Auvolat	d9a35359bf	Add metrics to web endpoint	2022-03-14 10:53:50 +01:00
Alex Auvolat	2a5609b292	Add metrics to API endpoint	2022-03-14 10:53:36 +01:00
Alex Auvolat	818daa5c78	Refactor how durations are measured	2022-03-14 10:53:35 +01:00
Alex Auvolat	f0d0cd9a20	Remove strum crate dependency; add protobuf nix dependency	2022-03-14 10:53:00 +01:00
Alex Auvolat	55d4471599	Remove ... at end of hex IDs	2022-03-14 10:52:31 +01:00
Alex Auvolat	bb04d94fa9	Update to Netapp 0.4 which supports distributed tracing	2022-03-14 10:52:30 +01:00
Alex Auvolat	8c2fb0c066	Add tracing integration with opentelemetry	2022-03-14 10:52:13 +01:00
Alex Auvolat	2cab84b1fe	Add many metrics in table/ and rpc/	2022-03-14 10:51:50 +01:00
Maximilien R	1e2cf26373	Implement basic metrics in table	2022-03-14 10:51:17 +01:00
mricher	e349af13a7	Update dependencies and add admin module with metrics ... - Global dependencies updated in Cargo.lock - New module created in src/admin to host: - the (future) admin REST API - the metric collection - add configuration block No metrics implemented yet	2022-03-14 10:51:12 +01:00
Max Audron	9d44127245	add support for kubernetes service discovery ... This commit adds support to discover garage instances running in kubernetes. Once enabled by setting `kubernetes_namespace` and `kubernetes_service_name` garage will create a Custom Resources `garagenodes.deuxfleurs.fr` with nodes public key as the resource name. and IP and Port information as spec in the namespace configured by `kubernetes_namespace`. For discovering nodes the resources are filtered with the optionally set `kubernetes_service_name` which sets a label `garage.deuxfleurs.fr/service` on the resources. This allows to separate multiple garage deployments in a single namespace. the `kubernetes_skip_crd` variable allows to disable the creation of the CRD by garage itself. The user must deploy this manually.	2022-03-12 13:05:52 +01:00
Quentin Dufour	c00b2c9948	Functional tests for admin commands	2022-03-07 17:32:07 +01:00
Quentin Dufour	8df1e186de	Functional tests for website endpoints	2022-03-07 17:32:07 +01:00
Quentin Dufour	2ef60b8417	Functional test for multipart endpoints	2022-03-07 17:32:07 +01:00
Quentin Dufour	1e639ec67c	Functional test for ListMultipartUploads	2022-03-07 17:32:07 +01:00
Quentin Dufour	cfea1e0315	Functional tests for bucket endpoints	2022-03-07 17:32:02 +01:00
Quentin Dufour	05eb79929e	Functional tests for object operations	2022-03-07 17:05:10 +01:00
Quentin Dufour	0f4e0e8bb9	Move ListObjects tests to Rust	2022-03-07 17:05:10 +01:00
Alex Auvolat	8a5bbc3b0b	More permissive OPTIONS on S3 API	2022-03-01 11:15:16 +01:00
Alex Auvolat	97f245f218	Add tracing output to signature calculation	2022-02-28 12:22:39 +01:00
Alex Auvolat	8129a98291	Process CORS earlier in pipeline	2022-02-28 12:22:39 +01:00
trinity-1686a	f6f8b7f1ad	Support for PostObject (#222) ... Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html) - [x] routing PostObject properly - [x] parsing multipart body - [x] validating signature - [x] validating policy - [x] validating content length - [x] actually saving data Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr> Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #222 Reviewed-by: Alex <alex@adnab.me> Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2022-02-21 23:02:30 +01:00
Jill	84613e66a2	garage(tests): Remove RNG stuff	2022-02-11 10:50:55 +01:00
Jill	c8b30ebc79	garage(tests): Remove superfluous test	2022-02-10 17:55:51 +01:00
Jill	d7decda3f4	garage(tests): Add random suffix to created buckets.	2022-02-10 17:55:50 +01:00
Jill	cd13ea461b	garage(tests): Add some unsafe-usage doc and tweaks	2022-02-10 17:55:50 +01:00
Quentin Dufour	5d19f3d2d7	Add integration tests to Drone	2022-02-10 17:55:50 +01:00
Jill	3baa841d6f	tests: Fix garage integration test	2022-02-10 17:55:49 +01:00
Jill	dd407e7041	tests: Add garage integration tests (base)	2022-02-10 17:55:49 +01:00
Alex Auvolat	af261e1789	Fix a bug when a migration is followed by a rebalance ... Nodes would stabilize on different encoding formats for the values, some having the pre-migration format and some having the post-migration format. This would be reflected in the Merkle trees never converging and thus having an infinite resync loop.	2022-02-10 17:38:27 +01:00
Alex Auvolat	ae2f32baf1	Hide deleted key in bucket info (fix #211)	2022-02-02 17:12:48 +01:00
Alex Auvolat	c99f55c420	Add restriction on part ordering in CompleteMultipartUpload	2022-01-25 12:45:00 +01:00
Alex Auvolat	acdf893362	Fix partnumber	2022-01-25 12:25:23 +01:00
Alex Auvolat	338b1b83ee	Implement part_number for GetObject	2022-01-24 21:04:42 +01:00
Alex Auvolat	6dab836f3a	Multipart improvements ... - support part_number for HeadObject - add checks in complete_multipart_upload	2022-01-24 21:04:40 +01:00
Alex Auvolat	513a6b15f9	Handle OPTIONS on website endpoint	2022-01-24 12:32:28 +01:00
Alex Auvolat	ea7fb901eb	Implement {Put,Get,Delete}BucketCors and CORS in general ... - OPTIONS request against API endpoint - Returning corresponding CORS headers on API calls - Returning corresponding CORS headers on website GET's	2022-01-24 11:58:00 +01:00
Trinity Pointard	820924534a	use clamp instead of min(max())	2022-01-24 11:56:59 +01:00
Quentin Dufour	440374524b	Implement ListParts	2022-01-21 10:42:30 +01:00
Quentin Dufour	fe003d6fbc	Add ListPartsResult structure	2022-01-20 16:38:55 +01:00
trinity-1686a	e55fa38c99	Add date verification to presigned urls (#196) ... fix #96 fix #162 by returning Forbidden instead Bad Request Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #196 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2022-01-18 12:22:31 +01:00
trinity-1686a	178e35f868	refactor s3_router and api_server to make unused Endpoint parameters more obvious	2022-01-17 15:50:24 +01:00
Jill	fdcddbe168	Fix Multipart Upload with WinSCP (#164) (#193) ... Closes #164. Reviewed-on: #193 Co-authored-by: Jill <kokakiwi@deuxfleurs.fr> Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>	2022-01-17 11:18:40 +01:00
Jill	b45dcc1925	Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156) ... Closes #64. Reviewed-on: #156 Co-authored-by: Jill <kokakiwi@deuxfleurs.fr> Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>	2022-01-17 10:55:31 +01:00
Alex Auvolat	d4dd2e2640	Make use of website config, return error document on error	2022-01-13 14:25:19 +01:00
Alex Auvolat	9eb211948e	Allow setting index document and error document on the CLI	2022-01-13 14:25:19 +01:00
Alex Auvolat	3ea8ca1b9e	Implement GetBucketWebsite	2022-01-13 14:23:52 +01:00
Alex Auvolat	f7349f4005	Add quotes in returned etags	2022-01-13 14:03:33 +01:00
Alex Auvolat	1ee8f596ee	Testing for UploadPartCopies and bugfixes in AWS signatures	2022-01-13 14:03:30 +01:00
Alex Auvolat	6617a72220	Implement UploadPartCopy	2022-01-13 13:58:47 +01:00
Alex Auvolat	3770a34e3d	Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187)	2022-01-13 13:56:55 +01:00
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
Alex Auvolat	3586c7257c	Clarify error message	2022-01-10 14:10:04 +01:00
Alex Auvolat	17ea28a438	Fix trivial bug in CLI	2022-01-10 12:38:33 +01:00
Alex Auvolat	7ee11f0eb6	Fix unit tests	2022-01-05 17:34:48 +01:00
Alex Auvolat	168a90dfb5	Fix some error codes	2022-01-05 17:07:36 +01:00
Alex Auvolat	fb1e31add0	Small CLI changes	2022-01-05 16:28:46 +01:00
Alex Auvolat	135858d067	Implement DeleteBucket	2022-01-05 16:28:19 +01:00
Alex Auvolat	8395030e48	Implement CreateBucket	2022-01-05 15:56:48 +01:00
Alex Auvolat	9431090b1e	Implement key allow|deny --create-bucket	2022-01-05 15:12:59 +01:00
Alex Auvolat	677ab60cc1	Small changes in key model and refactoring	2022-01-04 18:59:17 +01:00
Alex Auvolat	df35feba18	New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates	2022-01-04 12:53:14 +01:00
Alex Auvolat	1bcd6fabbd	New buckets for 0.6.0: small changes ... - Fix bucket delete - fix merge of bucket creation date - Replace deletable with option in aliases Rationale: if two aliases point to conflicting bucket, resolving by making an arbitrary choice risks making data accessible when it shouldn't be. We'd rather resolve to deleting the alias until someone puts it back.	2022-01-04 12:52:47 +01:00
Alex Auvolat	ba7f268b99	Rename and change query filters	2022-01-04 12:52:46 +01:00
Alex Auvolat	e59c23a69d	Refactor logic for setting/unsetting aliases	2022-01-04 12:52:46 +01:00
Alex Auvolat	2140cd7205	Remove website redirects	2022-01-04 12:52:46 +01:00
Alex Auvolat	beeef4758e	Some movement of helper code and refactoring of error handling	2022-01-04 12:52:46 +01:00
Alex Auvolat	d8ab5bdc3e	New buckets for 0.6.0: fix model and migration	2022-01-04 12:47:28 +01:00
Alex Auvolat	c7d5c73244	Add must_use to some CRDT functions	2022-01-04 12:47:28 +01:00
Alex Auvolat	b76d0580a0	Fix forgotten flag	2022-01-04 12:47:28 +01:00
Alex Auvolat	87121dce9d	New buckets for 0.6.0: documentation and build files	2022-01-04 12:47:06 +01:00
Alex Auvolat	b1cfd16913	New buckets for 0.6.0: small fixes, including: ... - ensure bucket names are correct aws s3 names - when making aliases, ensure timestamps of links in both ways are the same - fix small remarks by trinity - don't have a separate website_access field	2022-01-04 12:46:41 +01:00
Alex Auvolat	5db600e231	More complete output to bucket info and key info	2022-01-04 12:46:41 +01:00
Alex Auvolat	4d30e62db4	New buckets for 0.6.0: migration code and build files	2022-01-04 12:46:13 +01:00
Alex Auvolat	0bbb6673e7	Model changes	2022-01-04 12:45:52 +01:00
Alex Auvolat	53f71b3a57	Implement bucket alias and bucket unalias	2022-01-04 12:45:51 +01:00
Alex Auvolat	5b1117e582	New model for buckets	2022-01-04 12:45:46 +01:00
Alex Auvolat	8f6026de5e	Make table name a const in trait	2021-12-15 15:39:10 +01:00
trinity-1686a	ca7b438f3f	less strict ListBuckets (#178) ... fix #175 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #178 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 15:05:36 +01:00
trinity-1686a	1eb972b1ac	Add compression using zstd (#173) ... fix #27 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #173 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 11:26:43 +01:00
trinity-1686a	60d4459926	BucketWebsite (#174) ... fix #77 this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though. Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #174 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 10:41:39 +01:00
trinity-1686a	c4ac8835d3	add proper request router for s3 api (#163) ... fix #161 Current request router was organically grown, and is getting messier and messier with each addition. This router cover exaustively existing API endpoints (with exceptions listed in [#161(comment)](#161 (comment)) either because new and old api endpoint can't feasabily be differentied, or it's more lambda than s3). Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #163 Reviewed-by: Alex <alex@adnab.me> Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-06 15:17:47 +01:00
trinity-1686a	7f26ed55cd	Improved handling of HTTP ranges ... - correct HTTP code when range syntax is invalid (fix #140) - when multiple ranges are given, simply ignore and send whole file Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #157 Reviewed-by: Alex <alex@adnab.me> Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-11-29 11:52:42 +01:00
Quentin Dufour	8811bb08e6	In ListBuckets, hide entry if no perms	2021-11-22 12:10:28 +01:00
Alex Auvolat	c94406f428	Improve how node roles are assigned in Garage ... - change the terminology: the network configuration becomes the role table, the configuration of a nodes becomes a node's role - the modification of the role table takes place in two steps: first, changes are staged in a CRDT data structure. Then, once the user is happy with the changes, they can commit them all at once (or revert them). - update documentation - fix tests - implement smarter partition assignation algorithm This patch breaks the format of the network configuration: when migrating, the cluster will be in a state where no roles are assigned. All roles must be re-assigned and commited at once. This migration should not pose an issue.	2021-11-16 16:05:53 +01:00
Trinity Pointard	53888995bd	update doc and comments	2021-11-16 15:41:41 +01:00
Trinity Pointard	396fe4c702	clippy	2021-11-16 15:41:41 +01:00
Trinity Pointard	02158ee666	fix issue where list on vhost-bucket would list bucket instead of bucket content	2021-11-16 15:41:41 +01:00
Trinity Pointard	9c58ec28d3	add support for vhost-style s3 bucket	2021-11-16 15:41:41 +01:00
Alex Auvolat	ad7ab31411	Implement GC delay for table data	2021-11-08 15:47:47 +01:00
Alex Auvolat	74a7a550eb	Safety: never voluntarily delete block in 10min interval after RC reaches zero	2021-11-08 15:47:47 +01:00
Alex Auvolat	cc255d46cd	Refactor and comment table GC logic	2021-11-08 15:47:44 +01:00
Trinity Pointard	9d7535c3f5	allow missing bootstrap_peers in garage.toml	2021-11-05 16:36:25 +01:00
Alex Auvolat	e8811f7c9d	Request strategy: don't launch all 3 requests if not needed	2021-11-04 16:19:27 +01:00
Alex Auvolat	2090a6187f	Add tranquilizer mechanism to improve on token bucket mechanism	2021-11-04 13:26:59 +01:00
Alex Auvolat	6f13d083ab	Add semaphore to limit RAM used by buffered outgoing requests	2021-11-03 18:02:57 +01:00
Alex Auvolat	8c4f418fe8	Fix peer list persistence: do not forget previous peers	2021-11-03 17:34:44 +01:00
Jill	bef6d627b0	Add environment variables equivalents for some CLI options.	2021-11-03 16:00:57 +01:00
Alex Auvolat	69b89fb46d	Fix race in block resync	2021-10-27 12:01:12 +02:00
Alex Auvolat	6b47c294f5	Refactoring on repair commands	2021-10-27 11:14:55 +02:00
Trinity Pointard	28c015d9ff	add cli parameter to verify local bloc integrity ... reuse code for listing local blocks add disk i/o speed limit on integrity check	2021-10-27 10:31:03 +02:00
Alex Auvolat	3e7f766d95	CLI: default rpc_host	2021-10-26 11:36:30 +02:00
Alex Auvolat	43e13a501d	Use published netapp crate instead of git repo	2021-10-26 10:36:57 +02:00
Alex Auvolat	ada7899b24	Fix clippy lints (fix #121)	2021-10-26 10:20:05 +02:00
Alex Auvolat	df8a4068d9	Refactor block manager code, and hopefully fix deadlock	2021-10-25 14:21:51 +02:00
Alex Auvolat	de4276202a	Improve CLI, adapt tests, update documentation	2021-10-25 14:21:48 +02:00
Alex Auvolat	1b450c4b49	Improvements to CLI and various fixes for netapp version ... Discovery via consul, persist peer list to file	2021-10-22 16:55:24 +02:00
Alex Auvolat	4067797d01	First port of Garage to Netapp	2021-10-22 15:55:18 +02:00
Quentin Dufour	dc017a0cab	Build Garage with Nix	2021-10-19 16:56:07 +02:00
Alex Auvolat	1acf7e4c66	Fix git_version!() when not in git repo (fix #100)	2021-10-11 14:26:54 +02:00
Alex Auvolat	f6060b92aa	Fix HTTP return code for DeleteObject (fix #98)	2021-10-11 14:24:49 +02:00
Alex Auvolat	f3a097abdf	WIP: try to fix #93, and improve S3 ListObjects (v1 and v2) API calls	2021-10-11 11:15:47 +02:00
Alex Auvolat	fa394dcd27	Support pkcs8 private keys (allowing for ed25519 to be used for rpc)	2021-07-06 11:16:01 +02:00
trinity-1686a	30a7dee920	exit when inconsistent level of replication is detected (#92) ... fix #88 Authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #92 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-06-02 13:30:39 +02:00
Trinity Pointard	b568765c75	try parsing rpc-host command-line parameter	2021-06-01 23:53:58 +02:00
Trinity Pointard	e9c265e9dc	format garage status output better	2021-06-01 19:12:27 +02:00
Trinity Pointard	289521886b	make most changes suggested during install-party	2021-05-29 21:37:49 +02:00
Alex Auvolat	b9127dd6f8	Prepare for v0.3.0 and add migration path from v0.2.1.x	2021-05-28 15:29:58 +02:00
Alex Auvolat	ddb2b29bfd	Rename datacenters into zones (doc not yet updated)	2021-05-28 14:07:36 +02:00
Alex Auvolat	b490ebc7f6	Many improvements on ring/replication and its configuration: ... - Explicit "replication_mode" configuration parameters that takes either "none", "2" or "3" as values, instead of letting user configure replication factor themselves. These are presets whose corresponding replication/quorum values can be found in replication/mode.rs - Explicit support for single-node and two-node deployments (number of nodes must be at least "replication_mode", with "none" we can have only one node) - Ring is now stored much more compactly with 256*8 + n*32 bytes, instead of 256*32 bytes - Support for gateway-only nodes that do not store data (these nodes still need a metadata_directory to store the list of bucket and keys since those are stored on all nodes; it also technically needs a data_directory to start but it will stay empty unless we have bugs)	2021-05-28 14:07:36 +02:00
Quentin Dufour	5fdabf3e75	Add basic support for the "Versioning" command	2021-05-14 22:33:26 +02:00
Alex Auvolat	6ccffc3162	Improved XML serialization ... - Use quick_xml and serde for all XML response returned by the S3 API. - Include tests for all structs used to generate XML - Remove old manual XML escaping function which was unsafe	2021-05-06 22:37:15 +02:00
Trinity Pointard	e4b9e4e24d	rename types to CamelCase	2021-05-03 22:15:09 +02:00
Trinity Pointard	6644df6b96	fix clippy warnings on garage	2021-05-03 22:11:42 +02:00
Trinity Pointard	f8ae8fc4be	fix clippy warnings on web	2021-05-03 22:11:42 +02:00
Trinity Pointard	84856e84e5	fix clippy warnings on api	2021-05-03 22:11:41 +02:00
Trinity Pointard	4a1e079e8f	fix clippy warnings on model	2021-05-03 22:11:41 +02:00
Trinity Pointard	f5a0cf0414	fix clippy warnings on table	2021-05-03 22:11:41 +02:00
Trinity Pointard	f05bb111c2	fix clippy warnings on util and rpc	2021-05-03 22:11:41 +02:00
Quentin Dufour	631c36b3ff	S3 API: support ListBuckets	2021-05-03 21:55:30 +02:00
Alex Auvolat	ee2a3d363b	Remove STREAMING-AWS4-HMAC-SHA256-PAYLOAD (see #64)	2021-05-03 17:30:40 +02:00
Alex Auvolat	575726358c	Tune Sled configuration ... - Make sled cache size and flush interval configurable - Set less agressive default values: - cache size 128MB instead of 1GB - Flush interval 2 seconds instead of .5 seconds	2021-05-03 17:27:43 +02:00
Alex Auvolat	dcfc32cf85	Many S3 compatibility improvements: ... - return XML errors - implement AuthorizationHeaderMalformed error to redirect clients to correct location (used by minio client) - implement GetBucketLocation - fix DeleteObjects XML parsing and response	2021-04-28 01:05:40 +02:00
Alex Auvolat	642186c530	Fix #59 (& issue)	2021-04-27 23:10:43 +02:00
Alex Auvolat	119217f9f6	change a few comments	2021-04-27 16:53:47 +02:00
Trinity Pointard	2812a027ea	change some more comments and revert changes on TableSchema	2021-04-27 16:49:07 +02:00
Trinity Pointard	74373aebcf	make most requested changes	2021-04-27 16:47:08 +02:00
Alex Auvolat	16300bbd89	remove useless comment	2021-04-27 16:44:01 +02:00
Trinity Pointard	5c4b2cec3c	document garage crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	1e3df189d0	document api crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	a2e1617d84	document web crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	67585a4ffa	attempt at documenting model crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	b437610812	attempt at documenting table crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	f871689571	run cargo fmt on util and make missing doc warning	2021-04-27 16:37:10 +02:00
Trinity Pointard	8e0524ae15	document rpc crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	f9bd2d8fb7	document util crate	2021-04-27 16:37:10 +02:00
Alex Auvolat	6b2b400292	small simplify	2021-04-27 16:37:09 +02:00
Alex Auvolat	8c33d565d6	Merge discovery loop with consul	2021-04-27 16:37:09 +02:00
Alex Auvolat	948e44a3f6	cargo fmt	2021-04-27 16:37:09 +02:00
Alex Auvolat	3e2e38c830	Print stats	2021-04-27 16:37:09 +02:00
Alex Auvolat	2e53e31cdd	Cargo fmt	2021-04-27 16:37:09 +02:00
Alex Auvolat	64b91c2645	Keep old data	2021-04-27 16:37:09 +02:00
Alex Auvolat	e16077f40a	Persist directly and not in background	2021-04-27 16:37:09 +02:00
Alex Auvolat	9ced9f78dc	Improve bootstraping: do it regularly; persist peer list	2021-04-27 16:37:08 +02:00
Alex Auvolat	f859d15062	update to v0.2.1	2021-03-19 13:39:18 +01:00
Alex Auvolat	fd8f4caa81	Support old CPUs	2021-03-19 12:19:40 +01:00
Alex Auvolat	4c26a0b9c1	Update Cargo.toml files with AGPL license info	2021-03-18 21:59:17 +01:00
Alex Auvolat	a1014224d3	garage node configure --replace <old_node_id> <new_node_id>	2021-03-18 21:49:12 +01:00
Trinity Pointard	f17cb6c969	resolve domain to multiple addresses ... And warn instead of failling when a domain can't be resolved	2021-03-18 21:04:30 +01:00
Trinity Pointard	c8a7ce5cdf	remove domain resolution for *_bind_addr	2021-03-18 19:47:51 +01:00
Trinity Pointard	81e9db783f	simplify addresse deserialialiser and limit allocations	2021-03-18 19:47:51 +01:00
Trinity Pointard	ae3b7029a9	add support for using domain name in configuration	2021-03-18 19:47:51 +01:00
Alex Auvolat	6edbc65847	Add trinity's comment in the code	2021-03-18 19:46:43 +01:00
Alex	bfa0ff8f82	Merge pull request 'add support for caching headers' (#49) from trinity-1686a/garage:cache-headers into master ... Reviewed-on: #49	2021-03-18 19:45:02 +01:00
Alex Auvolat	dead945c8f	Prepare for release 0.2	2021-03-18 19:33:15 +01:00
Alex Auvolat	4eb16e8863	Allow to import keys from previous Garage instance	2021-03-18 19:24:59 +01:00
Trinity Pointard	b4c903371c	add support for caching headers	2021-03-18 15:46:33 +01:00
Alex Auvolat	390ab02f41	Todo make a test for the Merkle updater	2021-03-16 20:13:07 +01:00
Alex Auvolat	7b10245dfb	Leader-based GC	2021-03-16 18:42:33 +01:00
Alex Auvolat	08bcd51956	GC object table in a specific case	2021-03-16 16:51:15 +01:00
Alex Auvolat	3fadc5cbbd	Small changes	2021-03-16 16:35:10 +01:00
Alex Auvolat	f4346cc5f4	Update dependencies	2021-03-16 15:58:40 +01:00
Alex Auvolat	2a41b82384	Simpler Merkle & sync	2021-03-16 12:18:03 +01:00
Alex Auvolat	0aad2f2e06	some reordering	2021-03-16 11:47:39 +01:00
Alex Auvolat	515029d026	Refactor code	2021-03-16 11:43:58 +01:00
Alex Auvolat	1d9961e411	Simplify replication logic	2021-03-16 11:14:27 +01:00
Alex Auvolat	6a8439fd13	Some improvements in background worker but we terminate late	2021-03-15 23:14:12 +01:00
Alex Auvolat	0cd5b2ae19	WIP migrate to tokio 1	2021-03-15 22:36:41 +01:00
Alex Auvolat	4d4117f2b4	Refactor block resync loop; make workers infaillible	2021-03-15 20:09:44 +01:00
Alex Auvolat	667e4e72a8	Small fixes	2021-03-15 19:51:16 +01:00
Alex Auvolat	642bed601f	Make it case-insensitive	2021-03-15 19:16:42 +01:00
Alex Auvolat	5ee1d956b6	Allow manipulation of keys by their shorthand in the CLI	2021-03-15 19:14:26 +01:00
Alex Auvolat	537f652fec	Tiny things	2021-03-15 18:40:27 +01:00
Alex Auvolat	0290afe1f8	Make block rc code more understandable	2021-03-15 18:27:26 +01:00
Alex Auvolat	3bf2df622a	Time and metadata improvements	2021-03-15 16:21:41 +01:00
Alex Auvolat	097c339d98	Fix race condition	2021-03-15 15:26:29 +01:00
Alex Auvolat	bdcbdd1cd8	Fix list API bug	2021-03-15 14:46:37 +01:00
Alex Auvolat	9b118160a8	Optim & refactor	2021-03-12 22:06:56 +01:00
Alex Auvolat	831eb35763	cargo fmt	2021-03-12 21:52:19 +01:00
Alex Auvolat	c475471e7a	Implement table gc, currently for block_ref and version only	2021-03-12 19:57:37 +01:00
Alex Auvolat	f4aad8fe6e	cargo fmt	2021-03-12 18:16:03 +01:00
Alex Auvolat	5ab33fddac	Refactor CLI and prettify CLI outpu	2021-03-12 18:12:31 +01:00
Alex Auvolat	a1442f072a	Implement garage stats to get info on node contents	2021-03-12 15:40:54 +01:00
Alex Auvolat	cbe7e1a66a	Move table rpc client out of tableaux	2021-03-12 15:07:23 +01:00
Alex Auvolat	8860aa19b8	Make syncer have its own rpc client/server	2021-03-12 15:05:26 +01:00
Alex Auvolat	1fea257291	Don't sync at beginning	2021-03-12 14:51:17 +01:00
Alex Auvolat	7fdaf7aef0	Fix merkle updater not being notified; improved logging	2021-03-12 14:37:46 +01:00
Alex Auvolat	1ec49980ec	whoops	2021-03-11 19:30:24 +01:00
Alex Auvolat	3f7a496355	More security: don't delete stuff too easily	2021-03-11 19:06:27 +01:00
Alex Auvolat	f7c2cd1cd7	Add comment, and also whoops, this wasn't doing what we expected	2021-03-11 18:56:18 +01:00
Alex Auvolat	fae5104a2c	Add a nice warning	2021-03-11 18:50:32 +01:00
Alex Auvolat	db7a9d4948	Tiny changes	2021-03-11 18:45:26 +01:00
Alex Auvolat	046b649bcc	(not well tested) use merkle tree for sync	2021-03-11 18:28:27 +01:00
Alex Auvolat	94f3d28774	WIP big refactoring	2021-03-11 16:54:15 +01:00
Alex Auvolat	8d63738cb0	Checkpoint: add merkle tree in data table	2021-03-11 13:47:21 +01:00
Alex Auvolat	3214dd52dd	Very minor changes	2021-03-10 21:50:09 +01:00
Alex Auvolat	af7600f989	Correctly implement CompleteMultipartUpload with etag check of parts	2021-03-10 17:01:05 +01:00
Alex Auvolat	445912dc6a	Remove migration paths from 0.1 branch	2021-03-10 16:38:31 +01:00
Alex Auvolat	0fd7df8fa0	Switch to blake2 sum for identifying blocks by their data	2021-03-10 16:33:31 +01:00
Alex Auvolat	2afd2c81ba	Change hash function to blake2 for partition keys based on strings	2021-03-10 16:23:57 +01:00
Alex Auvolat	f319a7d374	Refactor model stuff, including cleaner CRDTs	2021-03-10 16:21:56 +01:00
Alex Auvolat	6a3dcf3974	Rename n_tokens into capacity	2021-03-10 14:52:03 +01:00
Alex Auvolat	7cda917b6b	update condition	2021-03-05 17:08:03 +01:00
Alex Auvolat	d7e005251d	Not fully tested: new multi-dc MagLev	2021-03-05 16:22:29 +01:00
Alex Auvolat	3882d5ba36	Remove epidemic propagation for fully replicated stuff: write directly to all nodes	2021-03-05 15:09:18 +01:00
Alex Auvolat	49c25a1509	Simulate stuff moving around	2021-02-25 10:53:33 +01:00
Alex Auvolat	5fe95ebae7	fix tracing	2021-02-24 12:18:01 +01:00
Alex Auvolat	13e2eda0c2	Arrange block manager	2021-02-24 11:58:03 +01:00
Alex Auvolat	09fd6ea7f0	I was tired yesterday	2021-02-24 11:05:59 +01:00
Alex Auvolat	a52ab69640	fix misuse of sled transactions	2021-02-23 22:45:36 +01:00
Alex Auvolat	20e6e9fa20	Update sled & try to debug deadlock (but its in sled...)	2021-02-23 21:27:28 +01:00
Alex Auvolat	bf25c95fe2	Make updated() be a sync function that doesn't fail	2021-02-23 20:25:15 +01:00
Alex Auvolat	28bc967c83	Handle correctly deletion dues to offloading	2021-02-23 19:59:43 +01:00
Alex Auvolat	55156cca9d	Several changes in table_sync: ... - separate path for case of offloading a partition we don't store - use sync::Mutex instead of tokio::Mutex, make less fn's async	2021-02-23 19:11:02 +01:00
Alex Auvolat	40763fd749	Cargo fmt	2021-02-23 18:46:25 +01:00
Alex Auvolat	6e6f7e8555	Replace some checksums where it makes sense	2021-02-23 18:14:37 +01:00
Alex Auvolat	e8e4418ca7	Add blake2 and xxhash hash functions	2021-02-23 17:52:28 +01:00
Alex Auvolat	b1b640ae8b	rename hash() to sha256sum(), we might want to change it at some places	2021-02-21 15:24:30 +01:00
Alex Auvolat	80892df8cc	Some refactoring	2021-02-21 13:11:10 +01:00
Alex Auvolat	3bcbbe1e31	More precise logging (warn only when returning a 500)	2021-02-20 00:30:39 +01:00
Alex Auvolat	10b983b8e7	Add verification of part numbers in CompleteMultipartUpload (WIP #30)	2021-02-20 00:13:07 +01:00
Alex Auvolat	1de96248e0	add application/xml header and missing xml escapes	2021-02-19 23:40:18 +01:00
Alex Auvolat	5d1fa591d9	Add compatibility list	2021-02-19 19:10:23 +01:00
Alex Auvolat	e64ecbdccd	S3 compatibility: return 404 instead of 400 on some multipart commands	2021-02-19 18:51:05 +01:00
Alex Auvolat	55a2a636ca	Implement ListObjectsV2	2021-02-19 16:44:06 +01:00
Alex Auvolat	02d512f3fd	Fix #28, extra headers being ignored (because of profound stupidity)	2021-02-19 12:38:22 +01:00
Alex Auvolat	76390085ef	Small improvements in the S3 put workflow	2021-02-19 12:11:02 +01:00
Alex Auvolat	36814be447	Fix S3 ListObjects result and replace println!s by debug!s	2021-01-16 16:05:54 +01:00
Alex Auvolat	e818f51073	Forgot a bump	2021-01-15 18:36:51 +01:00
Alex Auvolat	ceeb0732a2	Use 0.1.0b instead of 0.1.0 (for compatibility with new Error type)	2021-01-15 18:27:58 +01:00
Alex Auvolat	1d1d497e2b	Bump everything to 0.1.1	2021-01-15 17:54:48 +01:00
Alex Auvolat	851893a3f2	Do not accept domains such as [hello	2021-01-15 17:49:10 +01:00
Quentin	f8a40e8c4f	Explicitly set code path unreachable	2021-01-15 17:11:15 +01:00
Quentin	fad7bc405b	Behavior problem: do not panic anymore + add tests	2021-01-15 17:03:54 +01:00
Quentin	1e10c6a61c	Doc tests that do not compile/work must be tagged with ignore	2021-01-15 17:03:38 +01:00
Quentin	11a79a95dd	Simplify Error file	2021-01-15 16:25:44 +01:00
Quentin	c441a358cd	Remove unused dependencies	2021-01-15 16:16:32 +01:00
Quentin	f496e41ef4	Replace an already done check by unreachable!()	2021-01-15 15:44:44 +01:00
Quentin Dufour	2f4378a9c4	Fix formatting	2020-12-17 22:51:44 +01:00
Quentin Dufour	ccda9ab1ca	Merge branch 'master' into feature/website	2020-12-17 21:09:50 +01:00
Quentin Dufour	3132deca58	Web server access control	2020-12-17 20:43:14 +01:00
Quentin	011ff87b5f	Push update	2020-12-15 13:23:22 +01:00
Quentin	3bc4d57a0f	First implementation of the CLI	2020-12-15 12:48:24 +01:00
Quentin	a3566e49da	Start to implement Website CLI	2020-12-14 21:50:40 +01:00
Quentin	d0eb6a457f	Migrate RPC to new schema	2020-12-14 21:46:49 +01:00
Quentin	96388acf23	Implement migration	2020-12-12 21:35:29 +01:00
Alex Auvolat	8956db2a81	Make less things public	2020-12-12 17:58:19 +01:00
Alex Auvolat	5c6c067b0c	More documentation on CRDTs (we should probably extract this to a ... standalone crate!)	2020-12-12 17:06:40 +01:00
Quentin	e1ce2b228a	WIP table migration	2020-12-12 17:00:31 +01:00
Alex Auvolat	0b3084ca5f	Merge branch 'master' into doc/model	2020-12-12 16:05:28 +01:00
Quentin	e8c12072ce	Merge branch 'master' into feature/website	2020-12-10 20:12:56 +01:00
Quentin	51d0c14e44	CLI structure	2020-12-10 18:13:32 +01:00
Alex Auvolat	022b386a50	Improved compatibility on list API call	2020-12-06 15:39:03 +01:00
Quentin	986e15459a	Merge branch 'master' into feature/website	2020-12-06 15:21:09 +01:00
Alex Auvolat	4a5bbbb810	Propose ETag fix	2020-12-05 19:23:46 +01:00
Alex Auvolat	76b489f3d3	Reformulate patch	2020-12-05 19:20:07 +01:00
Quentin	bd7e3d1bd1	Fix Content-Length	2020-12-05 18:57:22 +01:00
Alex Auvolat	9f46fb699a	Content-range fix	2020-12-05 16:37:59 +01:00
Quentin	15f409d404	Merge branch 'master' into feature/website	2020-11-29 17:19:55 +01:00
Alex Auvolat	d54f15b2c6	Small optimisation	2020-11-29 17:07:14 +01:00
Alex Auvolat	fed97f37e1	ETag patch	2020-11-29 16:38:01 +01:00
Alex Auvolat	601ae25ad2	Small refactorings	2020-11-29 16:21:28 +01:00
Quentin	cbd10c1b0a	Add some doc on LWW	2020-11-23 18:17:48 +01:00
Quentin	8722e27600	CRDT doc	2020-11-23 17:49:21 +01:00
Quentin	aa320aa04a	Merge branch 'master' into feature/website	2020-11-22 19:54:47 +01:00
Quentin	17dc610f8a	Also check hash for < 3KB files	2020-11-22 11:14:46 +01:00
Quentin	435d5f9205	Fix base64/hex checksum comparison	2020-11-22 11:04:33 +01:00