86 commits

Author	SHA1	Message	Date
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
Alex Auvolat	9cb2e9e57c	Add documentation to migrate to 0.6	2022-01-10 14:42:58 +01:00
Alex Auvolat	8f39360f22	Update documentation	2022-01-07 17:14:37 +01:00
trinity-1686a	945b75dbf1	update s3 compatibility list (#177) ... Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #177 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 15:05:54 +01:00
trinity-1686a	1eb972b1ac	Add compression using zstd (#173) ... fix #27 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #173 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 11:26:43 +01:00
Quentin Dufour	3b3a1f275f	Add a second plot	2021-12-13 11:58:03 +01:00
Quentin Dufour	dba9af2968	Update benchmark	2021-12-09 18:42:45 +01:00
Quentin Dufour	e9358054ac	Typos and dead links	2021-12-08 14:40:14 +01:00
Quentin Dufour	f9e5520ffb	Add a benchmark page with a first benchmark	2021-12-08 11:30:07 +01:00
Alex Auvolat	4b369347c0	S3 compatibility target	2021-12-06 17:18:45 +01:00
Alex Auvolat	224c89ad6e	Reorganize and improve documentation	2021-12-06 16:33:01 +01:00
Quentin Dufour	7c2037ba87	WIP front page garage	2021-12-06 15:33:43 +01:00
Alex Auvolat	c94406f428	Improve how node roles are assigned in Garage ... - change the terminology: the network configuration becomes the role table, the configuration of a nodes becomes a node's role - the modification of the role table takes place in two steps: first, changes are staged in a CRDT data structure. Then, once the user is happy with the changes, they can commit them all at once (or revert them). - update documentation - fix tests - implement smarter partition assignation algorithm This patch breaks the format of the network configuration: when migrating, the cluster will be in a state where no roles are assigned. All roles must be re-assigned and commited at once. This migration should not pose an issue.	2021-11-16 16:05:53 +01:00
Trinity Pointard	53888995bd	update doc and comments	2021-11-16 15:41:41 +01:00
Trinity Pointard	57df9c6e2d	add s3_api.root_domain to doc book	2021-11-16 15:41:41 +01:00
adrien	cdeb5b4dbb	added link to RFID Garage talk (#155) ... Co-authored-by: ADRN <adrien@luxeylab.net> Reviewed-on: #155 Co-authored-by: adrien <adrien@luxeylab.net> Co-committed-by: adrien <adrien@luxeylab.net>	2021-11-16 15:39:08 +01:00
Quentin Dufour	100aad8bf4	Add rclone mount doc	2021-11-10 18:38:17 +01:00
Quentin Dufour	80a87929b0	Improve CLI documentation	2021-11-10 18:18:34 +01:00
Quentin Dufour	76d21be1b9	Add skeleton for backups, fuse and code sections	2021-11-10 18:05:07 +01:00
Quentin Dufour	1928f59d54	Add documentation for Gitea.	2021-11-10 12:41:09 +01:00
Quentin Dufour	323514be15	Documentation for Nix binary cache	2021-11-10 10:02:22 +01:00
Quentin Dufour	ad8d5139cf	hugo deploy does not build website, fix doc	2021-11-10 10:02:22 +01:00
Alex Auvolat	08b1e8a7ea	Move design draft to separate file; write about GC in internals	2021-11-09 12:25:33 +01:00
Quentin Dufour	8e25a37f0e	Add documentation for nginx	2021-11-08 12:20:40 +01:00
Quentin Dufour	e342db19aa	Add documentation about Gateways	2021-11-08 12:20:40 +01:00
Quentin Dufour	f3405b6378	Doc about exposing your website	2021-11-08 12:20:40 +01:00
Quentin Dufour	860ccf2811	Harden Garage's systemd service	2021-11-08 12:20:40 +01:00
Quentin Dufour	9df7559446	Documentation for hugo, jekyll and publii	2021-11-08 12:20:40 +01:00
Quentin Dufour	a97467075d	Add documentation for synapse-s3-storage-provider	2021-11-08 12:20:40 +01:00
Trinity Pointard	da6efb4b23	fix missing bootstrap_peers in doc	2021-11-05 11:21:50 +01:00
Quentin Dufour	e93d7fb228	Add Peertube + improve CLI instructions	2021-11-03 14:39:14 +01:00
Quentin Dufour	eaf54efb25	Add doc for Nextcloud	2021-11-03 14:07:55 +01:00
Quentin Dufour	93f8d59e4c	Extract toolchain build from the CI	2021-10-29 11:34:01 +02:00
ADRN	4e8af1d956	Modified the 'Funding' sentence to remove 'promise' since we actually got the first instalment	2021-10-26 13:34:28 +02:00
Alex Auvolat	3e7f766d95	CLI: default rpc_host	2021-10-26 11:36:30 +02:00
Alex Auvolat	b2c51844a1	Add download link on homepage	2021-10-25 15:55:30 +02:00
Alex Auvolat	de4276202a	Improve CLI, adapt tests, update documentation	2021-10-25 14:21:48 +02:00
Quentin Dufour	dc017a0cab	Build Garage with Nix	2021-10-19 16:56:07 +02:00
Alex Auvolat	1aed317818	Small changes on NGI kickoff talk	2021-10-07 11:12:34 +02:00
Quentin	c5574c8409	Add links and put logos in a flexbox	2021-09-28 10:21:10 +02:00
mricher	de0228ca2a	Doc: add funding disclaimer for NGI/EU grant ... PNG logs optimized, render to be checked by @quentin. Fix #106.	2021-09-25 17:21:07 +02:00
Alex Auvolat	df345e37db	Add sticker and NGI kickoff talk	2021-09-12 13:37:33 +02:00
Alex Auvolat	42f692b1e0	Documentation fixes (typo and small reorganization)	2021-05-31 23:55:51 +02:00
Alex Auvolat	14fd3df654	Write about S3 client configuration	2021-05-31 17:41:21 +02:00
Alex Auvolat	56ac9fd460	Updates to documentation	2021-05-31 17:23:35 +02:00
Alex Auvolat	d76a8576f4	Reorganize documentation	2021-05-31 17:13:36 +02:00
Trinity Pointard	289521886b	make most changes suggested during install-party	2021-05-29 21:37:49 +02:00
Alex Auvolat	ebd21b325e	Write documentation on configuration file and other improvements	2021-05-28 18:00:59 +02:00
Alex Auvolat	339c611789	Add links to git/matrix/drone on top of documentation	2021-05-03 10:45:45 +02:00
Alex Auvolat	a981244f11	Replace talk links with permalinks	2021-04-30 15:51:32 +02:00