25 Commits

Author	SHA1	Message	Date
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
Alex	d8ab5bdc3e	New buckets for 0.6.0: fix model and migration	2022-01-04 12:47:28 +01:00
Alex	4d30e62db4	New buckets for 0.6.0: migration code and build files	2022-01-04 12:46:13 +01:00
Alex	5b1117e582	New model for buckets	2022-01-04 12:45:46 +01:00
Alex	8f6026de5e	Make table name a const in trait	2021-12-15 15:39:10 +01:00
Trinity Pointard	e4b9e4e24d	rename types to CamelCase	2021-05-03 22:15:09 +02:00
Trinity Pointard	4a1e079e8f	fix clippy warnings on model	2021-05-03 22:11:41 +02:00
Alex	119217f9f6	change a few comments	2021-04-27 16:53:47 +02:00
Trinity Pointard	74373aebcf	make most requested changes	2021-04-27 16:47:08 +02:00
Trinity Pointard	67585a4ffa	attempt at documenting model crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	b437610812	attempt at documenting table crate	2021-04-27 16:37:10 +02:00
Alex	9ced9f78dc	Improve bootstraping: do it regularly; persist peer list	2021-04-27 16:37:08 +02:00
Alex	08bcd51956	GC object table in a specific case	2021-03-16 16:51:15 +01:00
Alex	6a8439fd13	Some improvements in background worker but we terminate late	2021-03-15 23:14:12 +01:00
Alex	0cd5b2ae19	WIP migrate to tokio 1	2021-03-15 22:36:41 +01:00
Alex	94f3d28774	WIP big refactoring	2021-03-11 16:54:15 +01:00
Alex	3214dd52dd	Very minor changes	2021-03-10 21:50:09 +01:00
Alex	445912dc6a	Remove migration paths from 0.1 branch	2021-03-10 16:38:31 +01:00
Alex	f319a7d374	Refactor model stuff, including cleaner CRDTs	2021-03-10 16:21:56 +01:00
Alex	bf25c95fe2	Make updated() be a sync function that doesn't fail	2021-02-23 20:25:15 +01:00
Alex	e9fd265ce6	Slight refactoring to make things clearer with DeletedFilter	2020-11-20 20:11:04 +01:00
Alex	44dba0e53c	cargo fmt	2020-07-08 17:34:37 +02:00
Alex	84bbbfaa7b	Add multiple headers to object model	2020-07-08 17:33:14 +02:00
Alex	a5fa2a136b	(WIP) New object table model, TODO: update API calls to use it	2020-07-08 16:46:47 +02:00
Alex	bec26a1312	Rename garage_core to garage_model	2020-07-07 13:59:22 +02:00

Renamed from src/core/object_table.rs (Browse further)