324 commits

Author	SHA1	Message	Date
Alex	c2a9f00a58	Merge pull request 'upgrading.md: Added small note about garage_build_info.' (#501) from jpds/garage:doc-upgrade-buildinfo-metric into main ... Reviewed-on: Deuxfleurs/garage#501	2023-02-06 14:20:00 +00:00
Jonathan Davies	179fda9fb6	upgrading.md: Added small note about garage_build_info.	2023-02-06 12:53:55 +00:00
Alex Auvolat	80e2326998	fixes for pr 499	2023-02-06 12:23:55 +01:00
Alex Auvolat	656b8d42de	secrets can be passed directly in config, as file, or as env	2023-02-03 15:27:39 +01:00
Alex Auvolat	8013a5cd58	Change talk links more	2023-01-30 18:51:48 +01:00
Alex Auvolat	2ba9463a8a	Raw links to presentations	2023-01-30 18:48:00 +01:00
Alex Auvolat	7f715ba94f	zero-downtime migration procedure	2023-01-30 18:41:04 +01:00
Alex Auvolat	44f8b1d71a	Reorder reference manual section, move metrics list to there	2023-01-30 18:00:01 +01:00
Alex Auvolat	56384677fa	Add links to presentations	2023-01-30 17:48:36 +01:00
Jonathan Davies	5f412abd4e	cookbook/reverse-proxy.md: Added on-demand TLS section.	2023-01-30 14:37:55 +00:00
Jonathan Davies	c753a9dfb6	cookbook/monitoring.md: Added new metrics (garage_build_info, ... garage_replication_factor, block_compression_level).	2023-01-30 12:54:42 +00:00
Jonathan Davies	ae9c7a2900	cookbook/_index.md: Added link to monitoring documentation.	2023-01-30 12:54:42 +00:00
Jonathan Davies	7ab27f84b8	configuration.md: Corrected OpenTelemetry.	2023-01-30 12:54:42 +00:00
Jonathan Davies	55c369137d	gateways.md: -z is a required flag for layout assign.	2023-01-30 12:54:38 +00:00
Jonathan Davies	0c618f8a89	reverse-proxy.md: Corrected web server ports in Caddy example.	2023-01-27 17:52:51 +00:00
Alex	a08e01f17a	Merge pull request 'Enable daemonset deployment using the helm chart' (#409) from kaiyou/garage:feat-k8s-daemonset into main ... Reviewed-on: Deuxfleurs/garage#409	2023-01-26 21:07:58 +00:00
Alex Auvolat	590a0a8450	Merge branch 'main' into k2v-watch-range-2	2023-01-26 16:46:40 +01:00
Jonathan Davies	f2492107d7	cookbook/real-world.md: Added note about mesh network options.	2023-01-25 12:00:01 +00:00
Alex	9adf5ca76d	Merge pull request 'Add talk made on 2023-01-18' (#482) from talk-tocatta-2023-01-18 into main ... Reviewed-on: Deuxfleurs/garage#482	2023-01-20 11:40:08 +00:00
Alex	18bf45061a	Merge pull request 'doc: Added observability.md.' (#477) from jpds/garage:observability-doc into main ... Reviewed-on: Deuxfleurs/garage#477	2023-01-19 12:34:14 +00:00
Alex Auvolat	3250be7c48	Update tocatta talk, add talks shell.nix and .envrc	2023-01-18 15:25:04 +01:00
Mike Coleman	fcc5033466	Change some integer types to int64 ... Modified integer types representing byte or object count to int64 to prevent overflow.	2023-01-16 23:57:23 -08:00
Jonathan Davies	97bb110219	doc: Added observability.md.	2023-01-13 14:32:10 +00:00
Alex Auvolat	0010f705ef	Talk for 2023-01-18 pretty much finished	2023-01-13 15:28:17 +01:00
Alex Auvolat	065d6e1e06	Talk about K2V specifics	2023-01-13 13:51:39 +01:00
Alex Auvolat	d44e8366e7	Reorder and add a hard problem	2023-01-13 13:16:55 +01:00
Alex Auvolat	cbb522e179	Different lattice figures	2023-01-13 12:33:27 +01:00
kaiyou	9bf94faaa1	Add docs about running pict-rs with garage	2023-01-12 20:46:17 +01:00
Alex Auvolat	1f5e3aaf8e	Add explanations about quorums	2023-01-12 17:39:12 +01:00
Alex Auvolat	f5a7bc3736	Add 12 lattice diagrams to explain CRDTs and quorums	2023-01-12 17:17:13 +01:00
Alex Auvolat	fe850f62c9	Talk 2023-01-18: some WIP talking about consensus	2023-01-12 16:27:02 +01:00
Alex Auvolat	7416ba97ef	Talk 2023-01-18 WIP	2023-01-12 13:25:09 +01:00
Alex Auvolat	12a4e1f303	Merge branch 'optimal-layout' into next	2023-01-11 17:50:42 +01:00
Alex Auvolat	dac254a6e7	Merge branch 'main' into k2v-watch-range-2	2023-01-11 17:09:37 +01:00
Alex Auvolat	399f137fd0	add precision in pollrange doc	2023-01-11 15:19:51 +01:00
Alex Auvolat	ba384e61c0	PollRange: return immediately if no seen marker is provided	2023-01-11 12:03:17 +01:00
Alex Auvolat	57eabe7879	Add proposal spec for PollRange API endpoint	2023-01-10 15:22:11 +01:00
Mendes	597d64b31a	change in gitignore	2023-01-09 16:06:47 +01:00
Mendes	e3cc7a89b0	First draft of t a preprint describing the layout computation algorithm	2023-01-09 16:05:20 +01:00
Felix Scheinost	f2106c2733	Implement rpc_secret_file	2023-01-04 18:35:10 +01:00
Alex	80e4abb98d	Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next ... Reviewed-on: Deuxfleurs/garage#459	2023-01-04 10:35:49 +00:00
Alex Auvolat	570e5e5bbb	Merge branch 'main' into next	2023-01-04 11:34:43 +01:00
Jonathan Davies	8be862aa19	Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time.	2023-01-03 11:11:12 +00:00
Alex	1649002e2b	Merge pull request 'Add a note about Peertube 5.0 private videos' (#456) from kaiyou/garage:docs-apps into main ... Reviewed-on: Deuxfleurs/garage#456	2023-01-02 12:49:14 +00:00
kaiyou	fd10200bec	Add a note about Peertube 5.0 private videos	2022-12-25 14:20:01 +01:00
kaiyou	0c7ed0b0af	Add some docs about using Python Minio SDK	2022-12-25 13:55:12 +01:00
kaiyou	e852c91d18	Fix documentation based on new deployment values	2022-12-25 13:30:14 +01:00
Alex Auvolat	2c2e65ad8b	Merge commit 'ec12d6c' into next	2022-12-11 18:41:15 +01:00
Alex	defd7d9e63	Merge pull request 'Implement /health admin API endpoint to check node health' (#440) from admin-health-api into main ... Reviewed-on: Deuxfleurs/garage#440	2022-12-11 17:25:28 +00:00
Alex Auvolat	5ea5fd2130	Always return 200 OK on /v0/health, reinstate admin api doc as draft and complete it	2022-12-11 18:11:28 +01:00
felix.scheinost	d5a2502b09	Fix typo in documentation	2022-12-07 12:43:49 +00:00
tompearson	a3afc761b6	Update 'doc/book/design/goals.md'	2022-12-04 16:27:46 +00:00
vincent	4d7b4d9d20	Add talk to the Capitole du Libre 2022	2022-11-27 11:36:01 +01:00
Alex Auvolat	379b2049f5	Fix #432: documentation issue	2022-11-24 15:33:33 +01:00
Alex Auvolat	555a54ec40	doc precisions and fixes	2022-11-16 13:40:49 +01:00
Alex Auvolat	fc8f795bba	Rename subsections and add docker compose file	2022-11-16 13:33:33 +01:00
Alex Auvolat	a7af0c8af9	Add best practices and doc of monitoring (fix #419)	2022-11-16 13:27:24 +01:00
Alex	bcc9772470	Merge pull request 'OpenAPI spec for admin API' (#379) from ecosystem/openapi into main ... Reviewed-on: Deuxfleurs/garage#379	2022-11-16 10:51:04 +00:00
Alex Auvolat	05547f2ba6	Move testing strategy to a dedicated doc section (fix #114)	2022-11-14 13:34:00 +01:00
Quentin Dufour	cf23aee183	Add a "build" section, doc for SDK	2022-11-13 16:48:52 +01:00
Quentin Dufour	74ea449f4b	Add missing parameter	2022-11-12 23:04:37 +01:00
Quentin Dufour	eabb37b53f	openapi validate fix	2022-11-12 22:37:42 +01:00
Quentin Dufour	e7824faa17	Finalize the specification of the admin API	2022-11-12 18:08:41 +01:00
Baptiste Jonglez	8dfc909759	Improve Nginx reverse proxy example ... By default, Nginx does proxy buffering and it may store big replies to a temporary file up to 1 GB. It also means that Nginx will read data as fast as possible from Garage, even if the client downloads slowly. Both behaviours are often not wanted, so disable this temporary file in the example. Ref: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering Also add an example of upstream with a "backup" server, which may be useful to only use remote servers as fallback.	2022-11-11 21:50:08 +01:00
Quentin Dufour	485109ea60	Bucket CRUD is defined	2022-11-11 18:32:35 +01:00
Quentin Dufour	ebe8a41f2d	Bucket skeleton	2022-11-11 17:10:41 +01:00
Quentin Dufour	a976c9190c	Use awscli in the getting started guide	2022-11-11 12:48:52 +01:00
Quentin Dufour	72a0f90070	Make capacity nullable to allow gateway config	2022-11-11 09:22:37 +01:00
Quentin Dufour	d814deb806	Error is nullable on AddNode	2022-11-11 09:22:37 +01:00
Quentin Dufour	6a09f16da7	Set required fields in the spec	2022-11-11 09:22:36 +01:00
Quentin Dufour	23207d18a0	Fix case of garage version	2022-11-11 09:22:36 +01:00
Quentin Dufour	3024405a65	Add operationId to entrypoints	2022-11-11 09:22:36 +01:00
Quentin Dufour	5f0928f89c	Declare Authorization scheme in OpenAPI	2022-11-11 09:22:36 +01:00
Quentin Dufour	0a01b34e81	Partial OpenAPI spec for admin API with a viewer	2022-11-11 09:22:36 +01:00
Alex Auvolat	28d7a49f63	Merge branch 'main' into optimal-layout	2022-11-07 12:20:59 +01:00
Baptiste Jonglez	26b3295aaa	Add documentation to run Mastodon on Garage	2022-11-06 14:07:31 +01:00
Alex Auvolat	8bc5caf7aa	Fix issue with 'http(s)://' prefix	2022-10-18 21:17:11 +02:00
Alex Auvolat	002b9fc50c	Add TLS support for Consul discovery + refactoring	2022-10-18 18:38:20 +02:00
Tobias Krischer	f1c96d108c	update k2v docs for status 204 changes	2022-10-18 13:50:56 +02:00
borgified	8e442001b9	Update 'doc/book/reference-manual/features.md' ... typo	2022-10-16 07:13:21 +00:00
Mendes	829f815a89	Merge remote-tracking branch 'origin/main' into optimal-layout	2022-10-04 18:14:49 +02:00
chemicstry	d2c937a931	Fix typo	2022-09-30 18:46:57 +02:00
chemicstry	744c3b4d94	Update docs	2022-09-30 18:46:57 +02:00
chemicstry	37a73d7d37	Move documentation to book	2022-09-30 18:46:57 +02:00
Alex Auvolat	0ab0d3cc29	Document db_engine	2022-09-27 16:52:36 +02:00
Alex	7291747a28	Merge pull request 'Documentation changes for v0.8' (#394) from doc-0.8 into main ... Reviewed-on: Deuxfleurs/garage#394	2022-09-27 11:37:12 +02:00
Alex Auvolat	194e8be1bb	Update docker image links	2022-09-26 18:01:17 +02:00
Alex Auvolat	69bcc813de	Add garage v0.8 migration guide	2022-09-26 17:46:38 +02:00
Mendes	7f3249a237	New version of the algorithm that calculate the layout. ... It takes as paramters the replication factor and the zone redundancy, computes the largest partition size reachable with these constraints, and among the possible assignation with this partition size, it computes the one that moves the least number of partitions compared to the previous assignation. This computation uses graph algorithms defined in graph_algo.rs	2022-09-21 14:39:59 +02:00
Alex Auvolat	a3758dc4c4	Update README	2022-09-21 12:53:02 +02:00
Alex Auvolat	1d0a610690	Finish writing about Garage features, and fix from-source instructions	2022-09-15 13:23:57 +02:00
Alex Auvolat	f6aebefcc9	Some work on documentation towards v0.8	2022-09-14 19:31:13 +02:00
Mendes	c4adbeed51	Added the section with description proofs of the parametric assignment computation in the optimal layout report	2022-09-10 13:51:12 +02:00
Mendes	d38fb6c250	ignore log files in commit	2022-09-08 12:43:33 +02:00
Alex Auvolat	bbb970965c	Document available build features	2022-09-06 17:16:45 +02:00
Mendes	81083dd415	Added a first draft version of the algorithm and analysis for the non-strict mode.	2022-08-19 21:21:41 +02:00
Quentin Dufour	532eca7ff9	Add some documentation for Caddy	2022-08-12 10:33:41 +02:00
Mendes	03e3a1bd15	Added the latex report on the optimal layout algorithm	2022-07-18 22:35:29 +02:00
Alex	b74b533b7b	Fix typo	2022-06-29 11:50:51 +02:00
Alex	996f2a6d58	Slides for talk at IMT Atlantique / STACK on 2022-06-23 (#333) ... Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: Deuxfleurs/garage#333 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-23 14:28:40 +02:00
Alex	77e3fd6db2	improve internal item counter mechanisms and implement bucket quotas (#326) ... - [x] Refactoring of internal counting API - [x] Repair procedure for counters (it's an offline procedure!!!) - [x] New counter for objects in buckets - [x] Add quotas to buckets struct - [x] Add CLI to manage bucket quotas - [x] Add admin API to manage bucket quotas - [x] Apply quotas by adding checks on put operations - [x] Proof-read Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: Deuxfleurs/garage#326 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-15 20:20:28 +02:00
Simon C	7eed3ceda9	docs: Add Trafik reverse proxy documentation	2022-06-07 16:16:52 +02:00
Simon C	4b8f48f3c5	docs: Fix title level	2022-06-07 13:32:52 +02:00
Simon C	7d3b5585f1	docs: Add link to facilitate navigation in the documentation	2022-06-07 09:38:59 +02:00
Quentin Dufour	a1abed0378	Remove useless MC_REGION env variable	2022-06-02 12:50:11 +02:00
Alex	93eab8eaa3	Fixes to S3 compatibility page (#314) ... Mention PostObject is implemented, fix english mistakes Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: Deuxfleurs/garage#314 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-25 16:54:44 +02:00
Quentin Dufour	43ddc933f9	Update Ceph S3 endpoints compatibility	2022-05-25 15:20:08 +02:00
Alex Auvolat	9f303f6308	Shorter page title	2022-05-24 15:47:42 +02:00
Alex Auvolat	3be43f3372	Add lost content for Restic with Garage ... Suggested-by: Quentin <quentin@deuxfleurs.fr>	2022-05-24 15:32:42 +02:00
Alex Auvolat	2da448b43f	Add documentation for new Admin API and a few infos on K2V	2022-05-24 15:28:37 +02:00
Alex	382e74c798	First version of admin API (#298) ... **Spec:** - [x] Start writing - [x] Specify all layout endpoints - [x] Specify all endpoints for operations on keys - [x] Specify all endpoints for operations on key/bucket permissions - [x] Specify all endpoints for operations on buckets - [x] Specify all endpoints for operations on bucket aliases View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md> **Code:** - [x] Refactor code for admin api to use common api code that was created for K2V **General endpoints:** - [x] Metrics - [x] GetClusterStatus - [x] ConnectClusterNodes - [x] GetClusterLayout - [x] UpdateClusterLayout - [x] ApplyClusterLayout - [x] RevertClusterLayout **Key-related endpoints:** - [x] ListKeys - [x] CreateKey - [x] ImportKey - [x] GetKeyInfo - [x] UpdateKey - [x] DeleteKey **Bucket-related endpoints:** - [x] ListBuckets - [x] CreateBucket - [x] GetBucketInfo - [x] DeleteBucket - [x] PutBucketWebsite - [x] DeleteBucketWebsite **Operations on key/bucket permissions:** - [x] BucketAllowKey - [x] BucketDenyKey **Operations on bucket aliases:** - [x] GlobalAliasBucket - [x] GlobalUnaliasBucket - [x] LocalAliasBucket - [x] LocalUnaliasBucket **And also:** - [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling) - [x] Add management of website access - [ ] Check that nothing is missing wrt what can be done using the CLI - [ ] Improve formatting of the spec - [x] Make sure everyone is cool with the API design Fix #231 Fix #295 Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: Deuxfleurs/garage#298 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-24 12:16:39 +02:00
Alex Auvolat	176715c5b2	Fix ReadIndex spec and add JSON5 remark to doc	2022-05-16 11:54:37 +02:00
Alex	5768bf3622	First implementation of K2V (#293) ... **Specification:** View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [x] Put K2V behind `k2v` feature flag Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: Deuxfleurs/garage#293 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-10 13:16:57 +02:00
Baptiste Jonglez	47e57518ec	Add documentation on running Kopia with Garage	2022-04-10 13:04:07 +02:00
Quentin Dufour	8e3ee82c3e	Be clearer on what upgrades are (not) supported	2022-04-06 21:45:59 +02:00
Quentin Dufour	a122a8cb46	Add an "upgrading" section, add a guide for 0.7	2022-04-05 10:08:31 +02:00
Quentin Dufour	9fd8ec1dee	Add documentation for winscp+sftpgo	2022-03-31 10:25:56 +02:00
Alex Auvolat	0091002ef2	New replication modes and their documentation	2022-03-28 16:26:04 +02:00
Alex Auvolat	dca2ffdf91	document administrative options	2022-03-28 12:26:08 +02:00
Alex Auvolat	822128e3c8	Talk a bit about capacity balancing between regions	2022-03-22 12:07:13 +01:00
Rune Henriksen	aea8b41728	document request routing logic	2022-03-21 12:03:57 +01:00
Rune Henriksen	71e6645e09	add short tutorial for duplicati usage with garage	2022-03-21 11:58:19 +01:00
Steam	15da2156f6	Change position of the node-id argument	2022-03-19 18:03:23 +01:00
Max Audron	9d44127245	add support for kubernetes service discovery ... This commit adds support to discover garage instances running in kubernetes. Once enabled by setting `kubernetes_namespace` and `kubernetes_service_name` garage will create a Custom Resources `garagenodes.deuxfleurs.fr` with nodes public key as the resource name. and IP and Port information as spec in the namespace configured by `kubernetes_namespace`. For discovering nodes the resources are filtered with the optionally set `kubernetes_service_name` which sets a label `garage.deuxfleurs.fr/service` on the resources. This allows to separate multiple garage deployments in a single namespace. the `kubernetes_skip_crd` variable allows to disable the creation of the CRD by garage itself. The user must deploy this manually.	2022-03-12 13:05:52 +01:00
Alex Auvolat	e312ba977e	Add FOSDEM talk and move all .pdf files to Git LFS	2022-02-16 20:01:36 +01:00
Alex Auvolat	2465163e39	documentation: add mention to install build-essential	2022-02-16 12:18:24 +01:00
Alex	4ae03aa774	Small documentation updates (#237) ... Fixes #234, among other things Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: Deuxfleurs/garage#237 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-02-10 15:58:09 +01:00
Quentin	3e1373fafc	Add a new S3 comparison Matrix to documentation (#220) ... Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: Deuxfleurs/garage#220 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-02-07 16:04:52 +01:00
Alex Auvolat	7d68b7060e	Fix anchors in links	2022-02-07 16:01:48 +01:00
Quentin Dufour	99ed67503c	Update quickstart to set endpoints ending in localhost	2022-02-07 16:01:48 +01:00
Quentin Dufour	5a1fb7cce7	Improve integration part of the doc	2022-02-07 16:01:45 +01:00
Alex	1c0ba930b8	Reorganize documentation for new website (#213) ... This PR should be merged after the new website is deployed. - [x] Rename files - [x] Add front matter section to all `.md` files in the book (necessary for Zola) - [x] Change all internal links to use Zola's linking system that checks broken links - [x] Some updates to documentation contents and organization Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: Deuxfleurs/garage#213 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-02-07 11:51:12 +01:00
Alex Auvolat	2760f1cb17	Add advice about --fast-list	2022-01-31 16:51:39 +01:00
Alex Auvolat	26849ed066	Add step to 0.6.0 migration guide	2022-01-27 14:31:25 +01:00
Alex Auvolat	ea7fb901eb	Implement {Put,Get,Delete}BucketCors and CORS in general ... - OPTIONS request against API endpoint - Returning corresponding CORS headers on API calls - Returning corresponding CORS headers on website GET's	2022-01-24 11:58:00 +01:00
Quentin Dufour	94f0e7c135	Test ListParts endpoint with awscli	2022-01-21 10:42:35 +01:00
Alex Auvolat	60c0033c8b	Update documentation	2022-01-13 14:25:22 +01:00
Alex Auvolat	6617a72220	Implement UploadPartCopy	2022-01-13 13:58:47 +01:00
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: Deuxfleurs/garage#171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
Alex Auvolat	9cb2e9e57c	Add documentation to migrate to 0.6	2022-01-10 14:42:58 +01:00
Alex Auvolat	8f39360f22	Update documentation	2022-01-07 17:14:37 +01:00
trinity-1686a	945b75dbf1	update s3 compatibility list (#177) ... Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: Deuxfleurs/garage#177 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 15:05:54 +01:00
trinity-1686a	1eb972b1ac	Add compression using zstd (#173) ... fix #27 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: Deuxfleurs/garage#173 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 11:26:43 +01:00
Quentin Dufour	3b3a1f275f	Add a second plot	2021-12-13 11:58:03 +01:00
Quentin Dufour	dba9af2968	Update benchmark	2021-12-09 18:42:45 +01:00
Quentin Dufour	e9358054ac	Typos and dead links	2021-12-08 14:40:14 +01:00
Quentin Dufour	f9e5520ffb	Add a benchmark page with a first benchmark	2021-12-08 11:30:07 +01:00
Alex Auvolat	4b369347c0	S3 compatibility target	2021-12-06 17:18:45 +01:00
Alex Auvolat	224c89ad6e	Reorganize and improve documentation	2021-12-06 16:33:01 +01:00
Quentin Dufour	7c2037ba87	WIP front page garage	2021-12-06 15:33:43 +01:00