256 commits

Author	SHA1	Message	Date
Alex	9adf5ca76d	Merge pull request 'Add talk made on 2023-01-18' (#482) from talk-tocatta-2023-01-18 into main ... Reviewed-on: #482	2023-01-20 11:40:08 +00:00
Alex	18bf45061a	Merge pull request 'doc: Added observability.md.' (#477) from jpds/garage:observability-doc into main ... Reviewed-on: #477	2023-01-19 12:34:14 +00:00
Alex Auvolat	3250be7c48	Update tocatta talk, add talks shell.nix and .envrc	2023-01-18 15:25:04 +01:00
Mike Coleman	fcc5033466	Change some integer types to int64 ... Modified integer types representing byte or object count to int64 to prevent overflow.	2023-01-16 23:57:23 -08:00
Jonathan Davies	97bb110219	doc: Added observability.md.	2023-01-13 14:32:10 +00:00
Alex Auvolat	0010f705ef	Talk for 2023-01-18 pretty much finished	2023-01-13 15:28:17 +01:00
Alex Auvolat	065d6e1e06	Talk about K2V specifics	2023-01-13 13:51:39 +01:00
Alex Auvolat	d44e8366e7	Reorder and add a hard problem	2023-01-13 13:16:55 +01:00
Alex Auvolat	cbb522e179	Different lattice figures	2023-01-13 12:33:27 +01:00
kaiyou	9bf94faaa1	Add docs about running pict-rs with garage	2023-01-12 20:46:17 +01:00
Alex Auvolat	1f5e3aaf8e	Add explanations about quorums	2023-01-12 17:39:12 +01:00
Alex Auvolat	f5a7bc3736	Add 12 lattice diagrams to explain CRDTs and quorums	2023-01-12 17:17:13 +01:00
Alex Auvolat	fe850f62c9	Talk 2023-01-18: some WIP talking about consensus	2023-01-12 16:27:02 +01:00
Alex Auvolat	7416ba97ef	Talk 2023-01-18 WIP	2023-01-12 13:25:09 +01:00
Alex Auvolat	12a4e1f303	Merge branch 'optimal-layout' into next	2023-01-11 17:50:42 +01:00
Alex Auvolat	dac254a6e7	Merge branch 'main' into k2v-watch-range-2	2023-01-11 17:09:37 +01:00
Alex Auvolat	399f137fd0	add precision in pollrange doc	2023-01-11 15:19:51 +01:00
Alex Auvolat	ba384e61c0	PollRange: return immediately if no seen marker is provided	2023-01-11 12:03:17 +01:00
Alex Auvolat	57eabe7879	Add proposal spec for PollRange API endpoint	2023-01-10 15:22:11 +01:00
Mendes	597d64b31a	change in gitignore	2023-01-09 16:06:47 +01:00
Mendes	e3cc7a89b0	First draft of t a preprint describing the layout computation algorithm	2023-01-09 16:05:20 +01:00
Felix Scheinost	f2106c2733	Implement rpc_secret_file	2023-01-04 18:35:10 +01:00
Alex	80e4abb98d	Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next ... Reviewed-on: #459	2023-01-04 10:35:49 +00:00
Alex Auvolat	570e5e5bbb	Merge branch 'main' into next	2023-01-04 11:34:43 +01:00
Jonathan Davies	8be862aa19	Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time.	2023-01-03 11:11:12 +00:00
Alex	1649002e2b	Merge pull request 'Add a note about Peertube 5.0 private videos' (#456) from kaiyou/garage:docs-apps into main ... Reviewed-on: #456	2023-01-02 12:49:14 +00:00
kaiyou	fd10200bec	Add a note about Peertube 5.0 private videos	2022-12-25 14:20:01 +01:00
kaiyou	0c7ed0b0af	Add some docs about using Python Minio SDK	2022-12-25 13:55:12 +01:00
kaiyou	e852c91d18	Fix documentation based on new deployment values	2022-12-25 13:30:14 +01:00
Alex Auvolat	2c2e65ad8b	Merge commit 'ec12d6c' into next	2022-12-11 18:41:15 +01:00
Alex	defd7d9e63	Merge pull request 'Implement /health admin API endpoint to check node health' (#440) from admin-health-api into main ... Reviewed-on: #440	2022-12-11 17:25:28 +00:00
Alex Auvolat	5ea5fd2130	Always return 200 OK on /v0/health, reinstate admin api doc as draft and complete it	2022-12-11 18:11:28 +01:00
felix.scheinost	d5a2502b09	Fix typo in documentation	2022-12-07 12:43:49 +00:00
tompearson	a3afc761b6	Update 'doc/book/design/goals.md'	2022-12-04 16:27:46 +00:00
vincent	4d7b4d9d20	Add talk to the Capitole du Libre 2022	2022-11-27 11:36:01 +01:00
Alex Auvolat	379b2049f5	Fix #432: documentation issue	2022-11-24 15:33:33 +01:00
Alex Auvolat	555a54ec40	doc precisions and fixes	2022-11-16 13:40:49 +01:00
Alex Auvolat	fc8f795bba	Rename subsections and add docker compose file	2022-11-16 13:33:33 +01:00
Alex Auvolat	a7af0c8af9	Add best practices and doc of monitoring (fix #419)	2022-11-16 13:27:24 +01:00
Alex	bcc9772470	Merge pull request 'OpenAPI spec for admin API' (#379) from ecosystem/openapi into main ... Reviewed-on: #379	2022-11-16 10:51:04 +00:00
Alex Auvolat	05547f2ba6	Move testing strategy to a dedicated doc section (fix #114)	2022-11-14 13:34:00 +01:00
Quentin Dufour	cf23aee183	Add a "build" section, doc for SDK	2022-11-13 16:48:52 +01:00
Quentin Dufour	74ea449f4b	Add missing parameter	2022-11-12 23:04:37 +01:00
Quentin Dufour	eabb37b53f	openapi validate fix	2022-11-12 22:37:42 +01:00
Quentin Dufour	e7824faa17	Finalize the specification of the admin API	2022-11-12 18:08:41 +01:00
Baptiste Jonglez	8dfc909759	Improve Nginx reverse proxy example ... By default, Nginx does proxy buffering and it may store big replies to a temporary file up to 1 GB. It also means that Nginx will read data as fast as possible from Garage, even if the client downloads slowly. Both behaviours are often not wanted, so disable this temporary file in the example. Ref: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering Also add an example of upstream with a "backup" server, which may be useful to only use remote servers as fallback.	2022-11-11 21:50:08 +01:00
Quentin Dufour	485109ea60	Bucket CRUD is defined	2022-11-11 18:32:35 +01:00
Quentin Dufour	ebe8a41f2d	Bucket skeleton	2022-11-11 17:10:41 +01:00
Quentin Dufour	a976c9190c	Use awscli in the getting started guide	2022-11-11 12:48:52 +01:00
Quentin Dufour	72a0f90070	Make capacity nullable to allow gateway config	2022-11-11 09:22:37 +01:00
Quentin Dufour	d814deb806	Error is nullable on AddNode	2022-11-11 09:22:37 +01:00
Quentin Dufour	6a09f16da7	Set required fields in the spec	2022-11-11 09:22:36 +01:00
Quentin Dufour	23207d18a0	Fix case of garage version	2022-11-11 09:22:36 +01:00
Quentin Dufour	3024405a65	Add operationId to entrypoints	2022-11-11 09:22:36 +01:00
Quentin Dufour	5f0928f89c	Declare Authorization scheme in OpenAPI	2022-11-11 09:22:36 +01:00
Quentin Dufour	0a01b34e81	Partial OpenAPI spec for admin API with a viewer	2022-11-11 09:22:36 +01:00
Alex Auvolat	28d7a49f63	Merge branch 'main' into optimal-layout	2022-11-07 12:20:59 +01:00
Baptiste Jonglez	26b3295aaa	Add documentation to run Mastodon on Garage	2022-11-06 14:07:31 +01:00
Alex Auvolat	8bc5caf7aa	Fix issue with 'http(s)://' prefix	2022-10-18 21:17:11 +02:00
Alex Auvolat	002b9fc50c	Add TLS support for Consul discovery + refactoring	2022-10-18 18:38:20 +02:00
Tobias Krischer	f1c96d108c	update k2v docs for status 204 changes	2022-10-18 13:50:56 +02:00
borgified	8e442001b9	Update 'doc/book/reference-manual/features.md' ... typo	2022-10-16 07:13:21 +00:00
Mendes	829f815a89	Merge remote-tracking branch 'origin/main' into optimal-layout	2022-10-04 18:14:49 +02:00
chemicstry	d2c937a931	Fix typo	2022-09-30 18:46:57 +02:00
chemicstry	744c3b4d94	Update docs	2022-09-30 18:46:57 +02:00
chemicstry	37a73d7d37	Move documentation to book	2022-09-30 18:46:57 +02:00
Alex Auvolat	0ab0d3cc29	Document db_engine	2022-09-27 16:52:36 +02:00
Alex	7291747a28	Merge pull request 'Documentation changes for v0.8' (#394) from doc-0.8 into main ... Reviewed-on: #394	2022-09-27 11:37:12 +02:00
Alex Auvolat	194e8be1bb	Update docker image links	2022-09-26 18:01:17 +02:00
Alex Auvolat	69bcc813de	Add garage v0.8 migration guide	2022-09-26 17:46:38 +02:00
Mendes	7f3249a237	New version of the algorithm that calculate the layout. ... It takes as paramters the replication factor and the zone redundancy, computes the largest partition size reachable with these constraints, and among the possible assignation with this partition size, it computes the one that moves the least number of partitions compared to the previous assignation. This computation uses graph algorithms defined in graph_algo.rs	2022-09-21 14:39:59 +02:00
Alex Auvolat	a3758dc4c4	Update README	2022-09-21 12:53:02 +02:00
Alex Auvolat	1d0a610690	Finish writing about Garage features, and fix from-source instructions	2022-09-15 13:23:57 +02:00
Alex Auvolat	f6aebefcc9	Some work on documentation towards v0.8	2022-09-14 19:31:13 +02:00
Mendes	c4adbeed51	Added the section with description proofs of the parametric assignment computation in the optimal layout report	2022-09-10 13:51:12 +02:00
Mendes	d38fb6c250	ignore log files in commit	2022-09-08 12:43:33 +02:00
Alex Auvolat	bbb970965c	Document available build features	2022-09-06 17:16:45 +02:00
Mendes	81083dd415	Added a first draft version of the algorithm and analysis for the non-strict mode.	2022-08-19 21:21:41 +02:00
Quentin Dufour	532eca7ff9	Add some documentation for Caddy	2022-08-12 10:33:41 +02:00
Mendes	03e3a1bd15	Added the latex report on the optimal layout algorithm	2022-07-18 22:35:29 +02:00
Alex	b74b533b7b	Fix typo	2022-06-29 11:50:51 +02:00
Alex	996f2a6d58	Slides for talk at IMT Atlantique / STACK on 2022-06-23 (#333) ... Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #333 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-23 14:28:40 +02:00
Alex	77e3fd6db2	improve internal item counter mechanisms and implement bucket quotas (#326) ... - [x] Refactoring of internal counting API - [x] Repair procedure for counters (it's an offline procedure!!!) - [x] New counter for objects in buckets - [x] Add quotas to buckets struct - [x] Add CLI to manage bucket quotas - [x] Add admin API to manage bucket quotas - [x] Apply quotas by adding checks on put operations - [x] Proof-read Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #326 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-15 20:20:28 +02:00
Simon C	7eed3ceda9	docs: Add Trafik reverse proxy documentation	2022-06-07 16:16:52 +02:00
Simon C	4b8f48f3c5	docs: Fix title level	2022-06-07 13:32:52 +02:00
Simon C	7d3b5585f1	docs: Add link to facilitate navigation in the documentation	2022-06-07 09:38:59 +02:00
Quentin Dufour	a1abed0378	Remove useless MC_REGION env variable	2022-06-02 12:50:11 +02:00
Alex	93eab8eaa3	Fixes to S3 compatibility page (#314) ... Mention PostObject is implemented, fix english mistakes Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #314 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-25 16:54:44 +02:00
Quentin Dufour	43ddc933f9	Update Ceph S3 endpoints compatibility	2022-05-25 15:20:08 +02:00
Alex Auvolat	9f303f6308	Shorter page title	2022-05-24 15:47:42 +02:00
Alex Auvolat	3be43f3372	Add lost content for Restic with Garage ... Suggested-by: Quentin <quentin@deuxfleurs.fr>	2022-05-24 15:32:42 +02:00
Alex Auvolat	2da448b43f	Add documentation for new Admin API and a few infos on K2V	2022-05-24 15:28:37 +02:00
Alex	382e74c798	First version of admin API (#298) ... **Spec:** - [x] Start writing - [x] Specify all layout endpoints - [x] Specify all endpoints for operations on keys - [x] Specify all endpoints for operations on key/bucket permissions - [x] Specify all endpoints for operations on buckets - [x] Specify all endpoints for operations on bucket aliases View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md> **Code:** - [x] Refactor code for admin api to use common api code that was created for K2V **General endpoints:** - [x] Metrics - [x] GetClusterStatus - [x] ConnectClusterNodes - [x] GetClusterLayout - [x] UpdateClusterLayout - [x] ApplyClusterLayout - [x] RevertClusterLayout **Key-related endpoints:** - [x] ListKeys - [x] CreateKey - [x] ImportKey - [x] GetKeyInfo - [x] UpdateKey - [x] DeleteKey **Bucket-related endpoints:** - [x] ListBuckets - [x] CreateBucket - [x] GetBucketInfo - [x] DeleteBucket - [x] PutBucketWebsite - [x] DeleteBucketWebsite **Operations on key/bucket permissions:** - [x] BucketAllowKey - [x] BucketDenyKey **Operations on bucket aliases:** - [x] GlobalAliasBucket - [x] GlobalUnaliasBucket - [x] LocalAliasBucket - [x] LocalUnaliasBucket **And also:** - [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling) - [x] Add management of website access - [ ] Check that nothing is missing wrt what can be done using the CLI - [ ] Improve formatting of the spec - [x] Make sure everyone is cool with the API design Fix #231 Fix #295 Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #298 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-24 12:16:39 +02:00
Alex Auvolat	176715c5b2	Fix ReadIndex spec and add JSON5 remark to doc	2022-05-16 11:54:37 +02:00
Alex	5768bf3622	First implementation of K2V (#293) ... **Specification:** View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [x] Put K2V behind `k2v` feature flag Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #293 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-10 13:16:57 +02:00
Baptiste Jonglez	47e57518ec	Add documentation on running Kopia with Garage	2022-04-10 13:04:07 +02:00
Quentin Dufour	8e3ee82c3e	Be clearer on what upgrades are (not) supported	2022-04-06 21:45:59 +02:00
Quentin Dufour	a122a8cb46	Add an "upgrading" section, add a guide for 0.7	2022-04-05 10:08:31 +02:00
Quentin Dufour	9fd8ec1dee	Add documentation for winscp+sftpgo	2022-03-31 10:25:56 +02:00
Alex Auvolat	0091002ef2	New replication modes and their documentation	2022-03-28 16:26:04 +02:00