jpds/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues 29 Pull requests Projects Releases Packages Wiki Activity
1368 commits 43 branches 48 tags 59 MiB
Commit graph

259 commits

Author SHA1 Message Date
Alex
a08e01f17a Merge pull request 'Enable daemonset deployment using the helm chart' (#409) from kaiyou/garage:feat-k8s-daemonset into main 
Reviewed-on: Deuxfleurs/garage#409
 2023-01-26 21:07:58 +00:00
Alex Auvolat
590a0a8450 Merge branch 'main' into k2v-watch-range-2 2023-01-26 16:46:40 +01:00
Jonathan Davies
f2492107d7 cookbook/real-world.md: Added note about mesh network options. 2023-01-25 12:00:01 +00:00
Alex
9adf5ca76d Merge pull request 'Add talk made on 2023-01-18' (#482) from talk-tocatta-2023-01-18 into main 
Reviewed-on: Deuxfleurs/garage#482
 2023-01-20 11:40:08 +00:00
Alex
18bf45061a Merge pull request 'doc: Added observability.md.' (#477) from jpds/garage:observability-doc into main 
Reviewed-on: Deuxfleurs/garage#477
 2023-01-19 12:34:14 +00:00
Alex Auvolat
3250be7c48 Update tocatta talk, add talks shell.nix and .envrc 2023-01-18 15:25:04 +01:00
Mike Coleman
fcc5033466 Change some integer types to int64 
Modified integer types representing byte or object count to int64 to prevent overflow.
 2023-01-16 23:57:23 -08:00
Jonathan Davies
97bb110219 doc: Added observability.md. 2023-01-13 14:32:10 +00:00
Alex Auvolat
0010f705ef
Talk for 2023-01-18 pretty much finished 2023-01-13 15:28:17 +01:00
Alex Auvolat
065d6e1e06
Talk about K2V specifics 2023-01-13 13:51:39 +01:00
Alex Auvolat
d44e8366e7
Reorder and add a hard problem 2023-01-13 13:16:55 +01:00
Alex Auvolat
cbb522e179
Different lattice figures 2023-01-13 12:33:27 +01:00
kaiyou
9bf94faaa1 Add docs about running pict-rs with garage 2023-01-12 20:46:17 +01:00
Alex Auvolat
1f5e3aaf8e
Add explanations about quorums 2023-01-12 17:39:12 +01:00
Alex Auvolat
f5a7bc3736
Add 12 lattice diagrams to explain CRDTs and quorums 2023-01-12 17:17:13 +01:00
Alex Auvolat
fe850f62c9
Talk 2023-01-18: some WIP talking about consensus 2023-01-12 16:27:02 +01:00
Alex Auvolat
7416ba97ef
Talk 2023-01-18 WIP 2023-01-12 13:25:09 +01:00
Alex Auvolat
12a4e1f303
Merge branch 'optimal-layout' into next 2023-01-11 17:50:42 +01:00
Alex Auvolat
dac254a6e7
Merge branch 'main' into k2v-watch-range-2 2023-01-11 17:09:37 +01:00
Alex Auvolat
399f137fd0
add precision in pollrange doc 2023-01-11 15:19:51 +01:00
Alex Auvolat
ba384e61c0
PollRange: return immediately if no seen marker is provided 2023-01-11 12:03:17 +01:00
Alex Auvolat
57eabe7879
Add proposal spec for PollRange API endpoint 2023-01-10 15:22:11 +01:00
Mendes
597d64b31a change in gitignore 2023-01-09 16:06:47 +01:00
Mendes
e3cc7a89b0 First draft of t a preprint describing the layout computation algorithm 2023-01-09 16:05:20 +01:00
Felix Scheinost
f2106c2733 Implement rpc_secret_file 2023-01-04 18:35:10 +01:00
Alex
80e4abb98d Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next 
Reviewed-on: Deuxfleurs/garage#459
 2023-01-04 10:35:49 +00:00
Alex Auvolat
570e5e5bbb
Merge branch 'main' into next 2023-01-04 11:34:43 +01:00
Jonathan Davies
8be862aa19 Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time. 2023-01-03 11:11:12 +00:00
Alex
1649002e2b Merge pull request 'Add a note about Peertube 5.0 private videos' (#456) from kaiyou/garage:docs-apps into main 
Reviewed-on: Deuxfleurs/garage#456
 2023-01-02 12:49:14 +00:00
kaiyou
fd10200bec Add a note about Peertube 5.0 private videos 2022-12-25 14:20:01 +01:00
kaiyou
0c7ed0b0af Add some docs about using Python Minio SDK 2022-12-25 13:55:12 +01:00
kaiyou
e852c91d18 Fix documentation based on new deployment values 2022-12-25 13:30:14 +01:00
Alex Auvolat
2c2e65ad8b
Merge commit 'ec12d6c' into next 2022-12-11 18:41:15 +01:00
Alex
defd7d9e63 Merge pull request 'Implement /health admin API endpoint to check node health' (#440) from admin-health-api into main 
Reviewed-on: Deuxfleurs/garage#440
 2022-12-11 17:25:28 +00:00
Alex Auvolat
5ea5fd2130
Always return 200 OK on /v0/health, reinstate admin api doc as draft and complete it 2022-12-11 18:11:28 +01:00
felix.scheinost
d5a2502b09 Fix typo in documentation 2022-12-07 12:43:49 +00:00
tompearson
a3afc761b6 Update 'doc/book/design/goals.md' 2022-12-04 16:27:46 +00:00
vincent
4d7b4d9d20 Add talk to the Capitole du Libre 2022 2022-11-27 11:36:01 +01:00
Alex Auvolat
379b2049f5
Fix #432: documentation issue 2022-11-24 15:33:33 +01:00
Alex Auvolat
555a54ec40
doc precisions and fixes 2022-11-16 13:40:49 +01:00
Alex Auvolat
fc8f795bba
Rename subsections and add docker compose file 2022-11-16 13:33:33 +01:00
Alex Auvolat
a7af0c8af9
Add best practices and doc of monitoring (fix #419) 2022-11-16 13:27:24 +01:00
Alex
bcc9772470 Merge pull request 'OpenAPI spec for admin API' (#379) from ecosystem/openapi into main 
Reviewed-on: Deuxfleurs/garage#379
 2022-11-16 10:51:04 +00:00
Alex Auvolat
05547f2ba6
Move testing strategy to a dedicated doc section (fix #114) 2022-11-14 13:34:00 +01:00
Quentin Dufour
cf23aee183
Add a "build" section, doc for SDK 2022-11-13 16:48:52 +01:00
Quentin Dufour
74ea449f4b
Add missing parameter 2022-11-12 23:04:37 +01:00
Quentin Dufour
eabb37b53f
openapi validate fix 2022-11-12 22:37:42 +01:00
Quentin Dufour
e7824faa17
Finalize the specification of the admin API 2022-11-12 18:08:41 +01:00
Baptiste Jonglez
8dfc909759 Improve Nginx reverse proxy example 
By default, Nginx does proxy buffering and it may store big replies to a
temporary file up to 1 GB.  It also means that Nginx will read data as
fast as possible from Garage, even if the client downloads slowly.  Both
behaviours are often not wanted, so disable this temporary file in the example.

Ref: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering

Also add an example of upstream with a "backup" server, which may be
useful to only use remote servers as fallback.
 2022-11-11 21:50:08 +01:00
Quentin Dufour
485109ea60
Bucket CRUD is defined 2022-11-11 18:32:35 +01:00
Quentin Dufour
ebe8a41f2d
Bucket skeleton 2022-11-11 17:10:41 +01:00
Quentin Dufour
a976c9190c
Use awscli in the getting started guide 2022-11-11 12:48:52 +01:00
Quentin Dufour
72a0f90070
Make capacity nullable to allow gateway config 2022-11-11 09:22:37 +01:00
Quentin Dufour
d814deb806
Error is nullable on AddNode 2022-11-11 09:22:37 +01:00
Quentin Dufour
6a09f16da7
Set required fields in the spec 2022-11-11 09:22:36 +01:00
Quentin Dufour
23207d18a0
Fix case of garage version 2022-11-11 09:22:36 +01:00
Quentin Dufour
3024405a65
Add operationId to entrypoints 2022-11-11 09:22:36 +01:00
Quentin Dufour
5f0928f89c
Declare Authorization scheme in OpenAPI 2022-11-11 09:22:36 +01:00
Quentin Dufour
0a01b34e81
Partial OpenAPI spec for admin API with a viewer 2022-11-11 09:22:36 +01:00
Alex Auvolat
28d7a49f63
Merge branch 'main' into optimal-layout 2022-11-07 12:20:59 +01:00
Baptiste Jonglez
26b3295aaa Add documentation to run Mastodon on Garage 2022-11-06 14:07:31 +01:00
Alex Auvolat
8bc5caf7aa
Fix issue with 'http(s)://' prefix 2022-10-18 21:17:11 +02:00
Alex Auvolat
002b9fc50c
Add TLS support for Consul discovery + refactoring 2022-10-18 18:38:20 +02:00
Tobias Krischer
f1c96d108c
update k2v docs for status 204 changes 2022-10-18 13:50:56 +02:00
borgified
8e442001b9 Update 'doc/book/reference-manual/features.md' 
typo
 2022-10-16 07:13:21 +00:00
Mendes
829f815a89 Merge remote-tracking branch 'origin/main' into optimal-layout 2022-10-04 18:14:49 +02:00
chemicstry
d2c937a931 Fix typo 2022-09-30 18:46:57 +02:00
chemicstry
744c3b4d94 Update docs 2022-09-30 18:46:57 +02:00
chemicstry
37a73d7d37 Move documentation to book 2022-09-30 18:46:57 +02:00
Alex Auvolat
0ab0d3cc29
Document db_engine 2022-09-27 16:52:36 +02:00
Alex
7291747a28 Merge pull request 'Documentation changes for v0.8' (#394) from doc-0.8 into main 
Reviewed-on: Deuxfleurs/garage#394
 2022-09-27 11:37:12 +02:00
Alex Auvolat
194e8be1bb
Update docker image links 2022-09-26 18:01:17 +02:00
Alex Auvolat
69bcc813de
Add garage v0.8 migration guide 2022-09-26 17:46:38 +02:00
Mendes
7f3249a237 New version of the algorithm that calculate the layout. 
It takes as paramters the replication factor and the zone redundancy, computes the
largest partition size reachable with these constraints, and among the possible
assignation with this partition size, it computes the one that moves the least number
of partitions compared to the previous assignation.
This computation uses graph algorithms defined in graph_algo.rs
 2022-09-21 14:39:59 +02:00
Alex Auvolat
a3758dc4c4
Update README 2022-09-21 12:53:02 +02:00
Alex Auvolat
1d0a610690
Finish writing about Garage features, and fix from-source instructions 2022-09-15 13:23:57 +02:00
Alex Auvolat
f6aebefcc9
Some work on documentation towards v0.8 2022-09-14 19:31:13 +02:00
Mendes
c4adbeed51 Added the section with description proofs of the parametric assignment computation in the optimal layout report 2022-09-10 13:51:12 +02:00
Mendes
d38fb6c250 ignore log files in commit 2022-09-08 12:43:33 +02:00
Alex Auvolat
bbb970965c
Document available build features 2022-09-06 17:16:45 +02:00
Mendes
81083dd415 Added a first draft version of the algorithm and analysis for the non-strict mode. 2022-08-19 21:21:41 +02:00
Quentin Dufour
532eca7ff9
Add some documentation for Caddy 2022-08-12 10:33:41 +02:00
Mendes
03e3a1bd15 Added the latex report on the optimal layout algorithm 2022-07-18 22:35:29 +02:00
Alex
b74b533b7b Fix typo 2022-06-29 11:50:51 +02:00
Alex
996f2a6d58 Slides for talk at IMT Atlantique / STACK on 2022-06-23 (#333) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#333
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-23 14:28:40 +02:00
Alex
77e3fd6db2 improve internal item counter mechanisms and implement bucket quotas (#326) 
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-15 20:20:28 +02:00
Simon C
7eed3ceda9 docs: Add Trafik reverse proxy documentation 2022-06-07 16:16:52 +02:00
Simon C
4b8f48f3c5 docs: Fix title level 2022-06-07 13:32:52 +02:00
Simon C
7d3b5585f1 docs: Add link to facilitate navigation in the documentation 2022-06-07 09:38:59 +02:00
Quentin Dufour
a1abed0378
Remove useless MC_REGION env variable 2022-06-02 12:50:11 +02:00
Alex
93eab8eaa3 Fixes to S3 compatibility page (#314) 
Mention PostObject is implemented, fix english mistakes

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#314
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-25 16:54:44 +02:00
Quentin Dufour
43ddc933f9
Update Ceph S3 endpoints compatibility 2022-05-25 15:20:08 +02:00
Alex Auvolat
9f303f6308
Shorter page title 2022-05-24 15:47:42 +02:00
Alex Auvolat
3be43f3372
Add lost content for Restic with Garage 
Suggested-by: Quentin <quentin@deuxfleurs.fr>
 2022-05-24 15:32:42 +02:00
Alex Auvolat
2da448b43f
Add documentation for new Admin API and a few infos on K2V 2022-05-24 15:28:37 +02:00
Alex
382e74c798 First version of admin API (#298) 
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:16:39 +02:00
Alex Auvolat
176715c5b2
Fix ReadIndex spec and add JSON5 remark to doc 2022-05-16 11:54:37 +02:00
Alex
5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Baptiste Jonglez
47e57518ec Add documentation on running Kopia with Garage 2022-04-10 13:04:07 +02:00
Quentin Dufour
8e3ee82c3e Be clearer on what upgrades are (not) supported 2022-04-06 21:45:59 +02:00