trinity-1686a/garage
2
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests 1 Projects Releases Wiki Activity
732 commits 13 branches 0 tags 59 MiB
Commit graph

116 commits

Author SHA1 Message Date
Mendes 7f3249a237 New version of the algorithm that calculate the layout. 
It takes as paramters the replication factor and the zone redundancy, computes the
largest partition size reachable with these constraints, and among the possible
assignation with this partition size, it computes the one that moves the least number
of partitions compared to the previous assignation.
This computation uses graph algorithms defined in graph_algo.rs
 2022-09-21 14:39:59 +02:00
Mendes c4adbeed51 Added the section with description proofs of the parametric assignment computation in the optimal layout report 2022-09-10 13:51:12 +02:00
Mendes d38fb6c250 ignore log files in commit 2022-09-08 12:43:33 +02:00
Mendes 81083dd415 Added a first draft version of the algorithm and analysis for the non-strict mode. 2022-08-19 21:21:41 +02:00
Mendes 03e3a1bd15 Added the latex report on the optimal layout algorithm 2022-07-18 22:35:29 +02:00
Baptiste Jonglez 47e57518ec Add documentation on running Kopia with Garage 2022-04-10 13:04:07 +02:00
Quentin 8e3ee82c3e Be clearer on what upgrades are (not) supported 2022-04-06 21:45:59 +02:00
Quentin a122a8cb46 Add an "upgrading" section, add a guide for 0.7 2022-04-05 10:08:31 +02:00
Quentin 9fd8ec1dee Add documentation for winscp+sftpgo 2022-03-31 10:25:56 +02:00
Alex 0091002ef2
New replication modes and their documentation 2022-03-28 16:26:04 +02:00
Alex dca2ffdf91
document administrative options 2022-03-28 12:26:08 +02:00
Alex 822128e3c8 Talk a bit about capacity balancing between regions 2022-03-22 12:07:13 +01:00
Rune Henriksen aea8b41728 document request routing logic 2022-03-21 12:03:57 +01:00
Rune Henriksen 71e6645e09 add short tutorial for duplicati usage with garage 2022-03-21 11:58:19 +01:00
Steam 15da2156f6 Change position of the node-id argument 2022-03-19 18:03:23 +01:00
Max Audron 9d44127245
add support for kubernetes service discovery 
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
 2022-03-12 13:05:52 +01:00
Alex e312ba977e
Add FOSDEM talk and move all .pdf files to Git LFS 2022-02-16 20:01:36 +01:00
Alex 2465163e39
documentation: add mention to install build-essential 2022-02-16 12:18:24 +01:00
Alex 4ae03aa774 Small documentation updates (#237) 
Fixes #234, among other things

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#237
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-02-10 15:58:09 +01:00
Quentin 3e1373fafc Add a new S3 comparison Matrix to documentation (#220) 
Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#220
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
 2022-02-07 16:04:52 +01:00
Alex 7d68b7060e
Fix anchors in links 2022-02-07 16:01:48 +01:00
Quentin 99ed67503c
Update quickstart to set endpoints ending in localhost 2022-02-07 16:01:48 +01:00
Quentin 5a1fb7cce7
Improve integration part of the doc 2022-02-07 16:01:45 +01:00
Alex 1c0ba930b8 Reorganize documentation for new website (#213) 
This PR should be merged after the new website is deployed.

- [x] Rename files
- [x] Add front matter section to all `.md` files in the book (necessary for Zola)
- [x] Change all internal links to use Zola's linking system that checks broken links
- [x] Some updates to documentation contents and organization

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#213
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-02-07 11:51:12 +01:00
Alex 2760f1cb17
Add advice about --fast-list 2022-01-31 16:51:39 +01:00
Alex 26849ed066
Add step to 0.6.0 migration guide 2022-01-27 14:31:25 +01:00
Alex ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general 
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
 2022-01-24 11:58:00 +01:00
Quentin 94f0e7c135 Test ListParts endpoint with awscli 2022-01-21 10:42:35 +01:00
Alex 60c0033c8b
Update documentation 2022-01-13 14:25:22 +01:00
Alex 6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
Quentin b4592a00fe Implement ListMultipartUploads (#171) 
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
 2022-01-12 19:04:55 +01:00
Alex 9cb2e9e57c
Add documentation to migrate to 0.6 2022-01-10 14:42:58 +01:00
Alex 8f39360f22
Update documentation 2022-01-07 17:14:37 +01:00
trinity-1686a 945b75dbf1 update s3 compatibility list (#177) 
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#177
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 15:05:54 +01:00
trinity-1686a 1eb972b1ac Add compression using zstd (#173) 
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 11:26:43 +01:00
Quentin 3b3a1f275f
Add a second plot 2021-12-13 11:58:03 +01:00
Quentin dba9af2968
Update benchmark 2021-12-09 18:42:45 +01:00
Quentin e9358054ac
Typos and dead links 2021-12-08 14:40:14 +01:00
Quentin f9e5520ffb
Add a benchmark page with a first benchmark 2021-12-08 11:30:07 +01:00
Alex 4b369347c0
S3 compatibility target 2021-12-06 17:18:45 +01:00
Alex 224c89ad6e
Reorganize and improve documentation 2021-12-06 16:33:01 +01:00
Quentin 7c2037ba87
WIP front page garage 2021-12-06 15:33:43 +01:00
Alex c94406f428
Improve how node roles are assigned in Garage 
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
 2021-11-16 16:05:53 +01:00
Trinity Pointard 53888995bd update doc and comments 2021-11-16 15:41:41 +01:00
Trinity Pointard 57df9c6e2d add s3_api.root_domain to doc book 2021-11-16 15:41:41 +01:00
adrien cdeb5b4dbb added link to RFID Garage talk (#155) 
Co-authored-by: ADRN <adrien@luxeylab.net>
Reviewed-on: Deuxfleurs/garage#155
Co-authored-by: adrien <adrien@luxeylab.net>
Co-committed-by: adrien <adrien@luxeylab.net>
 2021-11-16 15:39:08 +01:00
Quentin 100aad8bf4
Add rclone mount doc 2021-11-10 18:38:17 +01:00
Quentin 80a87929b0
Improve CLI documentation 2021-11-10 18:18:34 +01:00
Quentin 76d21be1b9
Add skeleton for backups, fuse and code sections 2021-11-10 18:05:07 +01:00
Quentin 1928f59d54
Add documentation for Gitea. 2021-11-10 12:41:09 +01:00