Commit graph

185 commits

Author SHA1 Message Date
f2106c2733 Implement rpc_secret_file 2023-01-04 18:35:10 +01:00
1649002e2b Merge pull request 'Add a note about Peertube 5.0 private videos' () from kaiyou/garage:docs-apps into main
Reviewed-on: 
2023-01-02 12:49:14 +00:00
fd10200bec Add a note about Peertube 5.0 private videos 2022-12-25 14:20:01 +01:00
0c7ed0b0af Add some docs about using Python Minio SDK 2022-12-25 13:55:12 +01:00
e852c91d18 Fix documentation based on new deployment values 2022-12-25 13:30:14 +01:00
d5a2502b09 Fix typo in documentation 2022-12-07 12:43:49 +00:00
a3afc761b6 Update 'doc/book/design/goals.md' 2022-12-04 16:27:46 +00:00
379b2049f5
Fix : documentation issue 2022-11-24 15:33:33 +01:00
555a54ec40
doc precisions and fixes 2022-11-16 13:40:49 +01:00
fc8f795bba
Rename subsections and add docker compose file 2022-11-16 13:33:33 +01:00
a7af0c8af9
Add best practices and doc of monitoring (fix ) 2022-11-16 13:27:24 +01:00
bcc9772470 Merge pull request 'OpenAPI spec for admin API' () from ecosystem/openapi into main
Reviewed-on: 
2022-11-16 10:51:04 +00:00
05547f2ba6
Move testing strategy to a dedicated doc section (fix ) 2022-11-14 13:34:00 +01:00
cf23aee183
Add a "build" section, doc for SDK 2022-11-13 16:48:52 +01:00
Baptiste Jonglez
8dfc909759 Improve Nginx reverse proxy example
By default, Nginx does proxy buffering and it may store big replies to a
temporary file up to 1 GB.  It also means that Nginx will read data as
fast as possible from Garage, even if the client downloads slowly.  Both
behaviours are often not wanted, so disable this temporary file in the example.

Ref: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering

Also add an example of upstream with a "backup" server, which may be
useful to only use remote servers as fallback.
2022-11-11 21:50:08 +01:00
a976c9190c
Use awscli in the getting started guide 2022-11-11 12:48:52 +01:00
Baptiste Jonglez
26b3295aaa Add documentation to run Mastodon on Garage 2022-11-06 14:07:31 +01:00
8bc5caf7aa
Fix issue with 'http(s)://' prefix 2022-10-18 21:17:11 +02:00
002b9fc50c
Add TLS support for Consul discovery + refactoring 2022-10-18 18:38:20 +02:00
8e442001b9 Update 'doc/book/reference-manual/features.md'
typo
2022-10-16 07:13:21 +00:00
d2c937a931 Fix typo 2022-09-30 18:46:57 +02:00
744c3b4d94 Update docs 2022-09-30 18:46:57 +02:00
37a73d7d37 Move documentation to book 2022-09-30 18:46:57 +02:00
0ab0d3cc29
Document db_engine 2022-09-27 16:52:36 +02:00
7291747a28 Merge pull request 'Documentation changes for v0.8' () from doc-0.8 into main
Reviewed-on: 
2022-09-27 11:37:12 +02:00
194e8be1bb
Update docker image links 2022-09-26 18:01:17 +02:00
69bcc813de
Add garage v0.8 migration guide 2022-09-26 17:46:38 +02:00
a3758dc4c4
Update README 2022-09-21 12:53:02 +02:00
1d0a610690
Finish writing about Garage features, and fix from-source instructions 2022-09-15 13:23:57 +02:00
f6aebefcc9
Some work on documentation towards v0.8 2022-09-14 19:31:13 +02:00
bbb970965c
Document available build features 2022-09-06 17:16:45 +02:00
532eca7ff9
Add some documentation for Caddy 2022-08-12 10:33:41 +02:00
b74b533b7b Fix typo 2022-06-29 11:50:51 +02:00
77e3fd6db2 improve internal item counter mechanisms and implement bucket quotas ()
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: 
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-15 20:20:28 +02:00
Simon C
7eed3ceda9 docs: Add Trafik reverse proxy documentation 2022-06-07 16:16:52 +02:00
Simon C
4b8f48f3c5 docs: Fix title level 2022-06-07 13:32:52 +02:00
Simon C
7d3b5585f1 docs: Add link to facilitate navigation in the documentation 2022-06-07 09:38:59 +02:00
a1abed0378
Remove useless MC_REGION env variable 2022-06-02 12:50:11 +02:00
93eab8eaa3 Fixes to S3 compatibility page ()
Mention PostObject is implemented, fix english mistakes

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: 
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-25 16:54:44 +02:00
43ddc933f9
Update Ceph S3 endpoints compatibility 2022-05-25 15:20:08 +02:00
9f303f6308
Shorter page title 2022-05-24 15:47:42 +02:00
3be43f3372
Add lost content for Restic with Garage
Suggested-by: Quentin <quentin@deuxfleurs.fr>
2022-05-24 15:32:42 +02:00
2da448b43f
Add documentation for new Admin API and a few infos on K2V 2022-05-24 15:28:37 +02:00
Baptiste Jonglez
47e57518ec Add documentation on running Kopia with Garage 2022-04-10 13:04:07 +02:00
8e3ee82c3e Be clearer on what upgrades are (not) supported 2022-04-06 21:45:59 +02:00
a122a8cb46 Add an "upgrading" section, add a guide for 0.7 2022-04-05 10:08:31 +02:00
9fd8ec1dee Add documentation for winscp+sftpgo 2022-03-31 10:25:56 +02:00
0091002ef2
New replication modes and their documentation 2022-03-28 16:26:04 +02:00
dca2ffdf91
document administrative options 2022-03-28 12:26:08 +02:00
822128e3c8 Talk a bit about capacity balancing between regions 2022-03-22 12:07:13 +01:00
Rune Henriksen
aea8b41728 document request routing logic 2022-03-21 12:03:57 +01:00
Rune Henriksen
71e6645e09 add short tutorial for duplicati usage with garage 2022-03-21 11:58:19 +01:00
15da2156f6 Change position of the node-id argument 2022-03-19 18:03:23 +01:00
9d44127245
add support for kubernetes service discovery
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
2022-03-12 13:05:52 +01:00
2465163e39
documentation: add mention to install build-essential 2022-02-16 12:18:24 +01:00
4ae03aa774 Small documentation updates ()
Fixes , among other things

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: 
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-02-10 15:58:09 +01:00
3e1373fafc Add a new S3 comparison Matrix to documentation ()
Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: 
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-02-07 16:04:52 +01:00
7d68b7060e
Fix anchors in links 2022-02-07 16:01:48 +01:00
99ed67503c
Update quickstart to set endpoints ending in localhost 2022-02-07 16:01:48 +01:00
5a1fb7cce7
Improve integration part of the doc 2022-02-07 16:01:45 +01:00
1c0ba930b8 Reorganize documentation for new website ()
This PR should be merged after the new website is deployed.

- [x] Rename files
- [x] Add front matter section to all `.md` files in the book (necessary for Zola)
- [x] Change all internal links to use Zola's linking system that checks broken links
- [x] Some updates to documentation contents and organization

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: 
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-02-07 11:51:12 +01:00
2760f1cb17
Add advice about --fast-list 2022-01-31 16:51:39 +01:00
26849ed066
Add step to 0.6.0 migration guide 2022-01-27 14:31:25 +01:00
ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
2022-01-24 11:58:00 +01:00
94f0e7c135 Test ListParts endpoint with awscli 2022-01-21 10:42:35 +01:00
60c0033c8b
Update documentation 2022-01-13 14:25:22 +01:00
6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
b4592a00fe Implement ListMultipartUploads ()
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: 
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
9cb2e9e57c
Add documentation to migrate to 0.6 2022-01-10 14:42:58 +01:00
8f39360f22
Update documentation 2022-01-07 17:14:37 +01:00
945b75dbf1 update s3 compatibility list ()
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: 
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:54 +01:00
1eb972b1ac Add compression using zstd ()
fix 

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: 
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 11:26:43 +01:00
3b3a1f275f
Add a second plot 2021-12-13 11:58:03 +01:00
dba9af2968
Update benchmark 2021-12-09 18:42:45 +01:00
e9358054ac
Typos and dead links 2021-12-08 14:40:14 +01:00
f9e5520ffb
Add a benchmark page with a first benchmark 2021-12-08 11:30:07 +01:00
4b369347c0
S3 compatibility target 2021-12-06 17:18:45 +01:00
224c89ad6e
Reorganize and improve documentation 2021-12-06 16:33:01 +01:00
7c2037ba87
WIP front page garage 2021-12-06 15:33:43 +01:00
c94406f428
Improve how node roles are assigned in Garage
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
2021-11-16 16:05:53 +01:00
53888995bd update doc and comments 2021-11-16 15:41:41 +01:00
57df9c6e2d add s3_api.root_domain to doc book 2021-11-16 15:41:41 +01:00
cdeb5b4dbb added link to RFID Garage talk ()
Co-authored-by: ADRN <adrien@luxeylab.net>
Reviewed-on: 
Co-authored-by: adrien <adrien@luxeylab.net>
Co-committed-by: adrien <adrien@luxeylab.net>
2021-11-16 15:39:08 +01:00
100aad8bf4
Add rclone mount doc 2021-11-10 18:38:17 +01:00
80a87929b0
Improve CLI documentation 2021-11-10 18:18:34 +01:00
76d21be1b9
Add skeleton for backups, fuse and code sections 2021-11-10 18:05:07 +01:00
1928f59d54
Add documentation for Gitea. 2021-11-10 12:41:09 +01:00
323514be15
Documentation for Nix binary cache 2021-11-10 10:02:22 +01:00
ad8d5139cf
hugo deploy does not build website, fix doc 2021-11-10 10:02:22 +01:00
08b1e8a7ea
Move design draft to separate file; write about GC in internals 2021-11-09 12:25:33 +01:00
8e25a37f0e
Add documentation for nginx 2021-11-08 12:20:40 +01:00
e342db19aa
Add documentation about Gateways 2021-11-08 12:20:40 +01:00
f3405b6378
Doc about exposing your website 2021-11-08 12:20:40 +01:00
860ccf2811
Harden Garage's systemd service 2021-11-08 12:20:40 +01:00
9df7559446
Documentation for hugo, jekyll and publii 2021-11-08 12:20:40 +01:00
a97467075d
Add documentation for synapse-s3-storage-provider 2021-11-08 12:20:40 +01:00
da6efb4b23 fix missing bootstrap_peers in doc 2021-11-05 11:21:50 +01:00
e93d7fb228
Add Peertube + improve CLI instructions 2021-11-03 14:39:14 +01:00
eaf54efb25
Add doc for Nextcloud 2021-11-03 14:07:55 +01:00
93f8d59e4c
Extract toolchain build from the CI 2021-10-29 11:34:01 +02:00