276 commits

Author	SHA1	Message	Date
Alex Auvolat	90b2d43eb4	Merge branch 'main' into next	2023-06-13 17:14:11 +02:00
Alex Auvolat	e7e164a280	Make fsync an option for meta and data	2023-06-09 16:23:21 +02:00
Roberto Hidalgo	32ad4538ee	fix references to old config names	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	2d46d24d06	update docs	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	b770504126	simplify code according to feedback	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	6b69404f1a	rename mode to consul_http_api	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	fd7dbea5b8	follow feedback, fold into existing feature	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	bd6485565e	allow additional ServiceMeta, docs	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	02ba9016ab	register consul services against local agent instead of catalog api	2023-05-22 08:57:15 -06:00
Jakub Jirutka	06caa12d49	doc: Add information about Alpine Linux package to Quick Start	2023-05-07 19:28:43 +02:00
Alex Auvolat	fa78d806e3	Merge branch 'main' into next	2023-04-25 12:34:26 +02:00
yuka	04a0063df9	cookbook/real-world: fix typo	2023-04-21 16:46:58 +00:00
arthurlutz	a2a35ac7a8	docs(book/quickstart): adapt aws s3 commands to example ... Signed-off-by: arthurlutz <arthurlutz@noreply.localhost>	2023-04-03 06:18:28 +00:00
Jonathan Davies	11b154b33b	cli.md: Pointed Cyberduck profile at upstream link.	2023-03-20 10:46:02 +00:00
Jonathan Davies	dc6be39833	doc: cli.md: Added s5cmd example.	2023-03-13 14:15:18 +00:00
Jonathan Davies	f056ad569d	binary-packages.md: Added.	2023-03-03 18:52:49 +00:00
Baptiste Jonglez	3b22da251d	Add documentation on community Ansible roles	2023-03-01 09:24:13 +01:00
wilson	a15eb115c8	docs: fix k2v spec link ... Signed-off-by: wilson <wilson@noreply.localhost>	2023-02-26 07:38:44 +00:00
Jonathan Davies	6b8d634cc2	cookbook/reverse-proxy.md: Fixed up Traefik section: ... * Renamed my_garage_service -> garage-s3-service. * Defined a web service for port 3902. * Added a garage-s3 router. * Pointed website definition at web service. * Use the /health endpoint for loadBalancer health check. * Renamed gzip_compress to just compression as traefik v3 will also do brotli compression.	2023-02-14 19:03:57 +00:00
Jonathan Davies	ee88ccf2b2	cookbook/reverse-proxy.md: Document how to use healthchecks for caddy.	2023-02-14 18:39:05 +00:00
Jonathan Davies	4c143776bf	backup.md: Added section for git-annex.	2023-02-08 22:54:56 +00:00
Alex	c2a9f00a58	Merge pull request 'upgrading.md: Added small note about garage_build_info.' (#501) from jpds/garage:doc-upgrade-buildinfo-metric into main ... Reviewed-on: #501	2023-02-06 14:20:00 +00:00
Jonathan Davies	179fda9fb6	upgrading.md: Added small note about garage_build_info.	2023-02-06 12:53:55 +00:00
Alex Auvolat	80e2326998	fixes for pr 499	2023-02-06 12:23:55 +01:00
Alex Auvolat	656b8d42de	secrets can be passed directly in config, as file, or as env	2023-02-03 15:27:39 +01:00
Alex Auvolat	8013a5cd58	Change talk links more	2023-01-30 18:51:48 +01:00
Alex Auvolat	2ba9463a8a	Raw links to presentations	2023-01-30 18:48:00 +01:00
Alex Auvolat	7f715ba94f	zero-downtime migration procedure	2023-01-30 18:41:04 +01:00
Alex Auvolat	44f8b1d71a	Reorder reference manual section, move metrics list to there	2023-01-30 18:00:01 +01:00
Alex Auvolat	56384677fa	Add links to presentations	2023-01-30 17:48:36 +01:00
Jonathan Davies	5f412abd4e	cookbook/reverse-proxy.md: Added on-demand TLS section.	2023-01-30 14:37:55 +00:00
Jonathan Davies	c753a9dfb6	cookbook/monitoring.md: Added new metrics (garage_build_info, ... garage_replication_factor, block_compression_level).	2023-01-30 12:54:42 +00:00
Jonathan Davies	ae9c7a2900	cookbook/_index.md: Added link to monitoring documentation.	2023-01-30 12:54:42 +00:00
Jonathan Davies	7ab27f84b8	configuration.md: Corrected OpenTelemetry.	2023-01-30 12:54:42 +00:00
Jonathan Davies	55c369137d	gateways.md: -z is a required flag for layout assign.	2023-01-30 12:54:38 +00:00
Jonathan Davies	0c618f8a89	reverse-proxy.md: Corrected web server ports in Caddy example.	2023-01-27 17:52:51 +00:00
Alex	a08e01f17a	Merge pull request 'Enable daemonset deployment using the helm chart' (#409) from kaiyou/garage:feat-k8s-daemonset into main ... Reviewed-on: #409	2023-01-26 21:07:58 +00:00
Jonathan Davies	f2492107d7	cookbook/real-world.md: Added note about mesh network options.	2023-01-25 12:00:01 +00:00
Jonathan Davies	97bb110219	doc: Added observability.md.	2023-01-13 14:32:10 +00:00
kaiyou	9bf94faaa1	Add docs about running pict-rs with garage	2023-01-12 20:46:17 +01:00
Felix Scheinost	f2106c2733	Implement rpc_secret_file	2023-01-04 18:35:10 +01:00
Jonathan Davies	8be862aa19	Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time.	2023-01-03 11:11:12 +00:00
Alex	1649002e2b	Merge pull request 'Add a note about Peertube 5.0 private videos' (#456) from kaiyou/garage:docs-apps into main ... Reviewed-on: #456	2023-01-02 12:49:14 +00:00
kaiyou	fd10200bec	Add a note about Peertube 5.0 private videos	2022-12-25 14:20:01 +01:00
kaiyou	0c7ed0b0af	Add some docs about using Python Minio SDK	2022-12-25 13:55:12 +01:00
kaiyou	e852c91d18	Fix documentation based on new deployment values	2022-12-25 13:30:14 +01:00
felix.scheinost	d5a2502b09	Fix typo in documentation	2022-12-07 12:43:49 +00:00
tompearson	a3afc761b6	Update 'doc/book/design/goals.md'	2022-12-04 16:27:46 +00:00
Alex Auvolat	379b2049f5	Fix #432: documentation issue	2022-11-24 15:33:33 +01:00
Alex Auvolat	555a54ec40	doc precisions and fixes	2022-11-16 13:40:49 +01:00
Alex Auvolat	fc8f795bba	Rename subsections and add docker compose file	2022-11-16 13:33:33 +01:00
Alex Auvolat	a7af0c8af9	Add best practices and doc of monitoring (fix #419)	2022-11-16 13:27:24 +01:00
Alex	bcc9772470	Merge pull request 'OpenAPI spec for admin API' (#379) from ecosystem/openapi into main ... Reviewed-on: #379	2022-11-16 10:51:04 +00:00
Alex Auvolat	05547f2ba6	Move testing strategy to a dedicated doc section (fix #114)	2022-11-14 13:34:00 +01:00
Quentin Dufour	cf23aee183	Add a "build" section, doc for SDK	2022-11-13 16:48:52 +01:00
Baptiste Jonglez	8dfc909759	Improve Nginx reverse proxy example ... By default, Nginx does proxy buffering and it may store big replies to a temporary file up to 1 GB. It also means that Nginx will read data as fast as possible from Garage, even if the client downloads slowly. Both behaviours are often not wanted, so disable this temporary file in the example. Ref: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering Also add an example of upstream with a "backup" server, which may be useful to only use remote servers as fallback.	2022-11-11 21:50:08 +01:00
Quentin Dufour	a976c9190c	Use awscli in the getting started guide	2022-11-11 12:48:52 +01:00
Baptiste Jonglez	26b3295aaa	Add documentation to run Mastodon on Garage	2022-11-06 14:07:31 +01:00
Alex Auvolat	8bc5caf7aa	Fix issue with 'http(s)://' prefix	2022-10-18 21:17:11 +02:00
Alex Auvolat	002b9fc50c	Add TLS support for Consul discovery + refactoring	2022-10-18 18:38:20 +02:00
borgified	8e442001b9	Update 'doc/book/reference-manual/features.md' ... typo	2022-10-16 07:13:21 +00:00
chemicstry	d2c937a931	Fix typo	2022-09-30 18:46:57 +02:00
chemicstry	744c3b4d94	Update docs	2022-09-30 18:46:57 +02:00
chemicstry	37a73d7d37	Move documentation to book	2022-09-30 18:46:57 +02:00
Alex Auvolat	0ab0d3cc29	Document db_engine	2022-09-27 16:52:36 +02:00
Alex	7291747a28	Merge pull request 'Documentation changes for v0.8' (#394) from doc-0.8 into main ... Reviewed-on: #394	2022-09-27 11:37:12 +02:00
Alex Auvolat	194e8be1bb	Update docker image links	2022-09-26 18:01:17 +02:00
Alex Auvolat	69bcc813de	Add garage v0.8 migration guide	2022-09-26 17:46:38 +02:00
Alex Auvolat	a3758dc4c4	Update README	2022-09-21 12:53:02 +02:00
Alex Auvolat	1d0a610690	Finish writing about Garage features, and fix from-source instructions	2022-09-15 13:23:57 +02:00
Alex Auvolat	f6aebefcc9	Some work on documentation towards v0.8	2022-09-14 19:31:13 +02:00
Alex Auvolat	bbb970965c	Document available build features	2022-09-06 17:16:45 +02:00
Quentin Dufour	532eca7ff9	Add some documentation for Caddy	2022-08-12 10:33:41 +02:00
Alex	b74b533b7b	Fix typo	2022-06-29 11:50:51 +02:00
Alex	77e3fd6db2	improve internal item counter mechanisms and implement bucket quotas (#326) ... - [x] Refactoring of internal counting API - [x] Repair procedure for counters (it's an offline procedure!!!) - [x] New counter for objects in buckets - [x] Add quotas to buckets struct - [x] Add CLI to manage bucket quotas - [x] Add admin API to manage bucket quotas - [x] Apply quotas by adding checks on put operations - [x] Proof-read Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #326 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-15 20:20:28 +02:00
Simon C	7eed3ceda9	docs: Add Trafik reverse proxy documentation	2022-06-07 16:16:52 +02:00
Simon C	4b8f48f3c5	docs: Fix title level	2022-06-07 13:32:52 +02:00
Simon C	7d3b5585f1	docs: Add link to facilitate navigation in the documentation	2022-06-07 09:38:59 +02:00
Quentin Dufour	a1abed0378	Remove useless MC_REGION env variable	2022-06-02 12:50:11 +02:00
Alex	93eab8eaa3	Fixes to S3 compatibility page (#314) ... Mention PostObject is implemented, fix english mistakes Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #314 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-25 16:54:44 +02:00
Quentin Dufour	43ddc933f9	Update Ceph S3 endpoints compatibility	2022-05-25 15:20:08 +02:00
Alex Auvolat	9f303f6308	Shorter page title	2022-05-24 15:47:42 +02:00
Alex Auvolat	3be43f3372	Add lost content for Restic with Garage ... Suggested-by: Quentin <quentin@deuxfleurs.fr>	2022-05-24 15:32:42 +02:00
Alex Auvolat	2da448b43f	Add documentation for new Admin API and a few infos on K2V	2022-05-24 15:28:37 +02:00
Baptiste Jonglez	47e57518ec	Add documentation on running Kopia with Garage	2022-04-10 13:04:07 +02:00
Quentin Dufour	8e3ee82c3e	Be clearer on what upgrades are (not) supported	2022-04-06 21:45:59 +02:00
Quentin Dufour	a122a8cb46	Add an "upgrading" section, add a guide for 0.7	2022-04-05 10:08:31 +02:00
Quentin Dufour	9fd8ec1dee	Add documentation for winscp+sftpgo	2022-03-31 10:25:56 +02:00
Alex Auvolat	0091002ef2	New replication modes and their documentation	2022-03-28 16:26:04 +02:00
Alex Auvolat	dca2ffdf91	document administrative options	2022-03-28 12:26:08 +02:00
Alex Auvolat	822128e3c8	Talk a bit about capacity balancing between regions	2022-03-22 12:07:13 +01:00
Rune Henriksen	aea8b41728	document request routing logic	2022-03-21 12:03:57 +01:00
Rune Henriksen	71e6645e09	add short tutorial for duplicati usage with garage	2022-03-21 11:58:19 +01:00
Steam	15da2156f6	Change position of the node-id argument	2022-03-19 18:03:23 +01:00
Max Audron	9d44127245	add support for kubernetes service discovery ... This commit adds support to discover garage instances running in kubernetes. Once enabled by setting `kubernetes_namespace` and `kubernetes_service_name` garage will create a Custom Resources `garagenodes.deuxfleurs.fr` with nodes public key as the resource name. and IP and Port information as spec in the namespace configured by `kubernetes_namespace`. For discovering nodes the resources are filtered with the optionally set `kubernetes_service_name` which sets a label `garage.deuxfleurs.fr/service` on the resources. This allows to separate multiple garage deployments in a single namespace. the `kubernetes_skip_crd` variable allows to disable the creation of the CRD by garage itself. The user must deploy this manually.	2022-03-12 13:05:52 +01:00
Alex Auvolat	2465163e39	documentation: add mention to install build-essential	2022-02-16 12:18:24 +01:00
Alex	4ae03aa774	Small documentation updates (#237) ... Fixes #234, among other things Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #237 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-02-10 15:58:09 +01:00
Quentin	3e1373fafc	Add a new S3 comparison Matrix to documentation (#220) ... Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #220 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-02-07 16:04:52 +01:00
Alex Auvolat	7d68b7060e	Fix anchors in links	2022-02-07 16:01:48 +01:00
Quentin Dufour	99ed67503c	Update quickstart to set endpoints ending in localhost	2022-02-07 16:01:48 +01:00
Quentin Dufour	5a1fb7cce7	Improve integration part of the doc	2022-02-07 16:01:45 +01:00
Alex	1c0ba930b8	Reorganize documentation for new website (#213) ... This PR should be merged after the new website is deployed. - [x] Rename files - [x] Add front matter section to all `.md` files in the book (necessary for Zola) - [x] Change all internal links to use Zola's linking system that checks broken links - [x] Some updates to documentation contents and organization Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #213 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-02-07 11:51:12 +01:00
Alex Auvolat	2760f1cb17	Add advice about --fast-list	2022-01-31 16:51:39 +01:00
Alex Auvolat	26849ed066	Add step to 0.6.0 migration guide	2022-01-27 14:31:25 +01:00
Alex Auvolat	ea7fb901eb	Implement {Put,Get,Delete}BucketCors and CORS in general ... - OPTIONS request against API endpoint - Returning corresponding CORS headers on API calls - Returning corresponding CORS headers on website GET's	2022-01-24 11:58:00 +01:00
Quentin Dufour	94f0e7c135	Test ListParts endpoint with awscli	2022-01-21 10:42:35 +01:00
Alex Auvolat	60c0033c8b	Update documentation	2022-01-13 14:25:22 +01:00
Alex Auvolat	6617a72220	Implement UploadPartCopy	2022-01-13 13:58:47 +01:00
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
Alex Auvolat	9cb2e9e57c	Add documentation to migrate to 0.6	2022-01-10 14:42:58 +01:00
Alex Auvolat	8f39360f22	Update documentation	2022-01-07 17:14:37 +01:00
trinity-1686a	945b75dbf1	update s3 compatibility list (#177) ... Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #177 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 15:05:54 +01:00
trinity-1686a	1eb972b1ac	Add compression using zstd (#173) ... fix #27 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #173 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 11:26:43 +01:00
Quentin Dufour	3b3a1f275f	Add a second plot	2021-12-13 11:58:03 +01:00
Quentin Dufour	dba9af2968	Update benchmark	2021-12-09 18:42:45 +01:00
Quentin Dufour	e9358054ac	Typos and dead links	2021-12-08 14:40:14 +01:00
Quentin Dufour	f9e5520ffb	Add a benchmark page with a first benchmark	2021-12-08 11:30:07 +01:00
Alex Auvolat	4b369347c0	S3 compatibility target	2021-12-06 17:18:45 +01:00
Alex Auvolat	224c89ad6e	Reorganize and improve documentation	2021-12-06 16:33:01 +01:00
Quentin Dufour	7c2037ba87	WIP front page garage	2021-12-06 15:33:43 +01:00
Alex Auvolat	c94406f428	Improve how node roles are assigned in Garage ... - change the terminology: the network configuration becomes the role table, the configuration of a nodes becomes a node's role - the modification of the role table takes place in two steps: first, changes are staged in a CRDT data structure. Then, once the user is happy with the changes, they can commit them all at once (or revert them). - update documentation - fix tests - implement smarter partition assignation algorithm This patch breaks the format of the network configuration: when migrating, the cluster will be in a state where no roles are assigned. All roles must be re-assigned and commited at once. This migration should not pose an issue.	2021-11-16 16:05:53 +01:00
Trinity Pointard	53888995bd	update doc and comments	2021-11-16 15:41:41 +01:00
Trinity Pointard	57df9c6e2d	add s3_api.root_domain to doc book	2021-11-16 15:41:41 +01:00
adrien	cdeb5b4dbb	added link to RFID Garage talk (#155) ... Co-authored-by: ADRN <adrien@luxeylab.net> Reviewed-on: #155 Co-authored-by: adrien <adrien@luxeylab.net> Co-committed-by: adrien <adrien@luxeylab.net>	2021-11-16 15:39:08 +01:00
Quentin Dufour	100aad8bf4	Add rclone mount doc	2021-11-10 18:38:17 +01:00
Quentin Dufour	80a87929b0	Improve CLI documentation	2021-11-10 18:18:34 +01:00
Quentin Dufour	76d21be1b9	Add skeleton for backups, fuse and code sections	2021-11-10 18:05:07 +01:00
Quentin Dufour	1928f59d54	Add documentation for Gitea.	2021-11-10 12:41:09 +01:00
Quentin Dufour	323514be15	Documentation for Nix binary cache	2021-11-10 10:02:22 +01:00
Quentin Dufour	ad8d5139cf	hugo deploy does not build website, fix doc	2021-11-10 10:02:22 +01:00
Alex Auvolat	08b1e8a7ea	Move design draft to separate file; write about GC in internals	2021-11-09 12:25:33 +01:00
Quentin Dufour	8e25a37f0e	Add documentation for nginx	2021-11-08 12:20:40 +01:00
Quentin Dufour	e342db19aa	Add documentation about Gateways	2021-11-08 12:20:40 +01:00
Quentin Dufour	f3405b6378	Doc about exposing your website	2021-11-08 12:20:40 +01:00
Quentin Dufour	860ccf2811	Harden Garage's systemd service	2021-11-08 12:20:40 +01:00
Quentin Dufour	9df7559446	Documentation for hugo, jekyll and publii	2021-11-08 12:20:40 +01:00
Quentin Dufour	a97467075d	Add documentation for synapse-s3-storage-provider	2021-11-08 12:20:40 +01:00
Trinity Pointard	da6efb4b23	fix missing bootstrap_peers in doc	2021-11-05 11:21:50 +01:00
Quentin Dufour	e93d7fb228	Add Peertube + improve CLI instructions	2021-11-03 14:39:14 +01:00
Quentin Dufour	eaf54efb25	Add doc for Nextcloud	2021-11-03 14:07:55 +01:00
Quentin Dufour	93f8d59e4c	Extract toolchain build from the CI	2021-10-29 11:34:01 +02:00
ADRN	4e8af1d956	Modified the 'Funding' sentence to remove 'promise' since we actually got the first instalment	2021-10-26 13:34:28 +02:00
Alex Auvolat	3e7f766d95	CLI: default rpc_host	2021-10-26 11:36:30 +02:00
Alex Auvolat	b2c51844a1	Add download link on homepage	2021-10-25 15:55:30 +02:00
Alex Auvolat	de4276202a	Improve CLI, adapt tests, update documentation	2021-10-25 14:21:48 +02:00
Quentin Dufour	dc017a0cab	Build Garage with Nix	2021-10-19 16:56:07 +02:00
Quentin	c5574c8409	Add links and put logos in a flexbox	2021-09-28 10:21:10 +02:00
mricher	de0228ca2a	Doc: add funding disclaimer for NGI/EU grant ... PNG logs optimized, render to be checked by @quentin. Fix #106.	2021-09-25 17:21:07 +02:00
Alex Auvolat	42f692b1e0	Documentation fixes (typo and small reorganization)	2021-05-31 23:55:51 +02:00
Alex Auvolat	14fd3df654	Write about S3 client configuration	2021-05-31 17:41:21 +02:00