1159 commits

Author	SHA1	Message	Date
Alex Auvolat	99ed18350f	block manager: refactor and fix monitoring/statistics	2023-09-07 12:41:36 +02:00
Alex Auvolat	f38a31b330	block manager: avoid incorrect data_dir configs and avoid losing files	2023-09-06 17:49:30 +02:00
Alex Auvolat	e30865984a	block manager: scrub checkpointing	2023-09-06 16:35:28 +02:00
Alex Auvolat	55c514999e	block manager: fixes in layout	2023-09-06 16:35:28 +02:00
Alex Auvolat	a44f486931	block manager: refactoring & increase max worker count to 8	2023-09-06 16:35:28 +02:00
Alex Auvolat	3a74844df0	block manager: fix dir_not_empty	2023-09-06 16:35:28 +02:00
Alex Auvolat	93114a9747	block manager: refactoring	2023-09-06 16:35:28 +02:00
Alex Auvolat	fd00a47ddc	table queue: increase batch size	2023-09-06 16:35:28 +02:00
Alex Auvolat	1b8c265c14	block manager: get rid of check_block_status	2023-09-06 16:35:28 +02:00
Alex Auvolat	a09f86729c	block manager: move blocks in write_block if necessary	2023-09-06 16:35:28 +02:00
Alex Auvolat	887b3233f4	block manager: use data paths from layout	2023-09-06 16:35:28 +02:00
Alex Auvolat	6c420c0880	block manager: multi-directory layout computation	2023-09-06 16:35:28 +02:00
Alex Auvolat	71c0188055	block manager: skeleton for multi-hdd support	2023-09-06 16:35:28 +02:00
Alex Auvolat	4b4f2000f4	lifecycle: fix SkipBucket bug	2023-09-06 16:34:07 +02:00
Alex Auvolat	51eac97260	update version to 0.8.4	2023-09-05 23:28:12 +02:00
Alex Auvolat	e78566591b	Revert netapp update, hold to version 0.5.2 that uses rmp-serde 0.15	2023-09-05 23:23:23 +02:00
Alex Auvolat	8e0c020bb9	lifecycle worker: correct small clippy lints	2023-09-04 11:33:44 +02:00
Alex Auvolat	1cdc321e28	lifecycle worker: don't get stuck on non-existent bucket	2023-08-31 11:36:30 +02:00
Alex Auvolat	f579d6d9b4	lifecycle worker: fix potential inifinite loop	2023-08-31 11:29:54 +02:00
Alex Auvolat	a00a52633f	lifecycle worker: add log message when starting	2023-08-31 11:25:14 +02:00
Alex Auvolat	adbf5925de	lifecycle worker: use queue_insert and process objects in batches	2023-08-31 11:19:26 +02:00
Alex Auvolat	1cfcc61de8	lifecycle worker: mitigate potential bugs + refactoring	2023-08-31 00:28:37 +02:00
Alex Auvolat	be03a4610f	s3api: remove redundant serde rename attribute	2023-08-31 00:00:26 +02:00
Alex Auvolat	b2f679675e	lifecycle worker: take into account disabled rules	2023-08-30 23:52:09 +02:00
Alex Auvolat	01c327a07a	lifecycle worker: avoid building chrono's serde feature	2023-08-30 23:46:15 +02:00
Alex Auvolat	f0a395e2e5	s3 bucket apis: remove redundant call	2023-08-30 23:39:28 +02:00
Alex Auvolat	a1d57283c0	bucket_table: bucketparams::new doesn't need to be pub	2023-08-30 20:07:14 +02:00
Alex Auvolat	d2e94e36d6	lifecycle config: add missing line in merge() and remove tracing	2023-08-30 20:05:53 +02:00
Alex Auvolat	75ccc5a95c	lifecycle config: store date as given, try to debug	2023-08-30 20:02:07 +02:00
Alex Auvolat	7200954318	lifecycle worker: add logging	2023-08-30 14:54:52 +02:00
Alex Auvolat	0f1849e1ac	lifecycle worker: launch with the rest of Garage	2023-08-30 14:51:08 +02:00
Alex Auvolat	da8b224e24	lifecycle worker: skip entire bucket when no lifecycle config is set	2023-08-30 14:38:19 +02:00
Alex Auvolat	2996dc875f	lifecycle worker: implement main functionality	2023-08-30 14:29:03 +02:00
Alex Auvolat	a2e0e34db5	lifecycle: skeleton for lifecycle worker	2023-08-30 12:41:11 +02:00
Alex Auvolat	f7b409f114	use a NaiveDate in data model, it serializes to string (iso 8601 format)	2023-08-30 11:24:01 +02:00
Alex Auvolat	abf011c290	lifecycle: implement validation into garage's internal data structure	2023-08-29 18:22:03 +02:00
Alex Auvolat	8041d9a827	s3: add xml structures to serialize/deserialize lifecycle configs	2023-08-29 17:44:17 +02:00
Alex Auvolat	0b83e0558e	bucket_table: data model for lifecycle configuration	2023-08-29 17:00:41 +02:00
Alex Auvolat	2e90e1c124	Merge branch 'main' into next	2023-08-29 11:32:42 +02:00
Alex Auvolat	06369c8f4a	add garage_db dependency in garage_rpc	2023-08-28 17:08:21 +02:00
Alex Auvolat	cece1be1bb	bump version to 0.8.3	2023-08-28 13:17:26 +02:00
Alex Auvolat	769b6fe054	fix test_website_check_domain	2023-08-28 12:40:28 +02:00
Alex Auvolat	e66c78d6ea	integration test: move json_body to root of crate	2023-08-28 12:32:57 +02:00
Alex Auvolat	9b4ce4a8ad	admin api: refactor caddy check api code	2023-08-28 12:17:10 +02:00
Alex	2bbe2da5ad	Merge pull request 'support index on path missing a trailing slash' (#612) from compat/index-without-trailing-slash into main ... Reviewed-on: #612	2023-08-28 10:15:01 +00:00
Alex Auvolat	c5cafa0000	web_server.rs: handle error properly and refactor	2023-08-28 12:05:14 +02:00
Jonathan Davies	d66d81ae2d	cargo: Updated gethostname v0.2.3 -> v0.4.3.	2023-08-28 09:30:27 +00:00
Jonathan Davies	7d8296ec59	cargo: Updated pretty_env_logger v0.4.0 -> v0.5.0.	2023-08-28 09:30:27 +00:00
Jonathan Davies	f607ac6792	garage/api: cargo: Updated idna dependency to 0.4.	2023-08-28 09:30:27 +00:00
Jonathan Davies	96d1d81ab7	garage/db: cargo: Updated rusqlite to 0.29.	2023-08-28 09:30:27 +00:00
Jonathan Davies	5185701aa8	cargo: Updated: ... * addr2line v0.19.0 -> v0.20.0 * async-compression v0.4.0 -> v0.4.1 * clap v4.3.8 -> v4.3.19 * hyper v0.14.26 -> v0.14.27 * ipnet v2.7.2 -> v2.8.0 * rmp v0.8.11 -> v0.8.12 * serde v1.0.164 -> v1.0.188 * tokio v1.29.0 -> v1.31.0 * zstd v0.12.3+zstd.1.5.2 -> v0.12.4 * Others in `cargo update`	2023-08-28 09:30:27 +00:00
Quentin Dufour	63da1d2443	support index on path missing a trailing slash	2023-08-08 15:28:57 +02:00
Quentin Dufour	24e533f262	support {s3,web}.root_domains in /check endpoint	2023-08-08 11:05:42 +02:00
Jonathan Davies	59bfc68f2e	post_object.rs: Fixed typos / grammar.	2023-08-01 15:31:39 +01:00
Florian Klink	c855284760	src/util: fix typo	2023-07-14 14:25:40 +03:00
Florian Klink	b1ca1784a1	src/garage/cli: fix typo	2023-07-14 14:25:33 +03:00
Jonathan Davies	aee0d97f22	cargo: Updated async-compression to 0.4.	2023-06-28 11:17:16 +01:00
Alex	e716320b0a	Merge pull request 'cargo: roxmltree-0.18 and aws-sdk-s3-0.28 bump' (#591) from jpds/garage:roxmltree-0.18 into main ... Reviewed-on: #591	2023-06-27 17:20:58 +00:00
Alex	e466edbaec	Merge pull request 'introduce dedicated return type for PollRange' (#590) from trinity-1686a/garage:k2v-client-poll-range-result into main ... Reviewed-on: #590	2023-06-27 08:28:26 +00:00
Jonathan Davies	a6da7e588f	tests/bucket.rs: Adjusted as previously used function is now private.	2023-06-26 18:03:43 +01:00
trinity-1686a	e5835704b7	don't build sqlite by default ... `bundled-libs` is enabled by default, and causes sqlite to be built too, even if the sqlite backend isn't enabled.	2023-06-26 11:15:11 +02:00
Jonathan Davies	7f8bf2d801	src/garage/tests: Updated types for aws-sdk-s3 bump.	2023-06-25 21:31:35 +01:00
Jonathan Davies	4297233d3e	garage/Cargo.toml: Updated aws-sdk-s3 to 0.28, added aws-config.	2023-06-25 21:17:15 +01:00
Jonathan Davies	b94ba47f29	api/Cargo.toml: Updated roxmltree to 0.18.	2023-06-24 14:15:26 +01:00
trinity-1686a	33b3cf8e22	introduce dedicated return type for PollRange	2023-06-24 10:17:20 +02:00
Alex Auvolat	8ef42c9609	admin docs: reformatting, key admin: add check	2023-06-14 17:19:25 +02:00
Alex Auvolat	a83a092c03	admin: uniformize layout api and improve code	2023-06-14 17:12:37 +02:00
Alex Auvolat	7895f99d3a	admin and cli: hide secret keys unless asked	2023-06-14 16:56:15 +02:00
Alex Auvolat	4a82f6380e	admin api: move all endpoints to v1/ by default (v0/ still supported)	2023-06-14 14:15:51 +02:00
Alex Auvolat	28cc9f178a	admin api: make name optionnal for CreateKey	2023-06-14 13:56:37 +02:00
Alex Auvolat	35c108b85d	admin api: switch GetClusterHealth to camelcase (fix #381 again)	2023-06-14 13:53:19 +02:00
Alex Auvolat	52376d47ca	admin api: change cluster status/layout to use lists and not maps (fix #377)	2023-06-14 13:45:27 +02:00
Alex Auvolat	5670367126	multipartupload in test: add forgotten timestamp	2023-06-13 23:10:46 +02:00
Alex Auvolat	cda957b4b1	update netapp's rmp-serde dependency to v1.1	2023-06-13 17:34:49 +02:00
Alex Auvolat	90b2d43eb4	Merge branch 'main' into next	2023-06-13 17:14:11 +02:00
Alex Auvolat	eb9cecf05c	Split garage/admin.rs into smaller files	2023-06-13 16:46:28 +02:00
Alex Auvolat	802ed75721	move admin.rs to admin/mod.rs, before splitting	2023-06-13 16:42:14 +02:00
Alex Auvolat	bf19a44fd9	admin API: add missing camelCase conversions (fix #381)	2023-06-13 16:15:50 +02:00
Alex Auvolat	7126f3e1d1	garage key import: add checks and --yes CLI flag (fix #278)	2023-06-13 15:56:48 +02:00
Alex Auvolat	942c1f1bfe	multipart uploads: save timestamp	2023-06-13 10:48:22 +02:00
Alex Auvolat	1ea4937c8b	fix timestamps wrapping around in garage block list-errors (fix #584)	2023-06-12 20:07:33 +02:00
Alex Auvolat	3d477906d4	properly delete multipart uploads after completion	2023-06-09 17:13:27 +02:00
Alex Auvolat	a6cc563bdd	UploadPart: automatic cleanup of version (and reference blocked) when interrupted	2023-06-09 16:23:37 +02:00
Alex Auvolat	c14d3735e5	Add test for multipart uploads and fix part renumbering	2023-06-09 16:23:37 +02:00
Alex Auvolat	53bf2f070c	undo sort_key() returning Cow	2023-06-09 16:23:37 +02:00
Alex Auvolat	412ab77b08	comments and clippy lint fixes	2023-06-09 16:23:37 +02:00
Alex Auvolat	511e07ecd4	fix mpu counter (add missing workers) and report info at appropriate places	2023-06-09 16:23:37 +02:00
Alex Auvolat	4ea53dc759	Add multipart upload repair	2023-06-09 16:23:37 +02:00
Alex Auvolat	058518c22b	refactor repair workers with a trait	2023-06-09 16:23:37 +02:00
Alex Auvolat	8644376ac2	fix test; simplify code	2023-06-09 16:23:37 +02:00
Alex Auvolat	7ad7dae5d4	fix s3 list test	2023-06-09 16:23:37 +02:00
Alex Auvolat	75a0e01372	fix online repair	2023-06-09 16:23:37 +02:00
Alex Auvolat	bb176ebcb8	cargo fmt	2023-06-09 16:23:37 +02:00
Alex Auvolat	87be8eeb93	updaet block admin for new multipartupload models	2023-06-09 16:23:37 +02:00
Alex Auvolat	82e75c0e29	Adapt S3 API code to use new multipart upload models ... - Create and PutPart - completemultipartupload - upload part copy - list_parts	2023-06-09 16:23:37 +02:00
Alex Auvolat	38d6ac4295	New multipart upload table layout	2023-06-09 16:23:37 +02:00
Alex Auvolat	6005491cd8	Use Cow<[u8]> for sort keys	2023-06-09 16:23:37 +02:00
Alex Auvolat	e7e164a280	Make fsync an option for meta and data	2023-06-09 16:23:21 +02:00
Alex Auvolat	1e466b11eb	Revert integration tests to using Sled as LMDB causes failures	2023-06-09 13:23:08 +02:00
Jonathan Davies	8a945ee996	payload.rs: Surround / in inverted commas.	2023-06-06 16:26:06 +01:00
Jonathan Davies	180992d0f1	payload.rs: Fixed typo in error message.	2023-06-06 16:25:29 +01:00
Alex Auvolat	8a74e1c2bd	Split garage/admin.rs into smaller files	2023-06-06 15:39:15 +02:00
Roberto Hidalgo	ef8a7add08	set default for [consul-services] api	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	b770504126	simplify code according to feedback	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	6b69404f1a	rename mode to consul_http_api	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	011f473048	revert rpc/Cargo.toml	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	fd7dbea5b8	follow feedback, fold into existing feature	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	bd6485565e	allow additional ServiceMeta, docs	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	4d6e6fc155	cargo fmt	2023-05-22 08:57:15 -06:00
Roberto Hidalgo	02ba9016ab	register consul services against local agent instead of catalog api	2023-05-22 08:57:15 -06:00
Alex Auvolat	c3d3b837eb	bump k2v-client to v0.0.4	2023-05-22 10:47:15 +02:00
Alex Auvolat	130e01505b	Fix k2v_client with unicode in partition keys	2023-05-22 10:45:09 +02:00
Alex Auvolat	e2ce5970c6	Add basic k2v_client integration tests	2023-05-22 10:45:06 +02:00
Alex Auvolat	644e872264	Port k2v-client to aws-sigv4 since rusoto_signature is deprecated	2023-05-19 12:08:29 +02:00
Alex Auvolat	4420db7310	add tracing to k2v-client	2023-05-18 11:18:21 +02:00
Alex Auvolat	746b0090e4	k2v signature verification: double urlencoding (see comment in source code)	2023-05-18 11:18:06 +02:00
Alex Auvolat	19639705e6	Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaults	2023-05-17 14:30:53 +02:00
Alex Auvolat	217d429937	fix clippy lint in format-table crate	2023-05-17 13:06:37 +02:00
Alex Auvolat	a1cec2cd60	Split format_table into separate crate and reduce k2v-client dependencies	2023-05-17 13:01:37 +02:00
Alex Auvolat	16f2a32bb7	cargo fmt	2023-05-16 19:46:57 +02:00
Alex Auvolat	472444ed8e	k2v-client 0.0.2	2023-05-16 19:46:57 +02:00
Alex Auvolat	bb03805b58	k2v-cli: fix sort_key being partition_key and specify which key	2023-05-16 19:46:57 +02:00
Alex Auvolat	e4f955d672	fix base64 uses	2023-05-16 19:46:56 +02:00
Alex	ea9b15f669	Merge pull request 'cargo: tokio-1.28 and hyper-0.14.26 update' (#569) from jpds/garage:tokio-1.28 into main ... Reviewed-on: #569	2023-05-11 10:16:33 +00:00
Jonathan Davies	c783194e8b	*: apply clippy recommendations.	2023-05-09 20:49:34 +01:00
Jonathan Davies	0f0795103d	block/Cargo.toml: Bump tokio-util to 0.7.	2023-05-09 14:33:21 +01:00
Jonathan Davies	c9d26e8c50	k2v-client/Cargo.toml: Make tokio dep match other packages.	2023-05-09 14:33:00 +01:00
Alex Auvolat	351d734e6c	Merge branch 'main' into next	2023-05-09 12:40:08 +02:00
Alex	b925f53dc3	Merge pull request 'move git-version dependency to main crate to reduce rebuilds' (#568) from move-git-version into main ... Reviewed-on: #568	2023-05-09 09:53:33 +00:00
Alex	2f495575d8	Merge pull request 'block/manager.rs: Prioritize raw blocks when no compression configured' (#566) from jpds/garage:skip-compressed-blocks-scrub-no-compression into main ... Reviewed-on: #566	2023-05-09 09:39:48 +00:00
Alex Auvolat	9e0a9c1c15	move git-version dependency to main crate to reduce rebuilds	2023-05-09 11:35:32 +02:00
Jonathan Davies	9c788059e2	block/manager.rs: In is_block_compressed - check which compression_level ... is configured on a node and check for raw block first if compression is disabled (to help reduce syscalls during a scrub).	2023-05-09 10:28:19 +01:00
Alex	5684e1990c	Merge pull request 'Really allow to disable sled feature' (#563) from jirutka/garage:workspace-deps into main ... Reviewed-on: #563	2023-05-09 09:08:35 +00:00
Alex	14c50f2f84	Merge pull request 'Fix undefined macro warn! on 32-bit' (#562) from jirutka/garage:fix-undefined-warn into main ... Reviewed-on: #562	2023-05-09 08:52:11 +00:00
Jakub Jirutka	d2deee0b8b	Declare garage crates using workspace.dependencies ... This will allow to really disable "sled" feature without declaring `default-features = false` in every Cargo.toml where garage_db and garage_model is used. See https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table	2023-05-09 08:46:15 +00:00
Jakub Jirutka	6d3ace1ea9	Fix undefined macro warn! on 32-bit ... Compiling garage_db v0.8.2 (garage-0.8.2/src/db) error: cannot find macro `warn` in this scope --> src/db/lmdb_adapter.rs:352:2 | 352 | warn!("LMDB is not recommended on 32-bit systems, database size will be limited"); | ^^^^ | = help: consider importing this macro: tracing::warn = note: `warn` is in scope, but it is an attribute: `#[warn]` error: could not compile `garage_db` due to previous error	2023-05-07 17:01:44 +02:00
Jakub Jirutka	833cf082da	Remove unnecessary/unused "timeago" features ... To decrease dependency bloat and binary size.	2023-05-07 01:03:54 +02:00
Alex Auvolat	fa78d806e3	Merge branch 'main' into next	2023-04-25 12:34:26 +02:00
Alex	a16eb7e4b8	Merge pull request 'api/Cargo.toml: Bumped quick-xml to version 0.26.' (#552) from jpds/garage:quick-xml-0.26 into main ... Reviewed-on: #552	2023-04-24 09:00:00 +00:00
Jonathan Davies	fb3bd11dce	block/repair.rs: Added log entries for scrub start/finish.	2023-04-23 22:22:26 +01:00
Jonathan Davies	c168383113	api/Cargo.toml: Bumped quick-xml to version 0.26.	2023-04-23 20:14:28 +01:00
Alex Auvolat	000006d689	obsolete clippy lints	2023-03-13 18:50:07 +01:00
Alex Auvolat	0a1ddcf630	Prepare for v0.8.2	2023-03-13 18:46:31 +01:00
Alex	d6ffa57f40	Merge pull request 'Increase Garage tests robustness' (#526) from tests/increase-robustness into main ... Reviewed-on: #526 Reviewed-by: Alex <alex@adnab.me> Reviewed-by: trinity-1686a <trinity.pointard@gmail.com>	2023-03-13 17:26:21 +00:00
Alex Auvolat	f37ec584b6	Merge branch 'main' into rustversion-label	2023-03-13 16:14:13 +01:00
Quentin Dufour	70b5424b99	use one key per context to isolate tests	2023-03-13 15:06:05 +01:00
Quentin Dufour	2687fb7fa8	do not assume Garage boots in 2sec during tests	2023-03-13 15:06:05 +01:00
teutat3s	8ad6efb338	Merge branch 'main' into pnet_datalink-0.33.0	2023-03-13 13:59:42 +01:00
Jonathan Davies	9ea154ae9c	admin/cluster.rs: Added rust_version.	2023-03-10 14:46:54 +00:00
Jonathan Davies	4421378023	garage/admin.rs: Display Rust version in stats output.	2023-03-10 14:46:54 +00:00
Jonathan Davies	25f2a46fc3	rpc/system_metrics.rs: Added rustversion label to garage_build_info metric.	2023-03-10 14:46:44 +00:00
Jonathan Davies	d218f475cb	block/manager.rs: Set defaults for scrub_persister.	2023-03-09 17:08:47 +00:00
Jonathan Davies	7b65dd24e2	block/repair.rs: Added a timestamp argument to ... randomize_next_scrub_run_time().	2023-03-09 16:38:41 +00:00
Jonathan Davies	b70cc0a940	block/repair.rs: Added migration for ScrubWorkerPersisted's time_next_run_scrub. ... Fixes: #520.	2023-03-09 16:38:36 +00:00
Alex	2dc80abbb1	Merge pull request 'block/repair.rs: Added a random element of 10 days to SCRUB_INTERVAL' (#516) from jpds/garage:scrub-randomize-window into main ... Reviewed-on: #516	2023-03-06 14:11:25 +00:00
Jonathan Davies	148b66b843	block/manager.rs: Display scrub-next-run.	2023-03-06 13:43:09 +00:00
Jonathan Davies	53d09eb00f	block/repair.rs: Added function and time_next_run_scrub with a random element of ... 10 days to SCRUB_INTERVAL to help balance scrub load across cluster.	2023-03-06 13:43:04 +00:00
Alex	00dcfc97a5	Merge pull request 'web_server.rs: Log X-Forwarded-For IP' (#504) from jpds/garage:web_server-log-x-forwarded-for into main ... Reviewed-on: #504	2023-03-06 12:33:06 +00:00
Jonathan Davies	4e0fc3d6c9	web/web_server.rs: Handle X-Forwarded-For here too.	2023-03-06 11:43:54 +00:00
Jonathan Davies	e4e5196066	api/generic_server.rs: Use new handle_forwarded_for_headers() function.	2023-03-06 11:43:35 +00:00
Alex Auvolat	b8123fb6cd	Clearer error message when LMDB has oom error (fix #517)	2023-03-06 11:38:49 +01:00
Jonathan Davies	ff70e09aa0	util/forwarded_headers.rs: Generalized handle_forwarded_for_headers() ... here.	2023-03-03 19:17:40 +00:00
teutat3s	f0717dd169	Bump pnet_datalink 0.28 -> 0.33 ... Motivation: building garage on illumos is only possible since pnet_datalink version 0.30 Changelog: https://github.com/libpnet/libpnet/compare/v0.28.0...v0.33.0	2023-02-28 16:06:43 +01:00
Alex	8b4d0adc75	Merge pull request 'generic_server.rs: Added support for logging X-Forwarded-For header.' (#500) from jpds/garage:generic_server-log-x-forwarded-for into main ... Reviewed-on: #500	2023-02-06 14:20:12 +00:00
Alex	d14678e0ac	Merge pull request 'Secrets can be passed directly in config, as file, or as env' (#499) from config-files-env into main ... Reviewed-on: #499	2023-02-06 14:18:58 +00:00
Alex Auvolat	80e2326998	fixes for pr 499	2023-02-06 12:23:55 +01:00
Jonathan Davies	94d70bec69	generic_server.rs: Added support for logging X-Forwarded-For header. ... Fixes: #460	2023-02-04 15:19:21 +00:00
Alex Auvolat	656b8d42de	secrets can be passed directly in config, as file, or as env	2023-02-03 15:27:39 +01:00
Jonathan Davies	1b6ec74748	error.rs: Corrected error messages to say unexpected scope.	2023-02-02 16:20:31 +00:00
Alex	f9573b6912	Merge pull request 'Fix duplicated content-type in error document' (#493) from baptiste/garage:fix_error_document_content_type into main ... Reviewed-on: #493	2023-01-30 10:56:35 +00:00
Baptiste Jonglez	e85a200189	Fix duplicated content-type in error document ... Fixes #492	2023-01-29 22:51:23 +01:00
Jonathan Davies	9c354f0a8f	Improved bucket authorization response strings.	2023-01-29 20:34:41 +00:00
Jonathan Davies	004bb5b4f1	api_server.rs: Adapted to use query string per Caddy upstream change.	2023-01-29 20:34:37 +00:00
Alex Auvolat	d6af95d205	fix cli display bug	2023-01-26 17:50:50 +01:00
Alex Auvolat	c56794655e	Fix fmt	2023-01-26 17:27:03 +01:00
Alex Auvolat	8e93d69974	More clippy fixes	2023-01-26 17:26:32 +01:00
Alex Auvolat	3113f6b5f2	more fixes	2023-01-26 17:14:17 +01:00
Alex Auvolat	1dff62564f	fix clippy	2023-01-26 17:05:31 +01:00
Alex Auvolat	590a0a8450	Merge branch 'main' into k2v-watch-range-2	2023-01-26 16:46:40 +01:00
Alex Auvolat	94d559ae00	Merge branch 'main' into report-disk-usage	2023-01-26 16:20:41 +01:00
Alex Auvolat	c7d0ad0aa0	Add local disk usage to exported prometheus metrics	2023-01-26 15:30:36 +01:00
Alex Auvolat	efb6b6e868	Disk space report ... Report available disk space on nodes and calculate cluster-wide available space in `garage stats` (fix #479)	2023-01-26 15:04:32 +01:00
Jonathan Davies	3dc655095f	db/Cargo.toml: Updated rusqlite from 0.27 to 0.28.	2023-01-26 11:13:11 +00:00
Jonathan Davies	20c1cdf662	Cargo.toml: Loosen tracing dependency to just 0.1.	2023-01-26 11:13:11 +00:00
Jonathan Davies	f952e37ba7	{model,util}/Cargo.toml: Updated blake2 from 0.9 to 0.10.	2023-01-26 11:13:11 +00:00
Jonathan Davies	fbafa76284	{db,util}/Cargo.toml: Updated mktemp from 0.4 to 0.5.	2023-01-26 11:13:11 +00:00
Jonathan Davies	63e22e71f2	api/Cargo.toml: Updated idna from 0.2 to 0.3.	2023-01-26 11:13:11 +00:00
Jonathan Davies	f6eaf3661c	garage/Cargo.toml: Updated timeage from 0.3 to 0.4.	2023-01-26 11:13:11 +00:00
Jonathan Davies	d3b2a68988	{garage,util}/Cargo.toml: Updated toml from 0.5 to 0.6.	2023-01-26 11:13:11 +00:00
Jonathan Davies	b4a1a6a32f	util/time.rs: Updated deprecated associated function to timestamp_opt().	2023-01-26 11:13:11 +00:00
Jonathan Davies	bcac889f9a	Cargo.toml: Updated clap from 3.1.18 to 4.1.	2023-01-26 11:13:11 +00:00
Jonathan Davies	9e08a05e69	k2v-client/Cargo.toml: Loosen dependencies.	2023-01-26 11:13:11 +00:00
Jonathan Davies	36944f1839	Cargo.toml: Updated base64 from 0.13 to 0.21.	2023-01-26 11:13:07 +00:00
Jonathan Davies	db56d4658f	util/Cargo.toml: Updated rmp-serde from 0.15 to 1.1.	2023-01-26 11:03:43 +00:00
Jonathan Davies	93c3f8fc8c	api/Cargo.toml: Updated url from 2.1 to 2.3.	2023-01-23 19:16:58 +00:00
Jonathan Davies	1c435fce09	api/Cargo.toml: Updated httpdate from 0.3 to 1.0.	2023-01-23 19:16:54 +00:00
Jonathan Davies	dead123892	api/Cargo.toml: Updated pin-project to 1.0.12.	2023-01-23 18:39:35 +00:00
Jonathan Davies	5c3075fe01	Cargo.toml: Updated zstd from 0.9 to 0.12.	2023-01-23 18:08:14 +00:00
Jonathan Davies	4962b88f8b	tests/s3/website.rs: Added website hosting authorization check tests.	2023-01-13 09:39:02 +00:00
Jonathan Davies	100b01e859	Implemented website hosting authorization endpoint. ... Fixes: #468	2023-01-13 09:38:58 +00:00
Alex Auvolat	84b4a868e3	Migration of cluster layout from v0.8 to v0.9	2023-01-11 17:47:46 +01:00
Alex Auvolat	dac254a6e7	Merge branch 'main' into k2v-watch-range-2	2023-01-11 17:09:37 +01:00
Alex	94d723f27c	Merge pull request 'Implement rpc_secret_file' (#466) from felix.scheinost/garage:feature/implement-rpc-secret-file into main ... Reviewed-on: #466	2023-01-11 16:04:35 +00:00
Alex Auvolat	638c5a3ce0	PollRange: add extra RPC delay after quorum is achieved, ... to give a chance to the 3rd node to respond	2023-01-11 16:12:07 +01:00
Alex Auvolat	5b5ca63cf6	Poll cleanup	2023-01-11 15:17:27 +01:00
Alex Auvolat	cbfae673e8	PollRange & PollItem: min timeout = 1 sec	2023-01-11 15:03:08 +01:00
Alex Auvolat	bba13f40fc	Correctly return bad requests when seeh marker is invalid	2023-01-11 12:27:19 +01:00
Alex Auvolat	ba384e61c0	PollRange: return immediately if no seen marker is provided	2023-01-11 12:03:17 +01:00
Alex Auvolat	09a3dad0f2	Lock once for insert_many	2023-01-11 11:35:36 +01:00
Alex Auvolat	32aab06929	k2v-client libary poll_range and CLI poll-range	2023-01-11 11:14:29 +01:00
Alex Auvolat	de1111076b	PollRange integration test	2023-01-11 10:04:41 +01:00
Alex Auvolat	b83517d521	Implement PollRange API endpoint	2023-01-10 15:22:25 +01:00
Alex Auvolat	43fd6c1526	PollRange RPC	2023-01-10 12:54:24 +01:00
Alex Auvolat	789540ca37	Type definition for range seen marker	2023-01-10 11:59:57 +01:00
Jonathan Davies	4cfb469d2b	block/metrics.rs: Added compression_level metric.	2023-01-10 10:40:03 +00:00
Jonathan Davies	df1d9a9873	system.rs: Integrated SystemMetrics into System implementation.	2023-01-10 10:39:50 +00:00
Jonathan Davies	aac348fe93	Added system_metrics.rs file.	2023-01-10 10:38:50 +00:00
Alex Auvolat	9f5419f465	Make K2V item timestamps globally increasing on each node	2023-01-10 11:03:52 +01:00
Alex Auvolat	a48e2e0cb2	K2V: Subscription to ranges of items	2023-01-10 10:30:59 +01:00
Felix Scheinost	d6ea0cbefa	Add tests for rpc_secret_file	2023-01-07 14:19:36 +01:00
Felix Scheinost	7b62fe3f0b	Error on both rpc_secret and rpc_secret_file	2023-01-07 13:49:03 +01:00
Jonathan Davies	cb07e6145c	Changed all instances of assignation to assignment.	2023-01-05 11:09:25 +00:00
Felix Scheinost	f2106c2733	Implement rpc_secret_file	2023-01-04 18:35:10 +01:00
Alex	02e8eb167e	Merge pull request 'PutObject: better cleanup when request is interrupted in the middle' (#462) from interrupted-cleanup into main ... Reviewed-on: #462	2023-01-04 14:43:45 +00:00
Alex Auvolat	29dbcb8278	bg var operation on all nodes at once	2023-01-04 13:25:57 +01:00
Alex Auvolat	f3f27293df	Uniform framework for bg variable management	2023-01-04 13:07:13 +01:00
Alex Auvolat	13c5549886	Remove token_bucket.rs	2023-01-04 11:47:56 +01:00
Alex	80e4abb98d	Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next ... Reviewed-on: #459	2023-01-04 10:35:49 +00:00
Alex Auvolat	570e5e5bbb	Merge branch 'main' into next	2023-01-04 11:34:43 +01:00
Alex Auvolat	936b6cb563	When saving block, delete .tmp file if we could not complete	2023-01-03 17:34:26 +01:00
Alex Auvolat	0650a43cf1	PutObject: better cleanup on Drop (incl. when request is interrupted in the middle)	2023-01-03 17:05:17 +01:00
Alex Auvolat	1fc220886a	Fix Consul & Kubernetes discovery with new way of doing background things	2023-01-03 16:55:59 +01:00
Alex Auvolat	1d5bdc17a4	use impossible enum type	2023-01-03 16:04:06 +01:00
Alex Auvolat	c106304b9c	more idiomatic and shorter	2023-01-03 16:00:19 +01:00
Alex Auvolat	33f25d26c7	fix doc and add tests for migrate.rs	2023-01-03 15:53:13 +01:00
Alex Auvolat	d6d571d512	cargo fmt	2023-01-03 15:30:21 +01:00
Alex Auvolat	a54b67740d	move debug_serialize to garage_util::encode	2023-01-03 15:29:29 +01:00
Alex Auvolat	8d5505514f	Make it explicit when using nonversioned encoding	2023-01-03 15:27:36 +01:00
Alex Auvolat	426d8784da	cleanup	2023-01-03 15:08:37 +01:00
Alex Auvolat	cdb2a591e9	Refactor how things are migrated	2023-01-03 14:44:47 +01:00
Jonathan Davies	8be862aa19	Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time.	2023-01-03 11:11:12 +00:00
Alex Auvolat	939a6d67e8	Merge branch 'main' into internals-rework	2023-01-02 15:07:44 +01:00
Alex Auvolat	6775569525	Bump everything to v0.8.1	2023-01-02 14:15:33 +01:00
Alex Auvolat	6b857a9b8c	cargo fmt	2023-01-02 13:50:42 +01:00
Alex	7f7d53cfa9	Merge pull request 'improvements to CLI and new debug features' (#448) from cli-improvements into main ... Reviewed-on: #448	2023-01-02 12:42:24 +00:00
Alex	1af4a5ed56	Merge pull request 'Fix router keyword handling (fix #442)' (#446) from router-keywords-fix into main ... Reviewed-on: #446	2022-12-15 08:40:26 +00:00
Alex Auvolat	1fcd0b371b	online repair workers: retry on error	2022-12-14 16:31:31 +01:00
Alex Auvolat	13c8662126	factorize	2022-12-14 16:16:55 +01:00
Alex Auvolat	e6f14ab5cf	better error message handling	2022-12-14 16:11:19 +01:00
Alex Auvolat	510b620108	Get rid of background::spawn	2022-12-14 16:08:05 +01:00
Alex Auvolat	dfc131850a	Simplified and more aggressive worker exit logic	2022-12-14 15:25:29 +01:00
Alex Auvolat	d4af27f920	Add missing notify	2022-12-14 13:54:21 +01:00
Alex Auvolat	a19bfef508	Improve error message on rpc connection failure	2022-12-14 12:57:33 +01:00
Alex Auvolat	d56c472712	Refactor background runner and get rid of job worker	2022-12-14 12:51:42 +01:00
Alex Auvolat	2183518edc	Spawn all background workers in a separate step	2022-12-14 12:28:07 +01:00
Alex Auvolat	83c8467e23	Proper queueing for delayed inserts, now backed to disk	2022-12-14 11:58:06 +01:00
Alex Auvolat	f8e528c15d	Small refactor of tables internals	2022-12-14 10:48:49 +01:00
Alex Auvolat	d1279e04f3	Fix error messages	2022-12-13 16:18:01 +01:00
Alex Auvolat	041b60ed1d	Add block.rc_size, table.size and table.merkle_tree_size metrics	2022-12-13 15:54:03 +01:00
Alex Auvolat	f8d5409894	cli: more info displayed on error in garage stats	2022-12-13 15:46:04 +01:00
Alex Auvolat	d6040e32a6	cli: prettier table in garage stats	2022-12-13 15:43:22 +01:00
Alex Auvolat	d7f90cabb0	Implement block retry-now and block purge	2022-12-13 15:02:42 +01:00
Alex Auvolat	687660b27f	Implement block list-errors and block info	2022-12-13 14:23:45 +01:00
Alex Auvolat	9d82196945	cli: new worker info command	2022-12-13 12:24:30 +01:00
Alex Auvolat	a51e8d94c6	cli: rename resync-n-workers into resync-worker-count	2022-12-13 11:44:11 +01:00
Alex Auvolat	de9d6cddf7	Prettier worker list table; remove useless CLI log messages	2022-12-12 17:17:05 +01:00
Alex Auvolat	0e61e3b6fb	Fix bucket creation tests to take permissions into account	2022-12-12 15:47:55 +01:00
Alex Auvolat	a0abf41762	Fix router keyword handling (fix #442)	2022-12-12 12:05:37 +01:00
Alex Auvolat	2ac75018a1	Properly enforce allow_create_bucket	2022-12-12 12:03:54 +01:00
Alex	6e44369cbc	Merge pull request 'Optimal layout assignation algorithm' (#296) from optimal-layout into next ... Reviewed-on: #296	2022-12-11 17:41:53 +00:00
Alex Auvolat	2c2e65ad8b	Merge commit 'ec12d6c' into next	2022-12-11 18:41:15 +01:00
Alex Auvolat	9d83364ad9	itertools .unique() doesn't require sorted items	2022-12-11 18:30:02 +01:00
Alex Auvolat	533afcf4e1	simplify	2022-12-11 18:17:08 +01:00
Alex Auvolat	5ea5fd2130	Always return 200 OK on /v0/health, reinstate admin api doc as draft and complete it	2022-12-11 18:11:28 +01:00
Alex Auvolat	d7868c48a4	Separate /health (simple text answer) and /v0/health (full json answer, authenticated)	2022-12-05 15:38:32 +01:00
Alex Auvolat	280d1be7b1	Refactor health check and add ability to return it in json	2022-12-05 15:28:57 +01:00
Alex Auvolat	2065f011ca	Implement /health admin API endpoint to check node health	2022-12-05 14:59:15 +01:00
Alex Auvolat	26121bb619	Fix logs appearing twice	2022-11-29 22:23:27 +01:00
Alex Auvolat	54e800ef8d	Tentative fix for issue #414	2022-11-21 17:13:41 +01:00
Alex Auvolat	c1fb65194c	Add sled default in garage_model also	2022-11-21 14:25:54 +01:00
Alex Auvolat	67941000ee	put sled as default feature in garage_db	2022-11-21 14:08:21 +01:00
Quentin Dufour	dc50fa3b34	Fix typo in admin API on BucketInfo	2022-11-11 16:56:56 +01:00
Alex Auvolat	ec12d6c8dd	Slightly simplify code at places	2022-11-08 16:15:45 +01:00
Alex Auvolat	217abdca18	Fix HTTP return code	2022-11-08 15:38:53 +01:00
Alex Auvolat	fc2729cd81	Fix integration test	2022-11-08 15:19:46 +01:00
Alex Auvolat	d75b37b018	Return more info when layout's .check() fails, fix compilation, fix test	2022-11-08 14:58:39 +01:00
Alex Auvolat	73a4ca8b15	Use bytes as capacity units	2022-11-07 21:12:11 +01:00
Alex Auvolat	fd5bc142b5	Ensure .sort() is called before counting unique items	2022-11-07 20:29:25 +01:00
Alex Auvolat	ea5afc2511	Style improvements	2022-11-07 20:11:30 +01:00
Alex Auvolat	28d7a49f63	Merge branch 'main' into optimal-layout	2022-11-07 12:20:59 +01:00
Alex Auvolat	e03d9062f7	Show a nice message and a backtrace when Garage panics	2022-11-04 16:39:02 +01:00
Alex Auvolat	8d3bbf5703	Clearer error messsages	2022-11-04 16:07:33 +01:00
Alex Auvolat	5b18fd8201	Add garage bucket cleanup-incomplete-uploads command	2022-11-04 11:55:59 +01:00
Alex Auvolat	57b5c2c754	Change reqwest rustls features	2022-10-18 22:11:27 +02:00
Alex Auvolat	8bc5caf7aa	Fix issue with 'http(s)://' prefix	2022-10-18 21:17:11 +02:00
Alex Auvolat	2da8786f54	move things around	2022-10-18 19:13:52 +02:00
Alex Auvolat	5d8d393054	Load TLS certificates only once	2022-10-18 19:11:16 +02:00
Alex Auvolat	002b9fc50c	Add TLS support for Consul discovery + refactoring	2022-10-18 18:38:20 +02:00
Alex	5670599372	Merge pull request 'Use status code 204 No Content for empty responses' (#403) from tobikris/garage:http-no-content into main ... Reviewed-on: #403	2022-10-18 14:20:44 +00:00
Tobias Krischer	7865003323	Use status code 204 No Content for empty responses	2022-10-17 10:55:26 +02:00