1243 commits

Author	SHA1	Message	Date
Alex Auvolat	6d38907dac	test: verify saved checksums in streaming putobject tests	2025-02-18 22:02:04 +01:00
Alex Auvolat	cfe8e8d45c	api: PutObject: save trailer checksum in metadata	2025-02-18 21:56:32 +01:00
Alex Auvolat	f6e805e7db	api: various fixes	2025-02-18 21:47:53 +01:00
Alex Auvolat	45e10e55f9	update aws-sdk-s3 in tests and fix wrong checksumming behavior in GetObject	2025-02-18 15:33:42 +01:00
Alex Auvolat	730bfee753	api: validate trailing checksum + add test for unsigned-paylad-trailer	2025-02-18 15:33:42 +01:00
Alex Auvolat	ccab0e4ae5	api: fix optional \n after trailer checksum header	2025-02-18 15:33:42 +01:00
Alex Auvolat	abb60dcf7e	api: remove content-encoding: aws-chunked for streaming payload	2025-02-18 15:33:42 +01:00
Alex Auvolat	f8b0817ddc	api: streaming signature: fix trailer parsing	2025-02-18 12:00:41 +01:00
Alex Auvolat	21c0dda16a	api: refactor: move checksumming code around again	2025-02-17 20:11:06 +01:00
Alex Auvolat	658541d812	api: use checksumming in api_common::signature for put/putpart	2025-02-17 19:54:25 +01:00
Alex Auvolat	c5df820e2c	api: start refactor of signature to calculate checksums earlier	2025-02-17 18:47:06 +01:00
Alex Auvolat	a04d6cd5b8	api: streaming: parse unsigned streaming bodies and payload trailers	2025-02-17 16:23:24 +01:00
Alex Auvolat	44a896f9b5	api: add logic to parse x-amz-content-sha256	2025-02-16 18:25:35 +01:00
Alex Auvolat	cee7560fc1	api: refactor: move checksum algorithms to common	2025-02-16 17:25:55 +01:00
Alex Auvolat	2f0c5ca220	signature: refactor: move constant defs to mod.rs	2025-02-16 16:34:18 +01:00
Alex	859b38b0d2	Merge pull request 'fix compilation warnings' (#959) from fixes into main ... Reviewed-on: #959	2025-02-14 17:32:30 +00:00
Alex Auvolat	2729a71d9d	fix warning in garage test	2025-02-14 18:27:00 +01:00
Alex Auvolat	c9d00f5f7b	garage_api_s3: remove unused field in ListPartsQuery	2025-02-14 18:25:23 +01:00
Alex Auvolat	24470377c9	garage_model: fix warning about dead code	2025-02-14 18:12:14 +01:00
Alex Auvolat	5b26545abf	fix deprecated uses of chrono in lifecycle worker	2025-02-14 18:08:23 +01:00
Alex Auvolat	165f9316e2	s3api: return Location in CompleteMultipartUpload (fix #852) ... NB. The location returned is not guaranteed to work in all cases. This already fixes the parse issue in #852.	2025-02-14 18:05:07 +01:00
Alex Auvolat	e4c9a8cd53	block manager: avoid deadlock in fix_block_location (fix #845)	2025-02-14 17:41:50 +01:00
Alex Auvolat	fdf4dad728	block resync: avoid saving blocks to draining nodes	2025-02-14 16:45:55 +01:00
Alex Auvolat	6820b69f30	block manager: improve read strategy to find blocks faster	2025-02-14 16:45:55 +01:00
Alex Auvolat	d0104b9f9b	block manager: write blocks only to currently active layout version (fix #815) ... avoid wastefully writing blocks to nodes that will discard them as soon as the layout migration is finished	2025-02-14 16:45:55 +01:00
Alex	3fe8db9e52	Merge pull request 'web_server.rs: Added bucket domain to observability' (#608) from jpds/garage:domain-web-requests into main ... Reviewed-on: #608	2025-02-14 14:26:08 +00:00
Alex	627a37fe9f	Merge pull request 's3 api: parse x-id query parameter and warn of any inconsistency (fix #822)' (#954) from fix-822 into main ... Reviewed-on: #954	2025-02-14 14:07:01 +00:00
Alex Auvolat	2f55889835	add configuration option to enable/disable monitoring bucket in web metrics	2025-02-14 14:59:00 +01:00
Jonathan Davies	8b9cc5ca3f	web_server.rs: Added bucket domain to observability.	2025-02-14 14:36:20 +01:00
Alex Auvolat	c1b39d9ba1	s3 api: parse x-id query parameter and warn of any inconsistency (fix #822)	2025-02-14 14:30:58 +01:00
Alex Auvolat	d84308c413	cli: return info of all nodes when doing garage stats -a (fix #814)	2025-02-14 14:11:41 +01:00
Baptiste Jonglez	a2e134f036	db-snapshot: propagate any node snapshot error through RPC call ... In particular, it means that "garage meta snapshot --all" will get an exit code of 1 if any node fails to snapshot. This makes sure that any external tool trying to snapshot nodes (e.g. from cron) will be aware of the failure. Fix #920	2025-02-07 00:29:43 +01:00
Baptiste Jonglez	06aa4b604f	db-snapshot: Fix error reporting when using "garage meta snapshot --all" ... Snapshot errors on remote nodes were not reported at all. We now get proper error output such as: 0fa0f35be69528ab error: Internal error: DB error: LMDB: No space left on device (os error 28) 88d92e2971d14bae ok Fix #920	2025-02-07 00:18:01 +01:00
Alex Auvolat	af67626ab2	remove async_trait for TableRepair	2025-02-05 20:45:07 +01:00
Alex Auvolat	5475da8ea8	remove async_trait used in generic_server.rs	2025-02-05 20:31:34 +01:00
Alex Auvolat	620dc58560	remove async_trait for traits declared in garage_net	2025-02-05 20:22:16 +01:00
Alex Auvolat	ec1a475923	build with rust 1.82.0	2025-02-03 17:46:48 +01:00
Alex Auvolat	390a5d97fe	nix, ci: build with Crane ... This removes our dependency on cargo2nix, which was causing us some issues. Whereas cargo2nix creates one Nix derivation per crate, Crane uses only two derivations: 1. Build dependencies only 2. Build the final binary This means that during the second step, no caching can be done. For instance, if we do a change in garage_model, we need to recompile all of the Garage crates including those that do not depend on garage_model. On the upside, this allows all of the Garage crates to be built at once using cargo build logic, which is optimized for high parallelism and better pipelining between all of the steps of the build. All in all, this makes most builds faster than cargo2nix. A few other changes have been made to the build scripts and CI: - Unit tests are now run within a Nix derivation. In fact, we have different derivations to run the tests using LMDB and Sqlite as metadata db engines. - For debug builds, most CI steps now run in parallel (with the notable exception of the smoke test that runs after the build, which is inevitable). - We no longer pass the GIT_VERSION argument when building debug builds and running the tests. This means that dev binaries and test binaries don't know the exact version of Garage they are from. That shouldn't be an issue in most cases. - The not-dynamic.sh scripts has been fixed to fail if the file does not exist.	2025-02-03 16:39:50 +01:00
Alex Auvolat	d18c5ad0ff	fix tests	2025-01-31 19:12:51 +01:00
Alex Auvolat	3d5e9a027e	cargo defs: simplify and fix descriptions	2025-01-31 18:54:29 +01:00
Alex Auvolat	4563313f87	use cargo-shear to remove many unused dependencies between crates	2025-01-31 18:47:30 +01:00
Alex Auvolat	afa28706e5	split s3/cors.rs into also common/cors.rs	2025-01-31 18:42:14 +01:00
Alex Auvolat	84f1db91c4	fix things up	2025-01-31 18:34:57 +01:00
Alex Auvolat	9fa20d45be	wip: split garage_api into garage_api_{common,s3,k2v,admin}	2025-01-31 18:18:29 +01:00
Alex Auvolat	83f6928ff7	table::insert_many: avoid failure with zero items (fix #915)	2025-01-30 18:06:47 +01:00
Alex Auvolat	9f3c7c3720	api: better handling of helper errors to distinguish error codes	2025-01-29 19:14:34 +01:00
Alex	a1d081ee84	Merge pull request 's3 api: make x-amz-meta-* headers lowercase (fix #844)' (#938) from fix-844 into main ... Reviewed-on: #938	2025-01-27 19:32:19 +00:00
Alex Auvolat	e8fa89e834	s3 api: make x-amz-meta-* headers lowercase (fix #844)	2025-01-27 19:58:06 +01:00
Baptiste Jonglez	43402c9619	snapshot: sqlite: use a subdirectory for consistency with LMDB ... Currently, taking a snapshot of the metadata database with sqlite creates a sqlite file without extension with the following format: snapshots/2025-01-26T15:29:17Z This makes it hard to understand what kind of data this is, and is not consistent with LMDB: snapshots/2025-01-26T15:29:17Z/data.mdb With this change, we now get a directory with a single db.sqlite file: snapshots/2025-01-26T15:29:17Z/db.sqlite	2025-01-27 19:06:52 +01:00
Baptiste Jonglez	59c153d280	db-snapshot: allow to set directory where snapshots are stored ... Fix #926	2025-01-27 18:33:55 +01:00