782 commits

Author	SHA1	Message	Date
Alex	80e4abb98d	Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next ... Reviewed-on: #459	2023-01-04 10:35:49 +00:00
Alex Auvolat	570e5e5bbb	Merge branch 'main' into next	2023-01-04 11:34:43 +01:00
Alex Auvolat	936b6cb563	When saving block, delete .tmp file if we could not complete	2023-01-03 17:34:26 +01:00
Alex Auvolat	0650a43cf1	PutObject: better cleanup on Drop (incl. when request is interrupted in the middle)	2023-01-03 17:05:17 +01:00
Alex Auvolat	1fc220886a	Fix Consul & Kubernetes discovery with new way of doing background things	2023-01-03 16:55:59 +01:00
Alex Auvolat	1d5bdc17a4	use impossible enum type	2023-01-03 16:04:06 +01:00
Alex Auvolat	c106304b9c	more idiomatic and shorter	2023-01-03 16:00:19 +01:00
Alex Auvolat	33f25d26c7	fix doc and add tests for migrate.rs	2023-01-03 15:53:13 +01:00
Alex Auvolat	d6d571d512	cargo fmt	2023-01-03 15:30:21 +01:00
Alex Auvolat	a54b67740d	move debug_serialize to garage_util::encode	2023-01-03 15:29:29 +01:00
Alex Auvolat	8d5505514f	Make it explicit when using nonversioned encoding	2023-01-03 15:27:36 +01:00
Alex Auvolat	426d8784da	cleanup	2023-01-03 15:08:37 +01:00
Alex Auvolat	cdb2a591e9	Refactor how things are migrated	2023-01-03 14:44:47 +01:00
Jonathan Davies	8be862aa19	Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time.	2023-01-03 11:11:12 +00:00
Alex Auvolat	939a6d67e8	Merge branch 'main' into internals-rework	2023-01-02 15:07:44 +01:00
Alex Auvolat	6775569525	Bump everything to v0.8.1	2023-01-02 14:15:33 +01:00
Alex Auvolat	6b857a9b8c	cargo fmt	2023-01-02 13:50:42 +01:00
Alex	7f7d53cfa9	Merge pull request 'improvements to CLI and new debug features' (#448) from cli-improvements into main ... Reviewed-on: #448	2023-01-02 12:42:24 +00:00
Alex	1af4a5ed56	Merge pull request 'Fix router keyword handling (fix #442)' (#446) from router-keywords-fix into main ... Reviewed-on: #446	2022-12-15 08:40:26 +00:00
Alex Auvolat	1fcd0b371b	online repair workers: retry on error	2022-12-14 16:31:31 +01:00
Alex Auvolat	13c8662126	factorize	2022-12-14 16:16:55 +01:00
Alex Auvolat	e6f14ab5cf	better error message handling	2022-12-14 16:11:19 +01:00
Alex Auvolat	510b620108	Get rid of background::spawn	2022-12-14 16:08:05 +01:00
Alex Auvolat	dfc131850a	Simplified and more aggressive worker exit logic	2022-12-14 15:25:29 +01:00
Alex Auvolat	d4af27f920	Add missing notify	2022-12-14 13:54:21 +01:00
Alex Auvolat	a19bfef508	Improve error message on rpc connection failure	2022-12-14 12:57:33 +01:00
Alex Auvolat	d56c472712	Refactor background runner and get rid of job worker	2022-12-14 12:51:42 +01:00
Alex Auvolat	2183518edc	Spawn all background workers in a separate step	2022-12-14 12:28:07 +01:00
Alex Auvolat	83c8467e23	Proper queueing for delayed inserts, now backed to disk	2022-12-14 11:58:06 +01:00
Alex Auvolat	f8e528c15d	Small refactor of tables internals	2022-12-14 10:48:49 +01:00
Alex Auvolat	d1279e04f3	Fix error messages	2022-12-13 16:18:01 +01:00
Alex Auvolat	041b60ed1d	Add block.rc_size, table.size and table.merkle_tree_size metrics	2022-12-13 15:54:03 +01:00
Alex Auvolat	f8d5409894	cli: more info displayed on error in garage stats	2022-12-13 15:46:04 +01:00
Alex Auvolat	d6040e32a6	cli: prettier table in garage stats	2022-12-13 15:43:22 +01:00
Alex Auvolat	d7f90cabb0	Implement block retry-now and block purge	2022-12-13 15:02:42 +01:00
Alex Auvolat	687660b27f	Implement block list-errors and block info	2022-12-13 14:23:45 +01:00
Alex Auvolat	9d82196945	cli: new worker info command	2022-12-13 12:24:30 +01:00
Alex Auvolat	a51e8d94c6	cli: rename resync-n-workers into resync-worker-count	2022-12-13 11:44:11 +01:00
Alex Auvolat	de9d6cddf7	Prettier worker list table; remove useless CLI log messages	2022-12-12 17:17:05 +01:00
Alex Auvolat	0e61e3b6fb	Fix bucket creation tests to take permissions into account	2022-12-12 15:47:55 +01:00
Alex Auvolat	a0abf41762	Fix router keyword handling (fix #442)	2022-12-12 12:05:37 +01:00
Alex Auvolat	2ac75018a1	Properly enforce allow_create_bucket	2022-12-12 12:03:54 +01:00
Alex	6e44369cbc	Merge pull request 'Optimal layout assignation algorithm' (#296) from optimal-layout into next ... Reviewed-on: #296	2022-12-11 17:41:53 +00:00
Alex Auvolat	2c2e65ad8b	Merge commit 'ec12d6c' into next	2022-12-11 18:41:15 +01:00
Alex Auvolat	9d83364ad9	itertools .unique() doesn't require sorted items	2022-12-11 18:30:02 +01:00
Alex Auvolat	533afcf4e1	simplify	2022-12-11 18:17:08 +01:00
Alex Auvolat	5ea5fd2130	Always return 200 OK on /v0/health, reinstate admin api doc as draft and complete it	2022-12-11 18:11:28 +01:00
Alex Auvolat	d7868c48a4	Separate /health (simple text answer) and /v0/health (full json answer, authenticated)	2022-12-05 15:38:32 +01:00
Alex Auvolat	280d1be7b1	Refactor health check and add ability to return it in json	2022-12-05 15:28:57 +01:00
Alex Auvolat	2065f011ca	Implement /health admin API endpoint to check node health	2022-12-05 14:59:15 +01:00
Alex Auvolat	26121bb619	Fix logs appearing twice	2022-11-29 22:23:27 +01:00
Alex Auvolat	54e800ef8d	Tentative fix for issue #414	2022-11-21 17:13:41 +01:00
Alex Auvolat	c1fb65194c	Add sled default in garage_model also	2022-11-21 14:25:54 +01:00
Alex Auvolat	67941000ee	put sled as default feature in garage_db	2022-11-21 14:08:21 +01:00
Quentin Dufour	dc50fa3b34	Fix typo in admin API on BucketInfo	2022-11-11 16:56:56 +01:00
Alex Auvolat	ec12d6c8dd	Slightly simplify code at places	2022-11-08 16:15:45 +01:00
Alex Auvolat	217abdca18	Fix HTTP return code	2022-11-08 15:38:53 +01:00
Alex Auvolat	fc2729cd81	Fix integration test	2022-11-08 15:19:46 +01:00
Alex Auvolat	d75b37b018	Return more info when layout's .check() fails, fix compilation, fix test	2022-11-08 14:58:39 +01:00
Alex Auvolat	73a4ca8b15	Use bytes as capacity units	2022-11-07 21:12:11 +01:00
Alex Auvolat	fd5bc142b5	Ensure .sort() is called before counting unique items	2022-11-07 20:29:25 +01:00
Alex Auvolat	ea5afc2511	Style improvements	2022-11-07 20:11:30 +01:00
Alex Auvolat	28d7a49f63	Merge branch 'main' into optimal-layout	2022-11-07 12:20:59 +01:00
Alex Auvolat	e03d9062f7	Show a nice message and a backtrace when Garage panics	2022-11-04 16:39:02 +01:00
Alex Auvolat	8d3bbf5703	Clearer error messsages	2022-11-04 16:07:33 +01:00
Alex Auvolat	5b18fd8201	Add garage bucket cleanup-incomplete-uploads command	2022-11-04 11:55:59 +01:00
Alex Auvolat	57b5c2c754	Change reqwest rustls features	2022-10-18 22:11:27 +02:00
Alex Auvolat	8bc5caf7aa	Fix issue with 'http(s)://' prefix	2022-10-18 21:17:11 +02:00
Alex Auvolat	2da8786f54	move things around	2022-10-18 19:13:52 +02:00
Alex Auvolat	5d8d393054	Load TLS certificates only once	2022-10-18 19:11:16 +02:00
Alex Auvolat	002b9fc50c	Add TLS support for Consul discovery + refactoring	2022-10-18 18:38:20 +02:00
Alex	5670599372	Merge pull request 'Use status code 204 No Content for empty responses' (#403) from tobikris/garage:http-no-content into main ... Reviewed-on: #403	2022-10-18 14:20:44 +00:00
Tobias Krischer	7865003323	Use status code 204 No Content for empty responses	2022-10-17 10:55:26 +02:00
Alex Auvolat	c050a59fd0	Fix conditional testing in garage_db	2022-10-14 18:27:18 +02:00
Alex Auvolat	fcaee3bea0	definitively expunge openssl from dependencies everywhere	2022-10-14 18:10:36 +02:00
Alex Auvolat	8d04ae7014	cargo2nix unstable (patched), rust 1.63.0, nixpkgs 22.05 (32-bit builds are broken)	2022-10-14 14:30:48 +02:00
Mendes	bcdd1e0c33	Added some comment	2022-10-11 18:29:21 +02:00
Mendes	e5664c9822	Improved the statistics displayed in layout show ... corrected a few bugs	2022-10-11 17:17:13 +02:00
Mendes	4abab246f1	cargo fmt	2022-10-10 17:21:13 +02:00
Mendes	fcf9ac674a	Tests written in layout.rs ... added staged_parameters to ClusterLayout removed the serde(default) -> will need a migration function	2022-10-10 17:19:25 +02:00
Mendes	911eb17bd9	corrected warnings of cargo clippy	2022-10-06 14:53:57 +02:00
Mendes	9407df60cc	Corrected two bugs: ... - self.node_id_vec was not properly updated when the previous ring was empty - ClusterLayout::merge was not considering changes in the layout parameters	2022-10-06 12:54:51 +02:00
Mendes	a951b6c452	Added a CLI command to update the parameters for the layout computation (for now, only the zone redundancy)	2022-10-05 16:04:19 +02:00
Mendes	ceac3713d6	modifications in several files to : ... - have consistent error return types - store the zone redundancy in a Lww - print the error and message in the CLI (TODO: for the server Api, should msg be returned in the body response?)	2022-10-05 15:29:48 +02:00
Mendes	829f815a89	Merge remote-tracking branch 'origin/main' into optimal-layout	2022-10-04 18:14:49 +02:00
Mendes	99f96b9564	deleted zone_redundancy from System struct	2022-10-04 18:09:24 +02:00
Alex Auvolat	ad917ffd3f	Fix instant substractions that might have panicked	2022-09-29 15:53:54 +02:00
Alex Auvolat	1f97ce37e6	Shutdown properly on SIGTERM/SIGHUP and on Windows signals	2022-09-28 10:41:59 +02:00
Alex Auvolat	1778e4b318	Fix span name for api server requests	2022-09-26 16:21:30 +02:00
Mendes	bd842e1388	Correction of a few bugs in the tests, modification of ClusterLayout::check	2022-09-22 19:30:01 +02:00
Mendes	7f3249a237	New version of the algorithm that calculate the layout. ... It takes as paramters the replication factor and the zone redundancy, computes the largest partition size reachable with these constraints, and among the possible assignation with this partition size, it computes the one that moves the least number of partitions compared to the previous assignation. This computation uses graph algorithms defined in graph_algo.rs	2022-09-21 14:39:59 +02:00
Alex Auvolat	782630fc27	Initialize metrics exporter earlier (fix #389)	2022-09-20 17:50:22 +02:00
Alex Auvolat	ded444f6c9	Ability to have custom timeouts in request strategy (not used)	2022-09-20 16:01:41 +02:00
Alex Auvolat	357b72f4ff	Merge branch 'main' into configurable-timeouts	2022-09-20 15:19:58 +02:00
Alex Auvolat	1f7b050b7d	Change a warn! into a debug!	2022-09-20 11:49:48 +02:00
Alex Auvolat	56592e1853	RPC performance changes ... - configurable ping timeout - single, much higher, configurable RPC timeout - no more concurrency semaphore	2022-09-19 20:31:00 +02:00
Alex Auvolat	5d4b6f2173	Faster GetObject workflow for getting entire objects	2022-09-19 12:19:59 +02:00
Alex Auvolat	76f42a1a2b	Properly return HTTP 204 when deleting non-existent object (fix #227)	2022-09-14 17:07:55 +02:00
Alex Auvolat	e46dc2a8ef	Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353)	2022-09-14 16:09:38 +02:00
Alex Auvolat	ab722cb40f	Add checks on replication_factor of layouts we use (fix #363, fix #364)	2022-09-13 16:22:23 +02:00
Alex Auvolat	38be811b1c	Fix clippy lint that says we should implement Eq	2022-09-13 16:08:00 +02:00
Alex Auvolat	44733474bb	Remove/change println! in server code (fix #358)	2022-09-13 16:01:55 +02:00
Alex Auvolat	07febd3ecd	Ensure data dir is created immediately when Garage starts (fix #349)	2022-09-13 15:57:27 +02:00
Alex	11bdc971e2	Merge pull request 'use netapp streaming body' (#343) from netapp-stream-body into main ... Reviewed-on: #343	2022-09-13 15:26:08 +02:00
Alex Auvolat	ff30891999	Use streaming block API for get with Range requests	2022-09-13 15:13:07 +02:00
Alex Auvolat	28a4af73ca	Use netapp 0.5 published from crates.io	2022-09-13 13:11:44 +02:00
Alex Auvolat	b823151a0b	improvements in block manager	2022-09-12 16:57:38 +02:00
Alex Auvolat	f91fab8582	Simplify+improve async hasher by using bounded channel	2022-09-12 16:23:43 +02:00
Alex Auvolat	7f54706b95	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-08 15:50:56 +02:00
Alex Auvolat	d9d199a6c9	Merge branch 'main' into lx-perf-improvements	2022-09-08 15:49:17 +02:00
Alex Auvolat	ceb1f0229a	Move version back into util	2022-09-07 18:36:46 +02:00
Alex Auvolat	f310fce34b	Inject GIT_VERSION even later	2022-09-07 18:30:15 +02:00
Alex Auvolat	06df301de5	Fix merge	2022-09-07 18:16:01 +02:00
Alex Auvolat	8adc654713	Merge branch 'main' into improve-deps	2022-09-07 18:13:27 +02:00
Alex Auvolat	107853334b	Fix build error	2022-09-07 18:10:19 +02:00
Alex Auvolat	1449204439	Add warnings when features are not included in build	2022-09-07 18:02:13 +02:00
Alex Auvolat	2e00809af5	Error messages when system-libs XOR bundled-libs != 1	2022-09-07 17:57:12 +02:00
Alex Auvolat	2559f63e9b	Make all HTTP services optionnal	2022-09-07 17:54:16 +02:00
Alex Auvolat	28d86e7602	Report build features in garage --help	2022-09-07 17:05:21 +02:00
Alex Auvolat	db61f41030	Move GIT_VERSION injection later in build chain to reduce build times	2022-09-07 11:59:56 +02:00
Alex Auvolat	907054775d	Faster copy, better get error message	2022-09-06 22:25:23 +02:00
Alex Auvolat	6b958979bd	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-06 22:13:01 +02:00
Alex Auvolat	c2cc08852b	Reenable node ordering	2022-09-06 19:31:42 +02:00
Alex Auvolat	6f02c36a89	cargo fmt	2022-09-06 17:59:41 +02:00
Alex Auvolat	0f5689c169	Include code from v0.5.1 directly to remove dependencies	2022-09-06 17:52:50 +02:00
Alex Auvolat	1e92e9f782	Disable k2v tests when feature is disabled	2022-09-06 17:29:46 +02:00
Alex Auvolat	431dee050f	Remove opentelemetry-otlp dep in api/	2022-09-06 17:25:44 +02:00
Alex Auvolat	b886c75450	Make all DB engines optional build features	2022-09-06 17:09:43 +02:00
Alex Auvolat	48ffaaadfc	Bump versions to 0.8.0 (compatibility is broken already)	2022-09-06 16:47:56 +02:00
Alex Auvolat	454d8474ef	Fix clippy	2022-09-06 15:43:50 +02:00
Jakub Jirutka	ea36b9ff90	Allow building without Prometheus exporter (/metrics endpoint) ... prometheus and opentelemetry-prometheus add 7 extra dependencies in total and increases the size of the garage binary by ~7 % (with fat LTO).	2022-09-06 01:15:09 +02:00
Jakub Jirutka	e7af006c1c	Make OTLP exporter optional via feature "telemetry-otlp" ... opentelemetry-otlp add 48 (!) extra dependencies and increases the size of the garage binary by ~11 % (with fat LTO).	2022-09-06 01:14:47 +02:00
Alex Auvolat	729a910e14	Remove Heed default features	2022-09-05 16:40:13 +02:00
Alex Auvolat	07e6bcde85	Merge branch 'main' into lx-perf-improvements	2022-09-05 12:40:17 +02:00
Jakub Jirutka	7511ba5530	Allow linking against system-provided libsqlite ... Unfortunately, rusqlite uses the opposite logic for enabling/disabling bundled libraries to others (libsodium-sys, zstd-sys). Cargo features are very limited and doesn't allow to enable feature A in a dependency iff feature B is disabled. Note, lmdb-rkv-sys doesn't need any special treatment because it automatically links against system liblmdb if found via pkgconf. Linux distros should build garage with `--no-default-features --features system-libs` to disable bundled-libs and enable system-libs.	2022-09-03 19:15:57 +02:00
Jakub Jirutka	a6e40b75ea	Add feature "system-libs" to enable linking against system libraries ... If this feature is enabled, libsodium-sys and zstd-sys will link dynamically against system-provided libraries instead of building and linking statically the bundled (possibly outdated and vulnerable) copies of them. This feature is intended mainly for linux package maintainers.	2022-09-03 18:44:34 +02:00
Alex Auvolat	e1751c8a9c	fix clippy	2022-09-02 17:24:26 +02:00
Alex Auvolat	5d4b937a00	Ability to have up to 4 concurrently working resync workers	2022-09-02 17:18:13 +02:00
Alex Auvolat	5e8baa433d	Make BlockManagerLocked fully private again	2022-09-02 16:52:22 +02:00
Alex Auvolat	47be652a1f	block manager: refactor: split resync into separate file	2022-09-02 16:47:15 +02:00
Alex Auvolat	943d76c583	Ability to dynamically set resync tranquility	2022-09-02 15:34:21 +02:00
Alex Auvolat	6226f5ceca	Update to netapp 0.4.5 - fixed ping	2022-09-02 14:33:12 +02:00
Alex Auvolat	13b5f28c7e	Make use of BytesBuf from new Netapp	2022-09-02 13:46:42 +02:00
Alex Auvolat	1ef87ac4cb	cargo fmt	2022-09-02 13:38:29 +02:00
Alex Auvolat	99b532b85b	Apply PRIO_SECONDARY to block data transfers	2022-09-01 16:35:43 +02:00
Alex Auvolat	df094bd807	Less strict timeouts	2022-09-01 16:30:44 +02:00
Alex Auvolat	bc977f9a7a	Update to Netapp with OrderTag support and exploit OrderTags	2022-09-01 12:58:20 +02:00
Alex Auvolat	70231d68b2	Fix bytes_read counter	2022-08-31 19:44:27 +02:00
Alex Auvolat	c9bc9d89de	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-08-31 17:42:31 +02:00
Alex Auvolat	efbca67ce4	Add env filter to tracing subscriber	2022-08-31 14:39:12 +02:00