1043 commits

Author	SHA1	Message	Date
Jonathan Davies	aac348fe93	Added system_metrics.rs file.	2023-01-10 10:38:50 +00:00
Alex Auvolat	9f5419f465	Make K2V item timestamps globally increasing on each node	2023-01-10 11:03:52 +01:00
Alex Auvolat	a48e2e0cb2	K2V: Subscription to ranges of items	2023-01-10 10:30:59 +01:00
Felix Scheinost	d6ea0cbefa	Add tests for rpc_secret_file	2023-01-07 14:19:36 +01:00
Felix Scheinost	7b62fe3f0b	Error on both rpc_secret and rpc_secret_file	2023-01-07 13:49:03 +01:00
Jonathan Davies	cb07e6145c	Changed all instances of assignation to assignment.	2023-01-05 11:09:25 +00:00
Felix Scheinost	f2106c2733	Implement rpc_secret_file	2023-01-04 18:35:10 +01:00
Alex	02e8eb167e	Merge pull request 'PutObject: better cleanup when request is interrupted in the middle' (#462) from interrupted-cleanup into main ... Reviewed-on: #462	2023-01-04 14:43:45 +00:00
Alex Auvolat	29dbcb8278	bg var operation on all nodes at once	2023-01-04 13:25:57 +01:00
Alex Auvolat	f3f27293df	Uniform framework for bg variable management	2023-01-04 13:07:13 +01:00
Alex Auvolat	13c5549886	Remove token_bucket.rs	2023-01-04 11:47:56 +01:00
Alex	80e4abb98d	Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next ... Reviewed-on: #459	2023-01-04 10:35:49 +00:00
Alex Auvolat	570e5e5bbb	Merge branch 'main' into next	2023-01-04 11:34:43 +01:00
Alex Auvolat	936b6cb563	When saving block, delete .tmp file if we could not complete	2023-01-03 17:34:26 +01:00
Alex Auvolat	0650a43cf1	PutObject: better cleanup on Drop (incl. when request is interrupted in the middle)	2023-01-03 17:05:17 +01:00
Alex Auvolat	1fc220886a	Fix Consul & Kubernetes discovery with new way of doing background things	2023-01-03 16:55:59 +01:00
Alex Auvolat	1d5bdc17a4	use impossible enum type	2023-01-03 16:04:06 +01:00
Alex Auvolat	c106304b9c	more idiomatic and shorter	2023-01-03 16:00:19 +01:00
Alex Auvolat	33f25d26c7	fix doc and add tests for migrate.rs	2023-01-03 15:53:13 +01:00
Alex Auvolat	d6d571d512	cargo fmt	2023-01-03 15:30:21 +01:00
Alex Auvolat	a54b67740d	move debug_serialize to garage_util::encode	2023-01-03 15:29:29 +01:00
Alex Auvolat	8d5505514f	Make it explicit when using nonversioned encoding	2023-01-03 15:27:36 +01:00
Alex Auvolat	426d8784da	cleanup	2023-01-03 15:08:37 +01:00
Alex Auvolat	cdb2a591e9	Refactor how things are migrated	2023-01-03 14:44:47 +01:00
Jonathan Davies	8be862aa19	Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time.	2023-01-03 11:11:12 +00:00
Alex Auvolat	939a6d67e8	Merge branch 'main' into internals-rework	2023-01-02 15:07:44 +01:00
Alex Auvolat	6775569525	Bump everything to v0.8.1	2023-01-02 14:15:33 +01:00
Alex Auvolat	6b857a9b8c	cargo fmt	2023-01-02 13:50:42 +01:00
Alex	7f7d53cfa9	Merge pull request 'improvements to CLI and new debug features' (#448) from cli-improvements into main ... Reviewed-on: #448	2023-01-02 12:42:24 +00:00
Alex	1af4a5ed56	Merge pull request 'Fix router keyword handling (fix #442)' (#446) from router-keywords-fix into main ... Reviewed-on: #446	2022-12-15 08:40:26 +00:00
Alex Auvolat	1fcd0b371b	online repair workers: retry on error	2022-12-14 16:31:31 +01:00
Alex Auvolat	13c8662126	factorize	2022-12-14 16:16:55 +01:00
Alex Auvolat	e6f14ab5cf	better error message handling	2022-12-14 16:11:19 +01:00
Alex Auvolat	510b620108	Get rid of background::spawn	2022-12-14 16:08:05 +01:00
Alex Auvolat	dfc131850a	Simplified and more aggressive worker exit logic	2022-12-14 15:25:29 +01:00
Alex Auvolat	d4af27f920	Add missing notify	2022-12-14 13:54:21 +01:00
Alex Auvolat	a19bfef508	Improve error message on rpc connection failure	2022-12-14 12:57:33 +01:00
Alex Auvolat	d56c472712	Refactor background runner and get rid of job worker	2022-12-14 12:51:42 +01:00
Alex Auvolat	2183518edc	Spawn all background workers in a separate step	2022-12-14 12:28:07 +01:00
Alex Auvolat	83c8467e23	Proper queueing for delayed inserts, now backed to disk	2022-12-14 11:58:06 +01:00
Alex Auvolat	f8e528c15d	Small refactor of tables internals	2022-12-14 10:48:49 +01:00
Alex Auvolat	d1279e04f3	Fix error messages	2022-12-13 16:18:01 +01:00
Alex Auvolat	041b60ed1d	Add block.rc_size, table.size and table.merkle_tree_size metrics	2022-12-13 15:54:03 +01:00
Alex Auvolat	f8d5409894	cli: more info displayed on error in garage stats	2022-12-13 15:46:04 +01:00
Alex Auvolat	d6040e32a6	cli: prettier table in garage stats	2022-12-13 15:43:22 +01:00
Alex Auvolat	d7f90cabb0	Implement block retry-now and block purge	2022-12-13 15:02:42 +01:00
Alex Auvolat	687660b27f	Implement block list-errors and block info	2022-12-13 14:23:45 +01:00
Alex Auvolat	9d82196945	cli: new worker info command	2022-12-13 12:24:30 +01:00
Alex Auvolat	a51e8d94c6	cli: rename resync-n-workers into resync-worker-count	2022-12-13 11:44:11 +01:00
Alex Auvolat	de9d6cddf7	Prettier worker list table; remove useless CLI log messages	2022-12-12 17:17:05 +01:00
Alex Auvolat	0e61e3b6fb	Fix bucket creation tests to take permissions into account	2022-12-12 15:47:55 +01:00
Alex Auvolat	a0abf41762	Fix router keyword handling (fix #442)	2022-12-12 12:05:37 +01:00
Alex Auvolat	2ac75018a1	Properly enforce allow_create_bucket	2022-12-12 12:03:54 +01:00
Alex	6e44369cbc	Merge pull request 'Optimal layout assignation algorithm' (#296) from optimal-layout into next ... Reviewed-on: #296	2022-12-11 17:41:53 +00:00
Alex Auvolat	2c2e65ad8b	Merge commit 'ec12d6c' into next	2022-12-11 18:41:15 +01:00
Alex Auvolat	9d83364ad9	itertools .unique() doesn't require sorted items	2022-12-11 18:30:02 +01:00
Alex Auvolat	533afcf4e1	simplify	2022-12-11 18:17:08 +01:00
Alex Auvolat	5ea5fd2130	Always return 200 OK on /v0/health, reinstate admin api doc as draft and complete it	2022-12-11 18:11:28 +01:00
Alex Auvolat	d7868c48a4	Separate /health (simple text answer) and /v0/health (full json answer, authenticated)	2022-12-05 15:38:32 +01:00
Alex Auvolat	280d1be7b1	Refactor health check and add ability to return it in json	2022-12-05 15:28:57 +01:00
Alex Auvolat	2065f011ca	Implement /health admin API endpoint to check node health	2022-12-05 14:59:15 +01:00
Alex Auvolat	26121bb619	Fix logs appearing twice	2022-11-29 22:23:27 +01:00
Alex Auvolat	54e800ef8d	Tentative fix for issue #414	2022-11-21 17:13:41 +01:00
Alex Auvolat	c1fb65194c	Add sled default in garage_model also	2022-11-21 14:25:54 +01:00
Alex Auvolat	67941000ee	put sled as default feature in garage_db	2022-11-21 14:08:21 +01:00
Quentin Dufour	dc50fa3b34	Fix typo in admin API on BucketInfo	2022-11-11 16:56:56 +01:00
Alex Auvolat	ec12d6c8dd	Slightly simplify code at places	2022-11-08 16:15:45 +01:00
Alex Auvolat	217abdca18	Fix HTTP return code	2022-11-08 15:38:53 +01:00
Alex Auvolat	fc2729cd81	Fix integration test	2022-11-08 15:19:46 +01:00
Alex Auvolat	d75b37b018	Return more info when layout's .check() fails, fix compilation, fix test	2022-11-08 14:58:39 +01:00
Alex Auvolat	73a4ca8b15	Use bytes as capacity units	2022-11-07 21:12:11 +01:00
Alex Auvolat	fd5bc142b5	Ensure .sort() is called before counting unique items	2022-11-07 20:29:25 +01:00
Alex Auvolat	ea5afc2511	Style improvements	2022-11-07 20:11:30 +01:00
Alex Auvolat	28d7a49f63	Merge branch 'main' into optimal-layout	2022-11-07 12:20:59 +01:00
Alex Auvolat	e03d9062f7	Show a nice message and a backtrace when Garage panics	2022-11-04 16:39:02 +01:00
Alex Auvolat	8d3bbf5703	Clearer error messsages	2022-11-04 16:07:33 +01:00
Alex Auvolat	5b18fd8201	Add garage bucket cleanup-incomplete-uploads command	2022-11-04 11:55:59 +01:00
Alex Auvolat	57b5c2c754	Change reqwest rustls features	2022-10-18 22:11:27 +02:00
Alex Auvolat	8bc5caf7aa	Fix issue with 'http(s)://' prefix	2022-10-18 21:17:11 +02:00
Alex Auvolat	2da8786f54	move things around	2022-10-18 19:13:52 +02:00
Alex Auvolat	5d8d393054	Load TLS certificates only once	2022-10-18 19:11:16 +02:00
Alex Auvolat	002b9fc50c	Add TLS support for Consul discovery + refactoring	2022-10-18 18:38:20 +02:00
Alex	5670599372	Merge pull request 'Use status code 204 No Content for empty responses' (#403) from tobikris/garage:http-no-content into main ... Reviewed-on: #403	2022-10-18 14:20:44 +00:00
Tobias Krischer	7865003323	Use status code 204 No Content for empty responses	2022-10-17 10:55:26 +02:00
Alex Auvolat	c050a59fd0	Fix conditional testing in garage_db	2022-10-14 18:27:18 +02:00
Alex Auvolat	fcaee3bea0	definitively expunge openssl from dependencies everywhere	2022-10-14 18:10:36 +02:00
Alex Auvolat	8d04ae7014	cargo2nix unstable (patched), rust 1.63.0, nixpkgs 22.05 (32-bit builds are broken)	2022-10-14 14:30:48 +02:00
Mendes	bcdd1e0c33	Added some comment	2022-10-11 18:29:21 +02:00
Mendes	e5664c9822	Improved the statistics displayed in layout show ... corrected a few bugs	2022-10-11 17:17:13 +02:00
Mendes	4abab246f1	cargo fmt	2022-10-10 17:21:13 +02:00
Mendes	fcf9ac674a	Tests written in layout.rs ... added staged_parameters to ClusterLayout removed the serde(default) -> will need a migration function	2022-10-10 17:19:25 +02:00
Mendes	911eb17bd9	corrected warnings of cargo clippy	2022-10-06 14:53:57 +02:00
Mendes	9407df60cc	Corrected two bugs: ... - self.node_id_vec was not properly updated when the previous ring was empty - ClusterLayout::merge was not considering changes in the layout parameters	2022-10-06 12:54:51 +02:00
Mendes	a951b6c452	Added a CLI command to update the parameters for the layout computation (for now, only the zone redundancy)	2022-10-05 16:04:19 +02:00
Mendes	ceac3713d6	modifications in several files to : ... - have consistent error return types - store the zone redundancy in a Lww - print the error and message in the CLI (TODO: for the server Api, should msg be returned in the body response?)	2022-10-05 15:29:48 +02:00
Mendes	829f815a89	Merge remote-tracking branch 'origin/main' into optimal-layout	2022-10-04 18:14:49 +02:00
Mendes	99f96b9564	deleted zone_redundancy from System struct	2022-10-04 18:09:24 +02:00
Alex Auvolat	ad917ffd3f	Fix instant substractions that might have panicked	2022-09-29 15:53:54 +02:00
Alex Auvolat	1f97ce37e6	Shutdown properly on SIGTERM/SIGHUP and on Windows signals	2022-09-28 10:41:59 +02:00
Alex Auvolat	1778e4b318	Fix span name for api server requests	2022-09-26 16:21:30 +02:00
Mendes	bd842e1388	Correction of a few bugs in the tests, modification of ClusterLayout::check	2022-09-22 19:30:01 +02:00
Mendes	7f3249a237	New version of the algorithm that calculate the layout. ... It takes as paramters the replication factor and the zone redundancy, computes the largest partition size reachable with these constraints, and among the possible assignation with this partition size, it computes the one that moves the least number of partitions compared to the previous assignation. This computation uses graph algorithms defined in graph_algo.rs	2022-09-21 14:39:59 +02:00
Alex Auvolat	782630fc27	Initialize metrics exporter earlier (fix #389)	2022-09-20 17:50:22 +02:00
Alex Auvolat	ded444f6c9	Ability to have custom timeouts in request strategy (not used)	2022-09-20 16:01:41 +02:00
Alex Auvolat	357b72f4ff	Merge branch 'main' into configurable-timeouts	2022-09-20 15:19:58 +02:00
Alex Auvolat	1f7b050b7d	Change a warn! into a debug!	2022-09-20 11:49:48 +02:00
Alex Auvolat	56592e1853	RPC performance changes ... - configurable ping timeout - single, much higher, configurable RPC timeout - no more concurrency semaphore	2022-09-19 20:31:00 +02:00
Alex Auvolat	5d4b6f2173	Faster GetObject workflow for getting entire objects	2022-09-19 12:19:59 +02:00
Alex Auvolat	76f42a1a2b	Properly return HTTP 204 when deleting non-existent object (fix #227)	2022-09-14 17:07:55 +02:00
Alex Auvolat	e46dc2a8ef	Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353)	2022-09-14 16:09:38 +02:00
Alex Auvolat	ab722cb40f	Add checks on replication_factor of layouts we use (fix #363, fix #364)	2022-09-13 16:22:23 +02:00
Alex Auvolat	38be811b1c	Fix clippy lint that says we should implement Eq	2022-09-13 16:08:00 +02:00
Alex Auvolat	44733474bb	Remove/change println! in server code (fix #358)	2022-09-13 16:01:55 +02:00
Alex Auvolat	07febd3ecd	Ensure data dir is created immediately when Garage starts (fix #349)	2022-09-13 15:57:27 +02:00
Alex	11bdc971e2	Merge pull request 'use netapp streaming body' (#343) from netapp-stream-body into main ... Reviewed-on: #343	2022-09-13 15:26:08 +02:00
Alex Auvolat	ff30891999	Use streaming block API for get with Range requests	2022-09-13 15:13:07 +02:00
Alex Auvolat	28a4af73ca	Use netapp 0.5 published from crates.io	2022-09-13 13:11:44 +02:00
Alex Auvolat	b823151a0b	improvements in block manager	2022-09-12 16:57:38 +02:00
Alex Auvolat	f91fab8582	Simplify+improve async hasher by using bounded channel	2022-09-12 16:23:43 +02:00
Alex Auvolat	7f54706b95	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-08 15:50:56 +02:00
Alex Auvolat	d9d199a6c9	Merge branch 'main' into lx-perf-improvements	2022-09-08 15:49:17 +02:00
Alex Auvolat	ceb1f0229a	Move version back into util	2022-09-07 18:36:46 +02:00
Alex Auvolat	f310fce34b	Inject GIT_VERSION even later	2022-09-07 18:30:15 +02:00
Alex Auvolat	06df301de5	Fix merge	2022-09-07 18:16:01 +02:00
Alex Auvolat	8adc654713	Merge branch 'main' into improve-deps	2022-09-07 18:13:27 +02:00
Alex Auvolat	107853334b	Fix build error	2022-09-07 18:10:19 +02:00
Alex Auvolat	1449204439	Add warnings when features are not included in build	2022-09-07 18:02:13 +02:00
Alex Auvolat	2e00809af5	Error messages when system-libs XOR bundled-libs != 1	2022-09-07 17:57:12 +02:00
Alex Auvolat	2559f63e9b	Make all HTTP services optionnal	2022-09-07 17:54:16 +02:00
Alex Auvolat	28d86e7602	Report build features in garage --help	2022-09-07 17:05:21 +02:00
Alex Auvolat	db61f41030	Move GIT_VERSION injection later in build chain to reduce build times	2022-09-07 11:59:56 +02:00
Alex Auvolat	907054775d	Faster copy, better get error message	2022-09-06 22:25:23 +02:00
Alex Auvolat	6b958979bd	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-06 22:13:01 +02:00
Alex Auvolat	c2cc08852b	Reenable node ordering	2022-09-06 19:31:42 +02:00
Alex Auvolat	6f02c36a89	cargo fmt	2022-09-06 17:59:41 +02:00
Alex Auvolat	0f5689c169	Include code from v0.5.1 directly to remove dependencies	2022-09-06 17:52:50 +02:00
Alex Auvolat	1e92e9f782	Disable k2v tests when feature is disabled	2022-09-06 17:29:46 +02:00
Alex Auvolat	431dee050f	Remove opentelemetry-otlp dep in api/	2022-09-06 17:25:44 +02:00
Alex Auvolat	b886c75450	Make all DB engines optional build features	2022-09-06 17:09:43 +02:00
Alex Auvolat	48ffaaadfc	Bump versions to 0.8.0 (compatibility is broken already)	2022-09-06 16:47:56 +02:00
Alex Auvolat	454d8474ef	Fix clippy	2022-09-06 15:43:50 +02:00
Jakub Jirutka	ea36b9ff90	Allow building without Prometheus exporter (/metrics endpoint) ... prometheus and opentelemetry-prometheus add 7 extra dependencies in total and increases the size of the garage binary by ~7 % (with fat LTO).	2022-09-06 01:15:09 +02:00
Jakub Jirutka	e7af006c1c	Make OTLP exporter optional via feature "telemetry-otlp" ... opentelemetry-otlp add 48 (!) extra dependencies and increases the size of the garage binary by ~11 % (with fat LTO).	2022-09-06 01:14:47 +02:00
Alex Auvolat	729a910e14	Remove Heed default features	2022-09-05 16:40:13 +02:00
Alex Auvolat	07e6bcde85	Merge branch 'main' into lx-perf-improvements	2022-09-05 12:40:17 +02:00
Jakub Jirutka	7511ba5530	Allow linking against system-provided libsqlite ... Unfortunately, rusqlite uses the opposite logic for enabling/disabling bundled libraries to others (libsodium-sys, zstd-sys). Cargo features are very limited and doesn't allow to enable feature A in a dependency iff feature B is disabled. Note, lmdb-rkv-sys doesn't need any special treatment because it automatically links against system liblmdb if found via pkgconf. Linux distros should build garage with `--no-default-features --features system-libs` to disable bundled-libs and enable system-libs.	2022-09-03 19:15:57 +02:00
Jakub Jirutka	a6e40b75ea	Add feature "system-libs" to enable linking against system libraries ... If this feature is enabled, libsodium-sys and zstd-sys will link dynamically against system-provided libraries instead of building and linking statically the bundled (possibly outdated and vulnerable) copies of them. This feature is intended mainly for linux package maintainers.	2022-09-03 18:44:34 +02:00
Alex Auvolat	e1751c8a9c	fix clippy	2022-09-02 17:24:26 +02:00
Alex Auvolat	5d4b937a00	Ability to have up to 4 concurrently working resync workers	2022-09-02 17:18:13 +02:00
Alex Auvolat	5e8baa433d	Make BlockManagerLocked fully private again	2022-09-02 16:52:22 +02:00
Alex Auvolat	47be652a1f	block manager: refactor: split resync into separate file	2022-09-02 16:47:15 +02:00
Alex Auvolat	943d76c583	Ability to dynamically set resync tranquility	2022-09-02 15:34:21 +02:00
Alex Auvolat	6226f5ceca	Update to netapp 0.4.5 - fixed ping	2022-09-02 14:33:12 +02:00
Alex Auvolat	13b5f28c7e	Make use of BytesBuf from new Netapp	2022-09-02 13:46:42 +02:00
Alex Auvolat	1ef87ac4cb	cargo fmt	2022-09-02 13:38:29 +02:00
Alex Auvolat	99b532b85b	Apply PRIO_SECONDARY to block data transfers	2022-09-01 16:35:43 +02:00
Alex Auvolat	df094bd807	Less strict timeouts	2022-09-01 16:30:44 +02:00
Alex Auvolat	bc977f9a7a	Update to Netapp with OrderTag support and exploit OrderTags	2022-09-01 12:58:20 +02:00
Alex Auvolat	70231d68b2	Fix bytes_read counter	2022-08-31 19:44:27 +02:00
Alex Auvolat	c9bc9d89de	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-08-31 17:42:31 +02:00
Alex Auvolat	efbca67ce4	Add env filter to tracing subscriber	2022-08-31 14:39:12 +02:00
Alex Auvolat	44cd98d2e4	Tracing-subscriber: write to stderr	2022-08-31 14:28:17 +02:00
Alex Auvolat	dd5304f6fc	Replace logging crate pretty_env_logger by tracing_subscriber::fmt	2022-08-31 14:24:41 +02:00
Alex Auvolat	322dafc761	Try to fix clippy	2022-08-29 17:32:45 +02:00
Alex Auvolat	1921f4f7e6	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-08-29 16:45:05 +02:00
Alex Auvolat	ebc20a8798	Merge branch 'main' into lx-perf-improvements	2022-08-29 16:44:13 +02:00
Quentin Dufour	2c7bae935a	Configure structopt to report the right version ... By default, structopt reports the value provided by the env var CARGO_PKG_VERSION, feeded by Cargo when reading Cargo.toml. However for Garage we use a versioning based on git, so we often report a version that is behind the real version. In this commit, we create garage_util::version::garage() that reports the right version and configure all structopt subcommands to call this function instead of using the env var.	2022-08-11 10:21:45 +02:00
Alex Auvolat	e935861854	Factor out node request order selection logic & use in manager	2022-07-29 12:25:03 +02:00
Alex Auvolat	68087ee13d	Fix clippy	2022-07-29 12:25:03 +02:00
Alex Auvolat	605a630333	Use streaming in block manager	2022-07-29 12:25:02 +02:00
Alex Auvolat	a35d4da721	update netapp to 0.5	2022-07-29 12:25:02 +02:00
Alex Auvolat	8e7e680afe	First adaptation to WIP netapp with streaming body	2022-07-29 12:25:02 +02:00
Alex Auvolat	16f6a1a65d	fix clippy	2022-07-29 12:24:49 +02:00
Alex Auvolat	ad35b18bb1	Faster chunker	2022-07-29 12:24:49 +02:00
Alex Auvolat	ff4771c36a	cargo fmt	2022-07-29 12:24:48 +02:00
Alex Auvolat	381eb9a5a1	Fix tests	2022-07-29 12:24:48 +02:00
Alex Auvolat	2cad656a03	More make clippy happy	2022-07-29 12:24:48 +02:00
Alex Auvolat	0176da3ad2	Make clippy happy	2022-07-29 12:24:48 +02:00
Alex Auvolat	2f111e6b3d	Performance improvements: ... - reduce contention on mutation_lock by having 256 of them - better lmdb defaults	2022-07-29 12:24:48 +02:00
Alex Auvolat	1b2e1296eb	Compute hashes on dedicated threads	2022-07-29 12:24:44 +02:00
Alex Auvolat	ac03fa7937	Uniformize tracing::* imports (hopefully fixes 32-bit build)	2022-07-15 18:31:19 +02:00
Alex	4f38cadf6e	Background task manager (#332) ... - [x] New background worker trait - [x] Adapt all current workers to use new API - [x] Command to list currently running workers, and whether they are active, idle, or dead - [x] Error reporting - Optimizations - [x] Merkle updater: several items per iteration - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on - scrub: - [x] have only one worker with a channel to start/pause/cancel - [x] automatic scrub - [x] ability to view and change tranquility from CLI - [x] persistence of a few info - [ ] Testing Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #332 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-07-08 13:30:26 +02:00
Alex Auvolat	aab34bfe54	add delays in k2v test_items_and_indices	2022-07-08 10:41:57 +02:00
Alex	fe3fa83de7	Publish k2v-client crate to crates.io (#337) ... Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #337 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-07-04 18:27:25 +02:00
Alex Auvolat	b6d59ec19a	Fix poll item when item didn't change	2022-07-04 14:00:02 +02:00
Alex	0850bac874	Add poll command to k2v-cli (#335) ... Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #335 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-07-04 12:45:32 +02:00
Alex	77e3fd6db2	improve internal item counter mechanisms and implement bucket quotas (#326) ... - [x] Refactoring of internal counting API - [x] Repair procedure for counters (it's an offline procedure!!!) - [x] New counter for objects in buckets - [x] Add quotas to buckets struct - [x] Add CLI to manage bucket quotas - [x] Add admin API to manage bucket quotas - [x] Apply quotas by adding checks on put operations - [x] Proof-read Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #326 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-15 20:20:28 +02:00
Quentin Dufour	d544a0e0e0	Send CORS headers for all requests	2022-06-13 10:19:52 +02:00
Alex Auvolat	138e13071b	Fix garage_db build on 32-bit systems	2022-06-09 14:55:20 +02:00
Alex	b44d3fc796	Abstract database behind generic interface and implement alternative drivers (#322) ... - [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ``` Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #322 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-08 10:01:44 +02:00
Alex Auvolat	ff06d3f082	Fix Content-Type headers for {admin,k2v} errors and admin responses ... Fix #315	2022-05-25 17:09:33 +02:00
Alex	b2a2d3859f	K2V client improvements (#307) ... - [x] Better distinguish error types - [x] Parse error messages received from server - [x] Remove `src/` folder layer, we don't have that for other crates Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #307 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-24 12:48:05 +02:00
Alex	382e74c798	First version of admin API (#298) ... **Spec:** - [x] Start writing - [x] Specify all layout endpoints - [x] Specify all endpoints for operations on keys - [x] Specify all endpoints for operations on key/bucket permissions - [x] Specify all endpoints for operations on buckets - [x] Specify all endpoints for operations on bucket aliases View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md> **Code:** - [x] Refactor code for admin api to use common api code that was created for K2V **General endpoints:** - [x] Metrics - [x] GetClusterStatus - [x] ConnectClusterNodes - [x] GetClusterLayout - [x] UpdateClusterLayout - [x] ApplyClusterLayout - [x] RevertClusterLayout **Key-related endpoints:** - [x] ListKeys - [x] CreateKey - [x] ImportKey - [x] GetKeyInfo - [x] UpdateKey - [x] DeleteKey **Bucket-related endpoints:** - [x] ListBuckets - [x] CreateBucket - [x] GetBucketInfo - [x] DeleteBucket - [x] PutBucketWebsite - [x] DeleteBucketWebsite **Operations on key/bucket permissions:** - [x] BucketAllowKey - [x] BucketDenyKey **Operations on bucket aliases:** - [x] GlobalAliasBucket - [x] GlobalUnaliasBucket - [x] LocalAliasBucket - [x] LocalUnaliasBucket **And also:** - [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling) - [x] Add management of website access - [ ] Check that nothing is missing wrt what can be done using the CLI - [ ] Improve formatting of the spec - [x] Make sure everyone is cool with the API design Fix #231 Fix #295 Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #298 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-24 12:16:39 +02:00
trinity-1686a	64c193e3db	Add a K2V client library and CLI (#303) ... lib.rs could use getting split in modules, but I'm not sure how exactly Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr> Reviewed-on: #303 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2022-05-18 22:24:09 +02:00
Alex Auvolat	c692f55d5c	K2V: Fix end parameter and add tests (fix #305)	2022-05-17 11:50:23 +02:00
Alex Auvolat	7b474855e3	Make background runner terminate correctly	2022-05-17 11:38:31 +02:00
Alex	5768bf3622	First implementation of K2V (#293) ... **Specification:** View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [x] Put K2V behind `k2v` feature flag Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #293 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-10 13:16:57 +02:00
Alex Auvolat	def78c5e6f	Update netapp to 0.4.4, fix #300	2022-05-09 12:08:47 +02:00
Alex	277a20ec44	Fix layout show to not show changes when there are no changes (#297) ... fixes #295, partially Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #297 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-09 11:14:55 +02:00
Alex Auvolat	617f28bfa4	Correct small formatting issue	2022-05-05 14:21:57 +02:00
Mendes	948ff93cf1	Corrected the warnings and errors issued by cargo clippy	2022-05-01 16:05:39 +02:00
Alex Auvolat	2aeaddd5e2	Apply cargo fmt	2022-05-01 09:57:05 +02:00
Alex Auvolat	c1d1646c4d	Change the way new layout assignations are computed. ... The function now computes an optimal assignation (with respect to partition size) that minimizes the distance to the former assignation, using flow algorithms. This commit was written by Mendes Oulamara <mendes.oulamara@pm.me>	2022-05-01 09:54:19 +02:00
Alex Auvolat	c9ef3e461b	fix clippy	2022-04-19 12:50:40 +02:00
Alex Auvolat	c93008d333	Prettier code for defragmentation	2022-04-19 12:50:40 +02:00
Alex Auvolat	e5341ca47b	Defragmentation in UploadPartCopy: first pass (not pretty but it compiles)	2022-04-19 12:50:36 +02:00
withinboredom	a4f9f19ac3	remove size limitation in UploadPartCopy (#280) ... This removes the >1mb s3_copy restriction. This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248). Co-authored-by: Rob Landers <landers.robert@gmail.com> Reviewed-on: #280 Co-authored-by: withinboredom <landers.robert@gmail.com> Co-committed-by: withinboredom <landers.robert@gmail.com>	2022-04-19 12:49:43 +02:00
Alex Auvolat	5d404dcd54	Add missing opentelemetry features	2022-04-08 14:21:04 +02:00
Alex Auvolat	94f1e48fff	Update to netapp 0.4.2 (a tiny fix)	2022-04-07 11:50:03 +02:00
Alex Auvolat	cb5836d53c	Bring maximum exponential backoff time down from 16h to 1h	2022-04-07 11:49:29 +02:00
Alex Auvolat	0091002ef2	New replication modes and their documentation	2022-03-28 16:26:04 +02:00
Alex Auvolat	8f9cf3a5d1	fix a clippy lint	2022-03-28 15:48:55 +02:00
Alex Auvolat	913f7754bb	Add blocks in errored state to garage stats	2022-03-28 15:47:23 +02:00
Alex Auvolat	42dde54126	Log admin GET requests at debug level instead of info ... to reduce noise in logs	2022-03-28 15:46:52 +02:00
Alex Auvolat	9d0ed78887	Add feature flag for Kubernetes discovery	2022-03-24 16:57:43 +01:00
Alex Auvolat	509d256c58	Make layout optimization work in relative terms	2022-03-24 15:27:14 +01:00
Alex Auvolat	2814d41842	Allow garage layout assign to assign to several nodes at once	2022-03-24 15:27:13 +01:00
Alex Auvolat	7e0e2ffda2	Slight change and add comment to layout assignation algo	2022-03-24 15:27:13 +01:00
Alex Auvolat	413ab0eaed	Small change to partition assignation algorithm ... This change helps ensure that nodes for each partition are spread over all datacenters, a property that wasn't ensured previously when going from a 2 DC deployment to a 3 DC deployment	2022-03-24 15:27:10 +01:00
Alex Auvolat	3dc9214172	Add lots of comments on how the resync queue works ... (I don't really want to change/refactor that code though)	2022-03-23 10:25:39 +01:00
Alex Auvolat	077dd1cde9	Clippy	2022-03-23 10:25:39 +01:00
Alex Auvolat	e480aaf338	Make background tranquility a configurable parameter	2022-03-23 10:25:19 +01:00
Alex Auvolat	8fd6745745	Move block RC code to separate rc.rs	2022-03-23 10:25:19 +01:00
Alex Auvolat	c3982a90b6	Move DataBlock out of manager.rs	2022-03-23 10:25:19 +01:00
Alex Auvolat	c1d9854d2c	Move block manager to separate module	2022-03-23 10:25:15 +01:00
trinity-1686a	8565f7dc31	cleanup	2022-03-23 10:22:37 +01:00
trinity-1686a	8db6b84559	add test for create bucket and put website with streaming signature	2022-03-23 10:22:37 +01:00
trinity-1686a	1eb7fdb08f	add test framework for arbitraty S3 requests ... and implement some basic test with it	2022-03-23 10:22:36 +01:00
KokaKiwi	e934934f14	garage_api: Update streaming payload stream unit tests	2022-03-23 10:22:36 +01:00
KokaKiwi	98545a16dd	garage_api: Handle streaming payload early in request handling	2022-03-23 10:22:36 +01:00
Alex Auvolat	db46cdef79	Update netapp to v0.4.1	2022-03-15 17:09:57 +01:00
Alex Auvolat	ba6b56ae68	Fix some new clippy lints	2022-03-14 12:27:49 +01:00
Alex Auvolat	0af314b295	Add comment for fsync	2022-03-14 11:54:00 +01:00
Alex Auvolat	d78bf379fb	Fix resync queue to not drop items	2022-03-14 11:51:37 +01:00
Alex Auvolat	f7e6f4616f	Spawn a single resync worker	2022-03-14 11:51:37 +01:00
Alex Auvolat	dc5ec4ecf9	Add appropriate fsync() calls in write_block ... to ensure that data is persisted properly	2022-03-14 11:51:32 +01:00
Alex Auvolat	fe62d01b7e	Implement exponential backoff for resync retries	2022-03-14 11:41:20 +01:00
Alex Auvolat	9b2b531f4d	Make admin server optional	2022-03-14 10:54:25 +01:00
Alex Auvolat	2377a92f6b	Add wrapper over sled tree to count items (used for big queues)	2022-03-14 10:54:25 +01:00
Alex Auvolat	203e8d2c34	Bump version to 0.7 because of incompatible Netapp	2022-03-14 10:54:24 +01:00
Alex Auvolat	f869ca625d	Add spans to table calls, change span names in RPC	2022-03-14 10:54:12 +01:00
Alex Auvolat	0cc31ee169	add missing netapp telemetry feature	2022-03-14 10:54:11 +01:00
Alex Auvolat	dc8d0496cc	Refactoring: rename config files, make modifications less invasive	2022-03-14 10:53:51 +01:00
Alex Auvolat	d9a35359bf	Add metrics to web endpoint	2022-03-14 10:53:50 +01:00
Alex Auvolat	2a5609b292	Add metrics to API endpoint	2022-03-14 10:53:36 +01:00
Alex Auvolat	818daa5c78	Refactor how durations are measured	2022-03-14 10:53:35 +01:00
Alex Auvolat	f0d0cd9a20	Remove strum crate dependency; add protobuf nix dependency	2022-03-14 10:53:00 +01:00
Alex Auvolat	55d4471599	Remove ... at end of hex IDs	2022-03-14 10:52:31 +01:00
Alex Auvolat	bb04d94fa9	Update to Netapp 0.4 which supports distributed tracing	2022-03-14 10:52:30 +01:00
Alex Auvolat	8c2fb0c066	Add tracing integration with opentelemetry	2022-03-14 10:52:13 +01:00
Alex Auvolat	2cab84b1fe	Add many metrics in table/ and rpc/	2022-03-14 10:51:50 +01:00
Maximilien R	1e2cf26373	Implement basic metrics in table	2022-03-14 10:51:17 +01:00
mricher	e349af13a7	Update dependencies and add admin module with metrics ... - Global dependencies updated in Cargo.lock - New module created in src/admin to host: - the (future) admin REST API - the metric collection - add configuration block No metrics implemented yet	2022-03-14 10:51:12 +01:00
Max Audron	9d44127245	add support for kubernetes service discovery ... This commit adds support to discover garage instances running in kubernetes. Once enabled by setting `kubernetes_namespace` and `kubernetes_service_name` garage will create a Custom Resources `garagenodes.deuxfleurs.fr` with nodes public key as the resource name. and IP and Port information as spec in the namespace configured by `kubernetes_namespace`. For discovering nodes the resources are filtered with the optionally set `kubernetes_service_name` which sets a label `garage.deuxfleurs.fr/service` on the resources. This allows to separate multiple garage deployments in a single namespace. the `kubernetes_skip_crd` variable allows to disable the creation of the CRD by garage itself. The user must deploy this manually.	2022-03-12 13:05:52 +01:00
Quentin Dufour	c00b2c9948	Functional tests for admin commands	2022-03-07 17:32:07 +01:00
Quentin Dufour	8df1e186de	Functional tests for website endpoints	2022-03-07 17:32:07 +01:00
Quentin Dufour	2ef60b8417	Functional test for multipart endpoints	2022-03-07 17:32:07 +01:00
Quentin Dufour	1e639ec67c	Functional test for ListMultipartUploads	2022-03-07 17:32:07 +01:00
Quentin Dufour	cfea1e0315	Functional tests for bucket endpoints	2022-03-07 17:32:02 +01:00
Quentin Dufour	05eb79929e	Functional tests for object operations	2022-03-07 17:05:10 +01:00
Quentin Dufour	0f4e0e8bb9	Move ListObjects tests to Rust	2022-03-07 17:05:10 +01:00
Alex Auvolat	8a5bbc3b0b	More permissive OPTIONS on S3 API	2022-03-01 11:15:16 +01:00
Alex Auvolat	97f245f218	Add tracing output to signature calculation	2022-02-28 12:22:39 +01:00
Alex Auvolat	8129a98291	Process CORS earlier in pipeline	2022-02-28 12:22:39 +01:00
trinity-1686a	f6f8b7f1ad	Support for PostObject (#222) ... Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html) - [x] routing PostObject properly - [x] parsing multipart body - [x] validating signature - [x] validating policy - [x] validating content length - [x] actually saving data Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr> Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #222 Reviewed-by: Alex <alex@adnab.me> Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2022-02-21 23:02:30 +01:00
Jill	84613e66a2	garage(tests): Remove RNG stuff	2022-02-11 10:50:55 +01:00
Jill	c8b30ebc79	garage(tests): Remove superfluous test	2022-02-10 17:55:51 +01:00
Jill	d7decda3f4	garage(tests): Add random suffix to created buckets.	2022-02-10 17:55:50 +01:00
Jill	cd13ea461b	garage(tests): Add some unsafe-usage doc and tweaks	2022-02-10 17:55:50 +01:00
Quentin Dufour	5d19f3d2d7	Add integration tests to Drone	2022-02-10 17:55:50 +01:00
Jill	3baa841d6f	tests: Fix garage integration test	2022-02-10 17:55:49 +01:00
Jill	dd407e7041	tests: Add garage integration tests (base)	2022-02-10 17:55:49 +01:00
Alex Auvolat	af261e1789	Fix a bug when a migration is followed by a rebalance ... Nodes would stabilize on different encoding formats for the values, some having the pre-migration format and some having the post-migration format. This would be reflected in the Merkle trees never converging and thus having an infinite resync loop.	2022-02-10 17:38:27 +01:00
Alex Auvolat	ae2f32baf1	Hide deleted key in bucket info (fix #211)	2022-02-02 17:12:48 +01:00
Alex Auvolat	c99f55c420	Add restriction on part ordering in CompleteMultipartUpload	2022-01-25 12:45:00 +01:00
Alex Auvolat	acdf893362	Fix partnumber	2022-01-25 12:25:23 +01:00
Alex Auvolat	338b1b83ee	Implement part_number for GetObject	2022-01-24 21:04:42 +01:00
Alex Auvolat	6dab836f3a	Multipart improvements ... - support part_number for HeadObject - add checks in complete_multipart_upload	2022-01-24 21:04:40 +01:00
Alex Auvolat	513a6b15f9	Handle OPTIONS on website endpoint	2022-01-24 12:32:28 +01:00
Alex Auvolat	ea7fb901eb	Implement {Put,Get,Delete}BucketCors and CORS in general ... - OPTIONS request against API endpoint - Returning corresponding CORS headers on API calls - Returning corresponding CORS headers on website GET's	2022-01-24 11:58:00 +01:00
Trinity Pointard	820924534a	use clamp instead of min(max())	2022-01-24 11:56:59 +01:00
Quentin Dufour	440374524b	Implement ListParts	2022-01-21 10:42:30 +01:00
Quentin Dufour	fe003d6fbc	Add ListPartsResult structure	2022-01-20 16:38:55 +01:00
trinity-1686a	e55fa38c99	Add date verification to presigned urls (#196) ... fix #96 fix #162 by returning Forbidden instead Bad Request Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #196 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2022-01-18 12:22:31 +01:00
trinity-1686a	178e35f868	refactor s3_router and api_server to make unused Endpoint parameters more obvious	2022-01-17 15:50:24 +01:00
Jill	fdcddbe168	Fix Multipart Upload with WinSCP (#164) (#193) ... Closes #164. Reviewed-on: #193 Co-authored-by: Jill <kokakiwi@deuxfleurs.fr> Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>	2022-01-17 11:18:40 +01:00
Jill	b45dcc1925	Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156) ... Closes #64. Reviewed-on: #156 Co-authored-by: Jill <kokakiwi@deuxfleurs.fr> Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>	2022-01-17 10:55:31 +01:00
Alex Auvolat	d4dd2e2640	Make use of website config, return error document on error	2022-01-13 14:25:19 +01:00
Alex Auvolat	9eb211948e	Allow setting index document and error document on the CLI	2022-01-13 14:25:19 +01:00
Alex Auvolat	3ea8ca1b9e	Implement GetBucketWebsite	2022-01-13 14:23:52 +01:00
Alex Auvolat	f7349f4005	Add quotes in returned etags	2022-01-13 14:03:33 +01:00
Alex Auvolat	1ee8f596ee	Testing for UploadPartCopies and bugfixes in AWS signatures	2022-01-13 14:03:30 +01:00
Alex Auvolat	6617a72220	Implement UploadPartCopy	2022-01-13 13:58:47 +01:00
Alex Auvolat	3770a34e3d	Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187)	2022-01-13 13:56:55 +01:00
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
Alex Auvolat	3586c7257c	Clarify error message	2022-01-10 14:10:04 +01:00
Alex Auvolat	17ea28a438	Fix trivial bug in CLI	2022-01-10 12:38:33 +01:00
Alex Auvolat	7ee11f0eb6	Fix unit tests	2022-01-05 17:34:48 +01:00
Alex Auvolat	168a90dfb5	Fix some error codes	2022-01-05 17:07:36 +01:00
Alex Auvolat	fb1e31add0	Small CLI changes	2022-01-05 16:28:46 +01:00
Alex Auvolat	135858d067	Implement DeleteBucket	2022-01-05 16:28:19 +01:00
Alex Auvolat	8395030e48	Implement CreateBucket	2022-01-05 15:56:48 +01:00
Alex Auvolat	9431090b1e	Implement key allow|deny --create-bucket	2022-01-05 15:12:59 +01:00
Alex Auvolat	677ab60cc1	Small changes in key model and refactoring	2022-01-04 18:59:17 +01:00
Alex Auvolat	df35feba18	New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates	2022-01-04 12:53:14 +01:00
Alex Auvolat	1bcd6fabbd	New buckets for 0.6.0: small changes ... - Fix bucket delete - fix merge of bucket creation date - Replace deletable with option in aliases Rationale: if two aliases point to conflicting bucket, resolving by making an arbitrary choice risks making data accessible when it shouldn't be. We'd rather resolve to deleting the alias until someone puts it back.	2022-01-04 12:52:47 +01:00
Alex Auvolat	ba7f268b99	Rename and change query filters	2022-01-04 12:52:46 +01:00
Alex Auvolat	e59c23a69d	Refactor logic for setting/unsetting aliases	2022-01-04 12:52:46 +01:00
Alex Auvolat	2140cd7205	Remove website redirects	2022-01-04 12:52:46 +01:00
Alex Auvolat	beeef4758e	Some movement of helper code and refactoring of error handling	2022-01-04 12:52:46 +01:00
Alex Auvolat	d8ab5bdc3e	New buckets for 0.6.0: fix model and migration	2022-01-04 12:47:28 +01:00
Alex Auvolat	c7d5c73244	Add must_use to some CRDT functions	2022-01-04 12:47:28 +01:00
Alex Auvolat	b76d0580a0	Fix forgotten flag	2022-01-04 12:47:28 +01:00
Alex Auvolat	87121dce9d	New buckets for 0.6.0: documentation and build files	2022-01-04 12:47:06 +01:00
Alex Auvolat	b1cfd16913	New buckets for 0.6.0: small fixes, including: ... - ensure bucket names are correct aws s3 names - when making aliases, ensure timestamps of links in both ways are the same - fix small remarks by trinity - don't have a separate website_access field	2022-01-04 12:46:41 +01:00
Alex Auvolat	5db600e231	More complete output to bucket info and key info	2022-01-04 12:46:41 +01:00
Alex Auvolat	4d30e62db4	New buckets for 0.6.0: migration code and build files	2022-01-04 12:46:13 +01:00
Alex Auvolat	0bbb6673e7	Model changes	2022-01-04 12:45:52 +01:00
Alex Auvolat	53f71b3a57	Implement bucket alias and bucket unalias	2022-01-04 12:45:51 +01:00
Alex Auvolat	5b1117e582	New model for buckets	2022-01-04 12:45:46 +01:00
Alex Auvolat	8f6026de5e	Make table name a const in trait	2021-12-15 15:39:10 +01:00
trinity-1686a	ca7b438f3f	less strict ListBuckets (#178) ... fix #175 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #178 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 15:05:36 +01:00
trinity-1686a	1eb972b1ac	Add compression using zstd (#173) ... fix #27 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #173 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 11:26:43 +01:00
trinity-1686a	60d4459926	BucketWebsite (#174) ... fix #77 this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though. Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #174 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 10:41:39 +01:00
trinity-1686a	c4ac8835d3	add proper request router for s3 api (#163) ... fix #161 Current request router was organically grown, and is getting messier and messier with each addition. This router cover exaustively existing API endpoints (with exceptions listed in [#161(comment)](#161 (comment)) either because new and old api endpoint can't feasabily be differentied, or it's more lambda than s3). Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #163 Reviewed-by: Alex <alex@adnab.me> Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-06 15:17:47 +01:00
trinity-1686a	7f26ed55cd	Improved handling of HTTP ranges ... - correct HTTP code when range syntax is invalid (fix #140) - when multiple ranges are given, simply ignore and send whole file Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #157 Reviewed-by: Alex <alex@adnab.me> Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-11-29 11:52:42 +01:00
Quentin Dufour	8811bb08e6	In ListBuckets, hide entry if no perms	2021-11-22 12:10:28 +01:00
Alex Auvolat	c94406f428	Improve how node roles are assigned in Garage ... - change the terminology: the network configuration becomes the role table, the configuration of a nodes becomes a node's role - the modification of the role table takes place in two steps: first, changes are staged in a CRDT data structure. Then, once the user is happy with the changes, they can commit them all at once (or revert them). - update documentation - fix tests - implement smarter partition assignation algorithm This patch breaks the format of the network configuration: when migrating, the cluster will be in a state where no roles are assigned. All roles must be re-assigned and commited at once. This migration should not pose an issue.	2021-11-16 16:05:53 +01:00
Trinity Pointard	53888995bd	update doc and comments	2021-11-16 15:41:41 +01:00
Trinity Pointard	396fe4c702	clippy	2021-11-16 15:41:41 +01:00
Trinity Pointard	02158ee666	fix issue where list on vhost-bucket would list bucket instead of bucket content	2021-11-16 15:41:41 +01:00
Trinity Pointard	9c58ec28d3	add support for vhost-style s3 bucket	2021-11-16 15:41:41 +01:00
Alex Auvolat	ad7ab31411	Implement GC delay for table data	2021-11-08 15:47:47 +01:00
Alex Auvolat	74a7a550eb	Safety: never voluntarily delete block in 10min interval after RC reaches zero	2021-11-08 15:47:47 +01:00
Alex Auvolat	cc255d46cd	Refactor and comment table GC logic	2021-11-08 15:47:44 +01:00
Trinity Pointard	9d7535c3f5	allow missing bootstrap_peers in garage.toml	2021-11-05 16:36:25 +01:00
Alex Auvolat	e8811f7c9d	Request strategy: don't launch all 3 requests if not needed	2021-11-04 16:19:27 +01:00
Alex Auvolat	2090a6187f	Add tranquilizer mechanism to improve on token bucket mechanism	2021-11-04 13:26:59 +01:00
Alex Auvolat	6f13d083ab	Add semaphore to limit RAM used by buffered outgoing requests	2021-11-03 18:02:57 +01:00
Alex Auvolat	8c4f418fe8	Fix peer list persistence: do not forget previous peers	2021-11-03 17:34:44 +01:00
Jill	bef6d627b0	Add environment variables equivalents for some CLI options.	2021-11-03 16:00:57 +01:00
Alex Auvolat	69b89fb46d	Fix race in block resync	2021-10-27 12:01:12 +02:00
Alex Auvolat	6b47c294f5	Refactoring on repair commands	2021-10-27 11:14:55 +02:00
Trinity Pointard	28c015d9ff	add cli parameter to verify local bloc integrity ... reuse code for listing local blocks add disk i/o speed limit on integrity check	2021-10-27 10:31:03 +02:00
Alex Auvolat	3e7f766d95	CLI: default rpc_host	2021-10-26 11:36:30 +02:00
Alex Auvolat	43e13a501d	Use published netapp crate instead of git repo	2021-10-26 10:36:57 +02:00
Alex Auvolat	ada7899b24	Fix clippy lints (fix #121)	2021-10-26 10:20:05 +02:00
Alex Auvolat	df8a4068d9	Refactor block manager code, and hopefully fix deadlock	2021-10-25 14:21:51 +02:00
Alex Auvolat	de4276202a	Improve CLI, adapt tests, update documentation	2021-10-25 14:21:48 +02:00
Alex Auvolat	1b450c4b49	Improvements to CLI and various fixes for netapp version ... Discovery via consul, persist peer list to file	2021-10-22 16:55:24 +02:00
Alex Auvolat	4067797d01	First port of Garage to Netapp	2021-10-22 15:55:18 +02:00
Quentin Dufour	dc017a0cab	Build Garage with Nix	2021-10-19 16:56:07 +02:00
Alex Auvolat	1acf7e4c66	Fix git_version!() when not in git repo (fix #100)	2021-10-11 14:26:54 +02:00
Alex Auvolat	f6060b92aa	Fix HTTP return code for DeleteObject (fix #98)	2021-10-11 14:24:49 +02:00
Alex Auvolat	f3a097abdf	WIP: try to fix #93, and improve S3 ListObjects (v1 and v2) API calls	2021-10-11 11:15:47 +02:00
Alex Auvolat	fa394dcd27	Support pkcs8 private keys (allowing for ed25519 to be used for rpc)	2021-07-06 11:16:01 +02:00
trinity-1686a	30a7dee920	exit when inconsistent level of replication is detected (#92) ... fix #88 Authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #92 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-06-02 13:30:39 +02:00
Trinity Pointard	b568765c75	try parsing rpc-host command-line parameter	2021-06-01 23:53:58 +02:00
Trinity Pointard	e9c265e9dc	format garage status output better	2021-06-01 19:12:27 +02:00
Trinity Pointard	289521886b	make most changes suggested during install-party	2021-05-29 21:37:49 +02:00
Alex Auvolat	b9127dd6f8	Prepare for v0.3.0 and add migration path from v0.2.1.x	2021-05-28 15:29:58 +02:00
Alex Auvolat	ddb2b29bfd	Rename datacenters into zones (doc not yet updated)	2021-05-28 14:07:36 +02:00
Alex Auvolat	b490ebc7f6	Many improvements on ring/replication and its configuration: ... - Explicit "replication_mode" configuration parameters that takes either "none", "2" or "3" as values, instead of letting user configure replication factor themselves. These are presets whose corresponding replication/quorum values can be found in replication/mode.rs - Explicit support for single-node and two-node deployments (number of nodes must be at least "replication_mode", with "none" we can have only one node) - Ring is now stored much more compactly with 256*8 + n*32 bytes, instead of 256*32 bytes - Support for gateway-only nodes that do not store data (these nodes still need a metadata_directory to store the list of bucket and keys since those are stored on all nodes; it also technically needs a data_directory to start but it will stay empty unless we have bugs)	2021-05-28 14:07:36 +02:00
Quentin Dufour	5fdabf3e75	Add basic support for the "Versioning" command	2021-05-14 22:33:26 +02:00
Alex Auvolat	6ccffc3162	Improved XML serialization ... - Use quick_xml and serde for all XML response returned by the S3 API. - Include tests for all structs used to generate XML - Remove old manual XML escaping function which was unsafe	2021-05-06 22:37:15 +02:00
Trinity Pointard	e4b9e4e24d	rename types to CamelCase	2021-05-03 22:15:09 +02:00
Trinity Pointard	6644df6b96	fix clippy warnings on garage	2021-05-03 22:11:42 +02:00
Trinity Pointard	f8ae8fc4be	fix clippy warnings on web	2021-05-03 22:11:42 +02:00
Trinity Pointard	84856e84e5	fix clippy warnings on api	2021-05-03 22:11:41 +02:00
Trinity Pointard	4a1e079e8f	fix clippy warnings on model	2021-05-03 22:11:41 +02:00
Trinity Pointard	f5a0cf0414	fix clippy warnings on table	2021-05-03 22:11:41 +02:00
Trinity Pointard	f05bb111c2	fix clippy warnings on util and rpc	2021-05-03 22:11:41 +02:00
Quentin Dufour	631c36b3ff	S3 API: support ListBuckets	2021-05-03 21:55:30 +02:00
Alex Auvolat	ee2a3d363b	Remove STREAMING-AWS4-HMAC-SHA256-PAYLOAD (see #64)	2021-05-03 17:30:40 +02:00
Alex Auvolat	575726358c	Tune Sled configuration ... - Make sled cache size and flush interval configurable - Set less agressive default values: - cache size 128MB instead of 1GB - Flush interval 2 seconds instead of .5 seconds	2021-05-03 17:27:43 +02:00
Alex Auvolat	dcfc32cf85	Many S3 compatibility improvements: ... - return XML errors - implement AuthorizationHeaderMalformed error to redirect clients to correct location (used by minio client) - implement GetBucketLocation - fix DeleteObjects XML parsing and response	2021-04-28 01:05:40 +02:00
Alex Auvolat	642186c530	Fix #59 (& issue)	2021-04-27 23:10:43 +02:00
Alex Auvolat	119217f9f6	change a few comments	2021-04-27 16:53:47 +02:00
Trinity Pointard	2812a027ea	change some more comments and revert changes on TableSchema	2021-04-27 16:49:07 +02:00
Trinity Pointard	74373aebcf	make most requested changes	2021-04-27 16:47:08 +02:00
Alex Auvolat	16300bbd89	remove useless comment	2021-04-27 16:44:01 +02:00
Trinity Pointard	5c4b2cec3c	document garage crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	1e3df189d0	document api crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	a2e1617d84	document web crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	67585a4ffa	attempt at documenting model crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	b437610812	attempt at documenting table crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	f871689571	run cargo fmt on util and make missing doc warning	2021-04-27 16:37:10 +02:00
Trinity Pointard	8e0524ae15	document rpc crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	f9bd2d8fb7	document util crate	2021-04-27 16:37:10 +02:00
Alex Auvolat	6b2b400292	small simplify	2021-04-27 16:37:09 +02:00
Alex Auvolat	8c33d565d6	Merge discovery loop with consul	2021-04-27 16:37:09 +02:00
Alex Auvolat	948e44a3f6	cargo fmt	2021-04-27 16:37:09 +02:00
Alex Auvolat	3e2e38c830	Print stats	2021-04-27 16:37:09 +02:00
Alex Auvolat	2e53e31cdd	Cargo fmt	2021-04-27 16:37:09 +02:00
Alex Auvolat	64b91c2645	Keep old data	2021-04-27 16:37:09 +02:00
Alex Auvolat	e16077f40a	Persist directly and not in background	2021-04-27 16:37:09 +02:00
Alex Auvolat	9ced9f78dc	Improve bootstraping: do it regularly; persist peer list	2021-04-27 16:37:08 +02:00
Alex Auvolat	f859d15062	update to v0.2.1	2021-03-19 13:39:18 +01:00
Alex Auvolat	fd8f4caa81	Support old CPUs	2021-03-19 12:19:40 +01:00
Alex Auvolat	4c26a0b9c1	Update Cargo.toml files with AGPL license info	2021-03-18 21:59:17 +01:00