1590 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Mendes	bd842e1388	Correction of a few bugs in the tests, modification of ClusterLayout::check	2022-09-22 19:30:01 +02:00
Mendes	7f3249a237	New version of the algorithm that calculate the layout. ... It takes as paramters the replication factor and the zone redundancy, computes the largest partition size reachable with these constraints, and among the possible assignation with this partition size, it computes the one that moves the least number of partitions compared to the previous assignation. This computation uses graph algorithms defined in graph_algo.rs	2022-09-21 14:39:59 +02:00
Alex	af2b2f26b4	Merge pull request 'Update README (fix #230)' (#391) from new-readme into main ... Reviewed-on: #391	2022-09-21 13:28:52 +02:00
Alex Auvolat	a3758dc4c4	Update README	2022-09-21 12:53:02 +02:00
Alex Auvolat	e89f880694	Enable k2v feature flag in CI	2022-09-20 17:54:41 +02:00
Alex	fc85508648	Merge pull request 'Initialize metrics exporter earlier (fix #389)' (#390) from fix-metrics into main ... Reviewed-on: #390	2022-09-20 17:53:46 +02:00
Alex Auvolat	782630fc27	Initialize metrics exporter earlier (fix #389)	2022-09-20 17:50:22 +02:00
Alex	7a901f7aab	Merge pull request 'RPC performance changes' (#387) from configurable-timeouts into main ... Reviewed-on: #387	2022-09-20 16:17:23 +02:00
Alex Auvolat	ded444f6c9	Ability to have custom timeouts in request strategy (not used)	2022-09-20 16:01:41 +02:00
Alex Auvolat	357b72f4ff	Merge branch 'main' into configurable-timeouts	2022-09-20 15:19:58 +02:00
Alex	2c312e9cbd	Merge pull request 'Change a warn! into a debug!' (#388) from less-noise into main ... Reviewed-on: #388	2022-09-20 11:57:52 +02:00
Alex Auvolat	1f7b050b7d	Change a warn! into a debug!	2022-09-20 11:49:48 +02:00
Alex Auvolat	56592e1853	RPC performance changes ... - configurable ping timeout - single, much higher, configurable RPC timeout - no more concurrency semaphore	2022-09-19 20:31:00 +02:00
Alex	fbd32933ea	Merge pull request 'Faster GetObject workflow for getting entire objects' (#386) from faster-get into main ... Reviewed-on: #386	2022-09-19 15:24:06 +02:00
Alex Auvolat	5d4b6f2173	Faster GetObject workflow for getting entire objects	2022-09-19 12:19:59 +02:00
Alex	4fba06d62e	Merge pull request 'updates to documentation for v0.8' (#385) from doc-0.8 into main ... Reviewed-on: #385	2022-09-19 10:45:10 +02:00
Alex Auvolat	1d0a610690	Finish writing about Garage features, and fix from-source instructions	2022-09-15 13:23:57 +02:00
Alex Auvolat	f6aebefcc9	Some work on documentation towards v0.8	2022-09-14 19:31:13 +02:00
Alex	89b8087ba8	Merge pull request 'Properly return HTTP 204 when deleting non-existent object (fix #227)' (#384) from deleteobject-204 into main ... Reviewed-on: #384	2022-09-14 17:16:39 +02:00
Alex Auvolat	76f42a1a2b	Properly return HTTP 204 when deleting non-existent object (fix #227)	2022-09-14 17:07:55 +02:00
Alex	82600acf77	Merge pull request 'Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353)' (#383) from resolve-peer-names into main ... Reviewed-on: #383	2022-09-14 16:37:18 +02:00
Alex Auvolat	e46dc2a8ef	Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353)	2022-09-14 16:09:38 +02:00
Alex	80fdbfb0aa	Merge pull request 'various fixes for v0.8.0' (#380) from various-fixes-for-0.8 into main ... Reviewed-on: #380	2022-09-13 16:49:05 +02:00
Alex Auvolat	ab722cb40f	Add checks on replication_factor of layouts we use (fix #363, fix #364)	2022-09-13 16:22:23 +02:00
Alex Auvolat	38be811b1c	Fix clippy lint that says we should implement Eq	2022-09-13 16:08:00 +02:00
Alex Auvolat	44733474bb	Remove/change println! in server code (fix #358)	2022-09-13 16:01:55 +02:00
Alex Auvolat	07febd3ecd	Ensure data dir is created immediately when Garage starts (fix #349)	2022-09-13 15:57:27 +02:00
Alex	11bdc971e2	Merge pull request 'use netapp streaming body' (#343) from netapp-stream-body into main ... Reviewed-on: #343	2022-09-13 15:26:08 +02:00
Alex Auvolat	ff30891999	Use streaming block API for get with Range requests	2022-09-13 15:13:07 +02:00
Alex Auvolat	28a4af73ca	Use netapp 0.5 published from crates.io	2022-09-13 13:11:44 +02:00
Alex Auvolat	b823151a0b	improvements in block manager	2022-09-12 16:57:38 +02:00
Alex	309d7aef3f	Merge pull request 'performance improvements' (#342) from lx-perf-improvements into main ... Performance improvements included in this PR: - [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies - [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized - [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking` - [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel - [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads Other changes included in this PR: - [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12 - [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event Reviewed-on: #342	2022-09-12 16:38:43 +02:00
Alex Auvolat	f91fab8582	Simplify+improve async hasher by using bounded channel	2022-09-12 16:23:43 +02:00
Mendes	c4adbeed51	Added the section with description proofs of the parametric assignment computation in the optimal layout report	2022-09-10 13:51:12 +02:00
Alex Auvolat	7f54706b95	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-08 15:50:56 +02:00
Alex Auvolat	d9d199a6c9	Merge branch 'main' into lx-perf-improvements	2022-09-08 15:49:17 +02:00
Alex	03c40a0b24	Merge pull request 'Reorganize dependencies' (#373) from improve-deps into main ... This PR includes work from @jirutka : - [x] Allow linking against system-provided libraries (libsodium, libsqlite, libzstd) #370 - [x] Make OTLP exporter optional and allow building without Prometheus exporter (/metrics) #372 And also: - [x] Update `.nix` files - [x] Remove heed default-features - [x] Bump versions of all Garage crates to 0.8.0 - [x] Make db engines (lmdb, sled, sqlite) optionnal - [x] Add documentation for available features - [x] Directly include code of previous versions used for migration in order to reduce dependencies - [x] Read variable `GIT_VERSION` from garage main instead of in crate garage_util to make builds faster - [x] Report features used in the build somewhere? (in `garage --version` or something) - [x] Check we `warn!` correctly if we try to use deactivated feature - [x] Allow not to launch S3 endpoint if not in config Reviewed-on: #373	2022-09-08 15:45:09 +02:00
Mendes	d38fb6c250	ignore log files in commit	2022-09-08 12:43:33 +02:00
Alex Auvolat	ceb1f0229a	Move version back into util	2022-09-07 18:36:46 +02:00
Alex Auvolat	f310fce34b	Inject GIT_VERSION even later	2022-09-07 18:30:15 +02:00
Alex Auvolat	06df301de5	Fix merge	2022-09-07 18:16:01 +02:00
Alex Auvolat	8adc654713	Merge branch 'main' into improve-deps	2022-09-07 18:13:27 +02:00
Alex Auvolat	107853334b	Fix build error	2022-09-07 18:10:19 +02:00
Alex Auvolat	1449204439	Add warnings when features are not included in build	2022-09-07 18:02:13 +02:00
Alex Auvolat	2e00809af5	Error messages when system-libs XOR bundled-libs != 1	2022-09-07 17:57:12 +02:00
Alex Auvolat	2559f63e9b	Make all HTTP services optionnal	2022-09-07 17:54:16 +02:00
Alex Auvolat	28d86e7602	Report build features in garage --help	2022-09-07 17:05:21 +02:00
Alex Auvolat	db61f41030	Move GIT_VERSION injection later in build chain to reduce build times	2022-09-07 11:59:56 +02:00
Alex Auvolat	907054775d	Faster copy, better get error message	2022-09-06 22:25:23 +02:00
Alex Auvolat	6b958979bd	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-06 22:13:01 +02:00