1360 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Alex	82600acf77	Merge pull request 'Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353)' (#383) from resolve-peer-names into main ... Reviewed-on: #383	2022-09-14 16:37:18 +02:00
Alex Auvolat	e46dc2a8ef	Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353)	2022-09-14 16:09:38 +02:00
Alex	80fdbfb0aa	Merge pull request 'various fixes for v0.8.0' (#380) from various-fixes-for-0.8 into main ... Reviewed-on: #380	2022-09-13 16:49:05 +02:00
Alex Auvolat	ab722cb40f	Add checks on replication_factor of layouts we use (fix #363, fix #364)	2022-09-13 16:22:23 +02:00
Alex Auvolat	38be811b1c	Fix clippy lint that says we should implement Eq	2022-09-13 16:08:00 +02:00
Alex Auvolat	44733474bb	Remove/change println! in server code (fix #358)	2022-09-13 16:01:55 +02:00
Alex Auvolat	07febd3ecd	Ensure data dir is created immediately when Garage starts (fix #349)	2022-09-13 15:57:27 +02:00
Alex	11bdc971e2	Merge pull request 'use netapp streaming body' (#343) from netapp-stream-body into main ... Reviewed-on: #343	2022-09-13 15:26:08 +02:00
Alex Auvolat	ff30891999	Use streaming block API for get with Range requests	2022-09-13 15:13:07 +02:00
Alex Auvolat	28a4af73ca	Use netapp 0.5 published from crates.io	2022-09-13 13:11:44 +02:00
Alex Auvolat	b823151a0b	improvements in block manager	2022-09-12 16:57:38 +02:00
Alex	309d7aef3f	Merge pull request 'performance improvements' (#342) from lx-perf-improvements into main ... Performance improvements included in this PR: - [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies - [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized - [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking` - [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel - [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads Other changes included in this PR: - [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12 - [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event Reviewed-on: #342	2022-09-12 16:38:43 +02:00
Alex Auvolat	f91fab8582	Simplify+improve async hasher by using bounded channel	2022-09-12 16:23:43 +02:00
Alex Auvolat	7f54706b95	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-08 15:50:56 +02:00
Alex Auvolat	d9d199a6c9	Merge branch 'main' into lx-perf-improvements	2022-09-08 15:49:17 +02:00
Alex	03c40a0b24	Merge pull request 'Reorganize dependencies' (#373) from improve-deps into main ... This PR includes work from @jirutka : - [x] Allow linking against system-provided libraries (libsodium, libsqlite, libzstd) #370 - [x] Make OTLP exporter optional and allow building without Prometheus exporter (/metrics) #372 And also: - [x] Update `.nix` files - [x] Remove heed default-features - [x] Bump versions of all Garage crates to 0.8.0 - [x] Make db engines (lmdb, sled, sqlite) optionnal - [x] Add documentation for available features - [x] Directly include code of previous versions used for migration in order to reduce dependencies - [x] Read variable `GIT_VERSION` from garage main instead of in crate garage_util to make builds faster - [x] Report features used in the build somewhere? (in `garage --version` or something) - [x] Check we `warn!` correctly if we try to use deactivated feature - [x] Allow not to launch S3 endpoint if not in config Reviewed-on: #373	2022-09-08 15:45:09 +02:00
Alex Auvolat	ceb1f0229a	Move version back into util	2022-09-07 18:36:46 +02:00
Alex Auvolat	f310fce34b	Inject GIT_VERSION even later	2022-09-07 18:30:15 +02:00
Alex Auvolat	06df301de5	Fix merge	2022-09-07 18:16:01 +02:00
Alex Auvolat	8adc654713	Merge branch 'main' into improve-deps	2022-09-07 18:13:27 +02:00
Alex Auvolat	107853334b	Fix build error	2022-09-07 18:10:19 +02:00
Alex Auvolat	1449204439	Add warnings when features are not included in build	2022-09-07 18:02:13 +02:00
Alex Auvolat	2e00809af5	Error messages when system-libs XOR bundled-libs != 1	2022-09-07 17:57:12 +02:00
Alex Auvolat	2559f63e9b	Make all HTTP services optionnal	2022-09-07 17:54:16 +02:00
Alex Auvolat	28d86e7602	Report build features in garage --help	2022-09-07 17:05:21 +02:00
Alex Auvolat	db61f41030	Move GIT_VERSION injection later in build chain to reduce build times	2022-09-07 11:59:56 +02:00
Alex Auvolat	907054775d	Faster copy, better get error message	2022-09-06 22:25:23 +02:00
Alex Auvolat	6b958979bd	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-06 22:13:01 +02:00
Alex Auvolat	d23b3a14fc	Merge branch 'main' into lx-perf-improvements	2022-09-06 21:53:37 +02:00
Alex Auvolat	4024822585	Update netapp to lastest git version with LAS scheduling	2022-09-06 19:45:00 +02:00
Alex Auvolat	c2cc08852b	Reenable node ordering	2022-09-06 19:31:42 +02:00
Alex Auvolat	6f02c36a89	cargo fmt	2022-09-06 17:59:41 +02:00
Alex Auvolat	0f5689c169	Include code from v0.5.1 directly to remove dependencies	2022-09-06 17:52:50 +02:00
Alex Auvolat	1e92e9f782	Disable k2v tests when feature is disabled	2022-09-06 17:29:46 +02:00
Alex Auvolat	431dee050f	Remove opentelemetry-otlp dep in api/	2022-09-06 17:25:44 +02:00
Alex Auvolat	2c2b93acdf	Update Nix files with optional db engines	2022-09-06 17:20:10 +02:00
Alex Auvolat	bbb970965c	Document available build features	2022-09-06 17:16:45 +02:00
Alex Auvolat	b886c75450	Make all DB engines optional build features	2022-09-06 17:09:43 +02:00
Alex Auvolat	48ffaaadfc	Bump versions to 0.8.0 (compatibility is broken already)	2022-09-06 16:47:56 +02:00
Alex Auvolat	7de53a4d66	Force disable pkg-config for libsodum-sys and libzstd-sys	2022-09-06 16:41:58 +02:00
Alex Auvolat	8d77a76df1	Update .nix files	2022-09-06 15:49:41 +02:00
Alex Auvolat	454d8474ef	Fix clippy	2022-09-06 15:43:50 +02:00
Alex	ed7796924b	Merge pull request 'Make OTLP exporter optional and allow building without Prometheus exporter (/metrics)' (#372) from jirutka/garage:telemetry-and-metrics into improve-deps ... Reviewed-on: #372 Reviewed-by: Alex <alex@adnab.me>	2022-09-06 15:11:30 +02:00
Jakub Jirutka	ea36b9ff90	Allow building without Prometheus exporter (/metrics endpoint) ... prometheus and opentelemetry-prometheus add 7 extra dependencies in total and increases the size of the garage binary by ~7 % (with fat LTO).	2022-09-06 01:15:09 +02:00
Jakub Jirutka	e7af006c1c	Make OTLP exporter optional via feature "telemetry-otlp" ... opentelemetry-otlp add 48 (!) extra dependencies and increases the size of the garage binary by ~11 % (with fat LTO).	2022-09-06 01:14:47 +02:00
Jakub Jirutka	db72812f01	Use the new cargo feature resolver "2" ... Garage currently uses the legacy resolver "1". The new one is used by default if the root package specifies 'edition = 2021', which Garage does not (yet). The problem with the legacy resolver is, among others, that features enabled by dev-dependencies are propagated to normal dependencies. This affects e.g. hyper - one of the dev-dependencies enables "http2" feature that adds many extra dependencies. If we build garage without opentelemetry-otlp (this is enabled in the following commit), there's no normal dependency enabling "http2" feature. See https://doc.rust-lang.org/cargo/reference/resolver.html#feature-resolver-version-2	2022-09-06 01:14:19 +02:00
Alex Auvolat	729a910e14	Remove Heed default features	2022-09-05 16:40:13 +02:00
Alex	9f5433db82	Merge pull request 'Update .drone.yml signature' (#374) from fix-drone-signature into main ... Reviewed-on: #374	2022-09-05 16:18:15 +02:00
Alex Auvolat	fd8074ad9b	Update .drone.yml signature	2022-09-05 16:09:01 +02:00
Alex Auvolat	07e6bcde85	Merge branch 'main' into lx-perf-improvements	2022-09-05 12:40:17 +02:00