506 commits

Author	SHA1	Message	Date
Alex Auvolat	2090a6187f	Add tranquilizer mechanism to improve on token bucket mechanism	2021-11-04 13:26:59 +01:00
Alex Auvolat	6f13d083ab	Add semaphore to limit RAM used by buffered outgoing requests	2021-11-03 18:02:57 +01:00
Alex Auvolat	8c4f418fe8	Fix peer list persistence: do not forget previous peers	2021-11-03 17:34:44 +01:00
Jill	bef6d627b0	Add environment variables equivalents for some CLI options.	2021-11-03 16:00:57 +01:00
Alex Auvolat	69b89fb46d	Fix race in block resync	2021-10-27 12:01:12 +02:00
Alex Auvolat	6b47c294f5	Refactoring on repair commands	2021-10-27 11:14:55 +02:00
Trinity Pointard	28c015d9ff	add cli parameter to verify local bloc integrity ... reuse code for listing local blocks add disk i/o speed limit on integrity check	2021-10-27 10:31:03 +02:00
Alex Auvolat	3e7f766d95	CLI: default rpc_host	2021-10-26 11:36:30 +02:00
Alex Auvolat	43e13a501d	Use published netapp crate instead of git repo	2021-10-26 10:36:57 +02:00
Alex Auvolat	ada7899b24	Fix clippy lints (fix #121)	2021-10-26 10:20:05 +02:00
Alex Auvolat	df8a4068d9	Refactor block manager code, and hopefully fix deadlock	2021-10-25 14:21:51 +02:00
Alex Auvolat	de4276202a	Improve CLI, adapt tests, update documentation	2021-10-25 14:21:48 +02:00
Alex Auvolat	1b450c4b49	Improvements to CLI and various fixes for netapp version ... Discovery via consul, persist peer list to file	2021-10-22 16:55:24 +02:00
Alex Auvolat	4067797d01	First port of Garage to Netapp	2021-10-22 15:55:18 +02:00
Quentin Dufour	dc017a0cab	Build Garage with Nix	2021-10-19 16:56:07 +02:00
Alex Auvolat	1acf7e4c66	Fix git_version!() when not in git repo (fix #100)	2021-10-11 14:26:54 +02:00
Alex Auvolat	f6060b92aa	Fix HTTP return code for DeleteObject (fix #98)	2021-10-11 14:24:49 +02:00
Alex Auvolat	f3a097abdf	WIP: try to fix #93, and improve S3 ListObjects (v1 and v2) API calls	2021-10-11 11:15:47 +02:00
Alex Auvolat	fa394dcd27	Support pkcs8 private keys (allowing for ed25519 to be used for rpc)	2021-07-06 11:16:01 +02:00
trinity-1686a	30a7dee920	exit when inconsistent level of replication is detected (#92) ... fix #88 Authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #92 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-06-02 13:30:39 +02:00
Trinity Pointard	b568765c75	try parsing rpc-host command-line parameter	2021-06-01 23:53:58 +02:00
Trinity Pointard	e9c265e9dc	format garage status output better	2021-06-01 19:12:27 +02:00
Trinity Pointard	289521886b	make most changes suggested during install-party	2021-05-29 21:37:49 +02:00
Alex Auvolat	b9127dd6f8	Prepare for v0.3.0 and add migration path from v0.2.1.x	2021-05-28 15:29:58 +02:00
Alex Auvolat	ddb2b29bfd	Rename datacenters into zones (doc not yet updated)	2021-05-28 14:07:36 +02:00
Alex Auvolat	b490ebc7f6	Many improvements on ring/replication and its configuration: ... - Explicit "replication_mode" configuration parameters that takes either "none", "2" or "3" as values, instead of letting user configure replication factor themselves. These are presets whose corresponding replication/quorum values can be found in replication/mode.rs - Explicit support for single-node and two-node deployments (number of nodes must be at least "replication_mode", with "none" we can have only one node) - Ring is now stored much more compactly with 256*8 + n*32 bytes, instead of 256*32 bytes - Support for gateway-only nodes that do not store data (these nodes still need a metadata_directory to store the list of bucket and keys since those are stored on all nodes; it also technically needs a data_directory to start but it will stay empty unless we have bugs)	2021-05-28 14:07:36 +02:00
Quentin Dufour	5fdabf3e75	Add basic support for the "Versioning" command	2021-05-14 22:33:26 +02:00
Alex Auvolat	6ccffc3162	Improved XML serialization ... - Use quick_xml and serde for all XML response returned by the S3 API. - Include tests for all structs used to generate XML - Remove old manual XML escaping function which was unsafe	2021-05-06 22:37:15 +02:00
Trinity Pointard	e4b9e4e24d	rename types to CamelCase	2021-05-03 22:15:09 +02:00
Trinity Pointard	6644df6b96	fix clippy warnings on garage	2021-05-03 22:11:42 +02:00
Trinity Pointard	f8ae8fc4be	fix clippy warnings on web	2021-05-03 22:11:42 +02:00
Trinity Pointard	84856e84e5	fix clippy warnings on api	2021-05-03 22:11:41 +02:00
Trinity Pointard	4a1e079e8f	fix clippy warnings on model	2021-05-03 22:11:41 +02:00
Trinity Pointard	f5a0cf0414	fix clippy warnings on table	2021-05-03 22:11:41 +02:00
Trinity Pointard	f05bb111c2	fix clippy warnings on util and rpc	2021-05-03 22:11:41 +02:00
Quentin Dufour	631c36b3ff	S3 API: support ListBuckets	2021-05-03 21:55:30 +02:00
Alex Auvolat	ee2a3d363b	Remove STREAMING-AWS4-HMAC-SHA256-PAYLOAD (see #64)	2021-05-03 17:30:40 +02:00
Alex Auvolat	575726358c	Tune Sled configuration ... - Make sled cache size and flush interval configurable - Set less agressive default values: - cache size 128MB instead of 1GB - Flush interval 2 seconds instead of .5 seconds	2021-05-03 17:27:43 +02:00
Alex Auvolat	dcfc32cf85	Many S3 compatibility improvements: ... - return XML errors - implement AuthorizationHeaderMalformed error to redirect clients to correct location (used by minio client) - implement GetBucketLocation - fix DeleteObjects XML parsing and response	2021-04-28 01:05:40 +02:00
Alex Auvolat	642186c530	Fix #59 (& issue)	2021-04-27 23:10:43 +02:00
Alex Auvolat	119217f9f6	change a few comments	2021-04-27 16:53:47 +02:00
Trinity Pointard	2812a027ea	change some more comments and revert changes on TableSchema	2021-04-27 16:49:07 +02:00
Trinity Pointard	74373aebcf	make most requested changes	2021-04-27 16:47:08 +02:00
Alex Auvolat	16300bbd89	remove useless comment	2021-04-27 16:44:01 +02:00
Trinity Pointard	5c4b2cec3c	document garage crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	1e3df189d0	document api crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	a2e1617d84	document web crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	67585a4ffa	attempt at documenting model crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	b437610812	attempt at documenting table crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	f871689571	run cargo fmt on util and make missing doc warning	2021-04-27 16:37:10 +02:00
Trinity Pointard	8e0524ae15	document rpc crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	f9bd2d8fb7	document util crate	2021-04-27 16:37:10 +02:00
Alex Auvolat	6b2b400292	small simplify	2021-04-27 16:37:09 +02:00
Alex Auvolat	8c33d565d6	Merge discovery loop with consul	2021-04-27 16:37:09 +02:00
Alex Auvolat	948e44a3f6	cargo fmt	2021-04-27 16:37:09 +02:00
Alex Auvolat	3e2e38c830	Print stats	2021-04-27 16:37:09 +02:00
Alex Auvolat	2e53e31cdd	Cargo fmt	2021-04-27 16:37:09 +02:00
Alex Auvolat	64b91c2645	Keep old data	2021-04-27 16:37:09 +02:00
Alex Auvolat	e16077f40a	Persist directly and not in background	2021-04-27 16:37:09 +02:00
Alex Auvolat	9ced9f78dc	Improve bootstraping: do it regularly; persist peer list	2021-04-27 16:37:08 +02:00
Alex Auvolat	f859d15062	update to v0.2.1	2021-03-19 13:39:18 +01:00
Alex Auvolat	fd8f4caa81	Support old CPUs	2021-03-19 12:19:40 +01:00
Alex Auvolat	4c26a0b9c1	Update Cargo.toml files with AGPL license info	2021-03-18 21:59:17 +01:00
Alex Auvolat	a1014224d3	garage node configure --replace <old_node_id> <new_node_id>	2021-03-18 21:49:12 +01:00
Trinity Pointard	f17cb6c969	resolve domain to multiple addresses ... And warn instead of failling when a domain can't be resolved	2021-03-18 21:04:30 +01:00
Trinity Pointard	c8a7ce5cdf	remove domain resolution for *_bind_addr	2021-03-18 19:47:51 +01:00
Trinity Pointard	81e9db783f	simplify addresse deserialialiser and limit allocations	2021-03-18 19:47:51 +01:00
Trinity Pointard	ae3b7029a9	add support for using domain name in configuration	2021-03-18 19:47:51 +01:00
Alex Auvolat	6edbc65847	Add trinity's comment in the code	2021-03-18 19:46:43 +01:00
Alex	bfa0ff8f82	Merge pull request 'add support for caching headers' (#49) from trinity-1686a/garage:cache-headers into master ... Reviewed-on: #49	2021-03-18 19:45:02 +01:00
Alex Auvolat	dead945c8f	Prepare for release 0.2	2021-03-18 19:33:15 +01:00
Alex Auvolat	4eb16e8863	Allow to import keys from previous Garage instance	2021-03-18 19:24:59 +01:00
Trinity Pointard	b4c903371c	add support for caching headers	2021-03-18 15:46:33 +01:00
Alex Auvolat	390ab02f41	Todo make a test for the Merkle updater	2021-03-16 20:13:07 +01:00
Alex Auvolat	7b10245dfb	Leader-based GC	2021-03-16 18:42:33 +01:00
Alex Auvolat	08bcd51956	GC object table in a specific case	2021-03-16 16:51:15 +01:00
Alex Auvolat	3fadc5cbbd	Small changes	2021-03-16 16:35:10 +01:00
Alex Auvolat	f4346cc5f4	Update dependencies	2021-03-16 15:58:40 +01:00
Alex Auvolat	2a41b82384	Simpler Merkle & sync	2021-03-16 12:18:03 +01:00
Alex Auvolat	0aad2f2e06	some reordering	2021-03-16 11:47:39 +01:00
Alex Auvolat	515029d026	Refactor code	2021-03-16 11:43:58 +01:00
Alex Auvolat	1d9961e411	Simplify replication logic	2021-03-16 11:14:27 +01:00
Alex Auvolat	6a8439fd13	Some improvements in background worker but we terminate late	2021-03-15 23:14:12 +01:00
Alex Auvolat	0cd5b2ae19	WIP migrate to tokio 1	2021-03-15 22:36:41 +01:00
Alex Auvolat	4d4117f2b4	Refactor block resync loop; make workers infaillible	2021-03-15 20:09:44 +01:00
Alex Auvolat	667e4e72a8	Small fixes	2021-03-15 19:51:16 +01:00
Alex Auvolat	642bed601f	Make it case-insensitive	2021-03-15 19:16:42 +01:00
Alex Auvolat	5ee1d956b6	Allow manipulation of keys by their shorthand in the CLI	2021-03-15 19:14:26 +01:00
Alex Auvolat	537f652fec	Tiny things	2021-03-15 18:40:27 +01:00
Alex Auvolat	0290afe1f8	Make block rc code more understandable	2021-03-15 18:27:26 +01:00
Alex Auvolat	3bf2df622a	Time and metadata improvements	2021-03-15 16:21:41 +01:00
Alex Auvolat	097c339d98	Fix race condition	2021-03-15 15:26:29 +01:00
Alex Auvolat	bdcbdd1cd8	Fix list API bug	2021-03-15 14:46:37 +01:00
Alex Auvolat	9b118160a8	Optim & refactor	2021-03-12 22:06:56 +01:00
Alex Auvolat	831eb35763	cargo fmt	2021-03-12 21:52:19 +01:00
Alex Auvolat	c475471e7a	Implement table gc, currently for block_ref and version only	2021-03-12 19:57:37 +01:00
Alex Auvolat	f4aad8fe6e	cargo fmt	2021-03-12 18:16:03 +01:00
Alex Auvolat	5ab33fddac	Refactor CLI and prettify CLI outpu	2021-03-12 18:12:31 +01:00
Alex Auvolat	a1442f072a	Implement garage stats to get info on node contents	2021-03-12 15:40:54 +01:00
Alex Auvolat	cbe7e1a66a	Move table rpc client out of tableaux	2021-03-12 15:07:23 +01:00
Alex Auvolat	8860aa19b8	Make syncer have its own rpc client/server	2021-03-12 15:05:26 +01:00
Alex Auvolat	1fea257291	Don't sync at beginning	2021-03-12 14:51:17 +01:00
Alex Auvolat	7fdaf7aef0	Fix merkle updater not being notified; improved logging	2021-03-12 14:37:46 +01:00
Alex Auvolat	1ec49980ec	whoops	2021-03-11 19:30:24 +01:00
Alex Auvolat	3f7a496355	More security: don't delete stuff too easily	2021-03-11 19:06:27 +01:00
Alex Auvolat	f7c2cd1cd7	Add comment, and also whoops, this wasn't doing what we expected	2021-03-11 18:56:18 +01:00
Alex Auvolat	fae5104a2c	Add a nice warning	2021-03-11 18:50:32 +01:00
Alex Auvolat	db7a9d4948	Tiny changes	2021-03-11 18:45:26 +01:00
Alex Auvolat	046b649bcc	(not well tested) use merkle tree for sync	2021-03-11 18:28:27 +01:00
Alex Auvolat	94f3d28774	WIP big refactoring	2021-03-11 16:54:15 +01:00
Alex Auvolat	8d63738cb0	Checkpoint: add merkle tree in data table	2021-03-11 13:47:21 +01:00
Alex Auvolat	3214dd52dd	Very minor changes	2021-03-10 21:50:09 +01:00
Alex Auvolat	af7600f989	Correctly implement CompleteMultipartUpload with etag check of parts	2021-03-10 17:01:05 +01:00
Alex Auvolat	445912dc6a	Remove migration paths from 0.1 branch	2021-03-10 16:38:31 +01:00
Alex Auvolat	0fd7df8fa0	Switch to blake2 sum for identifying blocks by their data	2021-03-10 16:33:31 +01:00
Alex Auvolat	2afd2c81ba	Change hash function to blake2 for partition keys based on strings	2021-03-10 16:23:57 +01:00
Alex Auvolat	f319a7d374	Refactor model stuff, including cleaner CRDTs	2021-03-10 16:21:56 +01:00
Alex Auvolat	6a3dcf3974	Rename n_tokens into capacity	2021-03-10 14:52:03 +01:00
Alex Auvolat	7cda917b6b	update condition	2021-03-05 17:08:03 +01:00
Alex Auvolat	d7e005251d	Not fully tested: new multi-dc MagLev	2021-03-05 16:22:29 +01:00
Alex Auvolat	3882d5ba36	Remove epidemic propagation for fully replicated stuff: write directly to all nodes	2021-03-05 15:09:18 +01:00
Alex Auvolat	49c25a1509	Simulate stuff moving around	2021-02-25 10:53:33 +01:00
Alex Auvolat	5fe95ebae7	fix tracing	2021-02-24 12:18:01 +01:00
Alex Auvolat	13e2eda0c2	Arrange block manager	2021-02-24 11:58:03 +01:00
Alex Auvolat	09fd6ea7f0	I was tired yesterday	2021-02-24 11:05:59 +01:00
Alex Auvolat	a52ab69640	fix misuse of sled transactions	2021-02-23 22:45:36 +01:00
Alex Auvolat	20e6e9fa20	Update sled & try to debug deadlock (but its in sled...)	2021-02-23 21:27:28 +01:00
Alex Auvolat	bf25c95fe2	Make updated() be a sync function that doesn't fail	2021-02-23 20:25:15 +01:00
Alex Auvolat	28bc967c83	Handle correctly deletion dues to offloading	2021-02-23 19:59:43 +01:00
Alex Auvolat	55156cca9d	Several changes in table_sync: ... - separate path for case of offloading a partition we don't store - use sync::Mutex instead of tokio::Mutex, make less fn's async	2021-02-23 19:11:02 +01:00
Alex Auvolat	40763fd749	Cargo fmt	2021-02-23 18:46:25 +01:00
Alex Auvolat	6e6f7e8555	Replace some checksums where it makes sense	2021-02-23 18:14:37 +01:00
Alex Auvolat	e8e4418ca7	Add blake2 and xxhash hash functions	2021-02-23 17:52:28 +01:00
Alex Auvolat	b1b640ae8b	rename hash() to sha256sum(), we might want to change it at some places	2021-02-21 15:24:30 +01:00
Alex Auvolat	80892df8cc	Some refactoring	2021-02-21 13:11:10 +01:00
Alex Auvolat	3bcbbe1e31	More precise logging (warn only when returning a 500)	2021-02-20 00:30:39 +01:00
Alex Auvolat	10b983b8e7	Add verification of part numbers in CompleteMultipartUpload (WIP #30)	2021-02-20 00:13:07 +01:00
Alex Auvolat	1de96248e0	add application/xml header and missing xml escapes	2021-02-19 23:40:18 +01:00
Alex Auvolat	5d1fa591d9	Add compatibility list	2021-02-19 19:10:23 +01:00
Alex Auvolat	e64ecbdccd	S3 compatibility: return 404 instead of 400 on some multipart commands	2021-02-19 18:51:05 +01:00
Alex Auvolat	55a2a636ca	Implement ListObjectsV2	2021-02-19 16:44:06 +01:00
Alex Auvolat	02d512f3fd	Fix #28, extra headers being ignored (because of profound stupidity)	2021-02-19 12:38:22 +01:00
Alex Auvolat	76390085ef	Small improvements in the S3 put workflow	2021-02-19 12:11:02 +01:00
Alex Auvolat	36814be447	Fix S3 ListObjects result and replace println!s by debug!s	2021-01-16 16:05:54 +01:00
Alex Auvolat	e818f51073	Forgot a bump	2021-01-15 18:36:51 +01:00
Alex Auvolat	ceeb0732a2	Use 0.1.0b instead of 0.1.0 (for compatibility with new Error type)	2021-01-15 18:27:58 +01:00
Alex Auvolat	1d1d497e2b	Bump everything to 0.1.1	2021-01-15 17:54:48 +01:00
Alex Auvolat	851893a3f2	Do not accept domains such as [hello	2021-01-15 17:49:10 +01:00
Quentin	f8a40e8c4f	Explicitly set code path unreachable	2021-01-15 17:11:15 +01:00
Quentin	fad7bc405b	Behavior problem: do not panic anymore + add tests	2021-01-15 17:03:54 +01:00
Quentin	1e10c6a61c	Doc tests that do not compile/work must be tagged with ignore	2021-01-15 17:03:38 +01:00
Quentin	11a79a95dd	Simplify Error file	2021-01-15 16:25:44 +01:00
Quentin	c441a358cd	Remove unused dependencies	2021-01-15 16:16:32 +01:00
Quentin	f496e41ef4	Replace an already done check by unreachable!()	2021-01-15 15:44:44 +01:00
Quentin Dufour	2f4378a9c4	Fix formatting	2020-12-17 22:51:44 +01:00
Quentin Dufour	ccda9ab1ca	Merge branch 'master' into feature/website	2020-12-17 21:09:50 +01:00
Quentin Dufour	3132deca58	Web server access control	2020-12-17 20:43:14 +01:00
Quentin	011ff87b5f	Push update	2020-12-15 13:23:22 +01:00
Quentin	3bc4d57a0f	First implementation of the CLI	2020-12-15 12:48:24 +01:00
Quentin	a3566e49da	Start to implement Website CLI	2020-12-14 21:50:40 +01:00
Quentin	d0eb6a457f	Migrate RPC to new schema	2020-12-14 21:46:49 +01:00
Quentin	96388acf23	Implement migration	2020-12-12 21:35:29 +01:00
Alex Auvolat	8956db2a81	Make less things public	2020-12-12 17:58:19 +01:00
Alex Auvolat	5c6c067b0c	More documentation on CRDTs (we should probably extract this to a ... standalone crate!)	2020-12-12 17:06:40 +01:00
Quentin	e1ce2b228a	WIP table migration	2020-12-12 17:00:31 +01:00
Alex Auvolat	0b3084ca5f	Merge branch 'master' into doc/model	2020-12-12 16:05:28 +01:00
Quentin	e8c12072ce	Merge branch 'master' into feature/website	2020-12-10 20:12:56 +01:00
Quentin	51d0c14e44	CLI structure	2020-12-10 18:13:32 +01:00
Alex Auvolat	022b386a50	Improved compatibility on list API call	2020-12-06 15:39:03 +01:00
Quentin	986e15459a	Merge branch 'master' into feature/website	2020-12-06 15:21:09 +01:00
Alex Auvolat	4a5bbbb810	Propose ETag fix	2020-12-05 19:23:46 +01:00
Alex Auvolat	76b489f3d3	Reformulate patch	2020-12-05 19:20:07 +01:00
Quentin	bd7e3d1bd1	Fix Content-Length	2020-12-05 18:57:22 +01:00
Alex Auvolat	9f46fb699a	Content-range fix	2020-12-05 16:37:59 +01:00
Quentin	15f409d404	Merge branch 'master' into feature/website	2020-11-29 17:19:55 +01:00
Alex Auvolat	d54f15b2c6	Small optimisation	2020-11-29 17:07:14 +01:00
Alex Auvolat	fed97f37e1	ETag patch	2020-11-29 16:38:01 +01:00
Alex Auvolat	601ae25ad2	Small refactorings	2020-11-29 16:21:28 +01:00
Quentin	cbd10c1b0a	Add some doc on LWW	2020-11-23 18:17:48 +01:00
Quentin	8722e27600	CRDT doc	2020-11-23 17:49:21 +01:00
Quentin	aa320aa04a	Merge branch 'master' into feature/website	2020-11-22 19:54:47 +01:00
Quentin	17dc610f8a	Also check hash for < 3KB files	2020-11-22 11:14:46 +01:00
Quentin	435d5f9205	Fix base64/hex checksum comparison	2020-11-22 11:04:33 +01:00
Quentin	fb18f5e17a	Fix wrong http status code	2020-11-21 18:14:02 +01:00
Quentin	28efe341cb	Merge branch 'master' into feature/website	2020-11-21 18:01:50 +01:00
Quentin	b7a377308b	Handle HEAD	2020-11-21 17:58:14 +01:00
Quentin	a88fd49f71	Use handle_get	2020-11-21 17:50:19 +01:00
Quentin	0f33231ee6	We are able to serve a file	2020-11-21 15:15:25 +01:00
Quentin	d4c7f4e374	Fix host to key	2020-11-21 12:01:02 +01:00
Quentin	2f6eca4ef3	Merge remote-tracking branch 'origin/master' into feature/website	2020-11-21 10:52:27 +01:00
Alex Auvolat	a8b3c8fd58	data hexdump in warning	2020-11-20 23:53:54 +01:00
Alex Auvolat	f9be964c3f	Warning when cannot decode entry (data format incompatibilities)	2020-11-20 23:53:54 +01:00
Alex Auvolat	4f7f1d1cb3	less type bounds	2020-11-20 23:53:54 +01:00
Alex Auvolat	68be5072e5	simplify	2020-11-20 23:53:54 +01:00
Alex Auvolat	173f0dbac9	oops	2020-11-20 23:53:54 +01:00
Alex Auvolat	f8a04852a2	Convert bucket table to better CRDT representation	2020-11-20 23:53:54 +01:00
Alex Auvolat	e02e9e035e	Begin improve model to use better CRDTs	2020-11-20 23:53:54 +01:00
Quentin	5b363626f4	Support punnycode	2020-11-20 21:23:32 +01:00
Alex Auvolat	2dc9a48022	Fix reconciliation logic	2020-11-20 20:12:32 +01:00
Alex Auvolat	e9fd265ce6	Slight refactoring to make things clearer with DeletedFilter	2020-11-20 20:11:04 +01:00