222 commits

Author	SHA1	Message	Date
Maximilien R	1e2cf26373	Implement basic metrics in table	2022-03-14 10:51:17 +01:00
mricher	e349af13a7	Update dependencies and add admin module with metrics ... - Global dependencies updated in Cargo.lock - New module created in src/admin to host: - the (future) admin REST API - the metric collection - add configuration block No metrics implemented yet	2022-03-14 10:51:12 +01:00
Quentin	c00b2c9948	Functional tests for admin commands	2022-03-07 17:32:07 +01:00
Quentin	8df1e186de	Functional tests for website endpoints	2022-03-07 17:32:07 +01:00
Quentin	2ef60b8417	Functional test for multipart endpoints	2022-03-07 17:32:07 +01:00
Quentin	1e639ec67c	Functional test for ListMultipartUploads	2022-03-07 17:32:07 +01:00
Quentin	cfea1e0315	Functional tests for bucket endpoints	2022-03-07 17:32:02 +01:00
Quentin	05eb79929e	Functional tests for object operations	2022-03-07 17:05:10 +01:00
Quentin	0f4e0e8bb9	Move ListObjects tests to Rust	2022-03-07 17:05:10 +01:00
Jill	84613e66a2	garage(tests): Remove RNG stuff	2022-02-11 10:50:55 +01:00
Jill	c8b30ebc79	garage(tests): Remove superfluous test	2022-02-10 17:55:51 +01:00
Jill	d7decda3f4	garage(tests): Add random suffix to created buckets.	2022-02-10 17:55:50 +01:00
Jill	cd13ea461b	garage(tests): Add some unsafe-usage doc and tweaks	2022-02-10 17:55:50 +01:00
Quentin	5d19f3d2d7	Add integration tests to Drone	2022-02-10 17:55:50 +01:00
Jill	3baa841d6f	tests: Fix garage integration test	2022-02-10 17:55:49 +01:00
Jill	dd407e7041	tests: Add garage integration tests (base)	2022-02-10 17:55:49 +01:00
Alex	ae2f32baf1	Hide deleted key in bucket info (fix #211)	2022-02-02 17:12:48 +01:00
Alex	9eb211948e	Allow setting index document and error document on the CLI	2022-01-13 14:25:19 +01:00
Quentin	b4592a00fe	Implement ListMultipartUploads (#171) ... Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: #171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>	2022-01-12 19:04:55 +01:00
Alex	17ea28a438	Fix trivial bug in CLI	2022-01-10 12:38:33 +01:00
Alex	fb1e31add0	Small CLI changes	2022-01-05 16:28:46 +01:00
Alex	8395030e48	Implement CreateBucket	2022-01-05 15:56:48 +01:00
Alex	9431090b1e	Implement key allow|deny --create-bucket	2022-01-05 15:12:59 +01:00
Alex	677ab60cc1	Small changes in key model and refactoring	2022-01-04 18:59:17 +01:00
Alex	df35feba18	New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates	2022-01-04 12:53:14 +01:00
Alex	1bcd6fabbd	New buckets for 0.6.0: small changes ... - Fix bucket delete - fix merge of bucket creation date - Replace deletable with option in aliases Rationale: if two aliases point to conflicting bucket, resolving by making an arbitrary choice risks making data accessible when it shouldn't be. We'd rather resolve to deleting the alias until someone puts it back.	2022-01-04 12:52:47 +01:00
Alex	e59c23a69d	Refactor logic for setting/unsetting aliases	2022-01-04 12:52:46 +01:00
Alex	2140cd7205	Remove website redirects	2022-01-04 12:52:46 +01:00
Alex	beeef4758e	Some movement of helper code and refactoring of error handling	2022-01-04 12:52:46 +01:00
Alex	d8ab5bdc3e	New buckets for 0.6.0: fix model and migration	2022-01-04 12:47:28 +01:00
Alex	b1cfd16913	New buckets for 0.6.0: small fixes, including: ... - ensure bucket names are correct aws s3 names - when making aliases, ensure timestamps of links in both ways are the same - fix small remarks by trinity - don't have a separate website_access field	2022-01-04 12:46:41 +01:00
Alex	5db600e231	More complete output to bucket info and key info	2022-01-04 12:46:41 +01:00
Alex	4d30e62db4	New buckets for 0.6.0: migration code and build files	2022-01-04 12:46:13 +01:00
Alex	0bbb6673e7	Model changes	2022-01-04 12:45:52 +01:00
Alex	53f71b3a57	Implement bucket alias and bucket unalias	2022-01-04 12:45:51 +01:00
Alex	5b1117e582	New model for buckets	2022-01-04 12:45:46 +01:00
Alex	8f6026de5e	Make table name a const in trait	2021-12-15 15:39:10 +01:00
Alex	c94406f428	Improve how node roles are assigned in Garage ... - change the terminology: the network configuration becomes the role table, the configuration of a nodes becomes a node's role - the modification of the role table takes place in two steps: first, changes are staged in a CRDT data structure. Then, once the user is happy with the changes, they can commit them all at once (or revert them). - update documentation - fix tests - implement smarter partition assignation algorithm This patch breaks the format of the network configuration: when migrating, the cluster will be in a state where no roles are assigned. All roles must be re-assigned and commited at once. This migration should not pose an issue.	2021-11-16 16:05:53 +01:00
Alex	74a7a550eb	Safety: never voluntarily delete block in 10min interval after RC reaches zero	2021-11-08 15:47:47 +01:00
Alex	2090a6187f	Add tranquilizer mechanism to improve on token bucket mechanism	2021-11-04 13:26:59 +01:00
Jill	bef6d627b0	Add environment variables equivalents for some CLI options.	2021-11-03 16:00:57 +01:00
Alex	6b47c294f5	Refactoring on repair commands	2021-10-27 11:14:55 +02:00
Trinity Pointard	28c015d9ff	add cli parameter to verify local bloc integrity ... reuse code for listing local blocks add disk i/o speed limit on integrity check	2021-10-27 10:31:03 +02:00
Alex	3e7f766d95	CLI: default rpc_host	2021-10-26 11:36:30 +02:00
Alex	43e13a501d	Use published netapp crate instead of git repo	2021-10-26 10:36:57 +02:00
Alex	ada7899b24	Fix clippy lints (fix #121)	2021-10-26 10:20:05 +02:00
Alex	de4276202a	Improve CLI, adapt tests, update documentation	2021-10-25 14:21:48 +02:00
Alex	1b450c4b49	Improvements to CLI and various fixes for netapp version ... Discovery via consul, persist peer list to file	2021-10-22 16:55:24 +02:00
Alex	4067797d01	First port of Garage to Netapp	2021-10-22 15:55:18 +02:00
Quentin	dc017a0cab	Build Garage with Nix	2021-10-19 16:56:07 +02:00
Alex	1acf7e4c66	Fix git_version!() when not in git repo (fix #100)	2021-10-11 14:26:54 +02:00
Trinity Pointard	b568765c75	try parsing rpc-host command-line parameter	2021-06-01 23:53:58 +02:00
Trinity Pointard	e9c265e9dc	format garage status output better	2021-06-01 19:12:27 +02:00
Trinity Pointard	289521886b	make most changes suggested during install-party	2021-05-29 21:37:49 +02:00
Alex	b9127dd6f8	Prepare for v0.3.0 and add migration path from v0.2.1.x	2021-05-28 15:29:58 +02:00
Alex	ddb2b29bfd	Rename datacenters into zones (doc not yet updated)	2021-05-28 14:07:36 +02:00
Alex	b490ebc7f6	Many improvements on ring/replication and its configuration: ... - Explicit "replication_mode" configuration parameters that takes either "none", "2" or "3" as values, instead of letting user configure replication factor themselves. These are presets whose corresponding replication/quorum values can be found in replication/mode.rs - Explicit support for single-node and two-node deployments (number of nodes must be at least "replication_mode", with "none" we can have only one node) - Ring is now stored much more compactly with 256*8 + n*32 bytes, instead of 256*32 bytes - Support for gateway-only nodes that do not store data (these nodes still need a metadata_directory to store the list of bucket and keys since those are stored on all nodes; it also technically needs a data_directory to start but it will stay empty unless we have bugs)	2021-05-28 14:07:36 +02:00
Trinity Pointard	e4b9e4e24d	rename types to CamelCase	2021-05-03 22:15:09 +02:00
Trinity Pointard	6644df6b96	fix clippy warnings on garage	2021-05-03 22:11:42 +02:00
Alex	575726358c	Tune Sled configuration ... - Make sled cache size and flush interval configurable - Set less agressive default values: - cache size 128MB instead of 1GB - Flush interval 2 seconds instead of .5 seconds	2021-05-03 17:27:43 +02:00
Trinity Pointard	74373aebcf	make most requested changes	2021-04-27 16:47:08 +02:00
Trinity Pointard	5c4b2cec3c	document garage crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	1e3df189d0	document api crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	a2e1617d84	document web crate	2021-04-27 16:37:10 +02:00
Trinity Pointard	8e0524ae15	document rpc crate	2021-04-27 16:37:10 +02:00
Alex	9ced9f78dc	Improve bootstraping: do it regularly; persist peer list	2021-04-27 16:37:08 +02:00
Alex	f859d15062	update to v0.2.1	2021-03-19 13:39:18 +01:00
Alex	4c26a0b9c1	Update Cargo.toml files with AGPL license info	2021-03-18 21:59:17 +01:00
Alex	a1014224d3	garage node configure --replace <old_node_id> <new_node_id>	2021-03-18 21:49:12 +01:00
Alex	dead945c8f	Prepare for release 0.2	2021-03-18 19:33:15 +01:00
Alex	4eb16e8863	Allow to import keys from previous Garage instance	2021-03-18 19:24:59 +01:00
Alex	3fadc5cbbd	Small changes	2021-03-16 16:35:10 +01:00
Alex	f4346cc5f4	Update dependencies	2021-03-16 15:58:40 +01:00
Alex	515029d026	Refactor code	2021-03-16 11:43:58 +01:00
Alex	6a8439fd13	Some improvements in background worker but we terminate late	2021-03-15 23:14:12 +01:00
Alex	0cd5b2ae19	WIP migrate to tokio 1	2021-03-15 22:36:41 +01:00
Alex	4d4117f2b4	Refactor block resync loop; make workers infaillible	2021-03-15 20:09:44 +01:00
Alex	667e4e72a8	Small fixes	2021-03-15 19:51:16 +01:00
Alex	5ee1d956b6	Allow manipulation of keys by their shorthand in the CLI	2021-03-15 19:14:26 +01:00
Alex	0290afe1f8	Make block rc code more understandable	2021-03-15 18:27:26 +01:00
Alex	3bf2df622a	Time and metadata improvements	2021-03-15 16:21:41 +01:00
Alex	097c339d98	Fix race condition	2021-03-15 15:26:29 +01:00
Alex	f4aad8fe6e	cargo fmt	2021-03-12 18:16:03 +01:00
Alex	5ab33fddac	Refactor CLI and prettify CLI outpu	2021-03-12 18:12:31 +01:00
Alex	a1442f072a	Implement garage stats to get info on node contents	2021-03-12 15:40:54 +01:00
Alex	046b649bcc	(not well tested) use merkle tree for sync	2021-03-11 18:28:27 +01:00
Alex	94f3d28774	WIP big refactoring	2021-03-11 16:54:15 +01:00
Alex	8d63738cb0	Checkpoint: add merkle tree in data table	2021-03-11 13:47:21 +01:00
Alex	445912dc6a	Remove migration paths from 0.1 branch	2021-03-10 16:38:31 +01:00
Alex	f319a7d374	Refactor model stuff, including cleaner CRDTs	2021-03-10 16:21:56 +01:00
Alex	6a3dcf3974	Rename n_tokens into capacity	2021-03-10 14:52:03 +01:00
Alex	20e6e9fa20	Update sled & try to debug deadlock (but its in sled...)	2021-02-23 21:27:28 +01:00
Alex	80892df8cc	Some refactoring	2021-02-21 13:11:10 +01:00
Alex	1d1d497e2b	Bump everything to 0.1.1	2021-01-15 17:54:48 +01:00
Quentin	f496e41ef4	Replace an already done check by unreachable!()	2021-01-15 15:44:44 +01:00
Quentin	011ff87b5f	Push update	2020-12-15 13:23:22 +01:00
Quentin	3bc4d57a0f	First implementation of the CLI	2020-12-15 12:48:24 +01:00
Quentin	a3566e49da	Start to implement Website CLI	2020-12-14 21:50:40 +01:00
Quentin	d0eb6a457f	Migrate RPC to new schema	2020-12-14 21:46:49 +01:00
Quentin	96388acf23	Implement migration	2020-12-12 21:35:29 +01:00
Quentin	e1ce2b228a	WIP table migration	2020-12-12 17:00:31 +01:00
Quentin	51d0c14e44	CLI structure	2020-12-10 18:13:32 +01:00
Quentin	28efe341cb	Merge branch 'master' into feature/website	2020-11-21 18:01:50 +01:00
Quentin	2f6eca4ef3	Merge remote-tracking branch 'origin/master' into feature/website	2020-11-21 10:52:27 +01:00
Alex	173f0dbac9	oops	2020-11-20 23:53:54 +01:00
Alex	f8a04852a2	Convert bucket table to better CRDT representation	2020-11-20 23:53:54 +01:00
Alex	e02e9e035e	Begin improve model to use better CRDTs	2020-11-20 23:53:54 +01:00
Alex	e9fd265ce6	Slight refactoring to make things clearer with DeletedFilter	2020-11-20 20:11:04 +01:00
Quentin	fc427b0b66	Merge branch 'master' into feature/website	2020-11-19 14:39:30 +01:00
Quentin	27795a390c	Fix formatting	2020-11-10 09:59:52 +01:00
Alex	a50fa70d45	Refactor error management in API part	2020-11-08 15:05:28 +01:00
Quentin	0d3bc169ee	It compiles!	2020-11-03 12:37:16 +01:00
Quentin	cea871d944	Skeleton to the new web API	2020-11-02 15:48:39 +01:00
Alex	86bf4dedac	Add support for model migrations	2020-07-08 16:10:53 +02:00
Alex	3b0b11085e	Add versions to dependencies	2020-07-07 14:18:47 +02:00
Alex	cc65cdc0fe	Add license, description and repository to .toml files	2020-07-07 14:14:58 +02:00
Alex	bec26a1312	Rename garage_core to garage_model	2020-07-07 13:59:22 +02:00
Alex	fbe8fe81f2	Add automatic peer discovery from Consul	2020-06-30 18:33:14 +02:00
Alex	fa13cf6996	Repair: do not mark deleted when upstream object is not found ... With the previous behaviour, repairing could see some data as absent and decide that the object or version was deleted, thus going on to delete the version and blocks. In the case where read_quorum + write_quorum <= replication_factor however, entries may not yet be returned by the get, thus data would have been deleted that should hot have been. The new behavior is more cautious and just skips the entry when the warning is emitted.	2020-05-04 13:30:42 +00:00
Alex	0a283e4e70	Fix deletion propagation	2020-04-26 18:59:17 +00:00
Alex	0e49e0c8b5	Add key table to repair procedure	2020-04-26 16:22:22 +00:00
Alex	d8f5e643bc	Split code for modular compilation	2020-04-24 10:10:01 +00:00