1bcd6fabbd
New buckets for 0.6.0: small changes
...
- Fix bucket delete
- fix merge of bucket creation date
- Replace deletable with option in aliases
Rationale: if two aliases point to conflicting bucket, resolving
by making an arbitrary choice risks making data accessible when it
shouldn't be. We'd rather resolve to deleting the alias until
someone puts it back.
2022-01-04 12:52:47 +01:00
ba7f268b99
Rename and change query filters
2022-01-04 12:52:46 +01:00
e59c23a69d
Refactor logic for setting/unsetting aliases
2022-01-04 12:52:46 +01:00
2140cd7205
Remove website redirects
2022-01-04 12:52:46 +01:00
beeef4758e
Some movement of helper code and refactoring of error handling
2022-01-04 12:52:46 +01:00
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration
2022-01-04 12:47:28 +01:00
c7d5c73244
Add must_use to some CRDT functions
2022-01-04 12:47:28 +01:00
b76d0580a0
Fix forgotten flag
2022-01-04 12:47:28 +01:00
87121dce9d
New buckets for 0.6.0: documentation and build files
2022-01-04 12:47:06 +01:00
b1cfd16913
New buckets for 0.6.0: small fixes, including:
...
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
same
- fix small remarks by trinity
- don't have a separate website_access field
2022-01-04 12:46:41 +01:00
5db600e231
More complete output to bucket info and key info
2022-01-04 12:46:41 +01:00
4d30e62db4
New buckets for 0.6.0: migration code and build files
2022-01-04 12:46:13 +01:00
0bbb6673e7
Model changes
2022-01-04 12:45:52 +01:00
53f71b3a57
Implement bucket alias and bucket unalias
2022-01-04 12:45:51 +01:00
5b1117e582
New model for buckets
2022-01-04 12:45:46 +01:00
8f6026de5e
Make table name a const in trait
2021-12-15 15:39:10 +01:00
ca7b438f3f
less strict ListBuckets ( #178 )
...
fix #175
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#178
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:36 +01:00
1eb972b1ac
Add compression using zstd ( #173 )
...
fix #27
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 11:26:43 +01:00
60d4459926
BucketWebsite ( #174 )
...
fix #77
this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though.
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#174
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 10:41:39 +01:00
c4ac8835d3
add proper request router for s3 api ( #163 )
...
fix #161
Current request router was organically grown, and is getting messier and messier with each addition.
This router cover exaustively existing API endpoints (with exceptions listed in [#161(comment)](Deuxfleurs/garage#161 (comment) ) either because new and old api endpoint can't feasabily be differentied, or it's more lambda than s3).
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#163
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-06 15:17:47 +01:00
7f26ed55cd
Improved handling of HTTP ranges
...
- correct HTTP code when range syntax is invalid (fix #140 )
- when multiple ranges are given, simply ignore and send whole file
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#157
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-11-29 11:52:42 +01:00
8811bb08e6
In ListBuckets, hide entry if no perms
2021-11-22 12:10:28 +01:00
c94406f428
Improve how node roles are assigned in Garage
...
- change the terminology: the network configuration becomes the role
table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
changes are staged in a CRDT data structure. Then, once the user is
happy with the changes, they can commit them all at once (or revert
them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm
This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
2021-11-16 16:05:53 +01:00
53888995bd
update doc and comments
2021-11-16 15:41:41 +01:00
396fe4c702
clippy
2021-11-16 15:41:41 +01:00
02158ee666
fix issue where list on vhost-bucket would list bucket instead of bucket content
2021-11-16 15:41:41 +01:00
9c58ec28d3
add support for vhost-style s3 bucket
2021-11-16 15:41:41 +01:00
ad7ab31411
Implement GC delay for table data
2021-11-08 15:47:47 +01:00
74a7a550eb
Safety: never voluntarily delete block in 10min interval after RC reaches zero
2021-11-08 15:47:47 +01:00
cc255d46cd
Refactor and comment table GC logic
2021-11-08 15:47:44 +01:00
9d7535c3f5
allow missing bootstrap_peers in garage.toml
2021-11-05 16:36:25 +01:00
e8811f7c9d
Request strategy: don't launch all 3 requests if not needed
2021-11-04 16:19:27 +01:00
2090a6187f
Add tranquilizer mechanism to improve on token bucket mechanism
2021-11-04 13:26:59 +01:00
6f13d083ab
Add semaphore to limit RAM used by buffered outgoing requests
2021-11-03 18:02:57 +01:00
8c4f418fe8
Fix peer list persistence: do not forget previous peers
2021-11-03 17:34:44 +01:00
bef6d627b0
Add environment variables equivalents for some CLI options.
2021-11-03 16:00:57 +01:00
69b89fb46d
Fix race in block resync
2021-10-27 12:01:12 +02:00
6b47c294f5
Refactoring on repair commands
2021-10-27 11:14:55 +02:00
28c015d9ff
add cli parameter to verify local bloc integrity
...
reuse code for listing local blocks
add disk i/o speed limit on integrity check
2021-10-27 10:31:03 +02:00
3e7f766d95
CLI: default rpc_host
2021-10-26 11:36:30 +02:00
43e13a501d
Use published netapp crate instead of git repo
2021-10-26 10:36:57 +02:00
ada7899b24
Fix clippy lints ( fix #121 )
2021-10-26 10:20:05 +02:00
df8a4068d9
Refactor block manager code, and hopefully fix deadlock
2021-10-25 14:21:51 +02:00
de4276202a
Improve CLI, adapt tests, update documentation
2021-10-25 14:21:48 +02:00
1b450c4b49
Improvements to CLI and various fixes for netapp version
...
Discovery via consul, persist peer list to file
2021-10-22 16:55:24 +02:00
4067797d01
First port of Garage to Netapp
2021-10-22 15:55:18 +02:00
dc017a0cab
Build Garage with Nix
2021-10-19 16:56:07 +02:00
1acf7e4c66
Fix git_version!() when not in git repo ( fix #100 )
2021-10-11 14:26:54 +02:00
f6060b92aa
Fix HTTP return code for DeleteObject ( fix #98 )
2021-10-11 14:24:49 +02:00
f3a097abdf
WIP: try to fix #93 , and improve S3 ListObjects (v1 and v2) API calls
2021-10-11 11:15:47 +02:00
fa394dcd27
Support pkcs8 private keys (allowing for ed25519 to be used for rpc)
2021-07-06 11:16:01 +02:00
30a7dee920
exit when inconsistent level of replication is detected ( #92 )
...
fix #88
Authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#92
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-06-02 13:30:39 +02:00
b568765c75
try parsing rpc-host command-line parameter
2021-06-01 23:53:58 +02:00
e9c265e9dc
format garage status output better
2021-06-01 19:12:27 +02:00
289521886b
make most changes suggested during install-party
2021-05-29 21:37:49 +02:00
b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x
2021-05-28 15:29:58 +02:00
ddb2b29bfd
Rename datacenters into zones (doc not yet updated)
2021-05-28 14:07:36 +02:00
b490ebc7f6
Many improvements on ring/replication and its configuration:
...
- Explicit "replication_mode" configuration parameters that takes
either "none", "2" or "3" as values, instead of letting user configure
replication factor themselves. These are presets whose corresponding
replication/quorum values can be found in replication/mode.rs
- Explicit support for single-node and two-node deployments
(number of nodes must be at least "replication_mode", with "none"
we can have only one node)
- Ring is now stored much more compactly with 256*8 + n*32 bytes,
instead of 256*32 bytes
- Support for gateway-only nodes that do not store data
(these nodes still need a metadata_directory to store the list
of bucket and keys since those are stored on all nodes; it also
technically needs a data_directory to start but it will stay
empty unless we have bugs)
2021-05-28 14:07:36 +02:00
5fdabf3e75
Add basic support for the "Versioning" command
2021-05-14 22:33:26 +02:00
6ccffc3162
Improved XML serialization
...
- Use quick_xml and serde for all XML response returned by the S3 API.
- Include tests for all structs used to generate XML
- Remove old manual XML escaping function which was unsafe
2021-05-06 22:37:15 +02:00
e4b9e4e24d
rename types to CamelCase
2021-05-03 22:15:09 +02:00
6644df6b96
fix clippy warnings on garage
2021-05-03 22:11:42 +02:00
f8ae8fc4be
fix clippy warnings on web
2021-05-03 22:11:42 +02:00
84856e84e5
fix clippy warnings on api
2021-05-03 22:11:41 +02:00
4a1e079e8f
fix clippy warnings on model
2021-05-03 22:11:41 +02:00
f5a0cf0414
fix clippy warnings on table
2021-05-03 22:11:41 +02:00
f05bb111c2
fix clippy warnings on util and rpc
2021-05-03 22:11:41 +02:00
631c36b3ff
S3 API: support ListBuckets
2021-05-03 21:55:30 +02:00
ee2a3d363b
Remove STREAMING-AWS4-HMAC-SHA256-PAYLOAD (see #64 )
2021-05-03 17:30:40 +02:00
575726358c
Tune Sled configuration
...
- Make sled cache size and flush interval configurable
- Set less agressive default values:
- cache size 128MB instead of 1GB
- Flush interval 2 seconds instead of .5 seconds
2021-05-03 17:27:43 +02:00
dcfc32cf85
Many S3 compatibility improvements:
...
- return XML errors
- implement AuthorizationHeaderMalformed error to redirect clients to
correct location (used by minio client)
- implement GetBucketLocation
- fix DeleteObjects XML parsing and response
2021-04-28 01:05:40 +02:00
642186c530
Fix #59 (& issue)
2021-04-27 23:10:43 +02:00
119217f9f6
change a few comments
2021-04-27 16:53:47 +02:00
2812a027ea
change some more comments and revert changes on TableSchema
2021-04-27 16:49:07 +02:00
74373aebcf
make most requested changes
2021-04-27 16:47:08 +02:00
16300bbd89
remove useless comment
2021-04-27 16:44:01 +02:00
5c4b2cec3c
document garage crate
2021-04-27 16:37:10 +02:00
1e3df189d0
document api crate
2021-04-27 16:37:10 +02:00
a2e1617d84
document web crate
2021-04-27 16:37:10 +02:00
67585a4ffa
attempt at documenting model crate
2021-04-27 16:37:10 +02:00
b437610812
attempt at documenting table crate
2021-04-27 16:37:10 +02:00
f871689571
run cargo fmt on util and make missing doc warning
2021-04-27 16:37:10 +02:00
8e0524ae15
document rpc crate
2021-04-27 16:37:10 +02:00
f9bd2d8fb7
document util crate
2021-04-27 16:37:10 +02:00
6b2b400292
small simplify
2021-04-27 16:37:09 +02:00
8c33d565d6
Merge discovery loop with consul
2021-04-27 16:37:09 +02:00
948e44a3f6
cargo fmt
2021-04-27 16:37:09 +02:00
3e2e38c830
Print stats
2021-04-27 16:37:09 +02:00
2e53e31cdd
Cargo fmt
2021-04-27 16:37:09 +02:00
64b91c2645
Keep old data
2021-04-27 16:37:09 +02:00
e16077f40a
Persist directly and not in background
2021-04-27 16:37:09 +02:00
9ced9f78dc
Improve bootstraping: do it regularly; persist peer list
2021-04-27 16:37:08 +02:00
f859d15062
update to v0.2.1
2021-03-19 13:39:18 +01:00
fd8f4caa81
Support old CPUs
2021-03-19 12:19:40 +01:00
4c26a0b9c1
Update Cargo.toml files with AGPL license info
2021-03-18 21:59:17 +01:00
a1014224d3
garage node configure --replace <old_node_id> <new_node_id>
2021-03-18 21:49:12 +01:00
f17cb6c969
resolve domain to multiple addresses
...
And warn instead of failling when a domain can't be resolved
2021-03-18 21:04:30 +01:00
c8a7ce5cdf
remove domain resolution for *_bind_addr
2021-03-18 19:47:51 +01:00
81e9db783f
simplify addresse deserialialiser and limit allocations
2021-03-18 19:47:51 +01:00
ae3b7029a9
add support for using domain name in configuration
2021-03-18 19:47:51 +01:00
6edbc65847
Add trinity's comment in the code
2021-03-18 19:46:43 +01:00
bfa0ff8f82
Merge pull request 'add support for caching headers' ( #49 ) from trinity-1686a/garage:cache-headers into master
...
Reviewed-on: Deuxfleurs/garage#49
2021-03-18 19:45:02 +01:00
dead945c8f
Prepare for release 0.2
2021-03-18 19:33:15 +01:00
4eb16e8863
Allow to import keys from previous Garage instance
2021-03-18 19:24:59 +01:00
b4c903371c
add support for caching headers
2021-03-18 15:46:33 +01:00
390ab02f41
Todo make a test for the Merkle updater
2021-03-16 20:13:07 +01:00
7b10245dfb
Leader-based GC
2021-03-16 18:42:33 +01:00
08bcd51956
GC object table in a specific case
2021-03-16 16:51:15 +01:00
3fadc5cbbd
Small changes
2021-03-16 16:35:10 +01:00
f4346cc5f4
Update dependencies
2021-03-16 15:58:40 +01:00
2a41b82384
Simpler Merkle & sync
2021-03-16 12:18:03 +01:00
0aad2f2e06
some reordering
2021-03-16 11:47:39 +01:00
515029d026
Refactor code
2021-03-16 11:43:58 +01:00
1d9961e411
Simplify replication logic
2021-03-16 11:14:27 +01:00
6a8439fd13
Some improvements in background worker but we terminate late
2021-03-15 23:14:12 +01:00
0cd5b2ae19
WIP migrate to tokio 1
2021-03-15 22:36:41 +01:00
4d4117f2b4
Refactor block resync loop; make workers infaillible
2021-03-15 20:09:44 +01:00
667e4e72a8
Small fixes
2021-03-15 19:51:16 +01:00
642bed601f
Make it case-insensitive
2021-03-15 19:16:42 +01:00
5ee1d956b6
Allow manipulation of keys by their shorthand in the CLI
2021-03-15 19:14:26 +01:00
537f652fec
Tiny things
2021-03-15 18:40:27 +01:00
0290afe1f8
Make block rc code more understandable
2021-03-15 18:27:26 +01:00
3bf2df622a
Time and metadata improvements
2021-03-15 16:21:41 +01:00
097c339d98
Fix race condition
2021-03-15 15:26:29 +01:00
bdcbdd1cd8
Fix list API bug
2021-03-15 14:46:37 +01:00
9b118160a8
Optim & refactor
2021-03-12 22:06:56 +01:00
831eb35763
cargo fmt
2021-03-12 21:52:19 +01:00
c475471e7a
Implement table gc, currently for block_ref and version only
2021-03-12 19:57:37 +01:00
f4aad8fe6e
cargo fmt
2021-03-12 18:16:03 +01:00
5ab33fddac
Refactor CLI and prettify CLI outpu
2021-03-12 18:12:31 +01:00
a1442f072a
Implement garage stats to get info on node contents
2021-03-12 15:40:54 +01:00
cbe7e1a66a
Move table rpc client out of tableaux
2021-03-12 15:07:23 +01:00
8860aa19b8
Make syncer have its own rpc client/server
2021-03-12 15:05:26 +01:00
1fea257291
Don't sync at beginning
2021-03-12 14:51:17 +01:00
7fdaf7aef0
Fix merkle updater not being notified; improved logging
2021-03-12 14:37:46 +01:00
1ec49980ec
whoops
2021-03-11 19:30:24 +01:00
3f7a496355
More security: don't delete stuff too easily
2021-03-11 19:06:27 +01:00
f7c2cd1cd7
Add comment, and also whoops, this wasn't doing what we expected
2021-03-11 18:56:18 +01:00
fae5104a2c
Add a nice warning
2021-03-11 18:50:32 +01:00
db7a9d4948
Tiny changes
2021-03-11 18:45:26 +01:00
046b649bcc
(not well tested) use merkle tree for sync
2021-03-11 18:28:27 +01:00
94f3d28774
WIP big refactoring
2021-03-11 16:54:15 +01:00
8d63738cb0
Checkpoint: add merkle tree in data table
2021-03-11 13:47:21 +01:00
3214dd52dd
Very minor changes
2021-03-10 21:50:09 +01:00
af7600f989
Correctly implement CompleteMultipartUpload with etag check of parts
2021-03-10 17:01:05 +01:00
445912dc6a
Remove migration paths from 0.1 branch
2021-03-10 16:38:31 +01:00
0fd7df8fa0
Switch to blake2 sum for identifying blocks by their data
2021-03-10 16:33:31 +01:00
2afd2c81ba
Change hash function to blake2 for partition keys based on strings
2021-03-10 16:23:57 +01:00
f319a7d374
Refactor model stuff, including cleaner CRDTs
2021-03-10 16:21:56 +01:00
6a3dcf3974
Rename n_tokens into capacity
2021-03-10 14:52:03 +01:00
