jpds/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues 3 Pull requests Packages Projects Releases Wiki Activity
1246 commits 43 branches 48 tags 62 MiB
Commit graph

172 commits

Author SHA1 Message Date
Max Audron 9d44127245
add support for kubernetes service discovery 
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
 2022-03-12 13:05:52 +01:00
Quentin 8df1e186de Functional tests for website endpoints 2022-03-07 17:32:07 +01:00
trinity-1686a f6f8b7f1ad Support for PostObject (#222) 
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html)

- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-02-21 23:02:30 +01:00
Jill dd407e7041
tests: Add garage integration tests (base) 2022-02-10 17:55:49 +01:00
Jill b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156) 
Closes #64.

Reviewed-on: Deuxfleurs/garage#156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
 2022-01-17 10:55:31 +01:00
Alex beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex b1cfd16913
New buckets for 0.6.0: small fixes, including: 
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
 2022-01-04 12:46:41 +01:00
Alex 4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
Alex 0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
Alex 5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
trinity-1686a 1eb972b1ac Add compression using zstd (#173) 
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 11:26:43 +01:00
Alex c94406f428
Improve how node roles are assigned in Garage 
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
 2021-11-16 16:05:53 +01:00
Trinity Pointard 9c58ec28d3 add support for vhost-style s3 bucket 2021-11-16 15:41:41 +01:00
Alex 43e13a501d
Use published netapp crate instead of git repo 2021-10-26 10:36:57 +02:00
Alex de4276202a
Improve CLI, adapt tests, update documentation 2021-10-25 14:21:48 +02:00
Alex 1b450c4b49
Improvements to CLI and various fixes for netapp version 
Discovery via consul, persist peer list to file
 2021-10-22 16:55:24 +02:00
Alex 4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
Alex b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x 2021-05-28 15:29:58 +02:00
Quentin 631c36b3ff S3 API: support ListBuckets 2021-05-03 21:55:30 +02:00
Alex f859d15062 update to v0.2.1 2021-03-19 13:39:18 +01:00
Alex fd8f4caa81 Support old CPUs 2021-03-19 12:19:40 +01:00
Alex dead945c8f Prepare for release 0.2 2021-03-18 19:33:15 +01:00
Alex f4346cc5f4 Update dependencies 2021-03-16 15:58:40 +01:00
Alex 0cd5b2ae19 WIP migrate to tokio 1 2021-03-15 22:36:41 +01:00
Alex 3bf2df622a Time and metadata improvements 2021-03-15 16:21:41 +01:00
Alex a1442f072a Implement garage stats to get info on node contents 2021-03-12 15:40:54 +01:00
Alex 94f3d28774 WIP big refactoring 2021-03-11 16:54:15 +01:00
Alex 445912dc6a Remove migration paths from 0.1 branch 2021-03-10 16:38:31 +01:00
Alex 20e6e9fa20 Update sled & try to debug deadlock (but its in sled...) 2021-02-23 21:27:28 +01:00
Alex e8e4418ca7 Add blake2 and xxhash hash functions 2021-02-23 17:52:28 +01:00
Alex 6a5add3386 Fix build 2021-01-15 19:12:08 +01:00
Alex e818f51073 Forgot a bump 2021-01-15 18:36:51 +01:00
Alex 1d1d497e2b Bump everything to 0.1.1 2021-01-15 17:54:48 +01:00
Quentin c441a358cd Remove unused dependencies 2021-01-15 16:16:32 +01:00
Quentin e8c12072ce Merge branch 'master' into feature/website 2020-12-10 20:12:56 +01:00
Alex 4a5bbbb810 Propose ETag fix 2020-12-05 19:23:46 +01:00
Quentin aa320aa04a Merge branch 'master' into feature/website 2020-11-22 19:54:47 +01:00
Quentin 435d5f9205 Fix base64/hex checksum comparison 2020-11-22 11:04:33 +01:00
Quentin 28efe341cb Merge branch 'master' into feature/website 2020-11-21 18:01:50 +01:00
Quentin a88fd49f71 Use handle_get 2020-11-21 17:50:19 +01:00
Quentin 0f33231ee6 We are able to serve a file 2020-11-21 15:15:25 +01:00
Alex a8b3c8fd58 data hexdump in warning 2020-11-20 23:53:54 +01:00
Quentin 5b363626f4 Support punnycode 2020-11-20 21:23:32 +01:00
Quentin 04f455ff7f Make it compile again 2020-11-19 14:56:00 +01:00
Quentin 4093833ae8 Extract bucket 2020-11-10 09:57:07 +01:00
Alex 54166d2a09 Update cargo.lock 2020-11-08 15:05:28 +01:00
Quentin 0d3bc169ee It compiles! 2020-11-03 12:37:16 +01:00
Quentin cea871d944 Skeleton to the new web API 2020-11-02 15:48:39 +01:00
Alex 6c7f9704ea Implement correct ETag for objects created with PutObject 2020-07-13 16:51:30 +02:00
Alex 86bf4dedac Add support for model migrations 2020-07-08 16:10:53 +02:00
Alex f22ecb60a8 Update to Hyper 0.13.6 that accepts non-Sync streams in wrap_stream. 
Simplifies code and makes it possible to publish on crates.io
 2020-07-07 17:15:53 +02:00
Alex bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00
Alex fbe8fe81f2 Add automatic peer discovery from Consul 2020-06-30 18:33:14 +02:00
Alex b46a7788d1 Implement HTTP ranges in get 2020-05-04 13:09:23 +00:00
Alex d867bbcfb5 Implement DeleteObjects 2020-05-01 15:52:35 +00:00
Alex 3324971701 Slightly improved S3 compatibility 
- ListBucket does not require any of the parameters (delimiter,
    prefix, max-keys, etc)
- URLs are properly percent_decoded
- PutObject and DeleteObject calls now answer correctly
    (empty body, version id in the x-amz-version-id: header)
 2020-05-01 14:30:50 +00:00
Alex f2e05986c4 Starting to be S3 compatible 2020-04-24 17:46:52 +00:00
Alex d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00
Alex e8214cb180 Better concurrency: 
Use Notify instead of stupid sleep in background worker
Use Semaphore to limit concurrent requests in rpc_client
Make more background tasks cancellable
 2020-04-22 16:51:52 +00:00
Alex c0335ac690 Remove a few features in dependencies 2020-04-21 20:37:02 +00:00
Alex cc4f2f1cfb Pretty logging 2020-04-21 12:54:55 +00:00
Alex e325c7f47a Add hostname to node info 2020-04-19 19:08:48 +02:00
Alex 69f1d8fef2 WIP 
TODOs:
- ensure sync goes both way
- finish sending blocks to other nodes when they need them before deleting
 2020-04-17 17:09:57 +02:00
Alex d2814b5c33 TLS works \o/ 
So, the issues were:
- webpki does not support IP addresses as DNS names in URLs,
  so I hacked the HttpsConnector to always provide a fixed string
  as the DNS name for server certificate validation
- the certificate requied a SAN section which was complicated to build
  but eventually the solution is there in genkeys.sh
 2020-04-12 19:00:30 +02:00
Alex d1e8f78b2c Trying to do TLS 2020-04-12 15:51:19 +02:00
Alex 1d786c2c66 Something works 2020-04-09 18:43:53 +02:00
Alex cc580da0ae Some work 2020-04-08 23:01:49 +02:00
Alex bacc76a057 Some work in actually storing things 2020-04-08 22:00:41 +02:00
Alex 90cdffb425 custom data type for hashes and identifiers 2020-04-07 18:10:20 +02:00
Alex 3c36b449a3 Some work 2020-04-06 21:02:15 +02:00
Alex 1a5e6e39af Some more basic work 2020-04-06 19:55:39 +02:00
Alex 7102db1d54 First commit: skeleton for something great 2020-04-05 23:33:42 +02:00