4b4f2000f4
lifecycle: fix SkipBucket bug
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build was killed
2023-09-06 16:34:07 +02:00
51eac97260
update version to 0.8.4
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is failing
2023-09-05 23:28:12 +02:00
8e0c020bb9
lifecycle worker: correct small clippy lints
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-09-04 11:33:44 +02:00
1cdc321e28
lifecycle worker: don't get stuck on non-existent bucket
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-08-31 11:36:30 +02:00
f579d6d9b4
lifecycle worker: fix potential inifinite loop
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-08-31 11:29:54 +02:00
a00a52633f
lifecycle worker: add log message when starting
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-31 11:25:14 +02:00
adbf5925de
lifecycle worker: use queue_insert and process objects in batches
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-31 11:19:26 +02:00
1cfcc61de8
lifecycle worker: mitigate potential bugs + refactoring
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-31 00:28:37 +02:00
b2f679675e
lifecycle worker: take into account disabled rules
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-30 23:52:09 +02:00
01c327a07a
lifecycle worker: avoid building chrono's serde feature
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-30 23:46:15 +02:00
a1d57283c0
bucket_table: bucketparams::new doesn't need to be pub
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-08-30 20:07:14 +02:00
d2e94e36d6
lifecycle config: add missing line in merge() and remove tracing
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-30 20:05:53 +02:00
75ccc5a95c
lifecycle config: store date as given, try to debug
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-30 20:02:07 +02:00
7200954318
lifecycle worker: add logging
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-08-30 14:54:52 +02:00
0f1849e1ac
lifecycle worker: launch with the rest of Garage
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-08-30 14:51:08 +02:00
da8b224e24
lifecycle worker: skip entire bucket when no lifecycle config is set
2023-08-30 14:38:19 +02:00
2996dc875f
lifecycle worker: implement main functionality
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-30 14:29:03 +02:00
a2e0e34db5
lifecycle: skeleton for lifecycle worker
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-08-30 12:41:11 +02:00
f7b409f114
use a NaiveDate in data model, it serializes to string (iso 8601 format)
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-08-30 11:24:01 +02:00
abf011c290
lifecycle: implement validation into garage's internal data structure
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-29 18:22:03 +02:00
0b83e0558e
bucket_table: data model for lifecycle configuration
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-29 17:00:41 +02:00
2e90e1c124
Merge branch 'main' into next
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build was killed
continuous-integration/drone Build is passing
2023-08-29 11:32:42 +02:00
cece1be1bb
bump version to 0.8.3
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
2023-08-28 13:17:26 +02:00
8ef42c9609
admin docs: reformatting, key admin: add check
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-06-14 17:19:25 +02:00
90b2d43eb4
Merge branch 'main' into next
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2023-06-13 17:14:11 +02:00
942c1f1bfe
multipart uploads: save timestamp
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2023-06-13 10:48:22 +02:00
3d477906d4
properly delete multipart uploads after completion
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-06-09 17:13:27 +02:00
412ab77b08
comments and clippy lint fixes
2023-06-09 16:23:37 +02:00
511e07ecd4
fix mpu counter (add missing workers) and report info at appropriate places
2023-06-09 16:23:37 +02:00
82e75c0e29
Adapt S3 API code to use new multipart upload models
...
- Create and PutPart
- completemultipartupload
- upload part copy
- list_parts
2023-06-09 16:23:37 +02:00
38d6ac4295
New multipart upload table layout
2023-06-09 16:23:37 +02:00
e7e164a280
Make fsync an option for meta and data
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2023-06-09 16:23:21 +02:00
19639705e6
Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaults
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2023-05-17 14:30:53 +02:00
Jonathan Davies
c783194e8b
*: apply clippy recommendations.
continuous-integration/drone/pr Build is passing
2023-05-09 20:49:34 +01:00
d2deee0b8b
Declare garage crates using workspace.dependencies
...
This will allow to really disable "sled" feature without declaring
`default-features = false` in every Cargo.toml where garage_db and
garage_model is used.
See https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table
2023-05-09 08:46:15 +00:00
000006d689
obsolete clippy lints
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build was killed
2023-03-13 18:50:07 +01:00
0a1ddcf630
Prepare for v0.8.2
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2023-03-13 18:46:31 +01:00
b8123fb6cd
Clearer error message when LMDB has oom error ( fix #517 )
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-03-06 11:38:49 +01:00
656b8d42de
secrets can be passed directly in config, as file, or as env
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-02-03 15:27:39 +01:00
c56794655e
Fix fmt
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-01-26 17:27:03 +01:00
8e93d69974
More clippy fixes
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2023-01-26 17:26:32 +01:00
590a0a8450
Merge branch 'main' into k2v-watch-range-2
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is failing
2023-01-26 16:46:40 +01:00
Jonathan Davies
20c1cdf662
Cargo.toml: Loosen tracing dependency to just 0.1.
2023-01-26 11:13:11 +00:00
Jonathan Davies
f952e37ba7
{model,util}/Cargo.toml: Updated blake2 from 0.9 to 0.10.
2023-01-26 11:13:11 +00:00
Jonathan Davies
36944f1839
Cargo.toml: Updated base64 from 0.13 to 0.21.
2023-01-26 11:13:07 +00:00
Jonathan Davies
5c3075fe01
Cargo.toml: Updated zstd from 0.9 to 0.12.
2023-01-23 18:08:14 +00:00
dac254a6e7
Merge branch 'main' into k2v-watch-range-2
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-01-11 17:09:37 +01:00
638c5a3ce0
PollRange: add extra RPC delay after quorum is achieved,
...
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
to give a chance to the 3rd node to respond
2023-01-11 16:12:07 +01:00
5b5ca63cf6
Poll cleanup
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-01-11 15:17:27 +01:00
bba13f40fc
Correctly return bad requests when seeh marker is invalid
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build was killed
2023-01-11 12:27:19 +01:00
ba384e61c0
PollRange: return immediately if no seen marker is provided
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-11 12:03:17 +01:00
09a3dad0f2
Lock once for insert_many
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-01-11 11:35:36 +01:00
b83517d521
Implement PollRange API endpoint
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-10 15:22:25 +01:00
43fd6c1526
PollRange RPC
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-10 12:54:24 +01:00
789540ca37
Type definition for range seen marker
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-01-10 11:59:57 +01:00
9f5419f465
Make K2V item timestamps globally increasing on each node
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-01-10 11:03:52 +01:00
a48e2e0cb2
K2V: Subscription to ranges of items
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-10 10:30:59 +01:00
f2106c2733
Implement rpc_secret_file
continuous-integration/drone/pr Build is passing
2023-01-04 18:35:10 +01:00
f3f27293df
Uniform framework for bg variable management
continuous-integration/drone/push Build is passing
2023-01-04 13:07:13 +01:00
8d5505514f
Make it explicit when using nonversioned encoding
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-03 15:27:36 +01:00
426d8784da
cleanup
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-01-03 15:08:37 +01:00
cdb2a591e9
Refactor how things are migrated
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2023-01-03 14:44:47 +01:00
939a6d67e8
Merge branch 'main' into internals-rework
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-02 15:07:44 +01:00
6775569525
Bump everything to v0.8.1
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-02 14:15:33 +01:00
d56c472712
Refactor background runner and get rid of job worker
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-12-14 12:51:42 +01:00
2183518edc
Spawn all background workers in a separate step
2022-12-14 12:28:07 +01:00
83c8467e23
Proper queueing for delayed inserts, now backed to disk
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-12-14 11:58:06 +01:00
f8e528c15d
Small refactor of tables internals
continuous-integration/drone/push Build is failing
2022-12-14 10:48:49 +01:00
de9d6cddf7
Prettier worker list table; remove useless CLI log messages
continuous-integration/drone/push Build is passing
2022-12-12 17:17:05 +01:00
280d1be7b1
Refactor health check and add ability to return it in json
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-12-05 15:28:57 +01:00
2065f011ca
Implement /health admin API endpoint to check node health
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-12-05 14:59:15 +01:00
c1fb65194c
Add sled default in garage_model also
continuous-integration/drone/push Build is passing
2022-11-21 14:25:54 +01:00
5b18fd8201
Add garage bucket cleanup-incomplete-uploads command
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build was killed
2022-11-04 11:55:59 +01:00
ded444f6c9
Ability to have custom timeouts in request strategy (not used)
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-20 16:01:41 +02:00
56592e1853
RPC performance changes
...
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build was killed
- configurable ping timeout
- single, much higher, configurable RPC timeout
- no more concurrency semaphore
2022-09-19 20:31:00 +02:00
ab722cb40f
Add checks on replication_factor of layouts we use ( fix #363 , fix #364 )
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-13 16:22:23 +02:00
38be811b1c
Fix clippy lint that says we should implement Eq
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-13 16:08:00 +02:00
07febd3ecd
Ensure data dir is created immediately when Garage starts ( fix #349 )
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-13 15:57:27 +02:00
28a4af73ca
Use netapp 0.5 published from crates.io
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-13 13:11:44 +02:00
7f54706b95
Merge branch 'lx-perf-improvements' into netapp-stream-body
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-08 15:50:56 +02:00
d9d199a6c9
Merge branch 'main' into lx-perf-improvements
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-08 15:49:17 +02:00
ceb1f0229a
Move version back into util
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-07 18:36:46 +02:00
f310fce34b
Inject GIT_VERSION even later
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-07 18:30:15 +02:00
8adc654713
Merge branch 'main' into improve-deps
2022-09-07 18:13:27 +02:00
28d86e7602
Report build features in garage --help
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2022-09-07 17:05:21 +02:00
db61f41030
Move GIT_VERSION injection later in build chain to reduce build times
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2022-09-07 11:59:56 +02:00
6b958979bd
Merge branch 'lx-perf-improvements' into netapp-stream-body
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-06 22:13:01 +02:00
6f02c36a89
cargo fmt
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-06 17:59:41 +02:00
0f5689c169
Include code from v0.5.1 directly to remove dependencies
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-09-06 17:52:50 +02:00
b886c75450
Make all DB engines optional build features
2022-09-06 17:09:43 +02:00
48ffaaadfc
Bump versions to 0.8.0 (compatibility is broken already)
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-09-06 16:47:56 +02:00
07e6bcde85
Merge branch 'main' into lx-perf-improvements
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-05 12:40:17 +02:00
943d76c583
Ability to dynamically set resync tranquility
continuous-integration/drone/push Build is passing
2022-09-02 15:34:21 +02:00
a35d4da721
update netapp to 0.5
2022-07-29 12:25:02 +02:00
8e7e680afe
First adaptation to WIP netapp with streaming body
2022-07-29 12:25:02 +02:00
2f111e6b3d
Performance improvements:
...
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
2022-07-29 12:24:48 +02:00
4f38cadf6e
Background task manager ( #332 )
...
continuous-integration/drone/push Build is passing
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
- [x] Merkle updater: several items per iteration
- [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
- [x] have only one worker with a channel to start/pause/cancel
- [x] automatic scrub
- [x] ability to view and change tranquility from CLI
- [x] persistence of a few info
- [ ] Testing
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-07-08 13:30:26 +02:00
77e3fd6db2
improve internal item counter mechanisms and implement bucket quotas ( #326 )
...
continuous-integration/drone/push Build is passing
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-15 20:20:28 +02:00
b44d3fc796
Abstract database behind generic interface and implement alternative drivers ( #322 )
...
continuous-integration/drone/push Build is passing
- [x] Design interface
- [x] Implement Sled backend
- [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
- [x] `.updated()` trigger on Garage tables
Fix #284
**Bugs**
- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work
**Known issues for various back-ends**
- Sled:
- Eats all my RAM and also all my disk space
- `.len()` has to traverse the whole table
- Is actually quite slow on some operations
- And is actually pretty bad code...
- Sqlite:
- Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
- (adapter uses a bunch of unsafe code)
- Heed (LMDB):
- Not suited for 32-bit machines as it has to map the whole DB in memory.
- (adpater uses a tiny bit of unsafe code)
**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.
**Converting databases** is actually quite easy. For example from Sled to LMDB:
```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```
Then, just add this to your `config.toml`:
```toml
db_engine = "lmdb"
```
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-08 10:01:44 +02:00
382e74c798
First version of admin API ( #298 )
...
continuous-integration/drone/push Build is passing
**Spec:**
- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases
View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md >
**Code:**
- [x] Refactor code for admin api to use common api code that was created for K2V
**General endpoints:**
- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout
**Key-related endpoints:**
- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey
**Bucket-related endpoints:**
- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite
**Operations on key/bucket permissions:**
- [x] BucketAllowKey
- [x] BucketDenyKey
**Operations on bucket aliases:**
- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket
**And also:**
- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design
Fix #231
Fix #295
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-24 12:16:39 +02:00