Commit graph

31 commits

Author SHA1 Message Date
2fe82be3bc RPC to ourself do not pass through serialization + HTTPS 2020-04-23 14:40:59 +00:00
e8214cb180 Better concurrency:
Use Notify instead of stupid sleep in background worker
Use Semaphore to limit concurrent requests in rpc_client
Make more background tasks cancellable
2020-04-22 16:51:52 +00:00
cc4f2f1cfb Pretty logging 2020-04-21 12:54:55 +00:00
ea75564851 More aggressive sync timings & improve other stuff 2020-04-19 17:59:59 +00:00
a6129d8626 Begin implement bucket management & admin commands 2020-04-19 17:15:48 +02:00
7131553c53 Refactor sharding logic; coming next: full replication with epidemic dissemination 2020-04-19 13:22:28 +02:00
4ba54ccfca Reorder imports.
Trying to separate:
1. Stuff for handling the swarm of nodes and generic table data replication
2. Stuff for the object store core application: metadata tables and block management
3. Stuff for the S3 API
2020-04-18 19:39:57 +02:00
bd1618e78e Remove proto.rs & move some definitions out of data.rs 2020-04-18 19:30:05 +02:00
f41583e1b7 Massive RPC refactoring 2020-04-18 19:21:34 +02:00
69f1d8fef2 WIP
TODOs:
- ensure sync goes both way
- finish sending blocks to other nodes when they need them before deleting
2020-04-17 17:09:57 +02:00
e41ce4d815 Implement getting missing blocks when RC increases
Issue: RC increases also when the block ref entry is first put by the actual client.
At that point the client is probably already sending us the block content,
so we don't need to do a get...
We should add a delay before the task is added or find something to do.
2020-04-17 15:40:13 +02:00
e8d750175d Implement ring comparison algorithm 2020-04-16 17:04:28 +02:00
f01c1e71b5 Begin work on sync... 2020-04-16 14:50:49 +02:00
2bea76ce16 Small refactorings 2020-04-12 22:24:53 +02:00
d1e8f78b2c Trying to do TLS 2020-04-12 15:51:19 +02:00
5967c5a5af Refactor a bit 2020-04-12 13:03:55 +02:00
9c931f5eda Keep network status & ring in a tokio::sync::watch
advantages
- reads don't prevent preparing writes
- can be followed from other parts of the system by cloning the receiver
2020-04-11 23:53:32 +02:00
5dd59e437d Local refcounting of blocks 2020-04-11 23:00:26 +02:00
dcf58499a4 table::insert_many, version_table::updated 2020-04-11 19:43:29 +02:00
53289b69e5 Background task runner that replaces tokio::spawn 2020-04-11 18:51:11 +02:00
cf8fd948fc Add block ref table 2020-04-10 23:11:52 +02:00
3477864142 Fix the Sync issue. Details:
So the HTTP client future of Hyper is not Sync, thus the stream
that read blocks wasn't either. However Hyper's default Body type
requires a stream to be Sync for wrap_stream. Solution: reimplement
a custom HTTP body type.
2020-04-10 22:01:48 +02:00
d66c0d6833 Why is it not Sync?? 2020-04-09 23:45:07 +02:00
1d786c2c66 Something works 2020-04-09 18:43:53 +02:00
101444abb3 Some progress 2020-04-09 17:32:28 +02:00
cc580da0ae Some work 2020-04-08 23:01:49 +02:00
bacc76a057 Some work in actually storing things 2020-04-08 22:00:41 +02:00
d50edcdb4f Basic workflow for client PUT, next is do actual storage 2020-04-08 00:39:07 +02:00
90cdffb425 custom data type for hashes and identifiers 2020-04-07 18:10:20 +02:00
82b7fcd280 Config management & ring 2020-04-07 17:00:48 +02:00
061e676136 Refactor; ability to update network config 2020-04-07 16:26:22 +02:00