Commit graph

1346 commits

Author SHA1 Message Date
91b2d1fcc1 Some basic S3 functionnality 2020-04-24 18:47:11 +00:00
f2e05986c4 Starting to be S3 compatible 2020-04-24 17:46:52 +00:00
d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00
51fb3799a1 Key management admin commands 2020-04-23 20:25:45 +00:00
4ef84a0558 Move repair to separate file 2020-04-23 18:36:12 +00:00
44a1089d95 Make table objects slightly more fool-proof; add key table 2020-04-23 18:16:52 +00:00
c9c6b0dbd4 Reorganize code 2020-04-23 17:05:46 +00:00
01a8acdeec Better error reporting 2020-04-23 16:23:06 +00:00
82f4cd8719 Continue pinging nodes when they are down ; overall better handling of down nodes 2020-04-23 16:06:11 +00:00
2fe82be3bc RPC to ourself do not pass through serialization + HTTPS 2020-04-23 14:40:59 +00:00
37f880bc09 RequestStrategy with possible interruption or not 2020-04-23 13:37:10 +00:00
73574ab43e Fix in rpc_client (see comment in code) 2020-04-22 20:42:23 +00:00
897fafa8db Improvements to block resync queue & worker 2020-04-22 20:32:58 +00:00
2556a1e383 I'm stupid though 2020-04-22 20:06:12 +00:00
231cb32955 Do not delete block if just a single replication error. Write TODO stuff. 2020-04-22 19:25:15 +00:00
8971f34c81 Well they still have to exit when we're exiting though 2020-04-22 17:04:33 +00:00
e8214cb180 Better concurrency:
Use Notify instead of stupid sleep in background worker
Use Semaphore to limit concurrent requests in rpc_client
Make more background tasks cancellable
2020-04-22 16:51:52 +00:00
c0335ac690 Remove a few features in dependencies 2020-04-21 20:37:02 +00:00
ec59e896c6 Make UUID & Hash Copy and remove some .clone() noise 2020-04-21 17:08:42 +00:00
8915224966 Return BadRequest codes for some admin_rpc failure cases 2020-04-21 16:45:32 +00:00
b1ddb933b0 Make the repair command accept subcommands to not do everything all the time 2020-04-21 16:40:17 +00:00
a04218047e Do full sync on node (re)start 2020-04-21 16:15:32 +00:00
2a84d965ab Improve table sync 2020-04-21 16:05:55 +00:00
0226561035 Do not insert deletion marker if there is no object to delete 2020-04-21 14:33:12 +00:00
be0a2bae81 Add node tags in configuration 2020-04-21 14:08:28 +00:00
cc4f2f1cfb Pretty logging 2020-04-21 12:54:55 +00:00
53cf4d1baa Log which workers are doing what 2020-04-19 21:33:38 +00:00
ec7f9f07e2 Implement repair object->version and version->block ref 2020-04-19 21:27:08 +00:00
04acaea231 Don't do version & block_ref updates in background on deletion 2020-04-19 20:52:20 +00:00
5ae32972ef Implement repair command 2020-04-19 20:36:36 +00:00
a54f3158f1 Less output 2020-04-19 19:38:45 +00:00
ea75564851 More aggressive sync timings & improve other stuff 2020-04-19 17:59:59 +00:00
e325c7f47a Add hostname to node info 2020-04-19 19:08:48 +02:00
a6129d8626 Begin implement bucket management & admin commands 2020-04-19 17:15:48 +02:00
302502f4c1 Add support for fully replicated tables with epidemic dissemination of updates 2020-04-19 15:14:23 +02:00
7131553c53 Refactor sharding logic; coming next: full replication with epidemic dissemination 2020-04-19 13:22:28 +02:00
4ba54ccfca Reorder imports.
Trying to separate:
1. Stuff for handling the swarm of nodes and generic table data replication
2. Stuff for the object store core application: metadata tables and block management
3. Stuff for the S3 API
2020-04-18 19:39:57 +02:00
bd1618e78e Remove proto.rs & move some definitions out of data.rs 2020-04-18 19:30:05 +02:00
f41583e1b7 Massive RPC refactoring 2020-04-18 19:21:34 +02:00
3f40ef149f Fix sync: use max root checksum level 2020-04-17 21:59:07 +02:00
f62b54f1df Fix add to resync on incref 2020-04-17 21:14:06 +02:00
ace07da7c1 Fix walk_ring_from 2020-04-17 21:08:43 +02:00
40c48e6a59 Several resync workers; add delay on retry resync 2020-04-17 20:58:10 +02:00
29a1e94f23 Implement missing handler for read_range 2020-04-17 19:38:47 +02:00
b4e96bdcf0 Fix paths :o 2020-04-17 19:20:17 +02:00
4abfb75509 Implement sending blocks to nodes that need them 2020-04-17 19:16:08 +02:00
db1c4222ce Don't send items...
...if syncer doesn't need them because he's going to delete the partition anyway.
Also, fix block resync queue
2020-04-17 18:51:29 +02:00
4bacaaf53f Resync block on read error 2020-04-17 18:38:11 +02:00
b780f6485d Make sync send data both ways 2020-04-17 18:27:29 +02:00
69f1d8fef2 WIP
TODOs:
- ensure sync goes both way
- finish sending blocks to other nodes when they need them before deleting
2020-04-17 17:09:57 +02:00