Merge pull request 'Fix venv sourcing' (#2 ) from baptiste/mknet:misc_fixes into main

Reviewed-on: Deuxfleurs/mknet#2
Merge pull request 'Add result with 30M objects on Grid'5000' (#3 ) from baptiste/mknet:gros_30M into main
2022-09-29 20:57:53 +02:00 · 2022-09-29 20:53:21 +02:00 · 2022-09-29 20:27:25 +02:00 · 2022-09-29 10:40:34 +02:00 · 2022-09-29 10:39:45 +02:00 · 2022-09-28 14:59:42 +02:00
99 changed files with 8348 additions and 408 deletions
--- a/.gitignore
+++ b/.gitignore
@ -1,3 +1,6 @@
 .current_state.yml
 __pycache__
 *.swp
+build
+mknet.egg-info
+.venv
--- a/.gitmodules
+++ b/.gitmodules
@ -0,0 +1,6 @@
+[submodule "benchmarks/warp"]
+	path = benchmarks/warp
+	url = https://github.com/minio/warp
+[submodule "analysis/quentin"]
+	path = analysis/quentin
+	url = https://git.deuxfleurs.fr/quentin/benchmarks
--- a/README.md
+++ b/README.md
@ -1,43 +1,77 @@
 # mknet

-## Installation
+mknet is a tool to simulate various network topologies
+locally thanks to network namespaces and traffic control (tc).
+
+## Prepare your environment
+
+Get the repository:

 ```bash
-sudo pip3 install git+https://git.deuxfleurs.fr/trinity-1686a/mknet
+git clone https://git.deuxfleurs.fr/Deuxfleurs/mknet.git
+cd mknet
 ```

-## Usage
+Run our configuration script:

 ```bash
-sudo rm -rf /tmp/garage-testnet/ # always start by deleting previous run
-sudo mknet create ./config.yml
-sudo mknet run-all example/deploy_garage.sh
-sudo mknet run dc1:dc1s1 garage -c /tmp/garage-testnet/dc1/dc1s1/garage.toml status
-sudo mknet destroy
+./prepare.py
 ```

-## Instrumented daemons
-
-If you want to use the scripts provided in the `example` folder,
-you must add to your path some tools.
-
-### Garage (`deploy_garage.sh`)
+Now, you are ready to launch an experiment:

 ```bash
-# see versions on https://garagehq.deuxfleurs.fr/_releases.html
-export GRG_ARCH=x86_64-unknown-linux-musl
-export GRG_VERSION=v0.7.2.1
-
-sudo wget https://garagehq.deuxfleurs.fr/_releases/${GRG_VERSION}/${GRG_ARCH}/garage -O /usr/local/bin/garage
-sudo chmod +x /usr/local/bin/garage
-
-garage help
+sudo -i
+source .venv/bin/activate
+./mknet scenario ./topo/50ms.yml ./scenarios/garage-s3lat garage-v0.8
 ```

-### Minio (`deploy_minio.py`)
+If a script crash, you must manually destroy the topology:

 ```
-sudo wget https://dl.min.io/server/minio/release/linux-amd64/minio -O /usr/local/bin/minio
-sudo chmod +x /usr/local/bin/minio
+./mknet destroy
 ```

+## Topologies
+
+All topologies:
+  - `./topo/dc.yml` - A 3 node topology connected with a 1Gbit/s link and a 1ms latency
+  - `./topo/with-vdsl.yml` - A topology mixing datacenters with fast internal connectivity and an isolated VDSL node
+  - `./topo/50ms.yml` - An artifical topology simulating nodes with high bandwidth but with a fixed 50ms latency, useful to quantify the impact of latency on a distributed software
+  - `./topo/multi-dc.yml` - Simulate 3 DC interconnected with 50ms latency WAN network and close to zero latency inside the DC
+  - `./topo/{1,5,10,50,100}mbps.yml` - Simulate 3 nodes interconnected with a bandwidth bottleneck.
+
+Feel free to write new topologies!
+
+## Scenarios
+
+All scenarios:
+ - `./scenarios/garage-s3lat [garage-v0.7|garage-v0.8]` - Run s3lat on Garage
+ - `./scenarios/garage-concurrent [garage-v0.7|garage-v0.8]` - Run s3concurrent on Garage
+ - `./scenarios/garage-warp [garage-v0.7|garage-v0.8] [default|fast]` - Run warp on Garage. 2 flavors are available: fast and default.
+ - `./scenarios/garage-s3billion [garage-v0.7|garage-v0.8]` - Run s3billion on Garage
+
+*Scenarios take optional flavors as input that modulate their behavioir. Passing them is not mandatory,
+a default one will be selected for you.*
+
+How to run them:
+
+```bash
+./mknet scenario <topology> <scenario> [flavors...]
+./mknet scenario ./topo/50ms.yml ./scenarios/garage-s3lat garage-v0.8
+```
+
+How to write good scenarios:
+  - If a scenario can be run with multiple different parameters, write one scenario with multiple flavors
+  - If the logic ran is different, write a new scenario
+  - A scenario code must remain short and looks like a DSL, abstract the logic in the `fragments/` module
+
+## Manual usage
+
+```bash
+./mknet create ./topo/with-vdsl.yml
+./mknet run-all ./scenarios/garage-manual
+./mknet run dc1:dc1s1 /tmp/mknet-bin/garage* -c /tmp/mknet-store/garage/dc1/dc1s1/garage.toml status
+./mknet run-all ./scenarios/clean
+./mknet destroy
+
--- a/analysis/quentin
+++ b/analysis/quentin
@ -0,0 +1 @@
+Subproject commit 902555b9d5db594fdad927f51c2d8984d6e9f6a5
--- a/artifacts/2022-09-19-s3concurrent/garage-v0.7_10mbps.csv
+++ b/artifacts/2022-09-19-s3concurrent/garage-v0.7_10mbps.csv
@ -0,0 +1,17 @@
+sent,success,elapsed,elapsed_per_req
+1,1,1733568406,1733568406
+2,2,3717781951,1858890975
+3,3,5554603016,1851534338
+4,4,7243764558,1810941139
+5,5,9088779023,1817755804
+6,6,10853379229,1808896538
+7,7,12687932044,1812561720
+8,8,14457291732,1807161466
+9,9,16271996844,1807999649
+10,10,18066058997,1806605899
+11,11,20370606012,1851873273
+12,12,11605742376,967145198
+13,13,13622546503,1047888192
+14,14,16059774589,1147126756
+15,15,20283463752,1352230916
+16,16,23829428368,1489339273
--- a/artifacts/2022-09-19-s3concurrent/garage-v0.7_5mbps.csv
+++ b/artifacts/2022-09-19-s3concurrent/garage-v0.7_5mbps.csv
@ -0,0 +1,17 @@
+sent,success,elapsed,elapsed_per_req
+1,1,3646698401,3646698401
+2,2,7245492636,3622746318
+3,3,10896710797,3632236932
+4,4,14469160703,3617290175
+5,4,18066381776,3613276355
+6,3,19209830571,3201638428
+7,0,186084824,26583546
+8,0,183207903,22900987
+9,0,199743038,22193670
+10,0,195703069,19570306
+11,0,198204447,18018586
+12,0,187247329,15603944
+13,0,193441556,14880119
+14,0,192003016,13714501
+15,0,140156357,9343757
+16,0,186228134,11639258
--- a/artifacts/2022-09-19-s3concurrent/garage-v0.8_10mbps.csv
+++ b/artifacts/2022-09-19-s3concurrent/garage-v0.8_10mbps.csv
@ -0,0 +1,17 @@
+sent,success,elapsed,elapsed_per_req
+1,1,1634632370,1634632370
+2,2,3799299300,1899649650
+3,3,5436258664,1812086221
+4,4,7242028242,1810507060
+5,5,9041569964,1808313992
+6,6,10852330277,1808721712
+7,7,12646683069,1806669009
+8,8,14468867472,1808608434
+9,9,16251724200,1805747133
+10,10,18190271787,1819027178
+11,11,19877017065,1807001551
+12,12,20526867033,1710572252
+13,13,24558838287,1889141406
+14,14,25274586429,1805327602
+15,15,27180202376,1812013491
+16,16,29437555977,1839847248
--- a/artifacts/2022-09-19-s3concurrent/garage-v0.8_5mbps.csv
+++ b/artifacts/2022-09-19-s3concurrent/garage-v0.8_5mbps.csv
@ -0,0 +1,17 @@
+sent,success,elapsed,elapsed_per_req
+1,1,3638584739,3638584739
+2,2,7246358601,3623179300
+3,3,11160920229,3720306743
+4,4,14454330626,3613582656
+5,5,18052975913,3610595182
+6,6,21690341725,3615056954
+7,7,25268069065,3609724152
+8,8,29007558135,3625944766
+9,9,32688763527,3632084836
+10,10,36101205725,3610120572
+11,11,28245933586,2567812144
+12,12,43983751108,3665312592
+13,13,53456719112,4112055316
+14,14,54398690114,3885620722
+15,15,54153860198,3610257346
+16,16,57732527899,3608282993
--- a/artifacts/2022-09-23-s3lat/garage-v0.7.csv
+++ b/artifacts/2022-09-23-s3lat/garage-v0.7.csv
@ -0,0 +1,501 @@
+endpoint,nanoseconds
+listbuckets,123446207
+listbuckets,131497917
+listbuckets,39054735
+listbuckets,105410142
+listbuckets,134197174
+listbuckets,92373770
+listbuckets,113362017
+listbuckets,109195200
+listbuckets,176558852
+listbuckets,27177881
+listbuckets,36252695
+listbuckets,106228157
+listbuckets,167364021
+listbuckets,6904816
+listbuckets,34264521
+listbuckets,83810002
+listbuckets,156429277
+listbuckets,200458223
+listbuckets,112529004
+listbuckets,35129564
+listbuckets,79573343
+listbuckets,140150702
+listbuckets,120555536
+listbuckets,29181815
+listbuckets,51198242
+listbuckets,56133478
+listbuckets,47130673
+listbuckets,4870606
+listbuckets,115299404
+listbuckets,199760726
+listbuckets,104304255
+listbuckets,165317856
+listbuckets,174747530
+listbuckets,69139944
+listbuckets,170345599
+listbuckets,133286913
+listbuckets,133428040
+listbuckets,152317101
+listbuckets,163419603
+listbuckets,11954339
+listbuckets,49001694
+listbuckets,1909515
+listbuckets,85317086
+listbuckets,193461440
+listbuckets,37110905
+listbuckets,170866722
+listbuckets,2820145
+listbuckets,136496793
+listbuckets,64132721
+listbuckets,125251903
+listbuckets,87449583
+listbuckets,156290592
+listbuckets,160402726
+listbuckets,200273480
+listbuckets,93021081
+listbuckets,68304026
+listbuckets,46970448
+listbuckets,165308307
+listbuckets,41980457
+listbuckets,166480048
+listbuckets,136400950
+listbuckets,111337721
+listbuckets,180455774
+listbuckets,119142367
+listbuckets,117091664
+listbuckets,126145949
+listbuckets,119181500
+listbuckets,55147467
+listbuckets,178508747
+listbuckets,101172269
+listbuckets,116156709
+listbuckets,53168650
+listbuckets,29044786
+listbuckets,47899914
+listbuckets,139056994
+listbuckets,26962604
+listbuckets,50780131
+listbuckets,36960440
+listbuckets,161531494
+listbuckets,9947611
+listbuckets,33134947
+listbuckets,81109551
+listbuckets,153415682
+listbuckets,199163150
+listbuckets,183053868
+listbuckets,148142646
+listbuckets,17965198
+listbuckets,161258658
+listbuckets,175383896
+listbuckets,42076378
+listbuckets,34970593
+listbuckets,14042201
+listbuckets,66182118
+listbuckets,42936091
+listbuckets,200730649
+listbuckets,75462399
+listbuckets,34967326
+listbuckets,99414279
+listbuckets,83041589
+listbuckets,9915941
+putobject,102659465
+putobject,92835679
+putobject,108731599
+putobject,201811799
+putobject,106443490
+putobject,92567983
+putobject,238871966
+putobject,104158815
+putobject,106243742
+putobject,116316799
+putobject,108757989
+putobject,97366623
+putobject,97898375
+putobject,96967168
+putobject,100256148
+putobject,107432215
+putobject,102248459
+putobject,251441345
+putobject,104946713
+putobject,110238211
+putobject,108091689
+putobject,113202108
+putobject,98765973
+putobject,104565383
+putobject,111568023
+putobject,112197090
+putobject,105911994
+putobject,95484988
+putobject,104482906
+putobject,234879244
+putobject,113432122
+putobject,100713850
+putobject,108868215
+putobject,103499519
+putobject,104655812
+putobject,115506648
+putobject,101683821
+putobject,108092129
+putobject,101964070
+putobject,106886313
+putobject,112577538
+putobject,261098453
+putobject,113249625
+putobject,108966089
+putobject,109411069
+putobject,104522489
+putobject,110678803
+putobject,105536735
+putobject,110833494
+putobject,112977212
+putobject,98197749
+putobject,110378043
+putobject,375687474
+putobject,203462479
+putobject,115639116
+putobject,118053054
+putobject,110270019
+putobject,109764686
+putobject,90674345
+putobject,98625115
+putobject,103500960
+putobject,102995627
+putobject,236288420
+putobject,107760181
+putobject,112772765
+putobject,99998427
+putobject,117912217
+putobject,105863219
+putobject,113652925
+putobject,100828684
+putobject,95361573
+putobject,95500334
+putobject,106068286
+putobject,99928444
+putobject,261266283
+putobject,106924492
+putobject,110718855
+putobject,87597876
+putobject,108996183
+putobject,99171595
+putobject,115447052
+putobject,103467696
+putobject,104448307
+putobject,97229488
+putobject,106477027
+putobject,105427947
+putobject,233050123
+putobject,115135303
+putobject,113237921
+putobject,110612835
+putobject,102891788
+putobject,112480159
+putobject,117117775
+putobject,196493186
+putobject,104291470
+putobject,118422258
+putobject,104951716
+putobject,244946550
+putobject,98641292
+putobject,119142327
+listobjects,227618086
+listobjects,218373723
+listobjects,220425637
+listobjects,212659074
+listobjects,194860492
+listobjects,221131339
+listobjects,227735226
+listobjects,212887826
+listobjects,202906841
+listobjects,221091271
+listobjects,216883738
+listobjects,213001179
+listobjects,217273734
+listobjects,211956305
+listobjects,213529174
+listobjects,375065283
+listobjects,218843356
+listobjects,217068015
+listobjects,223369781
+listobjects,219801804
+listobjects,225260921
+listobjects,207088252
+listobjects,229780152
+listobjects,227569658
+listobjects,211492577
+listobjects,200421226
+listobjects,224666076
+listobjects,217078671
+listobjects,220658029
+listobjects,220137146
+listobjects,205746397
+listobjects,222356274
+listobjects,210329960
+listobjects,219004606
+listobjects,226442588
+listobjects,374038605
+listobjects,221425877
+listobjects,226949625
+listobjects,198272437
+listobjects,215884986
+listobjects,210904573
+listobjects,231082876
+listobjects,220581992
+listobjects,215368028
+listobjects,204236634
+listobjects,230323641
+listobjects,221006783
+listobjects,197521967
+listobjects,217673459
+listobjects,210402664
+listobjects,222289473
+listobjects,216105460
+listobjects,233017185
+listobjects,217884686
+listobjects,216886613
+listobjects,215456984
+listobjects,342657645
+listobjects,218310540
+listobjects,346101165
+listobjects,227114600
+listobjects,207167945
+listobjects,217958955
+listobjects,256687056
+listobjects,216946794
+listobjects,211024233
+listobjects,221148255
+listobjects,214229333
+listobjects,224119805
+listobjects,217875907
+listobjects,216024912
+listobjects,220679148
+listobjects,218154613
+listobjects,222423060
+listobjects,212511440
+listobjects,215068766
+listobjects,207965636
+listobjects,374287848
+listobjects,231411797
+listobjects,344053841
+listobjects,218949902
+listobjects,208085171
+listobjects,202097107
+listobjects,220625984
+listobjects,232280185
+listobjects,214265437
+listobjects,216862289
+listobjects,219434755
+listobjects,217021028
+listobjects,214401512
+listobjects,207317610
+listobjects,215392944
+listobjects,217703786
+listobjects,215988628
+listobjects,198853351
+listobjects,222486834
+listobjects,213094404
+listobjects,211948650
+listobjects,215660888
+listobjects,222729552
+listobjects,357486752
+getobject,120789563
+getobject,257364122
+getobject,193090188
+getobject,105639883
+getobject,110039709
+getobject,106107405
+getobject,113226765
+getobject,228216027
+getobject,204992307
+getobject,115070175
+getobject,118444544
+getobject,116666047
+getobject,114052724
+getobject,109807911
+getobject,108123550
+getobject,245998795
+getobject,103858529
+getobject,104810357
+getobject,101301352
+getobject,104881290
+getobject,101871689
+getobject,245400756
+getobject,99811659
+getobject,107963659
+getobject,108133779
+getobject,109433413
+getobject,112918863
+getobject,99263685
+getobject,109925392
+getobject,404106741
+getobject,111264038
+getobject,107867546
+getobject,232065851
+getobject,104837346
+getobject,249346993
+getobject,107660264
+getobject,98665144
+getobject,93135947
+getobject,93305867
+getobject,103816208
+getobject,101191343
+getobject,104046332
+getobject,234664977
+getobject,109126683
+getobject,109256458
+getobject,111475958
+getobject,106229192
+getobject,111367102
+getobject,247160403
+getobject,114725281
+getobject,115208863
+getobject,115752038
+getobject,99434436
+getobject,114499775
+getobject,114262136
+getobject,237897437
+getobject,109039858
+getobject,116366349
+getobject,109513421
+getobject,119350461
+getobject,103568549
+getobject,268923073
+getobject,103935562
+getobject,114438288
+getobject,250696155
+getobject,100985713
+getobject,108943735
+getobject,107398809
+getobject,201770787
+getobject,258949074
+getobject,108722357
+getobject,106948669
+getobject,105753903
+getobject,106842969
+getobject,254243109
+getobject,109977076
+getobject,101122461
+getobject,106152776
+getobject,105146877
+getobject,115352192
+getobject,114154611
+getobject,106352783
+getobject,316675402
+getobject,109678912
+getobject,105611654
+getobject,111231964
+getobject,110622814
+getobject,103520055
+getobject,264080226
+getobject,115777874
+getobject,117219506
+getobject,237658128
+getobject,105861315
+getobject,118189958
+getobject,250123569
+getobject,210303951
+getobject,114146153
+getobject,114062677
+getobject,116419134
+getobject,110233217
+removeobject,230846328
+removeobject,352354358
+removeobject,213513979
+removeobject,213021069
+removeobject,201490202
+removeobject,212111001
+removeobject,215118297
+removeobject,206781539
+removeobject,490175308
+removeobject,207408693
+removeobject,201921334
+removeobject,214565432
+removeobject,223499977
+removeobject,219187737
+removeobject,353799701
+removeobject,226319108
+removeobject,204468762
+removeobject,206638858
+removeobject,227752042
+removeobject,227770838
+removeobject,368061761
+removeobject,304793510
+removeobject,231980092
+removeobject,227557765
+removeobject,216285686
+removeobject,204581431
+removeobject,332004595
+removeobject,209905432
+removeobject,208660772
+removeobject,298520566
+removeobject,221629301
+removeobject,216850310
+removeobject,346809979
+removeobject,225626686
+removeobject,335040322
+removeobject,217416498
+removeobject,221590626
+removeobject,211254506
+removeobject,218186009
+removeobject,333605972
+removeobject,214970419
+removeobject,400660245
+removeobject,226176422
+removeobject,210009344
+removeobject,214122397
+removeobject,217046195
+removeobject,348420624
+removeobject,222496374
+removeobject,200770472
+removeobject,206765238
+removeobject,212797886
+removeobject,221853199
+removeobject,366803902
+removeobject,369766134
+removeobject,254126638
+removeobject,206946940
+removeobject,214755369
+removeobject,222743167
+removeobject,214028196
+removeobject,365328074
+removeobject,222478296
+removeobject,200478669
+removeobject,228313711
+removeobject,351395352
+removeobject,347788382
+removeobject,370554938
+removeobject,220125163
+removeobject,211178604
+removeobject,478779432
+removeobject,226303414
+removeobject,212846913
+removeobject,232756403
+removeobject,352451452
+removeobject,204864755
+removeobject,386939750
+removeobject,236333214
+removeobject,213864532
+removeobject,337968992
+removeobject,208224425
+removeobject,215012709
+removeobject,220231159
+removeobject,200163319
+removeobject,201449085
+removeobject,203033356
+removeobject,362988387
+removeobject,352665831
+removeobject,255105686
+removeobject,209154819
+removeobject,218832617
+removeobject,220790763
+removeobject,345915165
+removeobject,213304301
+removeobject,218302224
+removeobject,218254023
+removeobject,204621450
+removeobject,213964266
+removeobject,383714183
+removeobject,312511671
+removeobject,207629778
+removeobject,203927189
--- a/artifacts/2022-09-23-s3lat/garage-v0.8.csv
+++ b/artifacts/2022-09-23-s3lat/garage-v0.8.csv
@ -0,0 +1,501 @@
+endpoint,nanoseconds
+listbuckets,142443772
+listbuckets,78416939
+listbuckets,58182021
+listbuckets,131770558
+listbuckets,23167382
+listbuckets,100239233
+listbuckets,115131496
+listbuckets,104406526
+listbuckets,12172321
+listbuckets,196757641
+listbuckets,154690693
+listbuckets,200549327
+listbuckets,97233951
+listbuckets,157416927
+listbuckets,91145569
+listbuckets,19144199
+listbuckets,64085612
+listbuckets,113225008
+listbuckets,166909251
+listbuckets,34198118
+listbuckets,174705394
+listbuckets,87295494
+listbuckets,57706664
+listbuckets,140442963
+listbuckets,14837772
+listbuckets,135331146
+listbuckets,111062526
+listbuckets,192533228
+listbuckets,78240866
+listbuckets,49209288
+listbuckets,139333059
+listbuckets,159433712
+listbuckets,168386408
+listbuckets,54060994
+listbuckets,182381749
+listbuckets,107235324
+listbuckets,19175648
+listbuckets,79408539
+listbuckets,102182087
+listbuckets,138252068
+listbuckets,200286419
+listbuckets,171574163
+listbuckets,139244520
+listbuckets,162405069
+listbuckets,191507551
+listbuckets,156309385
+listbuckets,45147822
+listbuckets,123534893
+listbuckets,114180389
+listbuckets,200631309
+listbuckets,183352440
+listbuckets,42285029
+listbuckets,185592518
+listbuckets,166551994
+listbuckets,63313864
+listbuckets,51172191
+listbuckets,162406210
+listbuckets,51252382
+listbuckets,5096417
+listbuckets,177549418
+listbuckets,21073149
+listbuckets,98302685
+listbuckets,173578013
+listbuckets,91105261
+listbuckets,105040220
+listbuckets,112304298
+listbuckets,65084455
+listbuckets,181875861
+listbuckets,142442935
+listbuckets,131125617
+listbuckets,76286586
+listbuckets,24552325
+listbuckets,121486988
+listbuckets,186709964
+listbuckets,5948226
+listbuckets,161299340
+listbuckets,40993441
+listbuckets,66113978
+listbuckets,42309585
+listbuckets,124003416
+listbuckets,97993931
+listbuckets,31175113
+listbuckets,98603492
+listbuckets,21409874
+listbuckets,194297813
+listbuckets,50059832
+listbuckets,148399545
+listbuckets,48887920
+listbuckets,152334411
+listbuckets,57077936
+listbuckets,26810977
+listbuckets,194375880
+listbuckets,138228951
+listbuckets,188415922
+listbuckets,59004763
+listbuckets,28981072
+listbuckets,41073893
+listbuckets,29776223
+listbuckets,140987798
+listbuckets,96201298
+putobject,96916217
+putobject,204175953
+putobject,116377273
+putobject,200815462
+putobject,189393367
+putobject,201157286
+putobject,209378110
+putobject,200861878
+putobject,127598269
+putobject,198228669
+putobject,206087830
+putobject,190252106
+putobject,213164125
+putobject,113273726
+putobject,184844831
+putobject,217834778
+putobject,213577904
+putobject,187314812
+putobject,197335012
+putobject,191781450
+putobject,245923315
+putobject,249794209
+putobject,154268417
+putobject,204865983
+putobject,200616863
+putobject,218347254
+putobject,225218951
+putobject,204122038
+putobject,187483799
+putobject,224081164
+putobject,313480753
+putobject,207639555
+putobject,231879901
+putobject,213694483
+putobject,221029143
+putobject,210575405
+putobject,203991790
+putobject,97757552
+putobject,202949604
+putobject,187003632
+putobject,192703458
+putobject,198853855
+putobject,187677763
+putobject,217450238
+putobject,201097498
+putobject,204520778
+putobject,209010932
+putobject,103217905
+putobject,208237561
+putobject,212482783
+putobject,223855484
+putobject,215215920
+putobject,196011159
+putobject,207887130
+putobject,203901248
+putobject,195527114
+putobject,194314145
+putobject,118554688
+putobject,174577871
+putobject,182678497
+putobject,214551635
+putobject,193939918
+putobject,210241591
+putobject,218407872
+putobject,212235655
+putobject,191045716
+putobject,199345769
+putobject,212445320
+putobject,213398017
+putobject,196443131
+putobject,111006983
+putobject,206131544
+putobject,227743928
+putobject,205929843
+putobject,210773624
+putobject,201460969
+putobject,178288219
+putobject,194333228
+putobject,200422249
+putobject,211491788
+putobject,277570466
+putobject,194155773
+putobject,202859287
+putobject,210188078
+putobject,200376501
+putobject,222446189
+putobject,216905202
+putobject,203768673
+putobject,211882053
+putobject,199813369
+putobject,202275734
+putobject,225392157
+putobject,108514537
+putobject,183571858
+putobject,220178028
+putobject,244245555
+putobject,246644462
+putobject,202754498
+putobject,125491965
+putobject,198043126
+listobjects,247336346
+listobjects,222421821
+listobjects,217409432
+listobjects,213489945
+listobjects,197741076
+listobjects,204144700
+listobjects,194520187
+listobjects,207039462
+listobjects,214175739
+listobjects,222630443
+listobjects,207519367
+listobjects,204852995
+listobjects,200389171
+listobjects,223342436
+listobjects,375412313
+listobjects,197168061
+listobjects,209902283
+listobjects,208728457
+listobjects,223192111
+listobjects,197001637
+listobjects,210692202
+listobjects,210082361
+listobjects,193242654
+listobjects,190713450
+listobjects,210137746
+listobjects,210096607
+listobjects,214155697
+listobjects,203814873
+listobjects,200252782
+listobjects,198504662
+listobjects,225529670
+listobjects,219412867
+listobjects,200873322
+listobjects,366185537
+listobjects,220358802
+listobjects,203736154
+listobjects,195030295
+listobjects,206763616
+listobjects,215264828
+listobjects,196677593
+listobjects,206296625
+listobjects,203169834
+listobjects,228228451
+listobjects,201871694
+listobjects,214160805
+listobjects,221120508
+listobjects,204784600
+listobjects,215667356
+listobjects,203478756
+listobjects,193957098
+listobjects,204028493
+listobjects,200296721
+listobjects,201366590
+listobjects,217724938
+listobjects,335372225
+listobjects,212399058
+listobjects,206802897
+listobjects,203512327
+listobjects,201467970
+listobjects,207952516
+listobjects,200169128
+listobjects,193368365
+listobjects,208873233
+listobjects,206348419
+listobjects,202684585
+listobjects,255082511
+listobjects,225277839
+listobjects,216147871
+listobjects,203396588
+listobjects,215594185
+listobjects,211893541
+listobjects,202693570
+listobjects,213428668
+listobjects,345450849
+listobjects,210503146
+listobjects,206714788
+listobjects,221670971
+listobjects,209193505
+listobjects,207268362
+listobjects,217422133
+listobjects,194262801
+listobjects,214257460
+listobjects,209999916
+listobjects,200992698
+listobjects,233030646
+listobjects,197622419
+listobjects,214531387
+listobjects,207687472
+listobjects,193650794
+listobjects,216352563
+listobjects,204419966
+listobjects,220824902
+listobjects,203824031
+listobjects,345556923
+listobjects,211586088
+listobjects,221054525
+listobjects,216966641
+listobjects,212481105
+listobjects,225077725
+listobjects,206661945
+getobject,106148474
+getobject,103474999
+getobject,110847021
+getobject,89406401
+getobject,107194067
+getobject,103967969
+getobject,99809582
+getobject,218870171
+getobject,116458559
+getobject,116438822
+getobject,114486708
+getobject,98523284
+getobject,107932449
+getobject,96947000
+getobject,111244400
+getobject,107520933
+getobject,96707408
+getobject,98561777
+getobject,100203555
+getobject,102302806
+getobject,238528526
+getobject,271793475
+getobject,103513430
+getobject,110508380
+getobject,88297618
+getobject,93017464
+getobject,92182439
+getobject,100926740
+getobject,90912953
+getobject,89262659
+getobject,106320567
+getobject,114727932
+getobject,93467784
+getobject,233897134
+getobject,99598463
+getobject,100682658
+getobject,196532397
+getobject,114136005
+getobject,107149842
+getobject,102095284
+getobject,101561076
+getobject,110858641
+getobject,112464019
+getobject,89475711
+getobject,104275969
+getobject,98529866
+getobject,241428685
+getobject,101153106
+getobject,119006084
+getobject,89105802
+getobject,95722098
+getobject,267246924
+getobject,106170312
+getobject,106624860
+getobject,101451128
+getobject,114344067
+getobject,98506251
+getobject,85363560
+getobject,109668504
+getobject,97277512
+getobject,217166462
+getobject,95365244
+getobject,105298136
+getobject,107433406
+getobject,91927105
+getobject,103556189
+getobject,101882261
+getobject,97490353
+getobject,109313364
+getobject,107477889
+getobject,105115091
+getobject,100009677
+getobject,96802735
+getobject,243376778
+getobject,116897654
+getobject,246725949
+getobject,104940522
+getobject,104899223
+getobject,99914467
+getobject,101929660
+getobject,96935645
+getobject,107399321
+getobject,94553040
+getobject,110316235
+getobject,112920198
+getobject,100314041
+getobject,239311246
+getobject,114578006
+getobject,106688650
+getobject,89102575
+getobject,82537327
+getobject,102124717
+getobject,107306736
+getobject,113530840
+getobject,102225458
+getobject,105090914
+getobject,109987193
+getobject,107637409
+getobject,99663933
+getobject,252948739
+removeobject,218637514
+removeobject,459986423
+removeobject,307487771
+removeobject,425595066
+removeobject,325293018
+removeobject,326633357
+removeobject,490052955
+removeobject,252844239
+removeobject,313024066
+removeobject,301593436
+removeobject,299154213
+removeobject,396128610
+removeobject,227674663
+removeobject,439796598
+removeobject,464313791
+removeobject,433487342
+removeobject,316319351
+removeobject,324010444
+removeobject,224091911
+removeobject,259658834
+removeobject,268735522
+removeobject,466588289
+removeobject,328431494
+removeobject,456136096
+removeobject,214305954
+removeobject,349200623
+removeobject,310132173
+removeobject,433264807
+removeobject,438724929
+removeobject,333912094
+removeobject,443030552
+removeobject,338569189
+removeobject,452852014
+removeobject,450187315
+removeobject,407122424
+removeobject,371514674
+removeobject,371585948
+removeobject,327859889
+removeobject,287348277
+removeobject,319214105
+removeobject,408538253
+removeobject,315233492
+removeobject,321187307
+removeobject,308503342
+removeobject,481438858
+removeobject,464326677
+removeobject,458208213
+removeobject,334169040
+removeobject,380953007
+removeobject,306156310
+removeobject,311735117
+removeobject,304652713
+removeobject,447482399
+removeobject,464054872
+removeobject,308287070
+removeobject,366031357
+removeobject,223529039
+removeobject,305332454
+removeobject,440680123
+removeobject,283803681
+removeobject,410491748
+removeobject,223265289
+removeobject,306268419
+removeobject,347366809
+removeobject,314563701
+removeobject,302068212
+removeobject,283520375
+removeobject,377963268
+removeobject,454052170
+removeobject,298085627
+removeobject,457265001
+removeobject,233934260
+removeobject,432713854
+removeobject,315891533
+removeobject,445005587
+removeobject,321635590
+removeobject,390314770
+removeobject,230655210
+removeobject,340190850
+removeobject,305314423
+removeobject,444872212
+removeobject,223437549
+removeobject,385529268
+removeobject,280271669
+removeobject,323868430
+removeobject,307544418
+removeobject,238040604
+removeobject,310293395
+removeobject,412163151
+removeobject,367516076
+removeobject,288358726
+removeobject,303789421
+removeobject,435084818
+removeobject,411008890
+removeobject,467616201
+removeobject,219465170
+removeobject,327976568
+removeobject,332048651
+removeobject,353130028
+removeobject,329197320
--- a/artifacts/2022-09-23-s3lat/minio-20220917.csv
+++ b/artifacts/2022-09-23-s3lat/minio-20220917.csv
@ -0,0 +1,501 @@
+endpoint,nanoseconds
+listbuckets,110526156
+listbuckets,111223674
+listbuckets,110350078
+listbuckets,108763355
+listbuckets,108151200
+listbuckets,110049558
+listbuckets,111261740
+listbuckets,106870917
+listbuckets,109675307
+listbuckets,114928865
+listbuckets,112976189
+listbuckets,100144534
+listbuckets,103672802
+listbuckets,115743698
+listbuckets,120312198
+listbuckets,114315912
+listbuckets,117664396
+listbuckets,113756128
+listbuckets,110819985
+listbuckets,108017540
+listbuckets,107455412
+listbuckets,117667963
+listbuckets,113112343
+listbuckets,117163158
+listbuckets,111710827
+listbuckets,108587376
+listbuckets,113677870
+listbuckets,100432934
+listbuckets,111720496
+listbuckets,113415267
+listbuckets,109700442
+listbuckets,116870615
+listbuckets,104193832
+listbuckets,104911949
+listbuckets,107990314
+listbuckets,111855197
+listbuckets,108076368
+listbuckets,109569833
+listbuckets,110641629
+listbuckets,105085431
+listbuckets,113240314
+listbuckets,109189490
+listbuckets,119818535
+listbuckets,111259357
+listbuckets,107390627
+listbuckets,101319090
+listbuckets,106351333
+listbuckets,117073689
+listbuckets,119127359
+listbuckets,117266560
+listbuckets,110077426
+listbuckets,107452728
+listbuckets,112721966
+listbuckets,114883973
+listbuckets,120073375
+listbuckets,103458296
+listbuckets,118342265
+listbuckets,107996502
+listbuckets,109767879
+listbuckets,112580238
+listbuckets,113636111
+listbuckets,111267829
+listbuckets,112798154
+listbuckets,107265512
+listbuckets,108961006
+listbuckets,120928218
+listbuckets,109095182
+listbuckets,110399063
+listbuckets,118328765
+listbuckets,112884542
+listbuckets,117913147
+listbuckets,120238848
+listbuckets,116074701
+listbuckets,109990943
+listbuckets,114816077
+listbuckets,113207212
+listbuckets,112350518
+listbuckets,106189705
+listbuckets,106836706
+listbuckets,114519552
+listbuckets,112357584
+listbuckets,113073218
+listbuckets,108251780
+listbuckets,115340245
+listbuckets,109100358
+listbuckets,116337818
+listbuckets,110881217
+listbuckets,109903972
+listbuckets,113243376
+listbuckets,111667494
+listbuckets,102401925
+listbuckets,109793240
+listbuckets,118365209
+listbuckets,117380902
+listbuckets,116149538
+listbuckets,109060437
+listbuckets,106900385
+listbuckets,101213710
+listbuckets,103102282
+listbuckets,115980177
+putobject,1181582068
+putobject,1208319832
+putobject,1139007207
+putobject,1172748692
+putobject,1213269319
+putobject,1222148262
+putobject,1160354410
+putobject,1186840670
+putobject,1178184375
+putobject,1199772934
+putobject,1243156199
+putobject,1162462450
+putobject,1160912226
+putobject,1181903350
+putobject,1178461918
+putobject,1155923039
+putobject,1198486430
+putobject,1128172593
+putobject,1161827240
+putobject,1191934325
+putobject,1241903028
+putobject,1154793146
+putobject,1182964786
+putobject,1192949284
+putobject,1211748830
+putobject,1153969786
+putobject,1173133011
+putobject,1159925140
+putobject,1151781019
+putobject,1212786711
+putobject,1136411443
+putobject,1161607447
+putobject,1194069897
+putobject,1181340894
+putobject,1154251172
+putobject,1142482758
+putobject,1184635609
+putobject,1401096568
+putobject,1193309386
+putobject,1155633130
+putobject,1185629277
+putobject,1150317409
+putobject,1189977499
+putobject,1172978955
+putobject,1167393215
+putobject,1176612406
+putobject,1220478477
+putobject,1214146462
+putobject,1201982436
+putobject,1187568105
+putobject,1233590518
+putobject,1254380879
+putobject,1195264797
+putobject,1212347722
+putobject,1265459581
+putobject,1199507629
+putobject,1213030838
+putobject,1261491944
+putobject,1175783260
+putobject,1171776028
+putobject,1175985781
+putobject,1204880209
+putobject,1185762634
+putobject,1205618309
+putobject,1156324684
+putobject,1212986779
+putobject,1153785732
+putobject,1186241798
+putobject,1169988181
+putobject,1169236282
+putobject,1159114879
+putobject,1169866154
+putobject,1185878817
+putobject,1303357322
+putobject,1198903604
+putobject,1180614532
+putobject,1198891504
+putobject,1186606007
+putobject,1178912791
+putobject,1181214681
+putobject,1148254776
+putobject,1193521548
+putobject,1172732700
+putobject,1224354922
+putobject,1189768028
+putobject,1187610090
+putobject,1158181354
+putobject,1168240861
+putobject,1184266557
+putobject,1211834285
+putobject,1133902391
+putobject,1169623406
+putobject,1153648781
+putobject,1260717643
+putobject,1213147555
+putobject,1196178431
+putobject,1193078015
+putobject,1192743109
+putobject,1231998132
+putobject,1169290675
+listobjects,445829270
+listobjects,448358224
+listobjects,234487499
+listobjects,331279803
+listobjects,239147274
+listobjects,356161983
+listobjects,231056963
+listobjects,428733347
+listobjects,234973895
+listobjects,349647146
+listobjects,238785967
+listobjects,430581787
+listobjects,325690302
+listobjects,337483670
+listobjects,335639646
+listobjects,327396437
+listobjects,230177176
+listobjects,347284630
+listobjects,230553012
+listobjects,338821210
+listobjects,343466400
+listobjects,346533869
+listobjects,311374291
+listobjects,341162492
+listobjects,235683323
+listobjects,356305631
+listobjects,234888137
+listobjects,335916950
+listobjects,312208053
+listobjects,345090441
+listobjects,449596366
+listobjects,347544888
+listobjects,361989101
+listobjects,235502161
+listobjects,232949965
+listobjects,337218790
+listobjects,447643000
+listobjects,238259618
+listobjects,235023776
+listobjects,420231514
+listobjects,341833697
+listobjects,235388450
+listobjects,232164722
+listobjects,422949063
+listobjects,501989025
+listobjects,240957275
+listobjects,337846216
+listobjects,231434805
+listobjects,430682948
+listobjects,231812947
+listobjects,465260992
+listobjects,433379275
+listobjects,236331095
+listobjects,332051421
+listobjects,237905130
+listobjects,343574528
+listobjects,242371130
+listobjects,311241850
+listobjects,239692903
+listobjects,360069478
+listobjects,317977063
+listobjects,324532475
+listobjects,246746304
+listobjects,339939390
+listobjects,238956624
+listobjects,338334961
+listobjects,236968116
+listobjects,338529503
+listobjects,235480073
+listobjects,334150022
+listobjects,234014534
+listobjects,336955875
+listobjects,245541026
+listobjects,321342105
+listobjects,241862995
+listobjects,342284075
+listobjects,223874385
+listobjects,330173393
+listobjects,230710360
+listobjects,338012760
+listobjects,243166839
+listobjects,326244217
+listobjects,217795705
+listobjects,343998726
+listobjects,234484396
+listobjects,326419671
+listobjects,228413611
+listobjects,345526317
+listobjects,242498473
+listobjects,322335836
+listobjects,325652509
+listobjects,353395591
+listobjects,241202612
+listobjects,316928525
+listobjects,415736209
+listobjects,336640460
+listobjects,235664816
+listobjects,318470444
+listobjects,233137178
+listobjects,338995281
+getobject,548766727
+getobject,544293094
+getobject,553996833
+getobject,524118176
+getobject,540256688
+getobject,521985460
+getobject,550010163
+getobject,522839295
+getobject,550206788
+getobject,556115384
+getobject,561480120
+getobject,529867231
+getobject,554654221
+getobject,522713545
+getobject,533201280
+getobject,559755253
+getobject,562719587
+getobject,530367590
+getobject,534634166
+getobject,528476593
+getobject,561330984
+getobject,548154277
+getobject,551287855
+getobject,543519973
+getobject,537798946
+getobject,543821201
+getobject,556706828
+getobject,562779335
+getobject,533088136
+getobject,543496053
+getobject,530090638
+getobject,557966646
+getobject,546652328
+getobject,556049330
+getobject,553087013
+getobject,552319581
+getobject,538039579
+getobject,524047395
+getobject,529321927
+getobject,561628538
+getobject,538975026
+getobject,537678052
+getobject,552261757
+getobject,553552417
+getobject,546538519
+getobject,539953289
+getobject,533392613
+getobject,543443065
+getobject,539024543
+getobject,537067523
+getobject,555793303
+getobject,610983302
+getobject,535123120
+getobject,528049669
+getobject,541554744
+getobject,564902500
+getobject,564639999
+getobject,534295582
+getobject,544951246
+getobject,540628827
+getobject,552900208
+getobject,541614941
+getobject,536825865
+getobject,535642848
+getobject,544302224
+getobject,549999591
+getobject,542030234
+getobject,545602568
+getobject,543278126
+getobject,524720136
+getobject,552230698
+getobject,539300237
+getobject,549856120
+getobject,540587629
+getobject,557880339
+getobject,555153802
+getobject,561113528
+getobject,556108255
+getobject,508374452
+getobject,560044395
+getobject,585922680
+getobject,548465899
+getobject,573308704
+getobject,554195559
+getobject,562441181
+getobject,550048971
+getobject,542032477
+getobject,587508303
+getobject,548308208
+getobject,529401594
+getobject,535337894
+getobject,566348968
+getobject,545058294
+getobject,560623443
+getobject,532580429
+getobject,555774911
+getobject,546448443
+getobject,526984769
+getobject,528538465
+getobject,551758307
+removeobject,1305475677
+removeobject,1328695106
+removeobject,1304086325
+removeobject,1314629714
+removeobject,1286472737
+removeobject,1323794460
+removeobject,1287026749
+removeobject,1308989986
+removeobject,1319191249
+removeobject,1302459188
+removeobject,1358027169
+removeobject,1298833331
+removeobject,1288044760
+removeobject,1309860548
+removeobject,1294431451
+removeobject,1323518263
+removeobject,1274537859
+removeobject,1306292281
+removeobject,1300283360
+removeobject,1236080169
+removeobject,1304088805
+removeobject,1319360876
+removeobject,1310348416
+removeobject,1306714462
+removeobject,1280353702
+removeobject,1313350545
+removeobject,1321192365
+removeobject,1304475647
+removeobject,1306586102
+removeobject,1294987618
+removeobject,1304176193
+removeobject,1296005363
+removeobject,1284994372
+removeobject,1317600899
+removeobject,1332743805
+removeobject,1352245087
+removeobject,1309279342
+removeobject,1307620458
+removeobject,1342551633
+removeobject,1321581956
+removeobject,1315162697
+removeobject,1336302765
+removeobject,1290969146
+removeobject,1320968235
+removeobject,1299056796
+removeobject,1288002161
+removeobject,1328125560
+removeobject,1316703843
+removeobject,1291268641
+removeobject,1327821693
+removeobject,1299867445
+removeobject,1296882414
+removeobject,1313288813
+removeobject,1298520791
+removeobject,1298185028
+removeobject,1274221533
+removeobject,1314194728
+removeobject,1336907835
+removeobject,1297413534
+removeobject,1283613694
+removeobject,1265657961
+removeobject,1329591094
+removeobject,1320682432
+removeobject,1317595366
+removeobject,1323562128
+removeobject,1298197589
+removeobject,1321950097
+removeobject,1281669970
+removeobject,1306586515
+removeobject,1296748831
+removeobject,1300070529
+removeobject,1282255502
+removeobject,1284141215
+removeobject,1297879969
+removeobject,1292837351
+removeobject,1317095287
+removeobject,1323647829
+removeobject,1283782591
+removeobject,1292866116
+removeobject,1283574839
+removeobject,1331580518
+removeobject,1355362975
+removeobject,1326405684
+removeobject,1297225883
+removeobject,1318610920
+removeobject,1298833112
+removeobject,1360833965
+removeobject,1311610613
+removeobject,1304567017
+removeobject,1325168434
+removeobject,1293403789
+removeobject,1293213442
+removeobject,1298548367
+removeobject,1303459493
+removeobject,1302979008
+removeobject,1336663252
+removeobject,1277657362
+removeobject,1308832733
+removeobject,1342049822
+removeobject,1307255424
--- a/artifacts/2022-09-23-s3lat/plot.R
+++ b/artifacts/2022-09-23-s3lat/plot.R
@ -0,0 +1,30 @@
+library(tidyverse)
+
+read_csv("garage-v0.7.csv") %>% add_column(daemon="garage 0.7.3") -> a
+read_csv("garage-v0.8.csv") %>% add_column(daemon="garage 0.8.0 beta") -> b
+read_csv("minio-20220917.csv") %>% add_column(daemon="minio RELEASE.2022-09-17") -> c
+bind_rows(a,b,c) %>% group_by(daemon,endpoint) %>% summarise(
+  time_mean = mean(nanoseconds) / 1000 / 1000,
+  time_max = max(nanoseconds) / 1000 / 1000,
+  time_min = min(nanoseconds) / 1000 / 1000
+) -> c
+
+st <- "100 measurements, 5 nodes, 50ms RTT + 10ms jitter between nodes\nno contention: latency is due to intra-cluster communications\ncolored bar = mean latency, error bar = min and max latency"
+
+
+ggplot(c, aes(x=endpoint,y=time_mean,fill=daemon,ymin=time_min,ymax=time_max)) +
+      geom_bar(stat="identity", position=position_dodge(),color="black") +
+      geom_errorbar(position=position_dodge(.9),width=.2) +
+      scale_y_continuous(expand=c(0,0))+
+      coord_flip() +
+      labs(
+        x="S3 Endpoint",
+        y="Request duration (ms)",
+        fill="Daemon",
+        caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+        title="S3 endpoint latency in a simulated geo-distributed cluster",
+        subtitle=st) +
+      theme_classic() +
+      theme(legend.position = c(.8, .3))
+
+ggsave("./plot.png", width=200, height=110, units="mm")
--- a/artifacts/2022-09-23-s3lat/plot.png
+++ b/artifacts/2022-09-23-s3lat/plot.png
--- a/artifacts/2022-09-23-s3ttfb/garage-v0.7.csv
+++ b/artifacts/2022-09-23-s3ttfb/garage-v0.7.csv
@ -0,0 +1,61 @@
+nanoseconds
+2043724295
+1819672397
+1856153241
+1814937942
+1818943177
+1814245475
+1828595614
+1817263829
+1857481054
+1816538404
+1816874169
+1856305403
+1857317361
+1814517488
+1817156893
+1815832311
+1857020274
+1822556545
+1816402439
+1815664395
+1818129711
+1855229810
+1817427508
+1817253968
+1855458740
+1818412087
+1816422102
+1814017661
+1821773596
+1814924392
+1819138108
+1814140079
+1814253338
+1814717178
+1823390172
+1856664915
+1817886566
+1815441938
+1815381667
+1864383216
+1815026715
+1816346397
+1814835227
+1855973303
+1814632968
+1823675273
+1817900846
+1816282804
+1817289580
+1817288129
+1860637032
+1855146585
+1816649086
+1822524427
+1814277922
+1901037764
+1818903030
+1815404774
+1815446844
+1815223628
--- a/artifacts/2022-09-23-s3ttfb/garage-v0.8.csv
+++ b/artifacts/2022-09-23-s3ttfb/garage-v0.8.csv
@ -0,0 +1,61 @@
+nanoseconds
+376764319
+268350943
+310303481
+266580460
+267172958
+46771899
+271623330
+270463024
+47401936
+45384570
+129505789
+46244307
+270594700
+267908944
+310163759
+271096844
+268930686
+309894956
+273312715
+269766963
+267947351
+270443270
+266292594
+215426567
+383550730
+84593093
+380970185
+419898767
+381574216
+422289246
+380535266
+268204508
+46449693
+270408730
+268572254
+308981862
+266303861
+46092770
+308073958
+383696451
+381725197
+379008890
+379902614
+382568588
+45370580
+83526768
+381230253
+267980416
+269187238
+84449786
+267524276
+46169934
+43509278
+308776528
+270626080
+271203190
+216167767
+239794865
+46230173
+267821837
--- a/artifacts/2022-09-23-s3ttfb/minio-20220917.csv
+++ b/artifacts/2022-09-23-s3ttfb/minio-20220917.csv
@ -0,0 +1,61 @@
+nanoseconds
+941737601
+970059899
+956848578
+981034132
+968456906
+986209313
+947371347
+961890275
+951011001
+947766957
+946381130
+1172968280
+944672409
+946216350
+960072933
+947966145
+947998544
+957730811
+960898554
+950611253
+953195848
+974654976
+982578891
+957419979
+947448876
+945953214
+936687543
+950116533
+979971727
+941427574
+931898843
+928494073
+927924296
+940995728
+927965457
+928098595
+933759012
+929902652
+953753466
+931448442
+934503899
+933297864
+934775226
+936615695
+950945991
+935054323
+1162136321
+940502403
+1158490674
+945337099
+948016443
+939863143
+938112935
+933271354
+946325434
+954355353
+948997521
+1124820589
+989829976
+938639125
--- a/artifacts/2022-09-23-s3ttfb/plot.R
+++ b/artifacts/2022-09-23-s3ttfb/plot.R
@ -0,0 +1,25 @@
+library(tidyverse)
+
+
+read_csv("garage-v0.7.csv") %>% add_column(daemon="garage 0.7.3") -> a
+read_csv("garage-v0.8.csv") %>% add_column(daemon="garage 0.8.0 beta") -> b
+read_csv("minio-20220917.csv") %>% add_column(daemon="minio RELEASE.2022-09-17") -> c
+bind_rows(a,b,c) %>% mutate(ms = nanoseconds / 1000 / 1000) -> d
+
+ggplot(d, aes(y = ms, group = daemon, fill=daemon)) +
+  #geom_violin(trim = TRUE, width=2) + 
+  geom_histogram(binwidth=50, color="black") +
+  scale_y_continuous(expand=c(0,0))+
+  scale_x_continuous(expand=c(0,0))+
+  coord_flip() + 
+  labs(
+    y="TTFB (ms)",
+    x="Count",
+    fill="Daemon",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="TTFB (Time To First Byte) on GetObject over a slow network (5 Mbps, 500 µs)",
+    subtitle="A 1MB file is uploaded and then fetched 60 times.\nExcept for Minio, the queried node does not store any data (gateway) to force net. communications.") +
+  theme_classic() +
+  theme(legend.position = c(.15, .8))
+
+ggsave("./plot.png", width=200, height=110, units="mm")
--- a/artifacts/2022-09-23-s3ttfb/plot.png
+++ b/artifacts/2022-09-23-s3ttfb/plot.png
--- a/artifacts/2022-09-23-warp/dc-garage-v0.7.zst
+++ b/artifacts/2022-09-23-warp/dc-garage-v0.7.zst
--- a/artifacts/2022-09-23-warp/plot.R
+++ b/artifacts/2022-09-23-warp/plot.R
@ -0,0 +1,21 @@
+library(tidyverse)
+
+
+read_csv("summary.csv")-> s
+
+hum_names <- as_labeller(c(`obj_per_sec` = "Objects/sec", `mbyte_per_sec` = "MByte/sec"))
+
+
+ggplot(s, aes(x=value,y=config)) +
+  geom_bar(stat="identity", fill="#00BFC4", color="black") +
+  facet_wrap(~measurement,dir="v", scales = "free", labeller = hum_names) +
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  labs(
+    y="Daemon",
+    x="Metric",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="\"minio/warp\" benchmark, \"cluster total\" result",
+    subtitle="Ran on a local machine (Ryzen 5 1400, 16GB RAM, SSD) with mknet\nDC topology (3 nodes, 1GB/s, 1ms lat)\nwarp in mixed mode, 5min bench, 5MB objects, initialized with 200 objects") +
+  theme_classic() 
+
+ggsave("./plot.png", width=200, height=120, units="mm")
--- a/artifacts/2022-09-23-warp/plot.png
+++ b/artifacts/2022-09-23-warp/plot.png
--- a/artifacts/2022-09-23-warp/summary.csv
+++ b/artifacts/2022-09-23-warp/summary.csv
@ -0,0 +1,11 @@
+config,measurement,value
+garage v0.7,mbyte_per_sec,8.60
+garage v0.7,obj_per_sec,3.02
+garage v0.8-beta1,mbyte_per_sec,12.41
+garage v0.8-beta1,obj_per_sec,4.36
+minio 20220917,mbyte_per_sec,20.22
+minio 20220917,obj_per_sec,7.11
+garage v0.8-beta2,mbyte_per_sec,14.56
+garage v0.8-beta2,obj_per_sec,5.19
+garage v0.8-no-fsync,mbyte_per_sec,17.31
+garage v0.8-no-fsync,obj_per_sec,6.11
--- a/artifacts/2022-09-24-s3billion/garage-regression.png
+++ b/artifacts/2022-09-24-s3billion/garage-regression.png
--- a/artifacts/2022-09-24-s3billion/garage-v0.8-beta2-lmdb.csv
+++ b/artifacts/2022-09-24-s3billion/garage-v0.8-beta2-lmdb.csv
@ -0,0 +1,129 @@
+total_objects,batch_dur_nanoseconds
+0,2018191689
+8192,1819596313
+16384,2102802010
+24576,2473935158
+32768,4510296173
+40960,6613958698
+49152,7977651423
+57344,13671435594
+65536,8060203586
+73728,3795495684
+81920,5794535775
+90112,5543820710
+98304,4519985809
+106496,5110068206
+114688,6708672963
+122880,7857479433
+131072,20642163540
+139264,5635269044
+147456,9351347336
+155648,5666222073
+163840,6501995906
+172032,5293598878
+180224,9350079270
+188416,13303962641
+196608,17428285824
+204800,11733919058
+212992,8372494836
+221184,26574754854
+229376,6314078661
+237568,9683594309
+245760,21850576414
+253952,8991130243
+262144,6567887673
+270336,9454781018
+278528,25552173846
+286720,9580711949
+294912,10257632597
+303104,7478191638
+311296,10501501215
+319488,3816776322
+327680,26795462810
+335872,9396360089
+344064,11208895014
+352256,21126662513
+360448,14198029001
+368640,5093580346
+376832,21004203835
+385024,14433396301
+393216,8933953570
+401408,5461923067
+409600,30844223516
+417792,8315342751
+425984,6166609079
+434176,29359253375
+442368,7642009005
+450560,7036982475
+458752,27168386971
+466944,10015226557
+475136,9245612551
+483328,24224252348
+491520,13827842768
+499712,9702435577
+507904,26083366912
+516096,7829114232
+524288,10792095771
+532480,28058503484
+540672,9984663485
+548864,10714515140
+557056,28472749292
+565248,10832278691
+573440,7594452219
+581632,23858923965
+589824,38909249537
+598016,15112735865
+606208,9644628857
+614400,28367134578
+622592,35180699079
+630784,14207640136
+638976,13236547125
+647168,37982462879
+655360,8448278050
+663552,11632520113
+671744,22661429490
+679936,29261192164
+688128,11982484321
+696320,30381827880
+704512,23530964587
+712704,8219614193
+720896,34002514929
+729088,21279203762
+737280,6880671212
+745472,10884265763
+753664,41999879903
+761856,13465319479
+770048,10792229369
+778240,29004881357
+786432,24372150617
+794624,13327280319
+802816,18478244110
+811008,48319469581
+819200,14174987827
+827392,14368184404
+835584,56647745264
+843776,9880099642
+851968,12808097118
+860160,28380242726
+868352,44970657202
+876544,18733551704
+884736,9340561806
+892928,25246663514
+901120,46462124425
+909312,12856639771
+917504,9028224897
+925696,38580954489
+933888,48184562190
+942080,11204051215
+950272,19754687230
+958464,36364975788
+966656,40759996520
+974848,10562565477
+983040,13905331718
+991232,54140254529
+999424,46643736817
+1007616,8362351299
+1015808,13514305332
+1024000,39389278472
+1032192,40328458712
+1040384,15958365846
--- a/artifacts/2022-09-24-s3billion/garage.png
+++ b/artifacts/2022-09-24-s3billion/garage.png
--- a/artifacts/2022-09-24-s3billion/minio.csv
+++ b/artifacts/2022-09-24-s3billion/minio.csv
@ -0,0 +1,129 @@
+total_objects,batch_dur_nanoseconds
+0,35604165299
+8192,30027091125
+16384,36769805630
+24576,44154399690
+32768,36841962377
+40960,52989512753
+49152,52536807234
+57344,53537348443
+65536,53276055435
+73728,55539526464
+81920,54013225676
+90112,68561851906
+98304,69298493247
+106496,114291303648
+114688,78507905949
+122880,126223986043
+131072,121999273534
+139264,153563457891
+147456,151032625349
+155648,149855834712
+163840,153190455466
+172032,150844647704
+180224,156256328509
+188416,151849908862
+196608,154337246115
+204800,154008292236
+212992,160001140992
+221184,154134240373
+229376,147943023290
+237568,150989793433
+245760,161064597934
+253952,150058583272
+262144,158784561065
+270336,150652453675
+278528,108917197085
+286720,158143657766
+294912,151991188276
+303104,150904775153
+311296,152538005225
+319488,158677889479
+327680,151948293662
+335872,131622851465
+344064,120774952190
+352256,160866374832
+360448,157913387369
+368640,157965094573
+376832,155015634674
+385024,152010190282
+393216,151353578675
+401408,152816629041
+409600,100685222932
+417792,151049307826
+425984,152092080473
+434176,156352511800
+442368,156617304416
+450560,156604667758
+458752,163954135953
+466944,102070359622
+475136,155481160507
+483328,157268905807
+491520,162446380203
+499712,159234723683
+507904,141918580163
+516096,133297515392
+524288,92684666697
+532480,153688292459
+540672,154151441890
+548864,157288990588
+557056,159081065544
+565248,153244088615
+573440,161424543833
+581632,174752406147
+589824,155541074973
+598016,159471780365
+606208,158247043033
+614400,157224045247
+622592,160337074095
+630784,157993307652
+638976,156821062432
+647168,158852305038
+655360,161012982349
+663552,156762111328
+671744,159782719621
+679936,156694048459
+688128,175969992462
+696320,159626684812
+704512,156680995342
+712704,155624836949
+720896,161583816024
+729088,156832016477
+737280,160834148361
+745472,157761321600
+753664,157553638650
+761856,176048273498
+770048,156697917513
+778240,162791069016
+786432,156496526158
+794624,161388816617
+802816,159569344379
+811008,159393281615
+819200,158389581978
+827392,164002225318
+835584,162722200475
+843776,160146392865
+851968,164551137708
+860160,177034374419
+868352,161672813995
+876544,169063897017
+884736,160829838695
+892928,157613804708
+901120,162687585988
+909312,166612325773
+917504,160649709927
+925696,163214320096
+933888,178447566516
+942080,155968188136
+950272,161252816115
+958464,159781104139
+966656,163854372812
+974848,157176626359
+983040,160325050560
+991232,165218492441
+999424,160172384289
+1007616,159316385255
+1015808,178044565059
+1024000,164817520071
+1032192,159428724027
+1040384,159973237521
--- a/artifacts/2022-09-24-s3billion/minio.png
+++ b/artifacts/2022-09-24-s3billion/minio.png
--- a/artifacts/2022-09-24-s3billion/plot.R
+++ b/artifacts/2022-09-24-s3billion/plot.R
@ -0,0 +1,65 @@
+library(tidyverse)
+library(ggpmisc)
+
+
+read_csv("garage-v0.8-beta2-lmdb.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000) %>% filter(total_objects != 0) -> s
+
+reg1 <- lm(s$batch_dur_sec~s$total_objects)
+reg2 <- lm(s$batch_dur_sec ~ log(s$total_objects))
+
+f1 <- y~log(x)
+f2 <- y~x
+
+ggplot(s, aes(x=total_objects, y=batch_dur_sec)) +
+  geom_point() +
+  #geom_smooth(method="lm",formula=f1,  se = FALSE, color="red") + 
+  #geom_smooth(method="lm",formula=f2,  se = FALSE, color="blue") +
+  #stat_poly_eq(formula = f1, label.y = 0.9, color = "red", aes(label=paste(..eq.label..,..rr.label..,..adj.rr.label..,..AIC.label..,..BIC.label.., sep = "~~~"))) +
+  #stat_poly_eq(formula = f2, label.y = 0.8, color="blue",aes(label=paste(..eq.label..,..rr.label..,..adj.rr.label..,..AIC.label..,..BIC.label.., sep = "~~~"))) +
+  #geom_smooth(method = "gam", se = FALSE) + 
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  coord_cartesian(ylim=c(0,60)) +
+  labs(
+    y="Time (in sec) spent sending a batch (8192 objects)",
+    x="Total number of objects stored in the cluster",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Storing 1M+ files in Garage! Impact of existing data on cluster interactiveness",
+    subtitle="Daemon: Garage v0.8 beta 2 with LMDB as db_engine\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
+  theme_classic()
+ggsave("./garage.png", width=200, height=120, units="mm")
+#ggsave("./garage-regression.png", width=200, height=120, units="mm")
+
+
+
+read_csv("minio.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000 ) -> s2
+ggplot(s2, aes(x=total_objects, y=batch_dur_sec)) +
+  geom_point() +
+  #geom_smooth(method = "gam", se = FALSE) + 
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  labs(
+    y="Time (in sec) spent sending a batch (8192 objects)",
+    x="Total number of objects stored in the cluster",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Storing 1M+ files in Minio! Impact of existing data on cluster interactiveness",
+    subtitle="Daemon: Minio RELEASE 20220917\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
+  theme_classic()
+ggsave("./minio.png", width=200, height=120, units="mm")
+
+bind_rows(s %>% add_column(daemon="garage 0.8 beta2 lmdb"), s2 %>% add_column(daemon="minio RELEASE 20220917")) -> sc
+ggplot(sc, aes(x=total_objects, y=batch_dur_sec, group=daemon, color=daemon)) +
+  geom_point() +
+  #geom_smooth(method = "gam", se = FALSE) + 
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  labs(
+    y="Time (in sec) spent sending a batch (8192 objects)",
+    x="Total number of objects stored in the cluster",
+    fill="Daemon",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Storing 1M+ files in Garage and Minio! Impact of existing data on cluster interactiveness",
+    subtitle="Daemon: Garage v0.8 beta 2 with LMDB as db_engine, Minio RELEASE 20220917\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
+  theme_classic() + 
+  theme(legend.position = c(.85, .55))
+ggsave("./plot.png", width=200, height=120, units="mm")
--- a/artifacts/2022-09-24-s3billion/plot.png
+++ b/artifacts/2022-09-24-s3billion/plot.png
--- a/artifacts/2022-09-24-warp-db-engine/lmdb.log
+++ b/artifacts/2022-09-24-warp-db-engine/lmdb.log
@ -0,0 +1,15 @@
+Mixed operations.
+Operation: DELETE, 10%, Concurrency: 20, Ran 4m59s.
+ * Throughput: 266.79 obj/s
+
+Operation: GET, 45%, Concurrency: 20, Ran 4m59s.
+ * Throughput: 0.29 MiB/s, 1200.51 obj/s
+
+Operation: PUT, 15%, Concurrency: 20, Ran 4m59s.
+ * Throughput: 0.10 MiB/s, 400.20 obj/s
+
+Operation: STAT, 30%, Concurrency: 20, Ran 4m59s.
+ * Throughput: 800.32 obj/s
+
+Cluster Total: 0.39 MiB/s, 2667.83 obj/s over 5m0s.
+Aggregated data saved to /tmp/mknet-store/warp.csv
--- a/artifacts/2022-09-24-warp-db-engine/plot.R
+++ b/artifacts/2022-09-24-warp-db-engine/plot.R
@ -0,0 +1,17 @@
+library(tidyverse)
+
+read_csv("summary.csv")-> s
+ggplot(s, aes(x=db_engine,y=rate)) +
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  geom_bar(stat='identity',fill="#00BFC4", color="black") +
+  coord_flip() +
+  facet_wrap(~operation, scales = "free", dir="v") +
+  labs(
+    x="Metadata Engine",
+    y="Object/sec",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Comparison of Garage's metadata engines with \"minio/warp\"",
+    subtitle="Daemon: Garage v0.8 no-fsync to avoid being impacted by block manager\nBenchmark: warp, mixed mode, 5min bench, 256B objects, initialized with 200 objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).",
+  ) +
+  theme_classic()
+ggsave("./plot.png", width=200, height=120, units="mm")
--- a/artifacts/2022-09-24-warp-db-engine/plot.png
+++ b/artifacts/2022-09-24-warp-db-engine/plot.png
--- a/artifacts/2022-09-24-warp-db-engine/sled-nosync.log
+++ b/artifacts/2022-09-24-warp-db-engine/sled-nosync.log
@ -0,0 +1,13 @@
+Operation: DELETE, 10%, Concurrency: 20, Ran 3m42s.
+ * Throughput: 161.98 obj/s
+
+Operation: GET, 45%, Concurrency: 20, Ran 3m42s.
+ * Throughput: 0.18 MiB/s, 729.07 obj/s
+
+Operation: PUT, 15%, Concurrency: 20, Ran 3m42s.
+ * Throughput: 0.06 MiB/s, 242.99 obj/s
+
+Operation: STAT, 30%, Concurrency: 20, Ran 3m42s.
+ * Throughput: 486.09 obj/s
+
+Cluster Total: 0.24 MiB/s, 1620.14 obj/s over 3m42s.
--- a/artifacts/2022-09-24-warp-db-engine/sled.log
+++ b/artifacts/2022-09-24-warp-db-engine/sled.log
@ -0,0 +1,15 @@
+Mixed operations.
+Operation: DELETE, 10%, Concurrency: 20, Ran 3m49s.
+ * Throughput: 129.83 obj/s
+
+Operation: GET, 45%, Concurrency: 20, Ran 3m49s.
+ * Throughput: 0.14 MiB/s, 584.32 obj/s
+
+Operation: PUT, 15%, Concurrency: 20, Ran 3m49s.
+ * Throughput: 0.05 MiB/s, 194.81 obj/s
+
+Operation: STAT, 30%, Concurrency: 20, Ran 3m49s.
+ * Throughput: 389.52 obj/s
+
+Cluster Total: 0.19 MiB/s, 1298.49 obj/s over 3m49s.
+Aggregated data saved to /tmp/mknet-store/warp.csv
--- a/artifacts/2022-09-24-warp-db-engine/sqlite.log
+++ b/artifacts/2022-09-24-warp-db-engine/sqlite.log
@ -0,0 +1,15 @@
+Mixed operations.
+Operation: DELETE, 10%, Concurrency: 20, Ran 4m55s.
+ * Throughput: 1.29 obj/s
+
+Operation: GET, 45%, Concurrency: 20, Ran 4m55s.
+ * Throughput: 0.00 MiB/s, 5.75 obj/s
+
+Operation: PUT, 15%, Concurrency: 20, Ran 4m55s.
+ * Throughput: 0.00 MiB/s, 1.93 obj/s
+
+Operation: STAT, 30%, Concurrency: 20, Ran 4m56s.
+ * Throughput: 3.86 obj/s
+
+Cluster Total: 0.00 MiB/s, 12.83 obj/s over 4m55s.
+Aggregated data saved to /tmp/mknet-store/warp.csv
--- a/artifacts/2022-09-24-warp-db-engine/summary.csv
+++ b/artifacts/2022-09-24-warp-db-engine/summary.csv
@ -0,0 +1,17 @@
+db_engine,operation,rate
+lmdb,delete,266.79
+lmdb,get,1200.51
+lmdb,put,400
+lmdb,stat,800
+sled,delete,129.83
+sled,get,584.32
+sled,put,194.81
+sled,stat,389.52
+sqlite,delete,1.29
+sqlite,get,5.75
+sqlite,put,1.93
+sqlite,stat,3.86
+sled-nosync,delete,161.98
+sled-nosync,get,729.07
+sled-nosync,put,242.99
+sled-nosync,stat,486.09
--- a/artifacts/2022-09-29-s3billion/garage-0.8-gros.png
+++ b/artifacts/2022-09-29-s3billion/garage-0.8-gros.png
--- a/artifacts/2022-09-29-s3billion/plot.R
+++ b/artifacts/2022-09-29-s3billion/plot.R
@ -0,0 +1,27 @@
+library(tidyverse)
+library(ggpmisc)
+library(scales)
+
+
+read_csv("s3billion-dc-gros-32threads-256batchsize-128000batchcount-v0.8-fix-time-30M.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000) %>% filter(total_objects != 0) -> s
+
+reg <- lm(s$batch_dur_sec~s$total_objects)
+
+f <- y~x
+
+ggplot(s, aes(x=total_objects, y=batch_dur_sec)) +
+  geom_point(shape = "bullet", alpha=0.4) +
+  geom_smooth(method="lm", formula=f, se = FALSE, color="blue") + 
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10), labels = unit_format(unit = "M", scale = 1e-6)) +
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10)) +
+  coord_cartesian(ylim=c(0,3)) +
+  labs(
+    y="Time (in sec) spent sending a batch (8192 objects)",
+    x="Total number of objects stored in the cluster",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Storing 30M small objects in Garage!",
+    subtitle="Daemon: Garage v0.8.0-rc1 (+ #398 pull request) with LMDB as db_engine\nBenchmark: 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet on Grid'5000 (gros) https://www.grid5000.fr/w/Nancy:Hardware#gros (SATA SSD).\nDC topo (3 logical nodes, 1Gb/s, 1ms latency).") +
+  theme_classic()
+ggsave("./garage-0.8-gros.png", width=200, height=120, units="mm")
+#ggsave("./garage-regression.png", width=200, height=120, units="mm")
+
--- a/artifacts/2022-09-29-s3billion/s3billion-dc-gros-32threads-256batchsize-128000batchcount-v0.8-fix-time-30M.csv
+++ b/artifacts/2022-09-29-s3billion/s3billion-dc-gros-32threads-256batchsize-128000batchcount-v0.8-fix-time-30M.csv
--- a/benchmarks/s3billion/.gitignore
+++ b/benchmarks/s3billion/.gitignore
@ -0,0 +1 @@
+s3billion
--- a/benchmarks/s3billion/README.md
+++ b/benchmarks/s3billion/README.md
@ -0,0 +1,22 @@
+# s3lat
+
+## installation
+
+```bash
+go get git.deuxfleurs.fr/quentin/s3lat@latest
+```
+
+## usage
+
+```bash
+export ENDPOINT=[fc00:9a7a:9e::1]:9000
+export AWS_ACCESS_KEY_ID=minioadmin
+export AWS_SECRET_ACCESS_KEY=minioadmin
+
+s3lat
+```
+
+## see also
+
+ - https://git.deuxfleurs.fr/quentin/benchmarks
+ - https://git.deuxfleurs.fr/trinity-1686a/mknet
--- a/benchmarks/s3billion/go.mod
+++ b/benchmarks/s3billion/go.mod
@ -0,0 +1,8 @@
+module git.deuxfleurs.fr/Deuxfleurs/mknet/benchmarks/s3billion
+
+go 1.16
+
+require (
+	github.com/google/uuid v1.1.1
+	github.com/minio/minio-go/v7 v7.0.16
+)
--- a/benchmarks/s3billion/go.sum
+++ b/benchmarks/s3billion/go.sum
@ -0,0 +1,78 @@
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
+github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
+github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
+github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
+github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
+github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
+github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
+github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
+github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
+github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
+github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
+github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
+github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
+github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
+github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
+github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae h1:Ih9Yo4hSPImZOpfGuA4bR/ORKTAbhZo2AbWNRCnevdo=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
+gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
--- a/benchmarks/s3billion/main.go
+++ b/benchmarks/s3billion/main.go
@ -0,0 +1,176 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"fmt"
+	"io"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+	"strconv"
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+)
+
+type PRNG struct {
+	rem int64
+}
+
+func (r *PRNG) Read(p []byte) (n int, err error) {
+	//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
+	if int64(len(p)) > r.rem {
+		p = p[:r.rem]
+	}
+
+	if int64(len(p)) > r.rem {
+		log.Fatal("LOGIC ERROR")
+	}
+
+	n, err = rand.Read(p)
+	if err != nil {
+		return
+	}
+	r.rem -= int64(n)
+	if r.rem <= 0 {
+		err = io.EOF
+		//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
+	}
+	return
+}
+
+func putObj(mc *minio.Client, buck string, size int64) error {
+	prng := new(PRNG)
+	prng.rem = size
+
+	key := uuid.New().String()
+
+	_, err := mc.PutObject(
+		context.Background(),
+		buck,
+		key,
+		prng,
+		size,
+		minio.PutObjectOptions{ContentType: "application/octet-stream"},
+	)
+
+	return err
+}
+
+func main() {
+	fmt.Printf("total_objects,batch_dur_nanoseconds\n")
+
+	minio.MaxRetry = 1
+
+	_, isSSL := os.LookupEnv("SSL")
+	opts := minio.Options{
+		Creds:  credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
+		Secure: isSSL,
+	}
+
+	if region, ok := os.LookupEnv("AWS_REGION"); ok {
+		opts.Region = region
+	}
+
+	if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
+		opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
+	}
+
+	mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
+	if err != nil {
+		log.Fatal("failed connect", err)
+		return
+	}
+
+	thread := 32
+	if env_thread, ok := os.LookupEnv("THREAD"); ok {
+		tmp, err := strconv.Atoi(env_thread)
+		if err != nil {
+			log.Fatalf("invalid value for THREAD: %v\n", env_thread)
+		}
+		thread = tmp
+	}
+
+	batch_size := 256
+	if env_batch_size, ok := os.LookupEnv("BATCH_SIZE"); ok {
+		tmp, err := strconv.Atoi(env_batch_size)
+		if err != nil {
+			log.Fatalf("invalid value for BATCH_SIZE: %v\n", env_batch_size)
+		}
+		batch_size = tmp
+	}
+
+	batch_count := 128
+	if env_batch_count, ok := os.LookupEnv("BATCH_COUNT"); ok {
+		tmp, err := strconv.Atoi(env_batch_count)
+		if err != nil {
+			log.Fatalf("invalid value for BATCH_COUNT: %v\n", env_batch_count)
+		}
+		batch_count = tmp
+	}
+
+	obj_size := 16
+	if env_obj_size, ok := os.LookupEnv("OBJ_SIZE"); ok {
+		tmp, err := strconv.Atoi(env_obj_size)
+		if err != nil {
+			log.Fatalf("invalid value for OBJ_SIZE: %v\n", env_obj_size)
+		}
+		obj_size = tmp
+	}
+
+	total_obj := thread * batch_size * batch_count
+	total_size := total_obj * obj_size
+	log.Printf("if bench succeed, %v objects (%v bytes) will be created\n", total_obj, total_size)
+
+	// Create Bucket
+	buck := uuid.New().String()
+	err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
+	if err != nil {
+		log.Fatal(err)
+		return
+	}
+	log.Printf("created bucket %s\n", buck)
+
+	// Start sending...
+	for bc := 0; bc < batch_count; bc++ {
+		log.Printf("batch %d/%d - start\n", bc+1, batch_count)
+
+		start := time.Now()
+		syn := make(chan error)
+
+		for tc := 0; tc < thread; tc++ {
+			go func() {
+				for bs := 0; bs < batch_size; bs++ {
+					err := putObj(mc, buck, int64(obj_size))
+					if err != nil {
+						syn <- err
+						return
+					}
+				}
+				syn <- nil
+
+			}()
+		}
+		log.Printf("batch %d/%d - all threads started\n", bc+1, batch_count)
+
+		errCount := 0
+		for tc := 0; tc < thread; tc++ {
+			cerr := <-syn
+			if cerr != nil {
+				errCount += 1
+				log.Printf("thread %d/%d failed with %s\n", tc, thread, cerr)
+			}
+		}
+		if errCount > 0 {
+			log.Fatal("Too many errors, exiting...")
+			return
+		}
+		elapsed := time.Since(start)
+		fmt.Printf("%d,%v\n", bc * thread * batch_size, elapsed.Nanoseconds())
+		log.Printf("batch %d/%d - all threads returned\n", bc+1, batch_count)
+	}
+}
--- a/benchmarks/s3concurrent/.gitignore
+++ b/benchmarks/s3concurrent/.gitignore
@ -0,0 +1 @@
+s3concurrent
--- a/benchmarks/s3concurrent/go.mod
+++ b/benchmarks/s3concurrent/go.mod
@ -0,0 +1,24 @@
+module git.deuxfleurs.fr/quentin/s3concurrent
+
+go 1.18
+
+require github.com/minio/minio-go/v7 v7.0.34
+
+require (
+	github.com/dustin/go-humanize v1.0.0 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/klauspost/compress v1.15.9 // indirect
+	github.com/klauspost/cpuid/v2 v2.1.0 // indirect
+	github.com/minio/md5-simd v1.1.2 // indirect
+	github.com/minio/sha256-simd v1.0.0 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/rs/xid v1.4.0 // indirect
+	github.com/sirupsen/logrus v1.9.0 // indirect
+	golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa // indirect
+	golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
+	golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
+	golang.org/x/text v0.3.7 // indirect
+	gopkg.in/ini.v1 v1.66.6 // indirect
+)
--- a/benchmarks/s3concurrent/go.sum
+++ b/benchmarks/s3concurrent/go.sum
@ -0,0 +1,52 @@
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
+github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
+github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
+github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY=
+github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
+github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0=
+github.com/klauspost/cpuid/v2 v2.1.0/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY=
+github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
+github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
+github.com/minio/minio-go/v7 v7.0.34 h1:JMfS5fudx1mN6V2MMNyCJ7UMrjEzZzIvMgfkWc1Vnjk=
+github.com/minio/minio-go/v7 v7.0.34/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw=
+github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g=
+github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
+github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY=
+github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
+github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0=
+github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c=
+golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI=
+gopkg.in/ini.v1 v1.66.6/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
--- a/benchmarks/s3concurrent/main.go
+++ b/benchmarks/s3concurrent/main.go
@ -0,0 +1,132 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"fmt"
+	"time"
+	"io"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+
+	"github.com/google/uuid"
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+)
+
+func buildMc() (*minio.Client, error) {
+	_, isSSL := os.LookupEnv("SSL")
+	opts := minio.Options{
+		Creds:  credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
+		Secure: isSSL,
+	}
+
+	if region, ok := os.LookupEnv("REGION"); ok {
+		opts.Region = region
+	}
+
+	if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
+		opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
+	}
+
+	mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
+	return mc, err
+}
+
+type PRNG struct {
+	rem int64
+}
+
+func (r *PRNG) Read(p []byte) (n int, err error) {
+	//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
+	if int64(len(p)) > r.rem {
+		p = p[:r.rem]
+	}
+
+	if int64(len(p)) > r.rem {
+		log.Fatal("LOGIC ERROR")
+	}
+
+	n, err = rand.Read(p)
+	if err != nil {
+		return
+	}
+	r.rem -= int64(n)
+	if r.rem <= 0 {
+		err = io.EOF
+		//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
+	}
+	return
+}
+
+func putObj(buck string, size int64) error {
+	mc, err := buildMc()
+	if err != nil {
+		return err
+	}
+
+	prng := new(PRNG)
+	prng.rem = size
+
+	key := uuid.New().String()
+
+	_, err = mc.PutObject(
+		context.Background(),
+		buck,
+		key,
+		prng,
+		size,
+		minio.PutObjectOptions{ContentType: "application/octet-stream"},
+	)
+
+	return err
+}
+
+func main() {
+	minio.MaxRetry = 1
+	mc, err := buildMc()
+	if err != nil {
+		log.Fatal("failed connect", err)
+		return
+	}
+
+	// Create Bucket
+	buck := uuid.New().String()
+	err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
+	if err != nil {
+		log.Fatal(err)
+		return
+	}
+	log.Printf("created bucket %s\n", buck)
+
+	fmt.Println("sent,success,elapsed,elapsed_per_req")
+
+	// Send to bucket
+	for i := 1; i <= 16; i++ {
+		log.Printf("start concurrent loop with %d coroutines\n", i)
+		syn := make(chan error)
+		
+		start := time.Now()
+		for j := 1; j <= i; j++ {
+			go func() {
+				syn <- putObj(buck, 1024*1024)
+			}()
+		}
+
+		errCount := 0
+		for j := 1; j <= i; j++ {
+			cerr := <-syn
+			if cerr != nil {
+				errCount += 1
+				log.Printf("%d/%d failed with %s\n", j, i, cerr)
+			}
+		}
+		elapsed := time.Since(start)
+		fmt.Printf("%d,%d,%v,%v\n", i, i - errCount, elapsed.Nanoseconds(), elapsed.Nanoseconds() / int64(i))
+		log.Printf("done, %d coroutines returned\n", i)
+	}
+
+	log.Println("bench done")
+}
--- a/benchmarks/s3lat/.gitignore
+++ b/benchmarks/s3lat/.gitignore
@ -0,0 +1 @@
+s3lat
--- a/benchmarks/s3lat/README.md
+++ b/benchmarks/s3lat/README.md
@ -0,0 +1,22 @@
+# s3lat
+
+## installation
+
+```bash
+go get git.deuxfleurs.fr/quentin/s3lat@latest
+```
+
+## usage
+
+```bash
+export ENDPOINT=[fc00:9a7a:9e::1]:9000
+export AWS_ACCESS_KEY_ID=minioadmin
+export AWS_SECRET_ACCESS_KEY=minioadmin
+
+s3lat
+```
+
+## see also
+
+ - https://git.deuxfleurs.fr/quentin/benchmarks
+ - https://git.deuxfleurs.fr/trinity-1686a/mknet
--- a/benchmarks/s3lat/go.mod
+++ b/benchmarks/s3lat/go.mod
@ -0,0 +1,8 @@
+module git.deuxfleurs.fr/quentin/s3lat
+
+go 1.16
+
+require (
+	github.com/google/uuid v1.1.1
+	github.com/minio/minio-go/v7 v7.0.16
+)
--- a/benchmarks/s3lat/go.sum
+++ b/benchmarks/s3lat/go.sum
@ -0,0 +1,78 @@
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
+github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
+github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
+github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
+github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
+github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
+github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
+github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
+github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
+github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
+github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
+github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
+github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
+github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
+github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
+github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae h1:Ih9Yo4hSPImZOpfGuA4bR/ORKTAbhZo2AbWNRCnevdo=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
+gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
--- a/benchmarks/s3lat/main.go
+++ b/benchmarks/s3lat/main.go
@ -0,0 +1,135 @@
+package main
+
+import (
+	"context"
+	"log"
+	"os"
+	"fmt"
+	"time"
+	"io"
+	"io/ioutil"
+	"strings"
+	"net/http"
+	"crypto/tls"
+	"strconv"
+
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+    "github.com/google/uuid"
+)
+
+func main() {
+	fmt.Printf("endpoint,nanoseconds\n")
+
+    // Initial setup
+	_, isSSL := os.LookupEnv("SSL");
+	opts := minio.Options {
+		Creds:  credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
+		Secure: isSSL,
+	}
+
+	if region, ok := os.LookupEnv("REGION"); ok {
+		opts.Region = region
+	}
+
+	if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
+		opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
+	}
+
+	mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
+
+	if err != nil {
+		log.Fatal("failed connect", err)
+		return
+	}
+
+    // Create Bucket
+    buck := uuid.New().String()
+    err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{ })
+    if err != nil {
+        log.Fatal(err)
+        return
+    }
+
+    // List Buckets
+	for i := 0; i < 100; i++ {
+		start := time.Now()
+		_, err := mc.ListBuckets(context.Background())
+		elapsed := time.Since(start)
+		if err != nil {
+			log.Fatal("failed listbucket: ", err)
+		return
+		}
+		fmt.Printf("listbuckets,%v\n", elapsed.Nanoseconds())
+	}
+
+    // PutObject
+	for i := 0; i < 100; i++ {
+		istr := strconv.Itoa(i)
+		content := istr + " hello world " + istr
+		start := time.Now()
+		_, err := mc.PutObject(context.Background(), buck, "element"+istr, strings.NewReader(content), int64(len(content)), minio.PutObjectOptions{ContentType:"application/octet-stream"})
+		elapsed := time.Since(start)
+		if err != nil {
+			log.Fatal("failed putObject: ",err)
+			return
+		}
+		fmt.Printf("putobject,%v\n", elapsed.Nanoseconds())
+	}
+
+    // ListObject
+	for i := 0; i < 100; i++ {
+		ctx, cancel := context.WithCancel(context.Background())
+		defer cancel()
+
+		start := time.Now()
+		objectCh := mc.ListObjects(ctx, buck, minio.ListObjectsOptions{
+			Recursive: true,
+		})
+		for object := range objectCh {
+			if object.Err != nil {
+				log.Fatal(object.Err)
+				return
+			}
+		}
+		elapsed := time.Since(start)
+		fmt.Printf("listobjects,%v\n", elapsed.Nanoseconds())
+	}
+
+    // GetObject
+	for i := 0; i < 100; i++ {
+		istr := strconv.Itoa(i)
+		start := time.Now()
+		object, err := mc.GetObject(context.Background(), buck, "element"+istr, minio.GetObjectOptions{})
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+		if _, err = io.Copy(ioutil.Discard, object) ; err != nil {
+			log.Fatal("failed getobject: ", err)
+			return
+		}
+		elapsed := time.Since(start)
+		fmt.Printf("getobject,%v\n", elapsed.Nanoseconds())
+	}
+
+    // RemoveObject
+	for i := 0; i < 100; i++ {
+		istr := strconv.Itoa(i)
+		start := time.Now()
+		err = mc.RemoveObject(context.Background(), buck, "element"+istr, minio.RemoveObjectOptions{})
+		elapsed := time.Since(start)
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+		fmt.Printf("removeobject,%v\n", elapsed.Nanoseconds())
+	}
+
+    // RemoveBucket
+    err = mc.RemoveBucket(context.Background(), buck)
+    if err != nil {
+        log.Fatal(err)
+        return
+    }
+}
--- a/benchmarks/s3ttfb/.gitignore
+++ b/benchmarks/s3ttfb/.gitignore
@ -0,0 +1 @@
+s3ttfb
--- a/benchmarks/s3ttfb/go.mod
+++ b/benchmarks/s3ttfb/go.mod
@ -0,0 +1,12 @@
+module git.deuxfleurs.fr/Deuxfleurs/mknet/benchmarks/s3ttfb
+
+go 1.16
+
+require (
+	github.com/aws/aws-sdk-go-v2 v1.16.16
+	github.com/aws/aws-sdk-go-v2/config v1.17.7
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11
+	github.com/google/uuid v1.1.1
+	github.com/minio/minio-go/v7 v7.0.16
+	golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd // indirect
+)
--- a/benchmarks/s3ttfb/go.sum
+++ b/benchmarks/s3ttfb/go.sum
@ -0,0 +1,123 @@
+github.com/aws/aws-sdk-go-v2 v1.16.16 h1:M1fj4FE2lB4NzRb9Y0xdWsn2P0+2UHVxwKyOa4YJNjk=
+github.com/aws/aws-sdk-go-v2 v1.16.16/go.mod h1:SwiyXi/1zTUZ6KIAmLK5V5ll8SiURNUYOqTerZPaF9k=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.8 h1:tcFliCWne+zOuUfKNRn8JdFBuWPDuISDH08wD2ULkhk=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.8/go.mod h1:JTnlBSot91steJeti4ryyu/tLd4Sk84O5W22L7O2EQU=
+github.com/aws/aws-sdk-go-v2/config v1.17.7 h1:odVM52tFHhpqZBKNjVW5h+Zt1tKHbhdTQRb+0WHrNtw=
+github.com/aws/aws-sdk-go-v2/config v1.17.7/go.mod h1:dN2gja/QXxFF15hQreyrqYhLBaQo1d9ZKe/v/uplQoI=
+github.com/aws/aws-sdk-go-v2/credentials v1.12.20 h1:9+ZhlDY7N9dPnUmf7CDfW9In4sW5Ff3bh7oy4DzS1IE=
+github.com/aws/aws-sdk-go-v2/credentials v1.12.20/go.mod h1:UKY5HyIux08bbNA7Blv4PcXQ8cTkGh7ghHMFklaviR4=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.17 h1:r08j4sbZu/RVi+BNxkBJwPMUYY3P8mgSDuKkZ/ZN1lE=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.17/go.mod h1:yIkQcCDYNsZfXpd5UX2Cy+sWA1jPgIhGTw9cOBzfVnQ=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.23 h1:s4g/wnzMf+qepSNgTvaQQHNxyMLKSawNhKCPNy++2xY=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.23/go.mod h1:2DFxAQ9pfIRy0imBCJv+vZ2X6RKxves6fbnEuSry6b4=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.17 h1:/K482T5A3623WJgWT8w1yRAFK4RzGzEl7y39yhtn9eA=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.17/go.mod h1:pRwaTYCJemADaqCbUAxltMoHKata7hmB5PjEXeu0kfg=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.3.24 h1:wj5Rwc05hvUSvKuOF29IYb9QrCLjU+rHAy/x/o0DK2c=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.3.24/go.mod h1:jULHjqqjDlbyTa7pfM7WICATnOv+iOhjletM3N0Xbu8=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.14 h1:ZSIPAkAsCCjYrhqfw2+lNzWDzxzHXEckFkTePL5RSWQ=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.14/go.mod h1:AyGgqiKv9ECM6IZeNQtdT8NnMvUb3/2wokeq2Fgryto=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.9 h1:Lh1AShsuIJTwMkoxVCAYPJgNG5H+eN6SmoUn8nOZ5wE=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.9/go.mod h1:a9j48l6yL5XINLHLcOKInjdvknN+vWqPBxqeIDw7ktw=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.18 h1:BBYoNQt2kUZUUK4bIPsKrCcjVPUMNsgQpNAwhznK/zo=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.18/go.mod h1:NS55eQ4YixUJPTC+INxi2/jCqe1y2Uw3rnh9wEOVJxY=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.17 h1:Jrd/oMh0PKQc6+BowB+pLEwLIgaQF29eYbe7E1Av9Ug=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.17/go.mod h1:4nYOrY41Lrbk2170/BGkcJKBhws9Pfn8MG3aGqjjeFI=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.17 h1:HfVVR1vItaG6le+Bpw6P4midjBDMKnjMyZnw9MXYUcE=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.17/go.mod h1:YqMdV+gEKCQ59NrB7rzrJdALeBIsYiVi8Inj3+KcqHI=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11 h1:3/gm/JTX9bX8CpzTgIlrtYpB3EVBDxyg/GY/QdcIEZw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11/go.mod h1:fmgDANqTUCxciViKl9hb/zD5LFbvPINFRgWhDbR+vZo=
+github.com/aws/aws-sdk-go-v2/service/sso v1.11.23 h1:pwvCchFUEnlceKIgPUouBJwK81aCkQ8UDMORfeFtW10=
+github.com/aws/aws-sdk-go-v2/service/sso v1.11.23/go.mod h1:/w0eg9IhFGjGyyncHIQrXtU8wvNsTJOP0R6PPj0wf80=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.5 h1:GUnZ62TevLqIoDyHeiWj2P7EqaosgakBKVvWriIdLQY=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.5/go.mod h1:csZuQY65DAdFBt1oIjO5hhBR49kQqop4+lcuCjf2arA=
+github.com/aws/aws-sdk-go-v2/service/sts v1.16.19 h1:9pPi0PsFNAGILFfPCk8Y0iyEBGc6lu6OQ97U7hmdesg=
+github.com/aws/aws-sdk-go-v2/service/sts v1.16.19/go.mod h1:h4J3oPZQbxLhzGnk+j9dfYHi5qIOVJ5kczZd658/ydM=
+github.com/aws/smithy-go v1.13.3 h1:l7LYxGuzK6/K+NzJ2mC+VvLUbae0sL3bXU//04MkmnA=
+github.com/aws/smithy-go v1.13.3/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
+github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
+github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
+github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
+github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
+github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
+github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
+github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
+github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
+github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
+github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
+github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
+github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
+github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
+github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
+github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
+github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
+github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
+github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
+golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e h1:fLOSk5Q00efkSvAm+4xcoXD+RRmLmmulPn5I3Y9F2EM=
+golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
+gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
--- a/benchmarks/s3ttfb/main.go
+++ b/benchmarks/s3ttfb/main.go
@ -0,0 +1,214 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"log"
+	"math/rand"
+	"net/http"
+	"net/http/httptrace"
+	"os"
+	"strconv"
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+
+	"github.com/aws/aws-sdk-go-v2/aws"
+	"github.com/aws/aws-sdk-go-v2/config"
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+)
+
+type PRNG struct {
+	rem int64
+}
+
+func (r *PRNG) Read(p []byte) (n int, err error) {
+	//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
+	if int64(len(p)) > r.rem {
+		p = p[:r.rem]
+	}
+
+	if int64(len(p)) > r.rem {
+		log.Fatal("LOGIC ERROR")
+	}
+
+	n, err = rand.Read(p)
+	if err != nil {
+		return
+	}
+	r.rem -= int64(n)
+	if r.rem <= 0 {
+		err = io.EOF
+		//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
+	}
+	return
+}
+
+func main() {
+	var err error
+	fmt.Printf("nanoseconds\n")
+
+	// Initial setup
+	_, isSSL := os.LookupEnv("SSL")
+	opts := minio.Options{
+		Creds:  credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
+		Secure: isSSL,
+	}
+
+	if region, ok := os.LookupEnv("AWS_REGION"); ok {
+		opts.Region = region
+	}
+
+	if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
+		opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
+	}
+
+	size := 1 * 1024 * 1024 // 1 MB
+	if env_size, ok := os.LookupEnv("SIZE"); ok {
+		size, err = strconv.Atoi(env_size)
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+	}
+
+	repeat := 60
+	if env_repeat, ok := os.LookupEnv("REPEAT"); ok {
+		repeat, err = strconv.Atoi(env_repeat)
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+	}
+
+	mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
+	if err != nil {
+		log.Fatal("failed connect", err)
+		return
+	}
+
+	// AWS SDK stuff
+	customResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
+        	return aws.Endpoint{
+            		PartitionID:   "aws",
+			URL:           "http://"+os.Getenv("ENDPOINT"),
+            		SigningRegion: region,
+        	}, nil
+	})
+
+
+	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithEndpointResolverWithOptions(customResolver))
+	if err != nil {
+		log.Fatalf("failed to load configuration, %v", err)
+	}
+
+	// Create an Amazon S3 service client
+	client := s3.NewFromConfig(cfg, func(opts *s3.Options) {
+		opts.UsePathStyle = true
+	})
+
+
+	// Create Bucket
+	buck := uuid.New().String()
+	err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
+	if err != nil {
+		log.Fatal(err)
+		return
+	}
+
+	// Send an object
+	prng := new(PRNG)
+	prng.rem = int64(size)
+
+
+	key := uuid.New().String()
+	_, err = mc.PutObject(
+		context.Background(),
+		buck,
+		key,
+		prng,
+		int64(size),
+		minio.PutObjectOptions{ContentType: "application/octet-stream"},
+	)
+	if err != nil {
+		log.Fatal("unable to upload our test object")
+		return
+	}
+
+	for i := 0; i < repeat; i++ {
+		// Now, try to fetch it *repeat* times (assume no caching)
+		var elapsed time.Duration
+
+		var start time.Time
+    		trace := &httptrace.ClientTrace{
+			GotFirstResponseByte: func() { elapsed = time.Since(start) },
+		}
+
+		start = time.Now()
+		/*object, err := mc.GetObject(httptrace.WithClientTrace(context.Background(), trace), buck, key, minio.GetObjectOptions{})
+		if err != nil {
+			log.Fatal(err)
+			return
+		}*/
+
+		out, err := client.GetObject(httptrace.WithClientTrace(context.Background(), trace), &s3.GetObjectInput {
+			Bucket: aws.String(buck),
+			Key: aws.String(key),
+		})
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+		object := out.Body
+
+
+		if _, err = io.Copy(ioutil.Discard, object) ; err != nil {
+			log.Fatal("failed getobject: ", err)
+			return
+		}
+
+		fmt.Printf("%v\n", elapsed.Nanoseconds())
+		//time.Sleep(10 * time.Second)
+	}
+}
+
+/*
+func timeGet(url string) {
+    req, _ := http.NewRequest("GET", url, nil)
+
+    var start, connect, dns, tlsHandshake time.Time
+
+    trace := &httptrace.ClientTrace{
+        DNSStart: func(dsi httptrace.DNSStartInfo) { dns = time.Now() },
+        DNSDone: func(ddi httptrace.DNSDoneInfo) {
+            fmt.Printf("DNS Done: %v\n", time.Since(dns))
+        },
+
+        TLSHandshakeStart: func() { tlsHandshake = time.Now() },
+        TLSHandshakeDone: func(cs tls.ConnectionState, err error) {
+            fmt.Printf("TLS Handshake: %v\n", time.Since(tlsHandshake))
+        },
+
+        ConnectStart: func(network, addr string) { connect = time.Now() },
+        ConnectDone: func(network, addr string, err error) {
+            fmt.Printf("Connect time: %v\n", time.Since(connect))
+        },
+
+        GotFirstResponseByte: func() {
+            fmt.Printf("Time from start to first byte: %v\n", time.Since(start))
+        },
+    }
+
+    req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
+    start = time.Now()
+    if _, err := http.DefaultTransport.RoundTrip(req); err != nil {
+        log.Fatal(err)
+    }
+    fmt.Printf("Total time: %v\n", time.Since(start))
+}
+*/
--- a/benchmarks/warp
+++ b/benchmarks/warp
@ -0,0 +1 @@
+Subproject commit df8f2cd7b7a8523755b4d3c04271a69571a455ca
--- a/example/deploy_garage.sh
+++ b/example/deploy_garage.sh
@ -1,67 +0,0 @@
-#!/usr/bin/env bash
-
-set -euo pipefail
-IFS=$'\n\t'
-
-GARAGE_PATH=garage
-STORAGE_PATH=/tmp/garage-testnet
-export RPC_SECRET=3e9abff5f9e480afbadb46a77b7a26fe0e404258f0dc3fd5386b0ba8e0ad2fba
-
-if [ -z "$ZONE" ]; then
-	NODE_STORAGE_PATH=${STORAGE_PATH}/${HOST}
-else
-	NODE_STORAGE_PATH=${STORAGE_PATH}/${ZONE}/${HOST}
-fi
-BOOTSTRAP_FILE=${STORAGE_PATH}/bootstrap_peer
-export GARAGE_CONFIG_FILE=${NODE_STORAGE_PATH}/garage.toml 
-
-
-mkdir -p ${NODE_STORAGE_PATH}
-cd ${NODE_STORAGE_PATH}
-rm ${BOOTSTRAP_FILE} 2>/dev/null || true
-
-cat > ${GARAGE_CONFIG_FILE} << EOF
-metadata_dir = "${NODE_STORAGE_PATH}/meta"
-data_dir = "${NODE_STORAGE_PATH}/data"
-
-block_size = 131072
-replication_mode = "3"
-
-rpc_bind_addr = "[::]:3901"
-rpc_public_addr = "[${IP}]:3901"
-rpc_secret = "${RPC_SECRET}"
-
-bootstrap_peers=[]
-
-[s3_api]
-s3_region = "garage"
-api_bind_addr = "[::]:3900"
-root_domain = ".s3.garage"
-
-[s3_web]
-bind_addr = "[::]:3902"
-root_domain = ".web.garage"
-index = "index.html"
-
-[admin]
-api_bind_addr = "[::]:3903"
-admin_token = "ae8cb40ea7368bbdbb6430af11cca7da833d3458a5f52086f4e805a570fb5c2a"
-trace_sink = "http://[fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff]:4317"
-EOF
-
-RUST_LOG=debug ${GARAGE_PATH} server 2>> ${NODE_STORAGE_PATH}/logs & disown
-sleep 2
-
-CONFIG_NODE_FPATH=$(find /tmp/garage-testnet/ -maxdepth 3 -name garage.toml|head -n 1)
-
-SELF_ID=$(${GARAGE_PATH} node id 2>/dev/null)
-SHORT_ID=$(echo ${SELF_ID} | cut -c-64)
-
-${GARAGE_PATH} -c ${CONFIG_NODE_FPATH} node connect ${SELF_ID}
-${GARAGE_PATH} -c ${CONFIG_NODE_FPATH} layout assign ${SHORT_ID} -z ${ZONE:-unzonned-${HOST}} -c 1 -t ${HOST}
-
-if [ ${CONFIG_NODE_FPATH} == ${GARAGE_CONFIG_FILE} ]; then
-	sleep 2
-	${GARAGE_PATH} layout show
-	${GARAGE_PATH} layout apply --version 1
-fi
--- a/example/deploy_minio.py
+++ b/example/deploy_minio.py
@ -1,62 +0,0 @@
-#!/usr/bin/env python3
-import json, os, sys, time, pathlib, socket, shutil
-
-STORAGE_PATH = os.path.join(os.getcwd(), '.minio-testnet')
-HOSTS_PATH = os.path.join(STORAGE_PATH, 'hosts.txt')
-UNIX_SOCK = os.path.join(STORAGE_PATH, 'deploy.sock')
-DATA_PATH = lambda nid: os.path.join(STORAGE_PATH, 'data'+str(nid))
-
-def main():
-    if int(os.environ['ID']) == 1: leader()
-    else: follower()
-
-def leader():
-    shutil.rmtree(STORAGE_PATH, ignore_errors=True)
-    os.makedirs(STORAGE_PATH)
-    print(STORAGE_PATH)
-
-    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
-    sock.bind(UNIX_SOCK)
-    sock.listen()
-
-    n_serv = int(os.environ['SERVER_COUNT'])
-    fl = [ co for co, addr in [ sock.accept() for i in range(n_serv - 1) ]]
-
-    identities = [ json.loads(co.makefile().readline()) for co in fl ] + [ { "ip": os.environ['IP'], "path": make_data() } ]
-    print(f"ident: {identities}")
-    msg = f"{json.dumps(identities)}\n".encode()
-    [ co.send(msg) for co in fl ]
-
-    run_minio(identities)
-
-def follower():
-    co = None
-    while True:
-        time.sleep(1)
-        try:
-            sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
-            sock.connect(UNIX_SOCK)
-            co = sock.makefile()
-            break
-        except Exception as err:
-            print('conn failed, wait,', err)
-    my_identity = json.dumps({ "ip": os.environ['IP'], "path": make_data() })
-    sock.send(f"{my_identity}\n".encode())
-    identities = json.loads(co.readline())
-
-    run_minio(identities)
-
-def make_data():
-    data_path = DATA_PATH(os.environ['ID'])
-    os.makedirs(data_path)
-    return data_path
-
-def run_minio(identities):
-    cmd = f"minio server --console-address ':9001' --address ':9000'"
-    for ident in identities:
-        cmd += f" http://[{ident['ip']}]:9000{ident['path']}"
-    cmd += f" > {os.path.join(STORAGE_PATH, 'minio'+os.environ['ID']+'.log')} 2>&1"
-    print("launch: ", cmd)
-    os.system(cmd)
-
-__name__ == '__main__' and main()
--- a/img/jaeger_s3_put.png
+++ b/img/jaeger_s3_put.png
--- a/liveness.md
+++ b/liveness.md
@ -1,245 +0,0 @@
-# Liveness issues
-
-We know that some people reported having timeouts when putting some load on their Garage cluster.
-On our production cluster that runs without pressure, we don't really observe this behaviour.
-
-But when I wanted to start a benchmark created by Minio developers, I hit the same limit.
-So I wanted to reproduce this behavior in a more controlled environment.
-
-## Reproducing the error in mknet
-
-I used mknet to emulate a simple network with close to zero latency but with a very small bandwidth: 1Mbit/s. The idea is that the network will be the bottleneck, but not the CPU, the memory or the disk, even on my low powered laptop. The following configuration describes the simulated network configuration I used:
-
-```yaml
-links:
-  - &slow
-    bandwidth: 1M
-    latency: 500us
-  - &1000
-    bandwidth: 1000M
-    latency: 100us
-
-servers:
-  - name: node1
-    <<: *slow
-  - name: node2
-    <<: *slow
-  - name: node3
-    <<: *slow
-
-global:
-  subnet:
-    base: 'fc00:9a7a:9e::'
-    local: 64
-    zone: 16
-  latency-offset: 3ms
-  upstream:
-    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
-    conn: *1000
-```
-
-After a while, we quickly observe that the cluster is not reacting very well:
-
-```
-[nix-shell:/home/quentin/Documents/dev/deuxfleurs/mknet]# warp get --host=[fc00:9a7a:9e::1]:3900 --obj.size 100MiB --obj.randsize --duration=10m --concurrent 8 --objects 200 --access-key=GKc1e16da48142bdb95d98a4e4 --secret-key=c4ef5d5f7ee24ccae12a98912bf5b1fda28120a7e3a8f90cb3710c8683478b31
-Creating Bucket "warp-benchmark-bucket"...Element { tag_name: {http://s3.amazonaws.com/doc/2006-03-01/}LocationConstraint, attributes: [], namespaces: [Namespace { name: None, uri: "http://s3.amazonaws.com/doc/2006-03-01/" }] }
-warp: <ERROR> upload error: Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Timeout", "Timeout"]
-warp: <ERROR> upload error: Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Netapp error: Not connected: 3cb7ed98f7c66a55", "Netapp error: Not connected: 92c7fb74ed89f289"]
-warp: <ERROR> upload error: Put "http://[fc00:9a7a:9e::1]:3900/warp-benchmark-bucket/xVdzjy23/1.KximayVLlhLwfE5f.rnd": dial tcp [fc00:9a7a:9e::1]:3900: i/o timeout
-warp: <ERROR> upload error: Put "http://[fc00:9a7a:9e::1]:3900/warp-benchmark-bucket/N4zQvKhs/1.XkkO6DJ%28hVpGIrMj.rnd": dial tcp [fc00:9a7a:9e::1]:3900: i/o timeout
-warp: <ERROR> upload error: Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Timeout", "Timeout"]
-warp: <ERROR> upload error: Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Netapp error: Not connected: 3cb7ed98f7c66a55", "Netapp error: Not connected: 92c7fb74ed89f289"]
-warp: <ERROR> upload error: Put "http://[fc00:9a7a:9e::1]:3900/warp-benchmark-bucket/GQrsevhN/1.7hglGIP%28mXTJMgFE.rnd": read tcp [fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff]:57008->[fc00:9a7a:9e::1]:3900: read: connection reset by peer
-
-warp: <ERROR> Error preparing server: upload error: Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Timeout", "Timeout"].
-```
-
-We observe many different types of error that I categorize as follow:
-  - [RPC] Timeout quorum errors, they are probably generated by a ping between nodes
-  - [RPC] Not connected error, after a timeout, a reconnection is not triggered directly
-  - [S3 Gateway] The gateway took to much time to answer and a timeout was triggered in the client
-  - [S3 Gateway] The S3 gateway closes the TCP connection before answering
-
-As a first conclusion, we started to clearly reduce the scope of the problem by identifying that this undesirable behavior is triggered by a network bottleneck. 
-
-Looking at Garage logs, we see that:
-  - node1, which is our S3 gateway, has many quorum errors / netapp not connected errors, which are the same than the ones reported earlier
-  - node2 and node3 which are only used as storage nodes, have no error/warn in their logs
-
-It starts to really look like a congestion control/control flow error/scheduler issue: our S3 gateway seems to receive more data than it can send over the network, which in turn trigger timeouts, that trigger disconnect, and breaks everything. 
-
-## Write a custom client exhibiting the issue
-
-We know how to trigger the issue with `warp`, Minio's benchmark tool but we don't yet understand well what kind of load it puts on the cluster except that it sends concurrently Multipart and PutObject requests concurrently. So, before investigating the issue more in depth, we want to know:
-  - If a single large PUT request can trigger this issue or not?
-  - How many parallel requests are needed to trigger this issue?
-  - Does Multipart transfer are more impacted by this issue?
-
-To get answer to our questions, we will write a specific benchmark.
-Named s3concurrent, it is available here: https://git.deuxfleurs.fr/quentin/s3concurrent
-The benchmark starts by sending 1 file, then 2 files concurrently,
-then 3, then 4, up to 16 (this is hardcoded for now).
-
-When ran on our mknet cluster, we start triggering issues as soon as we send 2 files at once:
-
-```
-$ ./s3concurrent
-2022/08/11 20:35:28 created bucket 3ffd6798-bdab-4218-b6d0-973a07e46ea9
-2022/08/11 20:35:28 start concurrent loop with 1 coroutines
-2022/08/11 20:35:55 done, 1 coroutines returned
-2022/08/11 20:35:55 start concurrent loop with 2 coroutines
-2022/08/11 20:36:34 1/2 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Timeout", "Timeout"]
-2022/08/11 20:36:37 done, 2 coroutines returned
-2022/08/11 20:36:37 start concurrent loop with 3 coroutines
-2022/08/11 20:37:13 1/3 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Netapp error: Not connected: 92c7fb74ed89f289", "Netapp error: Not connected: 3cb7ed98f7c66a55"]
-2022/08/11 20:37:51 2/3 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Netapp error: Not connected: 92c7fb74ed89f289", "Netapp error: Not connected: 3cb7ed98f7c66a55"]
-2022/08/11 20:37:51 3/3 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Netapp error: Not connected: 92c7fb74ed89f289", "Netapp error: Not connected: 3cb7ed98f7c66a55"]
-2022/08/11 20:37:51 done, 3 coroutines returned
-2022/08/11 20:37:51 start concurrent loop with 4 coroutines
-```
-
-We observe that Garage starts its timeout as soon as 2 coroutines are executed in parallel.
-
-Internally, Garage uses a library of our own named netapp. This library has an integrated scheduler that knows how to:
-  - handle priority of RPC packets
-  - multiplex packets of the same priority so that progress is made by everyone
-
-So in theory, this scheduler should be able to handle all our packets seamlessly.
-To better understand its behaviour, we observe the behaviour of Garage with a smaller block size, to see if it's a multiplexing problem. We select 128 KiB blocks instead of 1MiB ones (10 times smaller).
-
-This time, we can handle 2 coroutines at once but not 3:
-
-```
-2022/08/12 10:50:08 created bucket a565074b-0609-4f5f-8d46-389f86565197
-2022/08/12 10:50:08 start concurrent loop with 1 coroutines
-2022/08/12 10:50:32 done, 1 coroutines returned
-2022/08/12 10:50:32 start concurrent loop with 2 coroutines
-2022/08/12 10:51:18 done, 2 coroutines returned
-2022/08/12 10:51:18 start concurrent loop with 3 coroutines
-2022/08/12 10:51:35 1/3 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Timeout", "Timeout"]
-2022/08/12 10:51:45 2/3 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Netapp error: Not connected: 0b36b6d0de0a6393", "Netapp error: Not connected: b61e6a192c9462c9"]
-2022/08/12 10:51:45 3/3 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Timeout", "Timeout"]
-2022/08/12 10:51:45 done, 3 coroutines returned
-2022/08/12 10:51:45 start concurrent loop with 4 coroutines
-2022/08/12 10:52:09 1/4 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Timeout", "Timeout"]
-2022/08/12 10:52:13 2/4 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Netapp error: Not connected: 0b36b6d0de0a6393", "Netapp error: Not connected: b61e6a192c9462c9"]
-2022/08/12 10:52:13 3/4 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Netapp error: Not connected: 0b36b6d0de0a6393", "Netapp error: Not connected: b61e6a192c9462c9"]
-2022/08/12 10:52:15 4/4 failed with Internal error: Could not reach quorum of 2. 1 of 3 request succeeded, others returned errors: ["Timeout", "Timeout"]
-2022/08/12 10:52:15 done, 4 coroutines returned
-```
-
-Despite the fact we divided by 10 our block size, we did not improve by 10 our parallelism. As a conclusion, we need to question our design.
-
-## Making an hypothesis on netapp inner working
-
-First, we took a look at netapp failure detectors.
-It seems that, despite a closed TCP socket, it has no way to detect failures.
-So we have a second layer in Garage to detect failures based on timeouts on RPC commands. In our analysis we identified 2 critical RPC commands:
-  - Ping, that have high priority, and a timeout of 2 seconds
-  - BlockRW, that have a normal priority, and a timeout of 30 seconds.
-It appears that the timeout is triggered by the second RPC command.
-For a reason I don't understand yet, it appears that any timeout
-will trigger a disconnect/reconnect of the node (with a delay).
-
-So, here is my current mental model of our issue when we send multiple PutObject requests:
-  - Ping RPC commands are always handled in less than 2sec due to their high priority and their low number
-  - BlockRW commands are accumulating in the queue without any limit. They all have the same priority, so progress is slowly made on each of them at the same pace. Because they are so many, none of them complete in less than 30 seconds, thus triggering many timeouts in Garage.
-
-To check this hypothesis, I will start by logging netapp queues and their content.
-
-It appears that the problem is more complicated than it seemed first, as we have 2 ping logic, one at the netapp layer and one at the garage layer. And in both ways. And it seems that netapp pings are failing from the storage node to the gateway node.
-
-```
- WARN  netapp::peering::fullmesh    > Error pinging 90af93030366c0da: Ping timeout
- DEBUG netapp::peering::fullmesh    > Ping from 591ac2bffb05a3ec
- WARN  netapp::peering::fullmesh    > Error pinging 90af93030366c0da: Ping timeout
- WARN  netapp::peering::fullmesh    > Too many failed pings from 90af93030366c0da, closing connection.
- DEBUG netapp::netapp               > Closing connection to 90af93030366c0da ([fc00:9a7a:9e::1]:3901)
-```
-
-Currently Garage does not pipeline writes, so it waits that a chunk has been written before writing the next one. So in the end, we have not so many entries in the queue:
-  - the first chunk of upload 1
-  - the first chunk of upload 2
-  - the first chunk of upload 3, and so on, and so forth
-
-But we can see that problems can still occure with numerous uploads!
-And if we start pipeling sending, it will make the problem even worse!
-
-It seems that we could improve the situation by:
-  - Deleting Garage pings as netapp is handling them for us (even if it seems that they are used to measure an average ping - not sure of this point)
-  - Deleting timeouts on RPC blocks as failure detection is handled by netapp
-
-And to future proof our software:
-  - Putting a bound to the netapp queue to avoid multiplexing too many requests. It will be usefull especially when/if we start pipelining requests.
-  - Passing streams to netapp
-
-But before implementing these solutions, we must understand why netapp pings are failing, this is even more surprising as they have a 5 second timeout instead of a 2sec one on Garage... We should really reduce the number of timeouts we are handling...
-
-## Understanding why netapp ping timeout despite priorities
-
-*todo*
-
-## Overview of available tools to observe Garage internals
-
-Even if I have some theory on what is going wrong, I want to collect as many information as possible before making hypothesis,
-adding specific debug hooks, and so on. Hopefully, we have at least 3 different tools in Garage to gather information about its internals:
-  - Prometheus telemetry
-  - Opentelemetry traces
-  - Tokio traces (not sure)
-
-### Prometheus
-
-Nothing very interesting is returned by Prometheus telemetry, except we returned 503 and 500 errors:
-
-```
-# HELP api_s3_error_counter Number of API calls to the various S3 API endpoints that resulted in errors
-# TYPE api_s3_error_counter counter
-api_s3_error_counter{api_endpoint="PutObject",status_code="500"} 3
-api_s3_error_counter{api_endpoint="PutObject",status_code="503"} 2
-# HELP api_s3_request_counter Number of API calls to the various S3 API endpoints
-# TYPE api_s3_request_counter counter
-api_s3_request_counter{api_endpoint="CreateBucket"} 1
-api_s3_request_counter{api_endpoint="PutObject"} 6
-# HELP api_s3_request_duration Duration of API calls to the various S3 API endpoints
-# TYPE api_s3_request_duration histogram
-api_s3_request_duration_bucket{api_endpoint="CreateBucket",le="0.5"} 1
-api_s3_request_duration_bucket{api_endpoint="CreateBucket",le="0.9"} 1
-api_s3_request_duration_bucket{api_endpoint="CreateBucket",le="0.99"} 1
-api_s3_request_duration_bucket{api_endpoint="CreateBucket",le="+Inf"} 1
-api_s3_request_duration_sum{api_endpoint="CreateBucket"} 0.109302301
-api_s3_request_duration_count{api_endpoint="CreateBucket"} 1
-api_s3_request_duration_bucket{api_endpoint="PutObject",le="0.5"} 0
-api_s3_request_duration_bucket{api_endpoint="PutObject",le="0.9"} 0
-api_s3_request_duration_bucket{api_endpoint="PutObject",le="0.99"} 0
-api_s3_request_duration_bucket{api_endpoint="PutObject",le="+Inf"} 6
-api_s3_request_duration_sum{api_endpoint="PutObject"} 147.68400154399998
-api_s3_request_duration_count{api_endpoint="PutObject"} 6
-```
-
-### Traces with Jaeger and OLTP
-
-Based on Jaeger doc, I run:
-
-```
-docker run --name jaeger \
-  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-  -e COLLECTOR_OTLP_ENABLED=true \
-  -p 6831:6831/udp \
-  -p 6832:6832/udp \
-  -p 5778:5778 \
-  -p 16686:16686 \
-  -p 4317:4317 \
-  -p 4318:4318 \
-  -p 14250:14250 \
-  -p 14268:14268 \
-  -p 14269:14269 \
-  -p 9411:9411 \
-  jaegertracing/all-in-one:1.37
-```
-
-And then I observe:
-
-![Jaeger trace screenshot](img/jaeger_s3_put.png)
-
-We see many traces with exactly a 30s request.
-I suspect that we are hitting a timeout here but I am not sure.
--- a/31
+++ b/31
@ -87,19 +87,23 @@ class Bandwidth:
        

 class LinkInfo:
-    def __init__(self, bandwidth, latency, jitter = None, offset = None, **kwargs):
+    def __init__(self, bandwidth, latency, jitter = None, offset = None, limit = None, txqueuelen = None, **kwargs):
        self.bandwidth = Bandwidth(bandwidth)
        self.latency = Latency(latency, offset)
        self.jitter = Latency(jitter or 0)
+        self.limit = limit
+        self.txqueuelen = txqueuelen

    def __eq__(self, o):
        return (isinstance(o, LinkInfo) and 
            o.bandwidth == self.bandwidth and
            o.latency == self.latency and
-            o.jitter == self.jitter)
+            o.jitter == self.jitter and
+            o.limit == self.limit and
+            o.txqueuelen == self.txqueuelen)

    def __str__(self):
-        return f'LinkInfo{{bw: {self.bandwidth}, latency: {self.latency}, jitter: {self.jitter}}}'
+        return f'LinkInfo{{bw: {self.bandwidth}, latency: {self.latency}, jitter: {self.jitter}, limit: {self.limit}, txqueuelen: {self.txqueuelen}}}'

 class Server:
    def __init__(self, name, link):
@ -313,6 +317,7 @@ def runall(cmd):
    zones = parse(config).zones

    number = 1
+    wait_list = [ ]
    for zone in zones.values():
        if zone.is_zone():
            for server in zone.servers.values():
@ -323,7 +328,7 @@ def runall(cmd):
                env["ID"] = str(number)
                env["SERVER_COUNT"] = str(len(config['servers']))
                name = f'testnet-{zone.name}-{server.name}'
-                net.ns.run(name, cmd, env)
+                wait_list.append(net.ns.run(name, cmd, env))
                number +=1
        else:
            env = os.environ.copy()
@ -333,9 +338,12 @@ def runall(cmd):
            env["ID"] = str(number)
            env["SERVER_COUNT"] = str(len(config['servers']))
            name = f'testnet-{zone.name}-{zone.name}'
-            net.ns.run(name, cmd, env)
+            wait_list.append(net.ns.run(name, cmd, env))
            first = False
            number +=1
+    for p in wait_list:
+        p.wait()
+    print("runall done")

 def destroy():
    for ns in net.ns.list():
@ -343,22 +351,31 @@ def destroy():
    net.ns.forget("unconfined")
    os.remove(".current_state.yml")

+def scenario(config, cmd):
+    create(config)
+    runall(cmd)
+    destroy()
+
 if __name__ == "__main__":
    if len(sys.argv) < 2:
        progname = os.path.basename(sys.argv[0]) if len(sys.argv) > 0 else "mknet"
        print(f"""Usage:
-    {progname} create [config_path] # create a new network. config_path defailt to config.yml
+    {progname} scenario <config_path> <cmd> [args...] # all in one simulation: create a network, run the command on all nodes, destroy the network
+
+    {progname} create [config_path] # create a new network. config_path defailt to topo/with-vdsl.yml
    {progname} run-all <cmd> [args...] # run a command as each host. set the IP, NAME and ZONE environment variables
    {progname} run <name> [cmd [args...]] # run command in host named <name>. Use zonename:name if multiple zones hosts server with same name. If cmd is empty, run a shell
    {progname} destroy # destroy the current environment""")
        exit()
    cmd = sys.argv[1]
    if cmd == "create":
-        create(sys.argv[2] if len(sys.argv) > 2 else "config.yml")
+        create(sys.argv[2] if len(sys.argv) > 2 else "topo/with-vdsl.yml")
    elif cmd == "run":
        run(sys.argv[2], sys.argv[3:])
    elif cmd == "run-all":
        runall(sys.argv[2:])
+    elif cmd == "scenario":
+        scenario(sys.argv[2], sys.argv[3:])
    elif cmd == "destroy":
        destroy()    
    else:
--- a/net.py
+++ b/net.py
@ -37,7 +37,7 @@ class ns:
        run_netns("exec", name, "ip", "link", "set", "dev", "lo", "up")

    def run(name, cmd, env=None):
-        subprocess.Popen(_netns + ["exec", name] + cmd, env=env)
+        return subprocess.Popen(_netns + ["exec", name] + cmd, env=env)

 def create_bridge(name, namespace, ports=[]):
    run_netns("exec", namespace, "ip", "link", "add", "name", name, "type", "bridge")
@ -56,6 +56,9 @@ def create_veth(name1, ns1, name2, ns2, ip = None, subnet=0, link=None):
    run_netns("exec", ns2, "ip", "link", "set", "dev", name2, "up")
    
    if link:
+        if link.txqueuelen:
+            run_netns("exec", ns1, "ip", "link", "set", "dev", name1, "txqueuelen", str(link.txqueuelen))
+            run_netns("exec", ns2, "ip", "link", "set", "dev", name2, "txqueuelen", str(link.txqueuelen))
        tc(ns1, name1, link)
        tc(ns2, name2, link, True)

@ -66,4 +69,6 @@ def tc(namespace, name, link, invert=False):
        options += ["rate", str(link.bandwidth.down)]
    else:
        options += ["rate", str(link.bandwidth.up)]
+    if link.limit:
+        options += ["limit", str(link.limit)]
    run_netns("exec", namespace, "tc", "qdisc", "add", "dev", name, "root", "netem", *options)
--- a/prepare.py
+++ b/prepare.py
@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+from pathlib import Path
+from scenarios.fragments import shared
+import os, venv
+
+me = Path(os.path.dirname(__file__))
+print("--- git submodule ---")
+shared.exec("git submodule update --init")
+
+print("--- compile go benchmarks ---")
+os.chdir(me / "benchmarks" / "warp")
+shared.exec("go build")
+os.chdir(me / "benchmarks" / "s3concurrent")
+shared.exec("go build")
+os.chdir(me / "benchmarks" / "s3lat")
+shared.exec("go build")
+os.chdir(me / "benchmarks" / "s3ttfb")
+shared.exec("go build")
+os.chdir(me / "benchmarks" / "s3billion")
+shared.exec("go build")
+os.chdir(me)
+
+print("--- install python dependencies ---")
+venv.create(".venv", with_pip=True)
+shared.exec("""
+. .venv/bin/activate
+which python3
+python3 -m pip install .
+python3 -m pip install -r scenarios/requirements.txt
+""")
+
+print("--- download artifacts ---")
+from scenarios.fragments import flavor
+flavor.download()
+
+print(f"""done! now, run:
+> sudo -i
+> cd {os.getcwd()}
+> . ./.venv/bin/activate
+> ./mknet scenario ./topo/with-vdsl.yml ./scenarios/garage-s3lat garage-v0.8
+""")
--- a/scenarios/clean
+++ b/scenarios/clean
@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+from fragments import garage, shared, minio
+
+garage.destroy()
+minio.destroy()
+if len(shared.storage_path) > 8 and shared.id() == 1:
+    shared.exec(f"rm -r {shared.storage_path}")
+shared.log("clean done")
+
--- a/scenarios/fragments/.shared.py.swo
+++ b/scenarios/fragments/.shared.py.swo
--- a/scenarios/fragments/flavor.py
+++ b/scenarios/fragments/flavor.py
@ -0,0 +1,73 @@
+from pathlib import Path
+from . import shared
+from os.path import exists
+
+def grg_path(d):
+    for flav, desc in d.items():
+        if "path" in desc: continue
+        binary = f"garage-{desc['target']}-{desc['version']}"
+        desc['path'] = Path(shared.binary_path) / binary
+    return d
+
+def minio_path(d):
+    for flav, desc in d.items():
+        if "path" in desc: continue
+        desc['path'] = Path(shared.binary_path) / flav
+    return d
+
+garage = grg_path({
+    "garage-local": { "path": "./garage/target/release/garage" },
+    "garage-v0.7": { 
+        "version": "v0.7.3", 
+        "target": "x86_64-unknown-linux-musl", 
+    },
+    # no rpc optimization
+    "garage-v0.8-beta1": { 
+        "version": "89b8087ba81c508ba382aa6c9cb6bb3afa6a43c8", 
+        "target": "x86_64-unknown-linux-musl" 
+    },
+    # with rpc optimizations
+    # with fsync deactivated
+    "garage-v0.8-no-fsync": { 
+        "version": "v0.8.0-dangerous-no-fsync", 
+        "target": "x86_64-unknown-linux-musl" 
+    },
+    # with rpc optimizations
+    "garage-v0.8-beta2": {
+        "version": "af2b2f26b4bd9fcdeedf2cd03f9e1392e5781abc",
+        "target": "x86_64-unknown-linux-musl",
+    }
+})
+
+warp = {
+    "warp-fast": "mixed --obj.size 5M --objects 200 --duration=5m",
+    "warp-small-obj": "mixed --obj.size 256 --objects 200 --duration=5m",
+    "warp-default": "mixed"
+}
+
+minio = minio_path({
+    "minio-20220917": {
+        "version": "2022-09-17T00-09-45Z"
+    }
+})
+
+
+def download():
+    for flav, desc in garage.items():
+        if "version" not in desc: continue
+        if exists(desc['path']): continue
+
+        shared.exec(f"mkdir -p {shared.binary_path}")
+        shared.exec(f"wget https://garagehq.deuxfleurs.fr/_releases/{desc['version']}/{desc['target']}/garage -O {desc['path']}")
+        shared.exec(f"chmod +x {desc['path']}")
+        shared.exec(f"{desc['path']} --version")
+
+    for flav, desc in minio.items():
+        if "version" not in desc: continue
+        if exists(desc['path']): continue
+
+        shared.exec(f"mkdir -p {shared.binary_path}")
+        shared.exec(f"wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio.RELEASE.{desc['version']} -O {desc['path']}")
+        shared.exec(f"chmod +x {desc['path']}")
+        shared.exec(f"{desc['path']} --version")
+
--- a/scenarios/fragments/garage.py
+++ b/scenarios/fragments/garage.py
@ -0,0 +1,229 @@
+import glob, json, time
+from os.path import exists
+from os import environ as env
+from pathlib import Path
+from . import shared, flavor
+
+import garage_admin_sdk
+from garage_admin_sdk.api import nodes_api, layout_api, key_api
+from garage_admin_sdk.model.node_cluster_info import NodeClusterInfo
+from garage_admin_sdk.model.layout_version import LayoutVersion
+from garage_admin_sdk.model.add_key_request import AddKeyRequest
+from garage_admin_sdk.model.update_key_request import UpdateKeyRequest
+from garage_admin_sdk.model.update_key_request_allow import UpdateKeyRequestAllow
+
+storage_path = "./i/am/not/defined"
+rpc_secret = "3e9abff5f9e480afbadb46a77b7a26fe0e404258f0dc3fd5386b0ba8e0ad2fba"
+metrics = "cacce0b2de4bc2d9f5b5fdff551e01ac1496055aed248202d415398987e35f81"
+admin = "ae8cb40ea7368bbdbb6430af11cca7da833d3458a5f52086f4e805a570fb5c2a"
+key = None
+
+version = flavor.garage["garage-v0.7"]
+
+configuration = garage_admin_sdk.Configuration(
+    host = "http://localhost:3903/v0",
+    access_token = admin
+)
+api = garage_admin_sdk.ApiClient(configuration) 
+nodes = nodes_api.NodesApi(api)
+layout = layout_api.LayoutApi(api)
+keys = key_api.KeyApi(api)
+
+
+# Setup, launch on import
+if 'HOST' in env:
+    storage_path = Path(shared.storage_path) / "garage" / env['HOST']
+    if 'ZONE' in env and env['ZONE'] != "":
+        storage_path = Path(shared.storage_path) / "garage" / env['ZONE'] / env['HOST']
+    config = storage_path / "garage.toml"
+    env['GARAGE_CONFIG_FILE'] = str(config)
+
+def deploy_coord(gw=None, uroot={}, us3_api={}, us3_web={}, uadmin={}):
+    destroy()
+    shared.log("start daemon")
+    shared.exec(f"{version['path']} --version")
+    daemon(uroot,us3_api,us3_web,uadmin)
+    shared.log("discover nodes")
+    connect()
+    shared.log("build layout")
+    create_layout(gw=gw)
+    shared.log("create key")
+    create_key()
+    shared.log("ready")
+
+def deploy_follow(uroot={}, us3_api={}, us3_web={}, uadmin={}):
+    destroy()
+    shared.log("start daemon")
+    daemon(uroot,us3_api,us3_web,uadmin)
+    shared.log("wait for coord")
+    sync_on_key_up()
+    shared.log("ready")
+
+def to_toml(d): return "\n".join([ f"{k} = {v}" if type(v) is int else f"{k} = \"{v}\"" for k, v in d.items() ])
+def daemon(uroot={}, us3_api={}, us3_web={}, uadmin={}):
+    root = {
+        "metadata_dir": f"{storage_path}/meta",
+        "data_dir": f"{storage_path}/data",
+        "replication_mode": "3",
+        "rpc_bind_addr": "[::]:3901",
+        "rpc_public_addr": f"[{env['IP']}]:3901",
+        "rpc_secret": f"{rpc_secret}",
+    }
+    root.update(uroot)
+
+    s3_api = {
+        "s3_region": "garage",
+        "api_bind_addr": "[::]:3900",
+        "root_domain": ".s3.garage",
+    }
+    s3_api.update(us3_api)
+    
+    s3_web = {
+        "bind_addr": "[::]:3902",
+        "root_domain": ".web.garage",
+        "index": "index.html",
+    }
+    s3_web.update(us3_web)
+
+    sect_admin = {
+        "api_bind_addr": "0.0.0.0:3903",
+        "metrics_token": f"{metrics}",
+        "admin_token": f"{admin}",
+    }
+    sect_admin.update(uadmin)
+
+
+    shared.exec(f"mkdir -p {storage_path}")
+    with open(config, 'w+') as f:
+        f.write(f"""
+{to_toml(root)}
+bootstrap_peers = []
+
+[s3_api]
+{to_toml(s3_api)}
+
+[s3_web]
+{to_toml(s3_web)}
+
+[admin]
+{to_toml(sect_admin)}
+        """)
+
+    if shared.id() == 1:
+        shared.exec(f"{version['path']} --version")
+    shared.exec(f"{version['path']} server 2>> {storage_path}/logs.stderr 1>> {storage_path}/logs.stdout & echo $! > {storage_path}/daemon.pid")
+    time.sleep(1)
+
+    node_info = storage_path / "node_info"
+    node_id = shared.fn_retry(lambda: nodes.get_nodes().node)
+    with open(node_info, 'w+') as f:
+        f.write(json.dumps({
+            "node_addr": f"{node_id}@{env['IP']}:3901",
+            "node_id": node_id,
+            "zone": env['ZONE'] if 'ZONE' in env and env['ZONE'] != "" else env['HOST'],
+            "host": env['HOST'],
+        }))
+
+def destroy():
+    dpid = Path(storage_path) / "daemon.pid"
+    if exists(dpid):
+        try:
+            shared.exec(f"kill -9 $(cat {dpid})")
+        except:
+            pass
+        shared.exec(f"rm -f {dpid}")
+    if len(str(storage_path)) < 8: # arbitrary, stupid safe guard
+        print(storage_path)
+        raise Exception("You tried to clean a storage path that might be the root of your FS, panicking...")
+    shared.exec(f"rm -fr {storage_path}")
+
+# this function is ugly, sorry :s
+_cluster_info = None
+def cluster_info():
+    global _cluster_info
+    if _cluster_info is not None: return _cluster_info
+    shared.log("fetch cluster info")
+
+    while True:
+        node_files = glob.glob(f"{shared.storage_path}/**/node_info", recursive=True)
+        if len(node_files) >= shared.count(): break
+        shared.log(f"found {len(node_files)} over {shared.count()}, wait 1 sec.")
+        time.sleep(1)
+
+    _cluster_info = [ json.loads(Path(f).read_text()) for f in node_files ]
+    return _cluster_info
+
+
+def connect():
+    cinf = cluster_info()
+    shared.log("start connections...")
+    while True:
+        try:
+            ret = nodes.add_node([n['node_addr'] for n in cinf], _request_timeout=3)
+        except:
+            shared.log("not ready, retry in 1sec")
+            time.sleep(1)
+            continue
+
+        for st in ret:
+            if not st.success:
+                continue
+                #raise Exception("Node connect failed", ret)
+        break
+
+    shared.log("all nodes connected")
+
+def create_layout(gw=None):
+    if gw is None:
+        gw = []
+
+    v = layout.get_layout().version
+
+    cinf = cluster_info()
+    nlay = dict()
+    for n in cinf:
+        capa = 1
+        if n['host'] in gw:
+            capa = None
+
+        nlay[n['node_id']] = NodeClusterInfo(
+            zone = n['zone'],
+            capacity = capa,
+            tags = [ n['host'] ],
+        )
+    layout.add_layout(nlay)
+    layout.apply_layout(LayoutVersion(version=v+1))
+
+    shared.log(layout.get_layout())
+
+def create_key():
+    global key
+    kinfo = shared.fn_retry(lambda: keys.add_key(AddKeyRequest(name="mknet")))
+    allow_create = UpdateKeyRequestAllow(create_bucket=True)
+    keys.update_key(kinfo.access_key_id, UpdateKeyRequest(allow=allow_create))
+    key = kinfo
+
+
+def delete_key():
+    global key
+    keys.delete_key(key.access_key_id)
+    key = None
+
+def sync_on_key_up():
+    global key
+    while True:
+        try:
+            key = keys.search_key("mknet")
+            return key
+        except:
+            pass
+        time.sleep(1)
+
+def sync_on_key_down():
+    while True:
+        try:
+            keys.search_key("mknet")
+        except:
+            return
+        time.sleep(1)
+
--- a/scenarios/fragments/minio.py
+++ b/scenarios/fragments/minio.py
@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+import json, os, sys, time, pathlib, socket, shutil, urllib3
+import minio
+from os import environ as env
+from pathlib import Path
+from . import shared, flavor
+
+storage_path = "./i/am/not/defined"
+version = flavor.minio["minio-20220917"]
+unix_sock = str(Path(shared.storage_path) / "minio.sock")
+access_key = "minioadmin"
+secret_key = "minioadmin"
+client = minio.Minio(
+    f"[{env['IP']}]:9000",
+    access_key="minioadmin",
+    secret_key="minioadmin",
+    secure=False,
+    http_client=urllib3.PoolManager(
+        timeout=2,
+        retries=False,
+    )
+)
+
+if 'HOST' in env:
+    storage_path = str(Path(shared.storage_path) / "minio" / env['HOST'])
+    if 'ZONE' in env and env['ZONE'] != "":
+        storage_path = str(Path(shared.storage_path) / "minio" / env['ZONE'] / env['HOST'])
+
+stdout = Path(storage_path) / "minio.stdout"
+stderr = Path(storage_path) / "minio.stderr"
+pid = Path(storage_path) / "daemon.pid"
+
+def destroy():
+    if os.path.exists(pid):
+        try:
+            shared.exec(f"kill -9 `cat {pid}`")
+        except:
+            pass
+
+    if len(str(storage_path)) > 8:
+        shutil.rmtree(storage_path, ignore_errors=True)
+
+def deploy_coord():
+    destroy()
+    if os.path.exists(unix_sock):
+        os.unlink(unix_sock)
+
+    os.makedirs(storage_path)
+
+    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+    sock.bind(unix_sock)
+    sock.listen()
+
+    # Create sockets
+    fl = [ co for co, addr in [ sock.accept() for i in range(1, shared.count()) ]]
+
+    # Receive configurations, centralize them
+    me = [ { "ip": os.environ['IP'], "path": storage_path } ]
+    others = [ json.loads(co.makefile().readline()) for co in fl ]
+    identities = others + me
+    shared.log(f"ident: {identities}")
+
+    # Dispatch them
+    msg = f"{json.dumps(identities)}\n".encode()
+    [ co.send(msg) for co in fl ]
+
+    run_minio(identities)
+
+    while True:
+        try:
+            if client.bucket_exists("sync"): break
+            client.make_bucket("sync")
+            break
+        except Exception as e: 
+            shared.log("waiting for bootstrap...", e)
+            time.sleep(1)
+    shared.log("ready")
+
+def deploy_follow():
+    destroy()
+    os.makedirs(storage_path)
+
+    co = None
+    while True:
+        try:
+            sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+            sock.connect(unix_sock)
+            co = sock.makefile()
+            break
+        except Exception as err:
+            shared.log('conn failed, wait 1 sec, err is', err)
+            time.sleep(1)
+    
+    # send my identity
+    my_identity = json.dumps({ "ip": os.environ['IP'], "path": storage_path })
+    sock.send(f"{my_identity}\n".encode())
+
+    # get all
+    identities = json.loads(co.readline())
+
+    run_minio(identities)
+    sync_on_bucket_up()
+    shared.log("ready")
+
+def sync_on_bucket_up():
+    while True:
+        try:
+            if client.bucket_exists("sync"): break
+        except:
+            pass
+        shared.log("waiting for bucket 'sync'...")
+        time.sleep(1)
+
+def sync_on_bucket_down():
+    while True:
+        try:
+            if not client.bucket_exists("sync"): break
+            time.sleep(1)
+        except Exception as e:
+            shared.log("the cluster is probably already half shutdown, so errors are expected ->", e)
+            break
+
+def delete_sync_bucket():
+    client.remove_bucket("sync")
+
+def run_minio(identities):
+
+    # Required to prevent Minio error: "/tmp/mknet-store/minio/node1` is part of root drive, will not be used"
+    # https://github.com/minio/minio/issues/15720
+    env['CI'] = "true"
+    env['MINIO_CI_CD'] = "true"
+
+    cmd = f"{version['path']} server --console-address ':9001' --address ':9000'"
+    for ident in identities:
+        cmd += f" http://[{ident['ip']}]:9000{ident['path']}"
+    cmd += f" > {stdout} 2> {stderr}"
+    cmd += f" & echo $! > {pid}"
+
+    shared.log("launch: ", cmd)
+    os.system(cmd)
--- a/scenarios/fragments/s3billion.py
+++ b/scenarios/fragments/s3billion.py
@ -0,0 +1,24 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, minio, garage
+
+s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3billion/s3billion"
+
+def common():
+    out = Path(shared.storage_path) / "s3billion.csv"
+    shared.log(f"launching s3billion ({s3bin})")
+    shared.exec(f"{s3bin} > {out}")
+    shared.log(f"execution done, output written to {out}")
+
+def on_garage():
+    os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
+    os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
+    os.environ['ENDPOINT'] = f"[{os.environ['IP']}]:3900"
+    common()
+
+def on_minio():
+    os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
+    os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
+    os.environ['ENDPOINT'] = f"[{os.environ['IP']}]:9000"
+    common()
--- a/scenarios/fragments/s3concurrent.py
+++ b/scenarios/fragments/s3concurrent.py
@ -0,0 +1,16 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, garage
+
+s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3concurrent/s3concurrent"
+
+def on_garage():
+    os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
+    os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
+    os.environ['ENDPOINT'] = "localhost:3900"
+
+    out = Path(shared.storage_path) / "s3concurrent.csv"
+    shared.log(f"launching s3concurrent ({s3bin})")
+    shared.exec(f"{s3bin} > {out}")
+    shared.log(f"execution done, output written to {out}")
--- a/scenarios/fragments/s3lat.py
+++ b/scenarios/fragments/s3lat.py
@ -0,0 +1,24 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, minio, garage
+
+s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3lat/s3lat"
+
+def common():
+    out = Path(shared.storage_path) / "s3lat.csv"
+    shared.log(f"launching s3lat ({s3bin})")
+    shared.exec(f"{s3bin} > {out}")
+    shared.log(f"execution done, output written to {out}")
+
+def on_garage():
+    os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
+    os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
+    os.environ['ENDPOINT'] = "localhost:3900"
+    common()
+
+def on_minio():
+    os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
+    os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
+    os.environ['ENDPOINT'] = "localhost:9000"
+    common()
--- a/scenarios/fragments/s3ttfb.py
+++ b/scenarios/fragments/s3ttfb.py
@ -0,0 +1,26 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, minio, garage
+
+s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3ttfb/s3ttfb"
+
+def common():
+    out = Path(shared.storage_path) / "s3ttfb.csv"
+    shared.log(f"launching s3ttfb ({s3bin})")
+    shared.exec(f"{s3bin} > {out}")
+    shared.log(f"execution done, output written to {out}")
+
+def on_garage():
+    os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
+    os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
+    os.environ['ENDPOINT'] = "localhost:3900"
+    os.environ['AWS_REGION'] = "garage"
+    common()
+
+def on_minio():
+    os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
+    os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
+    os.environ['ENDPOINT'] = "localhost:9000"
+    os.environ['AWS_REGION'] = "us-east-1"
+    common()
--- a/scenarios/fragments/shared.py
+++ b/scenarios/fragments/shared.py
@ -0,0 +1,28 @@
+import os, time
+
+binary_path = "/tmp/mknet-bin"
+storage_path = "/tmp/mknet-store"
+
+def exec(s): 
+    if os.system(s) != 0:
+        raise Exception("Command terminated with an error")
+def exec_retry(s, cnt=16):
+    print(s)
+    for i in range(cnt):
+        time.sleep(i) # this is expected to sleep before running the command to reduce the noise
+        if os.system(s) == 0: return
+    raise Exception("Command terminated with an error too many times")
+def fn_retry(f, cnt=5):
+    for i in range(cnt):
+        try:
+            r = f()
+            return r
+        except Exception as e:
+            if i+1 == cnt: raise e
+            log(f"failed call, retry in {i} sec")
+            time.sleep(i)
+
+def id(): return int(os.environ['ID'])
+def count(): return int(os.environ['SERVER_COUNT'])
+def log(*args): print(f"[{id()}/{count()} - {os.environ['HOST']}]", *args)
+
--- a/scenarios/fragments/warp.py
+++ b/scenarios/fragments/warp.py
@ -0,0 +1,17 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, garage, flavor, minio
+
+warp_bin = Path(os.path.dirname(__file__)) / "../../benchmarks/warp/warp"
+bench = flavor.warp["warp-fast"]
+
+def common(port, ak, sk):
+    out = Path(shared.storage_path) / "warp.csv"
+    shared.log(f"launching warp {warp_bin}")
+    shared.exec(f"{warp_bin} {bench} --host=[{os.environ['IP']}]:{port} --analyze.out={out} --access-key={ak} --secret-key={sk}")
+    shared.log(f"execution done")
+
+def on_garage(): common(3900, garage.key.access_key_id, garage.key.secret_access_key)
+def on_minio(): common(9000, minio.access_key, minio.secret_key)
+
--- a/scenarios/garage-manual
+++ b/scenarios/garage-manual
@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+from fragments import garage, s3lat, shared
+
+if shared.id() == 1:
+    garage.deploy_coord()
+else:
+    garage.deploy_follow()
--- a/scenarios/garage-s3billion
+++ b/scenarios/garage-s3billion
@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+from fragments import garage, s3billion, shared, flavor
+import sys, os
+
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+
+#os.environ['BATCH_COUNT'] = "2"
+shared.exec("ulimit -n 65535")
+conf_root = {
+    "db_engine": "lmdb",
+}
+
+if shared.id() == 1:
+    garage.deploy_coord(uroot=conf_root)
+    s3billion.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow(uroot=conf_root)
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/garage-s3concurrent
+++ b/scenarios/garage-s3concurrent
@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+from fragments import garage, s3concurrent, shared, flavor
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+
+if shared.id() == 1:
+    garage.deploy_coord()
+    s3concurrent.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow()
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/garage-s3lat
+++ b/scenarios/garage-s3lat
@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+from fragments import garage, s3lat, shared, flavor
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+
+if shared.id() == 1:
+    garage.deploy_coord()
+    s3lat.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow()
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/garage-s3ttfb
+++ b/scenarios/garage-s3ttfb
@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+from fragments import garage, s3ttfb, shared, flavor
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+
+if shared.id() == 1:
+    garage.deploy_coord(gw = [ 'node1' ])
+    s3ttfb.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow()
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/garage-warp
+++ b/scenarios/garage-warp
@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+from fragments import garage, warp, shared, flavor
+import sys
+
+db_engines = [ 'sled', 'lmdb', 'sqlite' ]
+conf_root = {}
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+    elif fl in flavor.warp:
+        warp.bench = flavor.warp[fl]
+    elif fl in db_engines:
+        conf_root['db_engine'] = fl
+    elif 'sled_no_fsync' == fl:
+        conf_root['sled_flush_every_ms'] = 1000 * 60 * 10 # 10 minutes
+
+if shared.id() == 1:
+    garage.deploy_coord(uroot=conf_root)
+    warp.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow(uroot=conf_root)
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/minio-manual
+++ b/scenarios/minio-manual
@ -0,0 +1,12 @@
+#!/usr/bin/env python3
+from fragments import shared, flavor, minio
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+
+if shared.id() == 1:
+    minio.deploy_coord()
+else:
+    minio.deploy_follow()
--- a/scenarios/minio-s3billion
+++ b/scenarios/minio-s3billion
@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+from fragments import minio, s3billion, shared, flavor
+import sys, os
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+
+#os.environ['BATCH_COUNT'] = "2"
+shared.exec("ulimit -n 65535")
+
+if shared.id() == 1:
+    minio.deploy_coord()
+    s3billion.on_minio()
+    minio.delete_sync_bucket()
+    minio.destroy()
+else:
+    minio.deploy_follow()
+    minio.sync_on_bucket_down()
+    minio.destroy()
+
+shared.log("bye")
--- a/scenarios/minio-s3lat
+++ b/scenarios/minio-s3lat
@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+from fragments import minio, s3lat, shared, flavor
+import sys
+
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+
+if shared.id() == 1:
+    minio.deploy_coord()
+    s3lat.on_minio()
+    minio.delete_sync_bucket()
+    minio.destroy()
+else:
+    minio.deploy_follow()
+    minio.sync_on_bucket_down()
+    minio.destroy()
+shared.log("bye")
--- a/scenarios/minio-s3ttfb
+++ b/scenarios/minio-s3ttfb
@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+from fragments import minio, s3ttfb, shared, flavor
+import sys
+
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+
+if shared.id() == 1:
+    minio.deploy_coord()
+    s3ttfb.on_minio()
+    minio.delete_sync_bucket()
+    minio.destroy()
+else:
+    minio.deploy_follow()
+    minio.sync_on_bucket_down()
+    minio.destroy()
+shared.log("bye")
--- a/scenarios/minio-warp
+++ b/scenarios/minio-warp
@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+from fragments import minio, warp, shared, flavor
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+    if fl in flavor.warp:
+        warp.bench = flavor.warp[fl]
+
+if shared.id() == 1:
+    minio.deploy_coord()
+    warp.on_minio()
+    minio.delete_sync_bucket()
+    minio.destroy()
+else:
+    minio.deploy_follow()
+    minio.sync_on_bucket_down()
+    minio.destroy()
+shared.log("bye")
--- a/scenarios/requirements.txt
+++ b/scenarios/requirements.txt
@ -0,0 +1,2 @@
+git+https://git.deuxfleurs.fr/quentin/garage-admin-sdk@2b685b90be9aa848a9450dd582d1d322b25b5568#egg=garage-admin-sdk&subdirectory=python
+minio
--- a/setup.py
+++ b/setup.py
@ -3,4 +3,8 @@ setup(name='mknet',
      version='1.0',
      scripts=['mknet'],
      py_modules=['net'],
+      install_requires=[
+          'PyYAML',
+          'requests'
+      ],
      )
--- a/shell.nix
+++ b/shell.nix
@ -0,0 +1,12 @@
+{ pkgs ? import <nixpkgs> {} }:
+  pkgs.mkShell {
+    nativeBuildInputs = [ 
+      pkgs.python310
+      pkgs.python310Packages.pyyaml
+      pkgs.python310Packages.requests
+
+      # to test the pip setup
+      pkgs.python310Packages.pip
+      pkgs.python310Packages.setuptools
+    ];
+}
--- a/topo/100mbps.yml
+++ b/topo/100mbps.yml
@ -0,0 +1,25 @@
+links:
+  - &slow
+    bandwidth: 100M
+    latency: 500us
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/10mbps.yml
+++ b/topo/10mbps.yml
@ -1,6 +1,6 @@
 links:
  - &slow
-    bandwidth: 1M
+    bandwidth: 10M
    latency: 500us
  - &1000
    bandwidth: 1000M
--- a/topo/1mbps.yml
+++ b/topo/1mbps.yml
@ -0,0 +1,27 @@
+links:
+  - &slow
+    bandwidth: 1M
+    latency: 500us
+    #txqueuelen: 10
+    limit: 10
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/50mbps.yml
+++ b/topo/50mbps.yml
@ -0,0 +1,25 @@
+links:
+  - &slow
+    bandwidth: 50M
+    latency: 500us
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/50ms.yml
+++ b/topo/50ms.yml
@ -0,0 +1,34 @@
+links:
+  - &fiber
+    bandwidth: 100M
+    latency: 50ms
+    jitter: 10ms
+
+zones:
+  - &dc1
+    name: dc1
+    internal: *fiber
+    external: *fiber
+
+servers:
+  - name: dc1s1
+    zone: *dc1
+  - name: dc1s2
+    zone: *dc1
+  - name: dc1s3
+    zone: *dc1
+  - name: dc1s4
+    zone: *dc1
+  - name: dc1s5
+    zone: *dc1
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *fiber
+
--- a/topo/5mbps.yml
+++ b/topo/5mbps.yml
@ -0,0 +1,28 @@
+links:
+  - &slow
+    bandwidth: 5M
+    latency: 500us
+    limit: 5
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+  - name: node4
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/dc.yml
+++ b/topo/dc.yml
@ -0,0 +1,22 @@
+links:
+  - &1000
+    bandwidth: 1000M
+    latency: 1ms
+
+servers:
+  - name: node1
+    <<: *1000
+  - name: node2
+    <<: *1000
+  - name: node3
+    <<: *1000
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/multi-dc.yml
+++ b/topo/multi-dc.yml
@ -0,0 +1,46 @@
+links:
+  - &100
+    bandwidth: 100M
+    latency: 500us
+  - &wan
+    bandwidth: 100M
+    latency: 50ms
+    jitter: 10ms
+
+zones:
+  - &dc1
+    name: dc1
+    internal: *100
+    external: *wan
+  - &dc2
+    name: dc2
+    internal: *100
+    external: *wan
+  - &dc3
+    name: dc3
+    internal: *100
+    external: *wan
+
+servers:
+  - name: dc1s1
+    zone: *dc1
+  - name: dc1s2
+    zone: *dc1
+  - name: dc2s1
+    zone: *dc2
+  - name: dc2s2
+    zone: *dc2
+  - name: dc3s1
+    zone: *dc3
+  - name: dc3s2
+    zone: *dc3
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *wan
--- a/topo/with-vdsl.yml
+++ b/topo/with-vdsl.yml
@ -5,16 +5,19 @@ links:
  - &1000
    bandwidth: 1000M
    latency: 100us
+    txqueuelen: 10000
  - &vdsl
    bandwidth:
      up: 3M
      down: 55M
    latency: 50ms
    jitter: 10ms
+    txqueuelen: 100
  - &fiber
    bandwidth: 400M
    latency: 10ms
    jitter: 3ms
+    txqueuelen: 1000

 zones:
  - &dc1
Author	SHA1	Message	Date
Quentin	db88b939c3	Merge pull request 'Fix venv sourcing' (#2 ) from baptiste/mknet:misc_fixes into main Reviewed-on: Deuxfleurs/mknet#2	2022-09-29 20:57:53 +02:00
Quentin	6d964937b3	Merge pull request 'Add result with 30M objects on Grid'5000' (#3 ) from baptiste/mknet:gros_30M into main Reviewed-on: Deuxfleurs/mknet#3	2022-09-29 20:53:21 +02:00
Baptiste Jonglez	c752ff7997	Add result with 30M objects on Grid'5000	2022-09-29 20:27:25 +02:00
Baptiste Jonglez	37040718ea	Document s3billion	2022-09-29 10:40:34 +02:00
Baptiste Jonglez	d1c659924a	prepare: fix venv sourcing when /bin/sh is Dash (e.g. on Debian)	2022-09-29 10:39:45 +02:00
Quentin Dufour	12a50ef2db	Remove all regressions	2022-09-28 14:59:42 +02:00
Quentin Dufour	1ae69c8e2a	To fit or not to fit	2022-09-28 14:59:18 +02:00
Quentin Dufour	7011ec431f	Add sled no-sync	2022-09-27 11:50:08 +02:00
Quentin Dufour	c78e4f8005	Details	2022-09-26 17:09:53 +02:00
Quentin Dufour	71d0d82361	Add s3billion minio graphs	2022-09-26 17:05:56 +02:00
Quentin Dufour	b0b4b162dc	Minio s3billion benchmark	2022-09-26 16:39:19 +02:00
Quentin Dufour	b4c1f58a3c	DB engine comparison	2022-09-24 18:09:18 +02:00
Quentin Dufour	c84dc6bd35	Better scales	2022-09-24 17:10:33 +02:00
Quentin Dufour	3fc014ad3c	Add unit	2022-09-24 17:09:17 +02:00
Quentin Dufour	c181eee22e	Fix plot	2022-09-24 17:07:39 +02:00
Quentin Dufour	4d61d4a72f	Refactor + fix typo in 1M+ objects bench	2022-09-24 17:06:43 +02:00
Quentin Dufour	a869e3614b	LMDB s3 million objects	2022-09-24 17:01:34 +02:00
Quentin Dufour	a744f72f15	Bench PutObject with many parallel small obj	2022-09-24 15:13:40 +02:00
Quentin Dufour	ac4f100da1	Add a first warp plot	2022-09-24 11:43:30 +02:00
Quentin Dufour	0dc16e5e80	Support minio warp	2022-09-23 23:20:10 +02:00
Quentin Dufour	60df0fb95d	Add black outline	2022-09-23 22:56:05 +02:00
Quentin Dufour	fa61df5e07	Add latency	2022-09-23 22:53:35 +02:00
Quentin Dufour	a40c123542	Add S3TTFB	2022-09-23 22:50:50 +02:00
Quentin Dufour	700025e479	Add plot	2022-09-23 19:15:58 +02:00
Quentin Dufour	5b460dd10f	Done some measurements for s3lat with Garage	2022-09-23 18:47:13 +02:00
Quentin Dufour	4a02f16489	Add support for minio-s3lat	2022-09-23 18:09:54 +02:00
Quentin Dufour	c9cbe5fc52	Add some support for minio	2022-09-23 17:55:45 +02:00
Quentin Dufour	0664442648	Add support for txqueuelen	2022-09-19 18:02:33 +02:00
Quentin Dufour	549c548e7c	Commit WIP	2022-09-19 14:16:08 +02:00
Quentin Dufour	bd6a125399	Create bandwidth bottleneck topo	2022-09-19 12:44:52 +02:00
Quentin Dufour	813b35e479	WIP s3concurrent + fix prepare.py	2022-09-19 12:41:51 +02:00
Quentin Dufour	1ea2244c8c	Update s3concurrent	2022-09-19 12:37:25 +02:00
Quentin Dufour	12764a020a	Fix prepare script	2022-09-16 16:19:09 +02:00
Quentin Dufour	b95db09b96	Create a prepare script	2022-09-16 15:43:58 +02:00
Quentin Dufour	4bec653fe9	Add warp	2022-09-15 18:04:37 +02:00
Quentin Dufour	18b8cb0f2b	update examples	2022-09-15 17:34:35 +02:00
Quentin Dufour	6588e878b2	Add flavors	2022-09-15 17:33:42 +02:00
Quentin Dufour	9652978433	Improve README	2022-09-15 17:12:21 +02:00
Quentin Dufour	62e4ed55a4	Implement scenario subcommand	2022-09-15 17:00:02 +02:00
Quentin Dufour	cfd3e500fe	run-all wait all processes before returning	2022-09-15 16:53:21 +02:00
Quentin Dufour	68cc7d635b	More doc	2022-09-14 18:35:09 +02:00
Quentin Dufour	8bebbd3181	Fix doc	2022-09-14 18:32:07 +02:00
Quentin Dufour	f0db8ba560	Scenario based benchmarks will be ready soon	2022-09-14 18:22:46 +02:00
Quentin Dufour	c75b239018	Update warp submodule	2022-09-14 18:04:03 +02:00
Quentin Dufour	c77d8dcfa7	Working s3lat with Garage	2022-09-14 18:01:44 +02:00
Quentin Dufour	e3409ce6b7	New way to deploy Garage	2022-09-14 17:25:16 +02:00
trinity-1686a	30643ca50d	add support for limiting queue len	2022-08-31 21:29:18 +02:00
				`@ -0,0 +1 @@`
				`Subproject commit 902555b9d5db594fdad927f51c2d8984d6e9f6a5`
				`@ -0,0 +1 @@`
				`Subproject commit df8f2cd7b7a8523755b4d3c04271a69571a455ca`