Merge pull request 'Fix venv sourcing' (#2 ) from baptiste/mknet:misc_fixes into main

Reviewed-on: Deuxfleurs/mknet#2
Merge pull request 'Add result with 30M objects on Grid'5000' (#3 ) from baptiste/mknet:gros_30M into main
2022-09-29 20:57:53 +02:00 · 2022-09-29 20:53:21 +02:00 · 2022-09-29 20:27:25 +02:00 · 2022-09-29 10:40:34 +02:00 · 2022-09-29 10:39:45 +02:00 · 2022-09-28 14:59:42 +02:00
97 changed files with 8372 additions and 156 deletions
--- a/.gitignore
+++ b/.gitignore
@ -1,3 +1,6 @@
 .current_state.yml
 __pycache__
 *.swp
+build
+mknet.egg-info
+.venv
--- a/.gitmodules
+++ b/.gitmodules
@ -0,0 +1,6 @@
+[submodule "benchmarks/warp"]
+	path = benchmarks/warp
+	url = https://github.com/minio/warp
+[submodule "analysis/quentin"]
+	path = analysis/quentin
+	url = https://git.deuxfleurs.fr/quentin/benchmarks
--- a/README.md
+++ b/README.md
@ -1,43 +1,77 @@
 # mknet

-## Installation
+mknet is a tool to simulate various network topologies
+locally thanks to network namespaces and traffic control (tc).
+
+## Prepare your environment
+
+Get the repository:

 ```bash
-sudo pip3 install git+https://git.deuxfleurs.fr/trinity-1686a/mknet
+git clone https://git.deuxfleurs.fr/Deuxfleurs/mknet.git
+cd mknet
 ```

-## Usage
+Run our configuration script:

 ```bash
-sudo rm -rf /tmp/garage-testnet/ # always start by deleting previous run
-sudo mknet create ./config.yml
-sudo mknet run-all example/deploy_garage.sh
-sudo mknet run dc1:dc1s1 garage -c /tmp/garage-testnet/dc1/dc1s1/garage.toml status
-sudo mknet destroy
+./prepare.py
 ```

-## Instrumented daemons
-
-If you want to use the scripts provided in the `example` folder,
-you must add to your path some tools.
-
-### Garage (`deploy_garage.sh`)
+Now, you are ready to launch an experiment:

 ```bash
-# see versions on https://garagehq.deuxfleurs.fr/_releases.html
-export GRG_ARCH=x86_64-unknown-linux-musl
-export GRG_VERSION=v0.5.0
-
-sudo wget https://garagehq.deuxfleurs.fr/_releases/${GRG_VERSION}/${GRG_ARCH}/garage -O /usr/local/bin/garage
-sudo chmod +x /usr/local/bin/garage
-
-garage help
+sudo -i
+source .venv/bin/activate
+./mknet scenario ./topo/50ms.yml ./scenarios/garage-s3lat garage-v0.8
 ```

-### Minio (`deploy_minio.py`)
+If a script crash, you must manually destroy the topology:

 ```
-sudo wget https://dl.min.io/server/minio/release/linux-amd64/minio -O /usr/local/bin/minio
-sudo chmod +x /usr/local/bin/minio
+./mknet destroy
 ```

+## Topologies
+
+All topologies:
+  - `./topo/dc.yml` - A 3 node topology connected with a 1Gbit/s link and a 1ms latency
+  - `./topo/with-vdsl.yml` - A topology mixing datacenters with fast internal connectivity and an isolated VDSL node
+  - `./topo/50ms.yml` - An artifical topology simulating nodes with high bandwidth but with a fixed 50ms latency, useful to quantify the impact of latency on a distributed software
+  - `./topo/multi-dc.yml` - Simulate 3 DC interconnected with 50ms latency WAN network and close to zero latency inside the DC
+  - `./topo/{1,5,10,50,100}mbps.yml` - Simulate 3 nodes interconnected with a bandwidth bottleneck.
+
+Feel free to write new topologies!
+
+## Scenarios
+
+All scenarios:
+ - `./scenarios/garage-s3lat [garage-v0.7|garage-v0.8]` - Run s3lat on Garage
+ - `./scenarios/garage-concurrent [garage-v0.7|garage-v0.8]` - Run s3concurrent on Garage
+ - `./scenarios/garage-warp [garage-v0.7|garage-v0.8] [default|fast]` - Run warp on Garage. 2 flavors are available: fast and default.
+ - `./scenarios/garage-s3billion [garage-v0.7|garage-v0.8]` - Run s3billion on Garage
+
+*Scenarios take optional flavors as input that modulate their behavioir. Passing them is not mandatory,
+a default one will be selected for you.*
+
+How to run them:
+
+```bash
+./mknet scenario <topology> <scenario> [flavors...]
+./mknet scenario ./topo/50ms.yml ./scenarios/garage-s3lat garage-v0.8
+```
+
+How to write good scenarios:
+  - If a scenario can be run with multiple different parameters, write one scenario with multiple flavors
+  - If the logic ran is different, write a new scenario
+  - A scenario code must remain short and looks like a DSL, abstract the logic in the `fragments/` module
+
+## Manual usage
+
+```bash
+./mknet create ./topo/with-vdsl.yml
+./mknet run-all ./scenarios/garage-manual
+./mknet run dc1:dc1s1 /tmp/mknet-bin/garage* -c /tmp/mknet-store/garage/dc1/dc1s1/garage.toml status
+./mknet run-all ./scenarios/clean
+./mknet destroy
+
--- a/analysis/quentin
+++ b/analysis/quentin
@ -0,0 +1 @@
+Subproject commit 902555b9d5db594fdad927f51c2d8984d6e9f6a5
--- a/artifacts/2022-09-19-s3concurrent/garage-v0.7_10mbps.csv
+++ b/artifacts/2022-09-19-s3concurrent/garage-v0.7_10mbps.csv
@ -0,0 +1,17 @@
+sent,success,elapsed,elapsed_per_req
+1,1,1733568406,1733568406
+2,2,3717781951,1858890975
+3,3,5554603016,1851534338
+4,4,7243764558,1810941139
+5,5,9088779023,1817755804
+6,6,10853379229,1808896538
+7,7,12687932044,1812561720
+8,8,14457291732,1807161466
+9,9,16271996844,1807999649
+10,10,18066058997,1806605899
+11,11,20370606012,1851873273
+12,12,11605742376,967145198
+13,13,13622546503,1047888192
+14,14,16059774589,1147126756
+15,15,20283463752,1352230916
+16,16,23829428368,1489339273
--- a/artifacts/2022-09-19-s3concurrent/garage-v0.7_5mbps.csv
+++ b/artifacts/2022-09-19-s3concurrent/garage-v0.7_5mbps.csv
@ -0,0 +1,17 @@
+sent,success,elapsed,elapsed_per_req
+1,1,3646698401,3646698401
+2,2,7245492636,3622746318
+3,3,10896710797,3632236932
+4,4,14469160703,3617290175
+5,4,18066381776,3613276355
+6,3,19209830571,3201638428
+7,0,186084824,26583546
+8,0,183207903,22900987
+9,0,199743038,22193670
+10,0,195703069,19570306
+11,0,198204447,18018586
+12,0,187247329,15603944
+13,0,193441556,14880119
+14,0,192003016,13714501
+15,0,140156357,9343757
+16,0,186228134,11639258
--- a/artifacts/2022-09-19-s3concurrent/garage-v0.8_10mbps.csv
+++ b/artifacts/2022-09-19-s3concurrent/garage-v0.8_10mbps.csv
@ -0,0 +1,17 @@
+sent,success,elapsed,elapsed_per_req
+1,1,1634632370,1634632370
+2,2,3799299300,1899649650
+3,3,5436258664,1812086221
+4,4,7242028242,1810507060
+5,5,9041569964,1808313992
+6,6,10852330277,1808721712
+7,7,12646683069,1806669009
+8,8,14468867472,1808608434
+9,9,16251724200,1805747133
+10,10,18190271787,1819027178
+11,11,19877017065,1807001551
+12,12,20526867033,1710572252
+13,13,24558838287,1889141406
+14,14,25274586429,1805327602
+15,15,27180202376,1812013491
+16,16,29437555977,1839847248
--- a/artifacts/2022-09-19-s3concurrent/garage-v0.8_5mbps.csv
+++ b/artifacts/2022-09-19-s3concurrent/garage-v0.8_5mbps.csv
@ -0,0 +1,17 @@
+sent,success,elapsed,elapsed_per_req
+1,1,3638584739,3638584739
+2,2,7246358601,3623179300
+3,3,11160920229,3720306743
+4,4,14454330626,3613582656
+5,5,18052975913,3610595182
+6,6,21690341725,3615056954
+7,7,25268069065,3609724152
+8,8,29007558135,3625944766
+9,9,32688763527,3632084836
+10,10,36101205725,3610120572
+11,11,28245933586,2567812144
+12,12,43983751108,3665312592
+13,13,53456719112,4112055316
+14,14,54398690114,3885620722
+15,15,54153860198,3610257346
+16,16,57732527899,3608282993
--- a/artifacts/2022-09-23-s3lat/garage-v0.7.csv
+++ b/artifacts/2022-09-23-s3lat/garage-v0.7.csv
@ -0,0 +1,501 @@
+endpoint,nanoseconds
+listbuckets,123446207
+listbuckets,131497917
+listbuckets,39054735
+listbuckets,105410142
+listbuckets,134197174
+listbuckets,92373770
+listbuckets,113362017
+listbuckets,109195200
+listbuckets,176558852
+listbuckets,27177881
+listbuckets,36252695
+listbuckets,106228157
+listbuckets,167364021
+listbuckets,6904816
+listbuckets,34264521
+listbuckets,83810002
+listbuckets,156429277
+listbuckets,200458223
+listbuckets,112529004
+listbuckets,35129564
+listbuckets,79573343
+listbuckets,140150702
+listbuckets,120555536
+listbuckets,29181815
+listbuckets,51198242
+listbuckets,56133478
+listbuckets,47130673
+listbuckets,4870606
+listbuckets,115299404
+listbuckets,199760726
+listbuckets,104304255
+listbuckets,165317856
+listbuckets,174747530
+listbuckets,69139944
+listbuckets,170345599
+listbuckets,133286913
+listbuckets,133428040
+listbuckets,152317101
+listbuckets,163419603
+listbuckets,11954339
+listbuckets,49001694
+listbuckets,1909515
+listbuckets,85317086
+listbuckets,193461440
+listbuckets,37110905
+listbuckets,170866722
+listbuckets,2820145
+listbuckets,136496793
+listbuckets,64132721
+listbuckets,125251903
+listbuckets,87449583
+listbuckets,156290592
+listbuckets,160402726
+listbuckets,200273480
+listbuckets,93021081
+listbuckets,68304026
+listbuckets,46970448
+listbuckets,165308307
+listbuckets,41980457
+listbuckets,166480048
+listbuckets,136400950
+listbuckets,111337721
+listbuckets,180455774
+listbuckets,119142367
+listbuckets,117091664
+listbuckets,126145949
+listbuckets,119181500
+listbuckets,55147467
+listbuckets,178508747
+listbuckets,101172269
+listbuckets,116156709
+listbuckets,53168650
+listbuckets,29044786
+listbuckets,47899914
+listbuckets,139056994
+listbuckets,26962604
+listbuckets,50780131
+listbuckets,36960440
+listbuckets,161531494
+listbuckets,9947611
+listbuckets,33134947
+listbuckets,81109551
+listbuckets,153415682
+listbuckets,199163150
+listbuckets,183053868
+listbuckets,148142646
+listbuckets,17965198
+listbuckets,161258658
+listbuckets,175383896
+listbuckets,42076378
+listbuckets,34970593
+listbuckets,14042201
+listbuckets,66182118
+listbuckets,42936091
+listbuckets,200730649
+listbuckets,75462399
+listbuckets,34967326
+listbuckets,99414279
+listbuckets,83041589
+listbuckets,9915941
+putobject,102659465
+putobject,92835679
+putobject,108731599
+putobject,201811799
+putobject,106443490
+putobject,92567983
+putobject,238871966
+putobject,104158815
+putobject,106243742
+putobject,116316799
+putobject,108757989
+putobject,97366623
+putobject,97898375
+putobject,96967168
+putobject,100256148
+putobject,107432215
+putobject,102248459
+putobject,251441345
+putobject,104946713
+putobject,110238211
+putobject,108091689
+putobject,113202108
+putobject,98765973
+putobject,104565383
+putobject,111568023
+putobject,112197090
+putobject,105911994
+putobject,95484988
+putobject,104482906
+putobject,234879244
+putobject,113432122
+putobject,100713850
+putobject,108868215
+putobject,103499519
+putobject,104655812
+putobject,115506648
+putobject,101683821
+putobject,108092129
+putobject,101964070
+putobject,106886313
+putobject,112577538
+putobject,261098453
+putobject,113249625
+putobject,108966089
+putobject,109411069
+putobject,104522489
+putobject,110678803
+putobject,105536735
+putobject,110833494
+putobject,112977212
+putobject,98197749
+putobject,110378043
+putobject,375687474
+putobject,203462479
+putobject,115639116
+putobject,118053054
+putobject,110270019
+putobject,109764686
+putobject,90674345
+putobject,98625115
+putobject,103500960
+putobject,102995627
+putobject,236288420
+putobject,107760181
+putobject,112772765
+putobject,99998427
+putobject,117912217
+putobject,105863219
+putobject,113652925
+putobject,100828684
+putobject,95361573
+putobject,95500334
+putobject,106068286
+putobject,99928444
+putobject,261266283
+putobject,106924492
+putobject,110718855
+putobject,87597876
+putobject,108996183
+putobject,99171595
+putobject,115447052
+putobject,103467696
+putobject,104448307
+putobject,97229488
+putobject,106477027
+putobject,105427947
+putobject,233050123
+putobject,115135303
+putobject,113237921
+putobject,110612835
+putobject,102891788
+putobject,112480159
+putobject,117117775
+putobject,196493186
+putobject,104291470
+putobject,118422258
+putobject,104951716
+putobject,244946550
+putobject,98641292
+putobject,119142327
+listobjects,227618086
+listobjects,218373723
+listobjects,220425637
+listobjects,212659074
+listobjects,194860492
+listobjects,221131339
+listobjects,227735226
+listobjects,212887826
+listobjects,202906841
+listobjects,221091271
+listobjects,216883738
+listobjects,213001179
+listobjects,217273734
+listobjects,211956305
+listobjects,213529174
+listobjects,375065283
+listobjects,218843356
+listobjects,217068015
+listobjects,223369781
+listobjects,219801804
+listobjects,225260921
+listobjects,207088252
+listobjects,229780152
+listobjects,227569658
+listobjects,211492577
+listobjects,200421226
+listobjects,224666076
+listobjects,217078671
+listobjects,220658029
+listobjects,220137146
+listobjects,205746397
+listobjects,222356274
+listobjects,210329960
+listobjects,219004606
+listobjects,226442588
+listobjects,374038605
+listobjects,221425877
+listobjects,226949625
+listobjects,198272437
+listobjects,215884986
+listobjects,210904573
+listobjects,231082876
+listobjects,220581992
+listobjects,215368028
+listobjects,204236634
+listobjects,230323641
+listobjects,221006783
+listobjects,197521967
+listobjects,217673459
+listobjects,210402664
+listobjects,222289473
+listobjects,216105460
+listobjects,233017185
+listobjects,217884686
+listobjects,216886613
+listobjects,215456984
+listobjects,342657645
+listobjects,218310540
+listobjects,346101165
+listobjects,227114600
+listobjects,207167945
+listobjects,217958955
+listobjects,256687056
+listobjects,216946794
+listobjects,211024233
+listobjects,221148255
+listobjects,214229333
+listobjects,224119805
+listobjects,217875907
+listobjects,216024912
+listobjects,220679148
+listobjects,218154613
+listobjects,222423060
+listobjects,212511440
+listobjects,215068766
+listobjects,207965636
+listobjects,374287848
+listobjects,231411797
+listobjects,344053841
+listobjects,218949902
+listobjects,208085171
+listobjects,202097107
+listobjects,220625984
+listobjects,232280185
+listobjects,214265437
+listobjects,216862289
+listobjects,219434755
+listobjects,217021028
+listobjects,214401512
+listobjects,207317610
+listobjects,215392944
+listobjects,217703786
+listobjects,215988628
+listobjects,198853351
+listobjects,222486834
+listobjects,213094404
+listobjects,211948650
+listobjects,215660888
+listobjects,222729552
+listobjects,357486752
+getobject,120789563
+getobject,257364122
+getobject,193090188
+getobject,105639883
+getobject,110039709
+getobject,106107405
+getobject,113226765
+getobject,228216027
+getobject,204992307
+getobject,115070175
+getobject,118444544
+getobject,116666047
+getobject,114052724
+getobject,109807911
+getobject,108123550
+getobject,245998795
+getobject,103858529
+getobject,104810357
+getobject,101301352
+getobject,104881290
+getobject,101871689
+getobject,245400756
+getobject,99811659
+getobject,107963659
+getobject,108133779
+getobject,109433413
+getobject,112918863
+getobject,99263685
+getobject,109925392
+getobject,404106741
+getobject,111264038
+getobject,107867546
+getobject,232065851
+getobject,104837346
+getobject,249346993
+getobject,107660264
+getobject,98665144
+getobject,93135947
+getobject,93305867
+getobject,103816208
+getobject,101191343
+getobject,104046332
+getobject,234664977
+getobject,109126683
+getobject,109256458
+getobject,111475958
+getobject,106229192
+getobject,111367102
+getobject,247160403
+getobject,114725281
+getobject,115208863
+getobject,115752038
+getobject,99434436
+getobject,114499775
+getobject,114262136
+getobject,237897437
+getobject,109039858
+getobject,116366349
+getobject,109513421
+getobject,119350461
+getobject,103568549
+getobject,268923073
+getobject,103935562
+getobject,114438288
+getobject,250696155
+getobject,100985713
+getobject,108943735
+getobject,107398809
+getobject,201770787
+getobject,258949074
+getobject,108722357
+getobject,106948669
+getobject,105753903
+getobject,106842969
+getobject,254243109
+getobject,109977076
+getobject,101122461
+getobject,106152776
+getobject,105146877
+getobject,115352192
+getobject,114154611
+getobject,106352783
+getobject,316675402
+getobject,109678912
+getobject,105611654
+getobject,111231964
+getobject,110622814
+getobject,103520055
+getobject,264080226
+getobject,115777874
+getobject,117219506
+getobject,237658128
+getobject,105861315
+getobject,118189958
+getobject,250123569
+getobject,210303951
+getobject,114146153
+getobject,114062677
+getobject,116419134
+getobject,110233217
+removeobject,230846328
+removeobject,352354358
+removeobject,213513979
+removeobject,213021069
+removeobject,201490202
+removeobject,212111001
+removeobject,215118297
+removeobject,206781539
+removeobject,490175308
+removeobject,207408693
+removeobject,201921334
+removeobject,214565432
+removeobject,223499977
+removeobject,219187737
+removeobject,353799701
+removeobject,226319108
+removeobject,204468762
+removeobject,206638858
+removeobject,227752042
+removeobject,227770838
+removeobject,368061761
+removeobject,304793510
+removeobject,231980092
+removeobject,227557765
+removeobject,216285686
+removeobject,204581431
+removeobject,332004595
+removeobject,209905432
+removeobject,208660772
+removeobject,298520566
+removeobject,221629301
+removeobject,216850310
+removeobject,346809979
+removeobject,225626686
+removeobject,335040322
+removeobject,217416498
+removeobject,221590626
+removeobject,211254506
+removeobject,218186009
+removeobject,333605972
+removeobject,214970419
+removeobject,400660245
+removeobject,226176422
+removeobject,210009344
+removeobject,214122397
+removeobject,217046195
+removeobject,348420624
+removeobject,222496374
+removeobject,200770472
+removeobject,206765238
+removeobject,212797886
+removeobject,221853199
+removeobject,366803902
+removeobject,369766134
+removeobject,254126638
+removeobject,206946940
+removeobject,214755369
+removeobject,222743167
+removeobject,214028196
+removeobject,365328074
+removeobject,222478296
+removeobject,200478669
+removeobject,228313711
+removeobject,351395352
+removeobject,347788382
+removeobject,370554938
+removeobject,220125163
+removeobject,211178604
+removeobject,478779432
+removeobject,226303414
+removeobject,212846913
+removeobject,232756403
+removeobject,352451452
+removeobject,204864755
+removeobject,386939750
+removeobject,236333214
+removeobject,213864532
+removeobject,337968992
+removeobject,208224425
+removeobject,215012709
+removeobject,220231159
+removeobject,200163319
+removeobject,201449085
+removeobject,203033356
+removeobject,362988387
+removeobject,352665831
+removeobject,255105686
+removeobject,209154819
+removeobject,218832617
+removeobject,220790763
+removeobject,345915165
+removeobject,213304301
+removeobject,218302224
+removeobject,218254023
+removeobject,204621450
+removeobject,213964266
+removeobject,383714183
+removeobject,312511671
+removeobject,207629778
+removeobject,203927189
--- a/artifacts/2022-09-23-s3lat/garage-v0.8.csv
+++ b/artifacts/2022-09-23-s3lat/garage-v0.8.csv
@ -0,0 +1,501 @@
+endpoint,nanoseconds
+listbuckets,142443772
+listbuckets,78416939
+listbuckets,58182021
+listbuckets,131770558
+listbuckets,23167382
+listbuckets,100239233
+listbuckets,115131496
+listbuckets,104406526
+listbuckets,12172321
+listbuckets,196757641
+listbuckets,154690693
+listbuckets,200549327
+listbuckets,97233951
+listbuckets,157416927
+listbuckets,91145569
+listbuckets,19144199
+listbuckets,64085612
+listbuckets,113225008
+listbuckets,166909251
+listbuckets,34198118
+listbuckets,174705394
+listbuckets,87295494
+listbuckets,57706664
+listbuckets,140442963
+listbuckets,14837772
+listbuckets,135331146
+listbuckets,111062526
+listbuckets,192533228
+listbuckets,78240866
+listbuckets,49209288
+listbuckets,139333059
+listbuckets,159433712
+listbuckets,168386408
+listbuckets,54060994
+listbuckets,182381749
+listbuckets,107235324
+listbuckets,19175648
+listbuckets,79408539
+listbuckets,102182087
+listbuckets,138252068
+listbuckets,200286419
+listbuckets,171574163
+listbuckets,139244520
+listbuckets,162405069
+listbuckets,191507551
+listbuckets,156309385
+listbuckets,45147822
+listbuckets,123534893
+listbuckets,114180389
+listbuckets,200631309
+listbuckets,183352440
+listbuckets,42285029
+listbuckets,185592518
+listbuckets,166551994
+listbuckets,63313864
+listbuckets,51172191
+listbuckets,162406210
+listbuckets,51252382
+listbuckets,5096417
+listbuckets,177549418
+listbuckets,21073149
+listbuckets,98302685
+listbuckets,173578013
+listbuckets,91105261
+listbuckets,105040220
+listbuckets,112304298
+listbuckets,65084455
+listbuckets,181875861
+listbuckets,142442935
+listbuckets,131125617
+listbuckets,76286586
+listbuckets,24552325
+listbuckets,121486988
+listbuckets,186709964
+listbuckets,5948226
+listbuckets,161299340
+listbuckets,40993441
+listbuckets,66113978
+listbuckets,42309585
+listbuckets,124003416
+listbuckets,97993931
+listbuckets,31175113
+listbuckets,98603492
+listbuckets,21409874
+listbuckets,194297813
+listbuckets,50059832
+listbuckets,148399545
+listbuckets,48887920
+listbuckets,152334411
+listbuckets,57077936
+listbuckets,26810977
+listbuckets,194375880
+listbuckets,138228951
+listbuckets,188415922
+listbuckets,59004763
+listbuckets,28981072
+listbuckets,41073893
+listbuckets,29776223
+listbuckets,140987798
+listbuckets,96201298
+putobject,96916217
+putobject,204175953
+putobject,116377273
+putobject,200815462
+putobject,189393367
+putobject,201157286
+putobject,209378110
+putobject,200861878
+putobject,127598269
+putobject,198228669
+putobject,206087830
+putobject,190252106
+putobject,213164125
+putobject,113273726
+putobject,184844831
+putobject,217834778
+putobject,213577904
+putobject,187314812
+putobject,197335012
+putobject,191781450
+putobject,245923315
+putobject,249794209
+putobject,154268417
+putobject,204865983
+putobject,200616863
+putobject,218347254
+putobject,225218951
+putobject,204122038
+putobject,187483799
+putobject,224081164
+putobject,313480753
+putobject,207639555
+putobject,231879901
+putobject,213694483
+putobject,221029143
+putobject,210575405
+putobject,203991790
+putobject,97757552
+putobject,202949604
+putobject,187003632
+putobject,192703458
+putobject,198853855
+putobject,187677763
+putobject,217450238
+putobject,201097498
+putobject,204520778
+putobject,209010932
+putobject,103217905
+putobject,208237561
+putobject,212482783
+putobject,223855484
+putobject,215215920
+putobject,196011159
+putobject,207887130
+putobject,203901248
+putobject,195527114
+putobject,194314145
+putobject,118554688
+putobject,174577871
+putobject,182678497
+putobject,214551635
+putobject,193939918
+putobject,210241591
+putobject,218407872
+putobject,212235655
+putobject,191045716
+putobject,199345769
+putobject,212445320
+putobject,213398017
+putobject,196443131
+putobject,111006983
+putobject,206131544
+putobject,227743928
+putobject,205929843
+putobject,210773624
+putobject,201460969
+putobject,178288219
+putobject,194333228
+putobject,200422249
+putobject,211491788
+putobject,277570466
+putobject,194155773
+putobject,202859287
+putobject,210188078
+putobject,200376501
+putobject,222446189
+putobject,216905202
+putobject,203768673
+putobject,211882053
+putobject,199813369
+putobject,202275734
+putobject,225392157
+putobject,108514537
+putobject,183571858
+putobject,220178028
+putobject,244245555
+putobject,246644462
+putobject,202754498
+putobject,125491965
+putobject,198043126
+listobjects,247336346
+listobjects,222421821
+listobjects,217409432
+listobjects,213489945
+listobjects,197741076
+listobjects,204144700
+listobjects,194520187
+listobjects,207039462
+listobjects,214175739
+listobjects,222630443
+listobjects,207519367
+listobjects,204852995
+listobjects,200389171
+listobjects,223342436
+listobjects,375412313
+listobjects,197168061
+listobjects,209902283
+listobjects,208728457
+listobjects,223192111
+listobjects,197001637
+listobjects,210692202
+listobjects,210082361
+listobjects,193242654
+listobjects,190713450
+listobjects,210137746
+listobjects,210096607
+listobjects,214155697
+listobjects,203814873
+listobjects,200252782
+listobjects,198504662
+listobjects,225529670
+listobjects,219412867
+listobjects,200873322
+listobjects,366185537
+listobjects,220358802
+listobjects,203736154
+listobjects,195030295
+listobjects,206763616
+listobjects,215264828
+listobjects,196677593
+listobjects,206296625
+listobjects,203169834
+listobjects,228228451
+listobjects,201871694
+listobjects,214160805
+listobjects,221120508
+listobjects,204784600
+listobjects,215667356
+listobjects,203478756
+listobjects,193957098
+listobjects,204028493
+listobjects,200296721
+listobjects,201366590
+listobjects,217724938
+listobjects,335372225
+listobjects,212399058
+listobjects,206802897
+listobjects,203512327
+listobjects,201467970
+listobjects,207952516
+listobjects,200169128
+listobjects,193368365
+listobjects,208873233
+listobjects,206348419
+listobjects,202684585
+listobjects,255082511
+listobjects,225277839
+listobjects,216147871
+listobjects,203396588
+listobjects,215594185
+listobjects,211893541
+listobjects,202693570
+listobjects,213428668
+listobjects,345450849
+listobjects,210503146
+listobjects,206714788
+listobjects,221670971
+listobjects,209193505
+listobjects,207268362
+listobjects,217422133
+listobjects,194262801
+listobjects,214257460
+listobjects,209999916
+listobjects,200992698
+listobjects,233030646
+listobjects,197622419
+listobjects,214531387
+listobjects,207687472
+listobjects,193650794
+listobjects,216352563
+listobjects,204419966
+listobjects,220824902
+listobjects,203824031
+listobjects,345556923
+listobjects,211586088
+listobjects,221054525
+listobjects,216966641
+listobjects,212481105
+listobjects,225077725
+listobjects,206661945
+getobject,106148474
+getobject,103474999
+getobject,110847021
+getobject,89406401
+getobject,107194067
+getobject,103967969
+getobject,99809582
+getobject,218870171
+getobject,116458559
+getobject,116438822
+getobject,114486708
+getobject,98523284
+getobject,107932449
+getobject,96947000
+getobject,111244400
+getobject,107520933
+getobject,96707408
+getobject,98561777
+getobject,100203555
+getobject,102302806
+getobject,238528526
+getobject,271793475
+getobject,103513430
+getobject,110508380
+getobject,88297618
+getobject,93017464
+getobject,92182439
+getobject,100926740
+getobject,90912953
+getobject,89262659
+getobject,106320567
+getobject,114727932
+getobject,93467784
+getobject,233897134
+getobject,99598463
+getobject,100682658
+getobject,196532397
+getobject,114136005
+getobject,107149842
+getobject,102095284
+getobject,101561076
+getobject,110858641
+getobject,112464019
+getobject,89475711
+getobject,104275969
+getobject,98529866
+getobject,241428685
+getobject,101153106
+getobject,119006084
+getobject,89105802
+getobject,95722098
+getobject,267246924
+getobject,106170312
+getobject,106624860
+getobject,101451128
+getobject,114344067
+getobject,98506251
+getobject,85363560
+getobject,109668504
+getobject,97277512
+getobject,217166462
+getobject,95365244
+getobject,105298136
+getobject,107433406
+getobject,91927105
+getobject,103556189
+getobject,101882261
+getobject,97490353
+getobject,109313364
+getobject,107477889
+getobject,105115091
+getobject,100009677
+getobject,96802735
+getobject,243376778
+getobject,116897654
+getobject,246725949
+getobject,104940522
+getobject,104899223
+getobject,99914467
+getobject,101929660
+getobject,96935645
+getobject,107399321
+getobject,94553040
+getobject,110316235
+getobject,112920198
+getobject,100314041
+getobject,239311246
+getobject,114578006
+getobject,106688650
+getobject,89102575
+getobject,82537327
+getobject,102124717
+getobject,107306736
+getobject,113530840
+getobject,102225458
+getobject,105090914
+getobject,109987193
+getobject,107637409
+getobject,99663933
+getobject,252948739
+removeobject,218637514
+removeobject,459986423
+removeobject,307487771
+removeobject,425595066
+removeobject,325293018
+removeobject,326633357
+removeobject,490052955
+removeobject,252844239
+removeobject,313024066
+removeobject,301593436
+removeobject,299154213
+removeobject,396128610
+removeobject,227674663
+removeobject,439796598
+removeobject,464313791
+removeobject,433487342
+removeobject,316319351
+removeobject,324010444
+removeobject,224091911
+removeobject,259658834
+removeobject,268735522
+removeobject,466588289
+removeobject,328431494
+removeobject,456136096
+removeobject,214305954
+removeobject,349200623
+removeobject,310132173
+removeobject,433264807
+removeobject,438724929
+removeobject,333912094
+removeobject,443030552
+removeobject,338569189
+removeobject,452852014
+removeobject,450187315
+removeobject,407122424
+removeobject,371514674
+removeobject,371585948
+removeobject,327859889
+removeobject,287348277
+removeobject,319214105
+removeobject,408538253
+removeobject,315233492
+removeobject,321187307
+removeobject,308503342
+removeobject,481438858
+removeobject,464326677
+removeobject,458208213
+removeobject,334169040
+removeobject,380953007
+removeobject,306156310
+removeobject,311735117
+removeobject,304652713
+removeobject,447482399
+removeobject,464054872
+removeobject,308287070
+removeobject,366031357
+removeobject,223529039
+removeobject,305332454
+removeobject,440680123
+removeobject,283803681
+removeobject,410491748
+removeobject,223265289
+removeobject,306268419
+removeobject,347366809
+removeobject,314563701
+removeobject,302068212
+removeobject,283520375
+removeobject,377963268
+removeobject,454052170
+removeobject,298085627
+removeobject,457265001
+removeobject,233934260
+removeobject,432713854
+removeobject,315891533
+removeobject,445005587
+removeobject,321635590
+removeobject,390314770
+removeobject,230655210
+removeobject,340190850
+removeobject,305314423
+removeobject,444872212
+removeobject,223437549
+removeobject,385529268
+removeobject,280271669
+removeobject,323868430
+removeobject,307544418
+removeobject,238040604
+removeobject,310293395
+removeobject,412163151
+removeobject,367516076
+removeobject,288358726
+removeobject,303789421
+removeobject,435084818
+removeobject,411008890
+removeobject,467616201
+removeobject,219465170
+removeobject,327976568
+removeobject,332048651
+removeobject,353130028
+removeobject,329197320
--- a/artifacts/2022-09-23-s3lat/minio-20220917.csv
+++ b/artifacts/2022-09-23-s3lat/minio-20220917.csv
@ -0,0 +1,501 @@
+endpoint,nanoseconds
+listbuckets,110526156
+listbuckets,111223674
+listbuckets,110350078
+listbuckets,108763355
+listbuckets,108151200
+listbuckets,110049558
+listbuckets,111261740
+listbuckets,106870917
+listbuckets,109675307
+listbuckets,114928865
+listbuckets,112976189
+listbuckets,100144534
+listbuckets,103672802
+listbuckets,115743698
+listbuckets,120312198
+listbuckets,114315912
+listbuckets,117664396
+listbuckets,113756128
+listbuckets,110819985
+listbuckets,108017540
+listbuckets,107455412
+listbuckets,117667963
+listbuckets,113112343
+listbuckets,117163158
+listbuckets,111710827
+listbuckets,108587376
+listbuckets,113677870
+listbuckets,100432934
+listbuckets,111720496
+listbuckets,113415267
+listbuckets,109700442
+listbuckets,116870615
+listbuckets,104193832
+listbuckets,104911949
+listbuckets,107990314
+listbuckets,111855197
+listbuckets,108076368
+listbuckets,109569833
+listbuckets,110641629
+listbuckets,105085431
+listbuckets,113240314
+listbuckets,109189490
+listbuckets,119818535
+listbuckets,111259357
+listbuckets,107390627
+listbuckets,101319090
+listbuckets,106351333
+listbuckets,117073689
+listbuckets,119127359
+listbuckets,117266560
+listbuckets,110077426
+listbuckets,107452728
+listbuckets,112721966
+listbuckets,114883973
+listbuckets,120073375
+listbuckets,103458296
+listbuckets,118342265
+listbuckets,107996502
+listbuckets,109767879
+listbuckets,112580238
+listbuckets,113636111
+listbuckets,111267829
+listbuckets,112798154
+listbuckets,107265512
+listbuckets,108961006
+listbuckets,120928218
+listbuckets,109095182
+listbuckets,110399063
+listbuckets,118328765
+listbuckets,112884542
+listbuckets,117913147
+listbuckets,120238848
+listbuckets,116074701
+listbuckets,109990943
+listbuckets,114816077
+listbuckets,113207212
+listbuckets,112350518
+listbuckets,106189705
+listbuckets,106836706
+listbuckets,114519552
+listbuckets,112357584
+listbuckets,113073218
+listbuckets,108251780
+listbuckets,115340245
+listbuckets,109100358
+listbuckets,116337818
+listbuckets,110881217
+listbuckets,109903972
+listbuckets,113243376
+listbuckets,111667494
+listbuckets,102401925
+listbuckets,109793240
+listbuckets,118365209
+listbuckets,117380902
+listbuckets,116149538
+listbuckets,109060437
+listbuckets,106900385
+listbuckets,101213710
+listbuckets,103102282
+listbuckets,115980177
+putobject,1181582068
+putobject,1208319832
+putobject,1139007207
+putobject,1172748692
+putobject,1213269319
+putobject,1222148262
+putobject,1160354410
+putobject,1186840670
+putobject,1178184375
+putobject,1199772934
+putobject,1243156199
+putobject,1162462450
+putobject,1160912226
+putobject,1181903350
+putobject,1178461918
+putobject,1155923039
+putobject,1198486430
+putobject,1128172593
+putobject,1161827240
+putobject,1191934325
+putobject,1241903028
+putobject,1154793146
+putobject,1182964786
+putobject,1192949284
+putobject,1211748830
+putobject,1153969786
+putobject,1173133011
+putobject,1159925140
+putobject,1151781019
+putobject,1212786711
+putobject,1136411443
+putobject,1161607447
+putobject,1194069897
+putobject,1181340894
+putobject,1154251172
+putobject,1142482758
+putobject,1184635609
+putobject,1401096568
+putobject,1193309386
+putobject,1155633130
+putobject,1185629277
+putobject,1150317409
+putobject,1189977499
+putobject,1172978955
+putobject,1167393215
+putobject,1176612406
+putobject,1220478477
+putobject,1214146462
+putobject,1201982436
+putobject,1187568105
+putobject,1233590518
+putobject,1254380879
+putobject,1195264797
+putobject,1212347722
+putobject,1265459581
+putobject,1199507629
+putobject,1213030838
+putobject,1261491944
+putobject,1175783260
+putobject,1171776028
+putobject,1175985781
+putobject,1204880209
+putobject,1185762634
+putobject,1205618309
+putobject,1156324684
+putobject,1212986779
+putobject,1153785732
+putobject,1186241798
+putobject,1169988181
+putobject,1169236282
+putobject,1159114879
+putobject,1169866154
+putobject,1185878817
+putobject,1303357322
+putobject,1198903604
+putobject,1180614532
+putobject,1198891504
+putobject,1186606007
+putobject,1178912791
+putobject,1181214681
+putobject,1148254776
+putobject,1193521548
+putobject,1172732700
+putobject,1224354922
+putobject,1189768028
+putobject,1187610090
+putobject,1158181354
+putobject,1168240861
+putobject,1184266557
+putobject,1211834285
+putobject,1133902391
+putobject,1169623406
+putobject,1153648781
+putobject,1260717643
+putobject,1213147555
+putobject,1196178431
+putobject,1193078015
+putobject,1192743109
+putobject,1231998132
+putobject,1169290675
+listobjects,445829270
+listobjects,448358224
+listobjects,234487499
+listobjects,331279803
+listobjects,239147274
+listobjects,356161983
+listobjects,231056963
+listobjects,428733347
+listobjects,234973895
+listobjects,349647146
+listobjects,238785967
+listobjects,430581787
+listobjects,325690302
+listobjects,337483670
+listobjects,335639646
+listobjects,327396437
+listobjects,230177176
+listobjects,347284630
+listobjects,230553012
+listobjects,338821210
+listobjects,343466400
+listobjects,346533869
+listobjects,311374291
+listobjects,341162492
+listobjects,235683323
+listobjects,356305631
+listobjects,234888137
+listobjects,335916950
+listobjects,312208053
+listobjects,345090441
+listobjects,449596366
+listobjects,347544888
+listobjects,361989101
+listobjects,235502161
+listobjects,232949965
+listobjects,337218790
+listobjects,447643000
+listobjects,238259618
+listobjects,235023776
+listobjects,420231514
+listobjects,341833697
+listobjects,235388450
+listobjects,232164722
+listobjects,422949063
+listobjects,501989025
+listobjects,240957275
+listobjects,337846216
+listobjects,231434805
+listobjects,430682948
+listobjects,231812947
+listobjects,465260992
+listobjects,433379275
+listobjects,236331095
+listobjects,332051421
+listobjects,237905130
+listobjects,343574528
+listobjects,242371130
+listobjects,311241850
+listobjects,239692903
+listobjects,360069478
+listobjects,317977063
+listobjects,324532475
+listobjects,246746304
+listobjects,339939390
+listobjects,238956624
+listobjects,338334961
+listobjects,236968116
+listobjects,338529503
+listobjects,235480073
+listobjects,334150022
+listobjects,234014534
+listobjects,336955875
+listobjects,245541026
+listobjects,321342105
+listobjects,241862995
+listobjects,342284075
+listobjects,223874385
+listobjects,330173393
+listobjects,230710360
+listobjects,338012760
+listobjects,243166839
+listobjects,326244217
+listobjects,217795705
+listobjects,343998726
+listobjects,234484396
+listobjects,326419671
+listobjects,228413611
+listobjects,345526317
+listobjects,242498473
+listobjects,322335836
+listobjects,325652509
+listobjects,353395591
+listobjects,241202612
+listobjects,316928525
+listobjects,415736209
+listobjects,336640460
+listobjects,235664816
+listobjects,318470444
+listobjects,233137178
+listobjects,338995281
+getobject,548766727
+getobject,544293094
+getobject,553996833
+getobject,524118176
+getobject,540256688
+getobject,521985460
+getobject,550010163
+getobject,522839295
+getobject,550206788
+getobject,556115384
+getobject,561480120
+getobject,529867231
+getobject,554654221
+getobject,522713545
+getobject,533201280
+getobject,559755253
+getobject,562719587
+getobject,530367590
+getobject,534634166
+getobject,528476593
+getobject,561330984
+getobject,548154277
+getobject,551287855
+getobject,543519973
+getobject,537798946
+getobject,543821201
+getobject,556706828
+getobject,562779335
+getobject,533088136
+getobject,543496053
+getobject,530090638
+getobject,557966646
+getobject,546652328
+getobject,556049330
+getobject,553087013
+getobject,552319581
+getobject,538039579
+getobject,524047395
+getobject,529321927
+getobject,561628538
+getobject,538975026
+getobject,537678052
+getobject,552261757
+getobject,553552417
+getobject,546538519
+getobject,539953289
+getobject,533392613
+getobject,543443065
+getobject,539024543
+getobject,537067523
+getobject,555793303
+getobject,610983302
+getobject,535123120
+getobject,528049669
+getobject,541554744
+getobject,564902500
+getobject,564639999
+getobject,534295582
+getobject,544951246
+getobject,540628827
+getobject,552900208
+getobject,541614941
+getobject,536825865
+getobject,535642848
+getobject,544302224
+getobject,549999591
+getobject,542030234
+getobject,545602568
+getobject,543278126
+getobject,524720136
+getobject,552230698
+getobject,539300237
+getobject,549856120
+getobject,540587629
+getobject,557880339
+getobject,555153802
+getobject,561113528
+getobject,556108255
+getobject,508374452
+getobject,560044395
+getobject,585922680
+getobject,548465899
+getobject,573308704
+getobject,554195559
+getobject,562441181
+getobject,550048971
+getobject,542032477
+getobject,587508303
+getobject,548308208
+getobject,529401594
+getobject,535337894
+getobject,566348968
+getobject,545058294
+getobject,560623443
+getobject,532580429
+getobject,555774911
+getobject,546448443
+getobject,526984769
+getobject,528538465
+getobject,551758307
+removeobject,1305475677
+removeobject,1328695106
+removeobject,1304086325
+removeobject,1314629714
+removeobject,1286472737
+removeobject,1323794460
+removeobject,1287026749
+removeobject,1308989986
+removeobject,1319191249
+removeobject,1302459188
+removeobject,1358027169
+removeobject,1298833331
+removeobject,1288044760
+removeobject,1309860548
+removeobject,1294431451
+removeobject,1323518263
+removeobject,1274537859
+removeobject,1306292281
+removeobject,1300283360
+removeobject,1236080169
+removeobject,1304088805
+removeobject,1319360876
+removeobject,1310348416
+removeobject,1306714462
+removeobject,1280353702
+removeobject,1313350545
+removeobject,1321192365
+removeobject,1304475647
+removeobject,1306586102
+removeobject,1294987618
+removeobject,1304176193
+removeobject,1296005363
+removeobject,1284994372
+removeobject,1317600899
+removeobject,1332743805
+removeobject,1352245087
+removeobject,1309279342
+removeobject,1307620458
+removeobject,1342551633
+removeobject,1321581956
+removeobject,1315162697
+removeobject,1336302765
+removeobject,1290969146
+removeobject,1320968235
+removeobject,1299056796
+removeobject,1288002161
+removeobject,1328125560
+removeobject,1316703843
+removeobject,1291268641
+removeobject,1327821693
+removeobject,1299867445
+removeobject,1296882414
+removeobject,1313288813
+removeobject,1298520791
+removeobject,1298185028
+removeobject,1274221533
+removeobject,1314194728
+removeobject,1336907835
+removeobject,1297413534
+removeobject,1283613694
+removeobject,1265657961
+removeobject,1329591094
+removeobject,1320682432
+removeobject,1317595366
+removeobject,1323562128
+removeobject,1298197589
+removeobject,1321950097
+removeobject,1281669970
+removeobject,1306586515
+removeobject,1296748831
+removeobject,1300070529
+removeobject,1282255502
+removeobject,1284141215
+removeobject,1297879969
+removeobject,1292837351
+removeobject,1317095287
+removeobject,1323647829
+removeobject,1283782591
+removeobject,1292866116
+removeobject,1283574839
+removeobject,1331580518
+removeobject,1355362975
+removeobject,1326405684
+removeobject,1297225883
+removeobject,1318610920
+removeobject,1298833112
+removeobject,1360833965
+removeobject,1311610613
+removeobject,1304567017
+removeobject,1325168434
+removeobject,1293403789
+removeobject,1293213442
+removeobject,1298548367
+removeobject,1303459493
+removeobject,1302979008
+removeobject,1336663252
+removeobject,1277657362
+removeobject,1308832733
+removeobject,1342049822
+removeobject,1307255424
--- a/artifacts/2022-09-23-s3lat/plot.R
+++ b/artifacts/2022-09-23-s3lat/plot.R
@ -0,0 +1,30 @@
+library(tidyverse)
+
+read_csv("garage-v0.7.csv") %>% add_column(daemon="garage 0.7.3") -> a
+read_csv("garage-v0.8.csv") %>% add_column(daemon="garage 0.8.0 beta") -> b
+read_csv("minio-20220917.csv") %>% add_column(daemon="minio RELEASE.2022-09-17") -> c
+bind_rows(a,b,c) %>% group_by(daemon,endpoint) %>% summarise(
+  time_mean = mean(nanoseconds) / 1000 / 1000,
+  time_max = max(nanoseconds) / 1000 / 1000,
+  time_min = min(nanoseconds) / 1000 / 1000
+) -> c
+
+st <- "100 measurements, 5 nodes, 50ms RTT + 10ms jitter between nodes\nno contention: latency is due to intra-cluster communications\ncolored bar = mean latency, error bar = min and max latency"
+
+
+ggplot(c, aes(x=endpoint,y=time_mean,fill=daemon,ymin=time_min,ymax=time_max)) +
+      geom_bar(stat="identity", position=position_dodge(),color="black") +
+      geom_errorbar(position=position_dodge(.9),width=.2) +
+      scale_y_continuous(expand=c(0,0))+
+      coord_flip() +
+      labs(
+        x="S3 Endpoint",
+        y="Request duration (ms)",
+        fill="Daemon",
+        caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+        title="S3 endpoint latency in a simulated geo-distributed cluster",
+        subtitle=st) +
+      theme_classic() +
+      theme(legend.position = c(.8, .3))
+
+ggsave("./plot.png", width=200, height=110, units="mm")
--- a/artifacts/2022-09-23-s3lat/plot.png
+++ b/artifacts/2022-09-23-s3lat/plot.png
--- a/artifacts/2022-09-23-s3ttfb/garage-v0.7.csv
+++ b/artifacts/2022-09-23-s3ttfb/garage-v0.7.csv
@ -0,0 +1,61 @@
+nanoseconds
+2043724295
+1819672397
+1856153241
+1814937942
+1818943177
+1814245475
+1828595614
+1817263829
+1857481054
+1816538404
+1816874169
+1856305403
+1857317361
+1814517488
+1817156893
+1815832311
+1857020274
+1822556545
+1816402439
+1815664395
+1818129711
+1855229810
+1817427508
+1817253968
+1855458740
+1818412087
+1816422102
+1814017661
+1821773596
+1814924392
+1819138108
+1814140079
+1814253338
+1814717178
+1823390172
+1856664915
+1817886566
+1815441938
+1815381667
+1864383216
+1815026715
+1816346397
+1814835227
+1855973303
+1814632968
+1823675273
+1817900846
+1816282804
+1817289580
+1817288129
+1860637032
+1855146585
+1816649086
+1822524427
+1814277922
+1901037764
+1818903030
+1815404774
+1815446844
+1815223628
--- a/artifacts/2022-09-23-s3ttfb/garage-v0.8.csv
+++ b/artifacts/2022-09-23-s3ttfb/garage-v0.8.csv
@ -0,0 +1,61 @@
+nanoseconds
+376764319
+268350943
+310303481
+266580460
+267172958
+46771899
+271623330
+270463024
+47401936
+45384570
+129505789
+46244307
+270594700
+267908944
+310163759
+271096844
+268930686
+309894956
+273312715
+269766963
+267947351
+270443270
+266292594
+215426567
+383550730
+84593093
+380970185
+419898767
+381574216
+422289246
+380535266
+268204508
+46449693
+270408730
+268572254
+308981862
+266303861
+46092770
+308073958
+383696451
+381725197
+379008890
+379902614
+382568588
+45370580
+83526768
+381230253
+267980416
+269187238
+84449786
+267524276
+46169934
+43509278
+308776528
+270626080
+271203190
+216167767
+239794865
+46230173
+267821837
--- a/artifacts/2022-09-23-s3ttfb/minio-20220917.csv
+++ b/artifacts/2022-09-23-s3ttfb/minio-20220917.csv
@ -0,0 +1,61 @@
+nanoseconds
+941737601
+970059899
+956848578
+981034132
+968456906
+986209313
+947371347
+961890275
+951011001
+947766957
+946381130
+1172968280
+944672409
+946216350
+960072933
+947966145
+947998544
+957730811
+960898554
+950611253
+953195848
+974654976
+982578891
+957419979
+947448876
+945953214
+936687543
+950116533
+979971727
+941427574
+931898843
+928494073
+927924296
+940995728
+927965457
+928098595
+933759012
+929902652
+953753466
+931448442
+934503899
+933297864
+934775226
+936615695
+950945991
+935054323
+1162136321
+940502403
+1158490674
+945337099
+948016443
+939863143
+938112935
+933271354
+946325434
+954355353
+948997521
+1124820589
+989829976
+938639125
--- a/artifacts/2022-09-23-s3ttfb/plot.R
+++ b/artifacts/2022-09-23-s3ttfb/plot.R
@ -0,0 +1,25 @@
+library(tidyverse)
+
+
+read_csv("garage-v0.7.csv") %>% add_column(daemon="garage 0.7.3") -> a
+read_csv("garage-v0.8.csv") %>% add_column(daemon="garage 0.8.0 beta") -> b
+read_csv("minio-20220917.csv") %>% add_column(daemon="minio RELEASE.2022-09-17") -> c
+bind_rows(a,b,c) %>% mutate(ms = nanoseconds / 1000 / 1000) -> d
+
+ggplot(d, aes(y = ms, group = daemon, fill=daemon)) +
+  #geom_violin(trim = TRUE, width=2) + 
+  geom_histogram(binwidth=50, color="black") +
+  scale_y_continuous(expand=c(0,0))+
+  scale_x_continuous(expand=c(0,0))+
+  coord_flip() + 
+  labs(
+    y="TTFB (ms)",
+    x="Count",
+    fill="Daemon",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="TTFB (Time To First Byte) on GetObject over a slow network (5 Mbps, 500 µs)",
+    subtitle="A 1MB file is uploaded and then fetched 60 times.\nExcept for Minio, the queried node does not store any data (gateway) to force net. communications.") +
+  theme_classic() +
+  theme(legend.position = c(.15, .8))
+
+ggsave("./plot.png", width=200, height=110, units="mm")
--- a/artifacts/2022-09-23-s3ttfb/plot.png
+++ b/artifacts/2022-09-23-s3ttfb/plot.png
--- a/artifacts/2022-09-23-warp/dc-garage-v0.7.zst
+++ b/artifacts/2022-09-23-warp/dc-garage-v0.7.zst
--- a/artifacts/2022-09-23-warp/plot.R
+++ b/artifacts/2022-09-23-warp/plot.R
@ -0,0 +1,21 @@
+library(tidyverse)
+
+
+read_csv("summary.csv")-> s
+
+hum_names <- as_labeller(c(`obj_per_sec` = "Objects/sec", `mbyte_per_sec` = "MByte/sec"))
+
+
+ggplot(s, aes(x=value,y=config)) +
+  geom_bar(stat="identity", fill="#00BFC4", color="black") +
+  facet_wrap(~measurement,dir="v", scales = "free", labeller = hum_names) +
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  labs(
+    y="Daemon",
+    x="Metric",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="\"minio/warp\" benchmark, \"cluster total\" result",
+    subtitle="Ran on a local machine (Ryzen 5 1400, 16GB RAM, SSD) with mknet\nDC topology (3 nodes, 1GB/s, 1ms lat)\nwarp in mixed mode, 5min bench, 5MB objects, initialized with 200 objects") +
+  theme_classic() 
+
+ggsave("./plot.png", width=200, height=120, units="mm")
--- a/artifacts/2022-09-23-warp/plot.png
+++ b/artifacts/2022-09-23-warp/plot.png
--- a/artifacts/2022-09-23-warp/summary.csv
+++ b/artifacts/2022-09-23-warp/summary.csv
@ -0,0 +1,11 @@
+config,measurement,value
+garage v0.7,mbyte_per_sec,8.60
+garage v0.7,obj_per_sec,3.02
+garage v0.8-beta1,mbyte_per_sec,12.41
+garage v0.8-beta1,obj_per_sec,4.36
+minio 20220917,mbyte_per_sec,20.22
+minio 20220917,obj_per_sec,7.11
+garage v0.8-beta2,mbyte_per_sec,14.56
+garage v0.8-beta2,obj_per_sec,5.19
+garage v0.8-no-fsync,mbyte_per_sec,17.31
+garage v0.8-no-fsync,obj_per_sec,6.11
--- a/artifacts/2022-09-24-s3billion/garage-regression.png
+++ b/artifacts/2022-09-24-s3billion/garage-regression.png
--- a/artifacts/2022-09-24-s3billion/garage-v0.8-beta2-lmdb.csv
+++ b/artifacts/2022-09-24-s3billion/garage-v0.8-beta2-lmdb.csv
@ -0,0 +1,129 @@
+total_objects,batch_dur_nanoseconds
+0,2018191689
+8192,1819596313
+16384,2102802010
+24576,2473935158
+32768,4510296173
+40960,6613958698
+49152,7977651423
+57344,13671435594
+65536,8060203586
+73728,3795495684
+81920,5794535775
+90112,5543820710
+98304,4519985809
+106496,5110068206
+114688,6708672963
+122880,7857479433
+131072,20642163540
+139264,5635269044
+147456,9351347336
+155648,5666222073
+163840,6501995906
+172032,5293598878
+180224,9350079270
+188416,13303962641
+196608,17428285824
+204800,11733919058
+212992,8372494836
+221184,26574754854
+229376,6314078661
+237568,9683594309
+245760,21850576414
+253952,8991130243
+262144,6567887673
+270336,9454781018
+278528,25552173846
+286720,9580711949
+294912,10257632597
+303104,7478191638
+311296,10501501215
+319488,3816776322
+327680,26795462810
+335872,9396360089
+344064,11208895014
+352256,21126662513
+360448,14198029001
+368640,5093580346
+376832,21004203835
+385024,14433396301
+393216,8933953570
+401408,5461923067
+409600,30844223516
+417792,8315342751
+425984,6166609079
+434176,29359253375
+442368,7642009005
+450560,7036982475
+458752,27168386971
+466944,10015226557
+475136,9245612551
+483328,24224252348
+491520,13827842768
+499712,9702435577
+507904,26083366912
+516096,7829114232
+524288,10792095771
+532480,28058503484
+540672,9984663485
+548864,10714515140
+557056,28472749292
+565248,10832278691
+573440,7594452219
+581632,23858923965
+589824,38909249537
+598016,15112735865
+606208,9644628857
+614400,28367134578
+622592,35180699079
+630784,14207640136
+638976,13236547125
+647168,37982462879
+655360,8448278050
+663552,11632520113
+671744,22661429490
+679936,29261192164
+688128,11982484321
+696320,30381827880
+704512,23530964587
+712704,8219614193
+720896,34002514929
+729088,21279203762
+737280,6880671212
+745472,10884265763
+753664,41999879903
+761856,13465319479
+770048,10792229369
+778240,29004881357
+786432,24372150617
+794624,13327280319
+802816,18478244110
+811008,48319469581
+819200,14174987827
+827392,14368184404
+835584,56647745264
+843776,9880099642
+851968,12808097118
+860160,28380242726
+868352,44970657202
+876544,18733551704
+884736,9340561806
+892928,25246663514
+901120,46462124425
+909312,12856639771
+917504,9028224897
+925696,38580954489
+933888,48184562190
+942080,11204051215
+950272,19754687230
+958464,36364975788
+966656,40759996520
+974848,10562565477
+983040,13905331718
+991232,54140254529
+999424,46643736817
+1007616,8362351299
+1015808,13514305332
+1024000,39389278472
+1032192,40328458712
+1040384,15958365846
--- a/artifacts/2022-09-24-s3billion/garage.png
+++ b/artifacts/2022-09-24-s3billion/garage.png
--- a/artifacts/2022-09-24-s3billion/minio.csv
+++ b/artifacts/2022-09-24-s3billion/minio.csv
@ -0,0 +1,129 @@
+total_objects,batch_dur_nanoseconds
+0,35604165299
+8192,30027091125
+16384,36769805630
+24576,44154399690
+32768,36841962377
+40960,52989512753
+49152,52536807234
+57344,53537348443
+65536,53276055435
+73728,55539526464
+81920,54013225676
+90112,68561851906
+98304,69298493247
+106496,114291303648
+114688,78507905949
+122880,126223986043
+131072,121999273534
+139264,153563457891
+147456,151032625349
+155648,149855834712
+163840,153190455466
+172032,150844647704
+180224,156256328509
+188416,151849908862
+196608,154337246115
+204800,154008292236
+212992,160001140992
+221184,154134240373
+229376,147943023290
+237568,150989793433
+245760,161064597934
+253952,150058583272
+262144,158784561065
+270336,150652453675
+278528,108917197085
+286720,158143657766
+294912,151991188276
+303104,150904775153
+311296,152538005225
+319488,158677889479
+327680,151948293662
+335872,131622851465
+344064,120774952190
+352256,160866374832
+360448,157913387369
+368640,157965094573
+376832,155015634674
+385024,152010190282
+393216,151353578675
+401408,152816629041
+409600,100685222932
+417792,151049307826
+425984,152092080473
+434176,156352511800
+442368,156617304416
+450560,156604667758
+458752,163954135953
+466944,102070359622
+475136,155481160507
+483328,157268905807
+491520,162446380203
+499712,159234723683
+507904,141918580163
+516096,133297515392
+524288,92684666697
+532480,153688292459
+540672,154151441890
+548864,157288990588
+557056,159081065544
+565248,153244088615
+573440,161424543833
+581632,174752406147
+589824,155541074973
+598016,159471780365
+606208,158247043033
+614400,157224045247
+622592,160337074095
+630784,157993307652
+638976,156821062432
+647168,158852305038
+655360,161012982349
+663552,156762111328
+671744,159782719621
+679936,156694048459
+688128,175969992462
+696320,159626684812
+704512,156680995342
+712704,155624836949
+720896,161583816024
+729088,156832016477
+737280,160834148361
+745472,157761321600
+753664,157553638650
+761856,176048273498
+770048,156697917513
+778240,162791069016
+786432,156496526158
+794624,161388816617
+802816,159569344379
+811008,159393281615
+819200,158389581978
+827392,164002225318
+835584,162722200475
+843776,160146392865
+851968,164551137708
+860160,177034374419
+868352,161672813995
+876544,169063897017
+884736,160829838695
+892928,157613804708
+901120,162687585988
+909312,166612325773
+917504,160649709927
+925696,163214320096
+933888,178447566516
+942080,155968188136
+950272,161252816115
+958464,159781104139
+966656,163854372812
+974848,157176626359
+983040,160325050560
+991232,165218492441
+999424,160172384289
+1007616,159316385255
+1015808,178044565059
+1024000,164817520071
+1032192,159428724027
+1040384,159973237521
--- a/artifacts/2022-09-24-s3billion/minio.png
+++ b/artifacts/2022-09-24-s3billion/minio.png
--- a/artifacts/2022-09-24-s3billion/plot.R
+++ b/artifacts/2022-09-24-s3billion/plot.R
@ -0,0 +1,65 @@
+library(tidyverse)
+library(ggpmisc)
+
+
+read_csv("garage-v0.8-beta2-lmdb.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000) %>% filter(total_objects != 0) -> s
+
+reg1 <- lm(s$batch_dur_sec~s$total_objects)
+reg2 <- lm(s$batch_dur_sec ~ log(s$total_objects))
+
+f1 <- y~log(x)
+f2 <- y~x
+
+ggplot(s, aes(x=total_objects, y=batch_dur_sec)) +
+  geom_point() +
+  #geom_smooth(method="lm",formula=f1,  se = FALSE, color="red") + 
+  #geom_smooth(method="lm",formula=f2,  se = FALSE, color="blue") +
+  #stat_poly_eq(formula = f1, label.y = 0.9, color = "red", aes(label=paste(..eq.label..,..rr.label..,..adj.rr.label..,..AIC.label..,..BIC.label.., sep = "~~~"))) +
+  #stat_poly_eq(formula = f2, label.y = 0.8, color="blue",aes(label=paste(..eq.label..,..rr.label..,..adj.rr.label..,..AIC.label..,..BIC.label.., sep = "~~~"))) +
+  #geom_smooth(method = "gam", se = FALSE) + 
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  coord_cartesian(ylim=c(0,60)) +
+  labs(
+    y="Time (in sec) spent sending a batch (8192 objects)",
+    x="Total number of objects stored in the cluster",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Storing 1M+ files in Garage! Impact of existing data on cluster interactiveness",
+    subtitle="Daemon: Garage v0.8 beta 2 with LMDB as db_engine\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
+  theme_classic()
+ggsave("./garage.png", width=200, height=120, units="mm")
+#ggsave("./garage-regression.png", width=200, height=120, units="mm")
+
+
+
+read_csv("minio.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000 ) -> s2
+ggplot(s2, aes(x=total_objects, y=batch_dur_sec)) +
+  geom_point() +
+  #geom_smooth(method = "gam", se = FALSE) + 
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  labs(
+    y="Time (in sec) spent sending a batch (8192 objects)",
+    x="Total number of objects stored in the cluster",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Storing 1M+ files in Minio! Impact of existing data on cluster interactiveness",
+    subtitle="Daemon: Minio RELEASE 20220917\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
+  theme_classic()
+ggsave("./minio.png", width=200, height=120, units="mm")
+
+bind_rows(s %>% add_column(daemon="garage 0.8 beta2 lmdb"), s2 %>% add_column(daemon="minio RELEASE 20220917")) -> sc
+ggplot(sc, aes(x=total_objects, y=batch_dur_sec, group=daemon, color=daemon)) +
+  geom_point() +
+  #geom_smooth(method = "gam", se = FALSE) + 
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  labs(
+    y="Time (in sec) spent sending a batch (8192 objects)",
+    x="Total number of objects stored in the cluster",
+    fill="Daemon",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Storing 1M+ files in Garage and Minio! Impact of existing data on cluster interactiveness",
+    subtitle="Daemon: Garage v0.8 beta 2 with LMDB as db_engine, Minio RELEASE 20220917\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
+  theme_classic() + 
+  theme(legend.position = c(.85, .55))
+ggsave("./plot.png", width=200, height=120, units="mm")
--- a/artifacts/2022-09-24-s3billion/plot.png
+++ b/artifacts/2022-09-24-s3billion/plot.png
--- a/artifacts/2022-09-24-warp-db-engine/lmdb.log
+++ b/artifacts/2022-09-24-warp-db-engine/lmdb.log
@ -0,0 +1,15 @@
+Mixed operations.
+Operation: DELETE, 10%, Concurrency: 20, Ran 4m59s.
+ * Throughput: 266.79 obj/s
+
+Operation: GET, 45%, Concurrency: 20, Ran 4m59s.
+ * Throughput: 0.29 MiB/s, 1200.51 obj/s
+
+Operation: PUT, 15%, Concurrency: 20, Ran 4m59s.
+ * Throughput: 0.10 MiB/s, 400.20 obj/s
+
+Operation: STAT, 30%, Concurrency: 20, Ran 4m59s.
+ * Throughput: 800.32 obj/s
+
+Cluster Total: 0.39 MiB/s, 2667.83 obj/s over 5m0s.
+Aggregated data saved to /tmp/mknet-store/warp.csv
--- a/artifacts/2022-09-24-warp-db-engine/plot.R
+++ b/artifacts/2022-09-24-warp-db-engine/plot.R
@ -0,0 +1,17 @@
+library(tidyverse)
+
+read_csv("summary.csv")-> s
+ggplot(s, aes(x=db_engine,y=rate)) +
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
+  geom_bar(stat='identity',fill="#00BFC4", color="black") +
+  coord_flip() +
+  facet_wrap(~operation, scales = "free", dir="v") +
+  labs(
+    x="Metadata Engine",
+    y="Object/sec",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Comparison of Garage's metadata engines with \"minio/warp\"",
+    subtitle="Daemon: Garage v0.8 no-fsync to avoid being impacted by block manager\nBenchmark: warp, mixed mode, 5min bench, 256B objects, initialized with 200 objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).",
+  ) +
+  theme_classic()
+ggsave("./plot.png", width=200, height=120, units="mm")
--- a/artifacts/2022-09-24-warp-db-engine/plot.png
+++ b/artifacts/2022-09-24-warp-db-engine/plot.png
--- a/artifacts/2022-09-24-warp-db-engine/sled-nosync.log
+++ b/artifacts/2022-09-24-warp-db-engine/sled-nosync.log
@ -0,0 +1,13 @@
+Operation: DELETE, 10%, Concurrency: 20, Ran 3m42s.
+ * Throughput: 161.98 obj/s
+
+Operation: GET, 45%, Concurrency: 20, Ran 3m42s.
+ * Throughput: 0.18 MiB/s, 729.07 obj/s
+
+Operation: PUT, 15%, Concurrency: 20, Ran 3m42s.
+ * Throughput: 0.06 MiB/s, 242.99 obj/s
+
+Operation: STAT, 30%, Concurrency: 20, Ran 3m42s.
+ * Throughput: 486.09 obj/s
+
+Cluster Total: 0.24 MiB/s, 1620.14 obj/s over 3m42s.
--- a/artifacts/2022-09-24-warp-db-engine/sled.log
+++ b/artifacts/2022-09-24-warp-db-engine/sled.log
@ -0,0 +1,15 @@
+Mixed operations.
+Operation: DELETE, 10%, Concurrency: 20, Ran 3m49s.
+ * Throughput: 129.83 obj/s
+
+Operation: GET, 45%, Concurrency: 20, Ran 3m49s.
+ * Throughput: 0.14 MiB/s, 584.32 obj/s
+
+Operation: PUT, 15%, Concurrency: 20, Ran 3m49s.
+ * Throughput: 0.05 MiB/s, 194.81 obj/s
+
+Operation: STAT, 30%, Concurrency: 20, Ran 3m49s.
+ * Throughput: 389.52 obj/s
+
+Cluster Total: 0.19 MiB/s, 1298.49 obj/s over 3m49s.
+Aggregated data saved to /tmp/mknet-store/warp.csv
--- a/artifacts/2022-09-24-warp-db-engine/sqlite.log
+++ b/artifacts/2022-09-24-warp-db-engine/sqlite.log
@ -0,0 +1,15 @@
+Mixed operations.
+Operation: DELETE, 10%, Concurrency: 20, Ran 4m55s.
+ * Throughput: 1.29 obj/s
+
+Operation: GET, 45%, Concurrency: 20, Ran 4m55s.
+ * Throughput: 0.00 MiB/s, 5.75 obj/s
+
+Operation: PUT, 15%, Concurrency: 20, Ran 4m55s.
+ * Throughput: 0.00 MiB/s, 1.93 obj/s
+
+Operation: STAT, 30%, Concurrency: 20, Ran 4m56s.
+ * Throughput: 3.86 obj/s
+
+Cluster Total: 0.00 MiB/s, 12.83 obj/s over 4m55s.
+Aggregated data saved to /tmp/mknet-store/warp.csv
--- a/artifacts/2022-09-24-warp-db-engine/summary.csv
+++ b/artifacts/2022-09-24-warp-db-engine/summary.csv
@ -0,0 +1,17 @@
+db_engine,operation,rate
+lmdb,delete,266.79
+lmdb,get,1200.51
+lmdb,put,400
+lmdb,stat,800
+sled,delete,129.83
+sled,get,584.32
+sled,put,194.81
+sled,stat,389.52
+sqlite,delete,1.29
+sqlite,get,5.75
+sqlite,put,1.93
+sqlite,stat,3.86
+sled-nosync,delete,161.98
+sled-nosync,get,729.07
+sled-nosync,put,242.99
+sled-nosync,stat,486.09
--- a/artifacts/2022-09-29-s3billion/garage-0.8-gros.png
+++ b/artifacts/2022-09-29-s3billion/garage-0.8-gros.png
--- a/artifacts/2022-09-29-s3billion/plot.R
+++ b/artifacts/2022-09-29-s3billion/plot.R
@ -0,0 +1,27 @@
+library(tidyverse)
+library(ggpmisc)
+library(scales)
+
+
+read_csv("s3billion-dc-gros-32threads-256batchsize-128000batchcount-v0.8-fix-time-30M.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000) %>% filter(total_objects != 0) -> s
+
+reg <- lm(s$batch_dur_sec~s$total_objects)
+
+f <- y~x
+
+ggplot(s, aes(x=total_objects, y=batch_dur_sec)) +
+  geom_point(shape = "bullet", alpha=0.4) +
+  geom_smooth(method="lm", formula=f, se = FALSE, color="blue") + 
+  scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10), labels = unit_format(unit = "M", scale = 1e-6)) +
+  scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10)) +
+  coord_cartesian(ylim=c(0,3)) +
+  labs(
+    y="Time (in sec) spent sending a batch (8192 objects)",
+    x="Total number of objects stored in the cluster",
+    caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
+    title="Storing 30M small objects in Garage!",
+    subtitle="Daemon: Garage v0.8.0-rc1 (+ #398 pull request) with LMDB as db_engine\nBenchmark: 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet on Grid'5000 (gros) https://www.grid5000.fr/w/Nancy:Hardware#gros (SATA SSD).\nDC topo (3 logical nodes, 1Gb/s, 1ms latency).") +
+  theme_classic()
+ggsave("./garage-0.8-gros.png", width=200, height=120, units="mm")
+#ggsave("./garage-regression.png", width=200, height=120, units="mm")
+
--- a/artifacts/2022-09-29-s3billion/s3billion-dc-gros-32threads-256batchsize-128000batchcount-v0.8-fix-time-30M.csv
+++ b/artifacts/2022-09-29-s3billion/s3billion-dc-gros-32threads-256batchsize-128000batchcount-v0.8-fix-time-30M.csv
--- a/benchmarks/s3billion/.gitignore
+++ b/benchmarks/s3billion/.gitignore
@ -0,0 +1 @@
+s3billion
--- a/benchmarks/s3billion/README.md
+++ b/benchmarks/s3billion/README.md
@ -0,0 +1,22 @@
+# s3lat
+
+## installation
+
+```bash
+go get git.deuxfleurs.fr/quentin/s3lat@latest
+```
+
+## usage
+
+```bash
+export ENDPOINT=[fc00:9a7a:9e::1]:9000
+export AWS_ACCESS_KEY_ID=minioadmin
+export AWS_SECRET_ACCESS_KEY=minioadmin
+
+s3lat
+```
+
+## see also
+
+ - https://git.deuxfleurs.fr/quentin/benchmarks
+ - https://git.deuxfleurs.fr/trinity-1686a/mknet
--- a/benchmarks/s3billion/go.mod
+++ b/benchmarks/s3billion/go.mod
@ -0,0 +1,8 @@
+module git.deuxfleurs.fr/Deuxfleurs/mknet/benchmarks/s3billion
+
+go 1.16
+
+require (
+	github.com/google/uuid v1.1.1
+	github.com/minio/minio-go/v7 v7.0.16
+)
--- a/benchmarks/s3billion/go.sum
+++ b/benchmarks/s3billion/go.sum
@ -0,0 +1,78 @@
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
+github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
+github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
+github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
+github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
+github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
+github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
+github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
+github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
+github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
+github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
+github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
+github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
+github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
+github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
+github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae h1:Ih9Yo4hSPImZOpfGuA4bR/ORKTAbhZo2AbWNRCnevdo=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
+gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
--- a/benchmarks/s3billion/main.go
+++ b/benchmarks/s3billion/main.go
@ -0,0 +1,176 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"fmt"
+	"io"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+	"strconv"
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+)
+
+type PRNG struct {
+	rem int64
+}
+
+func (r *PRNG) Read(p []byte) (n int, err error) {
+	//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
+	if int64(len(p)) > r.rem {
+		p = p[:r.rem]
+	}
+
+	if int64(len(p)) > r.rem {
+		log.Fatal("LOGIC ERROR")
+	}
+
+	n, err = rand.Read(p)
+	if err != nil {
+		return
+	}
+	r.rem -= int64(n)
+	if r.rem <= 0 {
+		err = io.EOF
+		//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
+	}
+	return
+}
+
+func putObj(mc *minio.Client, buck string, size int64) error {
+	prng := new(PRNG)
+	prng.rem = size
+
+	key := uuid.New().String()
+
+	_, err := mc.PutObject(
+		context.Background(),
+		buck,
+		key,
+		prng,
+		size,
+		minio.PutObjectOptions{ContentType: "application/octet-stream"},
+	)
+
+	return err
+}
+
+func main() {
+	fmt.Printf("total_objects,batch_dur_nanoseconds\n")
+
+	minio.MaxRetry = 1
+
+	_, isSSL := os.LookupEnv("SSL")
+	opts := minio.Options{
+		Creds:  credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
+		Secure: isSSL,
+	}
+
+	if region, ok := os.LookupEnv("AWS_REGION"); ok {
+		opts.Region = region
+	}
+
+	if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
+		opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
+	}
+
+	mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
+	if err != nil {
+		log.Fatal("failed connect", err)
+		return
+	}
+
+	thread := 32
+	if env_thread, ok := os.LookupEnv("THREAD"); ok {
+		tmp, err := strconv.Atoi(env_thread)
+		if err != nil {
+			log.Fatalf("invalid value for THREAD: %v\n", env_thread)
+		}
+		thread = tmp
+	}
+
+	batch_size := 256
+	if env_batch_size, ok := os.LookupEnv("BATCH_SIZE"); ok {
+		tmp, err := strconv.Atoi(env_batch_size)
+		if err != nil {
+			log.Fatalf("invalid value for BATCH_SIZE: %v\n", env_batch_size)
+		}
+		batch_size = tmp
+	}
+
+	batch_count := 128
+	if env_batch_count, ok := os.LookupEnv("BATCH_COUNT"); ok {
+		tmp, err := strconv.Atoi(env_batch_count)
+		if err != nil {
+			log.Fatalf("invalid value for BATCH_COUNT: %v\n", env_batch_count)
+		}
+		batch_count = tmp
+	}
+
+	obj_size := 16
+	if env_obj_size, ok := os.LookupEnv("OBJ_SIZE"); ok {
+		tmp, err := strconv.Atoi(env_obj_size)
+		if err != nil {
+			log.Fatalf("invalid value for OBJ_SIZE: %v\n", env_obj_size)
+		}
+		obj_size = tmp
+	}
+
+	total_obj := thread * batch_size * batch_count
+	total_size := total_obj * obj_size
+	log.Printf("if bench succeed, %v objects (%v bytes) will be created\n", total_obj, total_size)
+
+	// Create Bucket
+	buck := uuid.New().String()
+	err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
+	if err != nil {
+		log.Fatal(err)
+		return
+	}
+	log.Printf("created bucket %s\n", buck)
+
+	// Start sending...
+	for bc := 0; bc < batch_count; bc++ {
+		log.Printf("batch %d/%d - start\n", bc+1, batch_count)
+
+		start := time.Now()
+		syn := make(chan error)
+
+		for tc := 0; tc < thread; tc++ {
+			go func() {
+				for bs := 0; bs < batch_size; bs++ {
+					err := putObj(mc, buck, int64(obj_size))
+					if err != nil {
+						syn <- err
+						return
+					}
+				}
+				syn <- nil
+
+			}()
+		}
+		log.Printf("batch %d/%d - all threads started\n", bc+1, batch_count)
+
+		errCount := 0
+		for tc := 0; tc < thread; tc++ {
+			cerr := <-syn
+			if cerr != nil {
+				errCount += 1
+				log.Printf("thread %d/%d failed with %s\n", tc, thread, cerr)
+			}
+		}
+		if errCount > 0 {
+			log.Fatal("Too many errors, exiting...")
+			return
+		}
+		elapsed := time.Since(start)
+		fmt.Printf("%d,%v\n", bc * thread * batch_size, elapsed.Nanoseconds())
+		log.Printf("batch %d/%d - all threads returned\n", bc+1, batch_count)
+	}
+}
--- a/benchmarks/s3concurrent/.gitignore
+++ b/benchmarks/s3concurrent/.gitignore
@ -0,0 +1 @@
+s3concurrent
--- a/benchmarks/s3concurrent/go.mod
+++ b/benchmarks/s3concurrent/go.mod
@ -0,0 +1,24 @@
+module git.deuxfleurs.fr/quentin/s3concurrent
+
+go 1.18
+
+require github.com/minio/minio-go/v7 v7.0.34
+
+require (
+	github.com/dustin/go-humanize v1.0.0 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/klauspost/compress v1.15.9 // indirect
+	github.com/klauspost/cpuid/v2 v2.1.0 // indirect
+	github.com/minio/md5-simd v1.1.2 // indirect
+	github.com/minio/sha256-simd v1.0.0 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/rs/xid v1.4.0 // indirect
+	github.com/sirupsen/logrus v1.9.0 // indirect
+	golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa // indirect
+	golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
+	golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
+	golang.org/x/text v0.3.7 // indirect
+	gopkg.in/ini.v1 v1.66.6 // indirect
+)
--- a/benchmarks/s3concurrent/go.sum
+++ b/benchmarks/s3concurrent/go.sum
@ -0,0 +1,52 @@
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
+github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
+github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
+github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY=
+github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
+github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0=
+github.com/klauspost/cpuid/v2 v2.1.0/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY=
+github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
+github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
+github.com/minio/minio-go/v7 v7.0.34 h1:JMfS5fudx1mN6V2MMNyCJ7UMrjEzZzIvMgfkWc1Vnjk=
+github.com/minio/minio-go/v7 v7.0.34/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw=
+github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g=
+github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
+github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY=
+github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
+github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0=
+github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c=
+golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI=
+gopkg.in/ini.v1 v1.66.6/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
--- a/benchmarks/s3concurrent/main.go
+++ b/benchmarks/s3concurrent/main.go
@ -0,0 +1,132 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"fmt"
+	"time"
+	"io"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+
+	"github.com/google/uuid"
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+)
+
+func buildMc() (*minio.Client, error) {
+	_, isSSL := os.LookupEnv("SSL")
+	opts := minio.Options{
+		Creds:  credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
+		Secure: isSSL,
+	}
+
+	if region, ok := os.LookupEnv("REGION"); ok {
+		opts.Region = region
+	}
+
+	if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
+		opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
+	}
+
+	mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
+	return mc, err
+}
+
+type PRNG struct {
+	rem int64
+}
+
+func (r *PRNG) Read(p []byte) (n int, err error) {
+	//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
+	if int64(len(p)) > r.rem {
+		p = p[:r.rem]
+	}
+
+	if int64(len(p)) > r.rem {
+		log.Fatal("LOGIC ERROR")
+	}
+
+	n, err = rand.Read(p)
+	if err != nil {
+		return
+	}
+	r.rem -= int64(n)
+	if r.rem <= 0 {
+		err = io.EOF
+		//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
+	}
+	return
+}
+
+func putObj(buck string, size int64) error {
+	mc, err := buildMc()
+	if err != nil {
+		return err
+	}
+
+	prng := new(PRNG)
+	prng.rem = size
+
+	key := uuid.New().String()
+
+	_, err = mc.PutObject(
+		context.Background(),
+		buck,
+		key,
+		prng,
+		size,
+		minio.PutObjectOptions{ContentType: "application/octet-stream"},
+	)
+
+	return err
+}
+
+func main() {
+	minio.MaxRetry = 1
+	mc, err := buildMc()
+	if err != nil {
+		log.Fatal("failed connect", err)
+		return
+	}
+
+	// Create Bucket
+	buck := uuid.New().String()
+	err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
+	if err != nil {
+		log.Fatal(err)
+		return
+	}
+	log.Printf("created bucket %s\n", buck)
+
+	fmt.Println("sent,success,elapsed,elapsed_per_req")
+
+	// Send to bucket
+	for i := 1; i <= 16; i++ {
+		log.Printf("start concurrent loop with %d coroutines\n", i)
+		syn := make(chan error)
+		
+		start := time.Now()
+		for j := 1; j <= i; j++ {
+			go func() {
+				syn <- putObj(buck, 1024*1024)
+			}()
+		}
+
+		errCount := 0
+		for j := 1; j <= i; j++ {
+			cerr := <-syn
+			if cerr != nil {
+				errCount += 1
+				log.Printf("%d/%d failed with %s\n", j, i, cerr)
+			}
+		}
+		elapsed := time.Since(start)
+		fmt.Printf("%d,%d,%v,%v\n", i, i - errCount, elapsed.Nanoseconds(), elapsed.Nanoseconds() / int64(i))
+		log.Printf("done, %d coroutines returned\n", i)
+	}
+
+	log.Println("bench done")
+}
--- a/benchmarks/s3lat/.gitignore
+++ b/benchmarks/s3lat/.gitignore
@ -0,0 +1 @@
+s3lat
--- a/benchmarks/s3lat/README.md
+++ b/benchmarks/s3lat/README.md
@ -0,0 +1,22 @@
+# s3lat
+
+## installation
+
+```bash
+go get git.deuxfleurs.fr/quentin/s3lat@latest
+```
+
+## usage
+
+```bash
+export ENDPOINT=[fc00:9a7a:9e::1]:9000
+export AWS_ACCESS_KEY_ID=minioadmin
+export AWS_SECRET_ACCESS_KEY=minioadmin
+
+s3lat
+```
+
+## see also
+
+ - https://git.deuxfleurs.fr/quentin/benchmarks
+ - https://git.deuxfleurs.fr/trinity-1686a/mknet
--- a/benchmarks/s3lat/go.mod
+++ b/benchmarks/s3lat/go.mod
@ -0,0 +1,8 @@
+module git.deuxfleurs.fr/quentin/s3lat
+
+go 1.16
+
+require (
+	github.com/google/uuid v1.1.1
+	github.com/minio/minio-go/v7 v7.0.16
+)
--- a/benchmarks/s3lat/go.sum
+++ b/benchmarks/s3lat/go.sum
@ -0,0 +1,78 @@
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
+github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
+github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
+github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
+github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
+github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
+github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
+github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
+github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
+github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
+github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
+github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
+github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
+github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
+github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
+github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae h1:Ih9Yo4hSPImZOpfGuA4bR/ORKTAbhZo2AbWNRCnevdo=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
+gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
--- a/benchmarks/s3lat/main.go
+++ b/benchmarks/s3lat/main.go
@ -0,0 +1,135 @@
+package main
+
+import (
+	"context"
+	"log"
+	"os"
+	"fmt"
+	"time"
+	"io"
+	"io/ioutil"
+	"strings"
+	"net/http"
+	"crypto/tls"
+	"strconv"
+
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+    "github.com/google/uuid"
+)
+
+func main() {
+	fmt.Printf("endpoint,nanoseconds\n")
+
+    // Initial setup
+	_, isSSL := os.LookupEnv("SSL");
+	opts := minio.Options {
+		Creds:  credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
+		Secure: isSSL,
+	}
+
+	if region, ok := os.LookupEnv("REGION"); ok {
+		opts.Region = region
+	}
+
+	if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
+		opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
+	}
+
+	mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
+
+	if err != nil {
+		log.Fatal("failed connect", err)
+		return
+	}
+
+    // Create Bucket
+    buck := uuid.New().String()
+    err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{ })
+    if err != nil {
+        log.Fatal(err)
+        return
+    }
+
+    // List Buckets
+	for i := 0; i < 100; i++ {
+		start := time.Now()
+		_, err := mc.ListBuckets(context.Background())
+		elapsed := time.Since(start)
+		if err != nil {
+			log.Fatal("failed listbucket: ", err)
+		return
+		}
+		fmt.Printf("listbuckets,%v\n", elapsed.Nanoseconds())
+	}
+
+    // PutObject
+	for i := 0; i < 100; i++ {
+		istr := strconv.Itoa(i)
+		content := istr + " hello world " + istr
+		start := time.Now()
+		_, err := mc.PutObject(context.Background(), buck, "element"+istr, strings.NewReader(content), int64(len(content)), minio.PutObjectOptions{ContentType:"application/octet-stream"})
+		elapsed := time.Since(start)
+		if err != nil {
+			log.Fatal("failed putObject: ",err)
+			return
+		}
+		fmt.Printf("putobject,%v\n", elapsed.Nanoseconds())
+	}
+
+    // ListObject
+	for i := 0; i < 100; i++ {
+		ctx, cancel := context.WithCancel(context.Background())
+		defer cancel()
+
+		start := time.Now()
+		objectCh := mc.ListObjects(ctx, buck, minio.ListObjectsOptions{
+			Recursive: true,
+		})
+		for object := range objectCh {
+			if object.Err != nil {
+				log.Fatal(object.Err)
+				return
+			}
+		}
+		elapsed := time.Since(start)
+		fmt.Printf("listobjects,%v\n", elapsed.Nanoseconds())
+	}
+
+    // GetObject
+	for i := 0; i < 100; i++ {
+		istr := strconv.Itoa(i)
+		start := time.Now()
+		object, err := mc.GetObject(context.Background(), buck, "element"+istr, minio.GetObjectOptions{})
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+		if _, err = io.Copy(ioutil.Discard, object) ; err != nil {
+			log.Fatal("failed getobject: ", err)
+			return
+		}
+		elapsed := time.Since(start)
+		fmt.Printf("getobject,%v\n", elapsed.Nanoseconds())
+	}
+
+    // RemoveObject
+	for i := 0; i < 100; i++ {
+		istr := strconv.Itoa(i)
+		start := time.Now()
+		err = mc.RemoveObject(context.Background(), buck, "element"+istr, minio.RemoveObjectOptions{})
+		elapsed := time.Since(start)
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+		fmt.Printf("removeobject,%v\n", elapsed.Nanoseconds())
+	}
+
+    // RemoveBucket
+    err = mc.RemoveBucket(context.Background(), buck)
+    if err != nil {
+        log.Fatal(err)
+        return
+    }
+}
--- a/benchmarks/s3ttfb/.gitignore
+++ b/benchmarks/s3ttfb/.gitignore
@ -0,0 +1 @@
+s3ttfb
--- a/benchmarks/s3ttfb/go.mod
+++ b/benchmarks/s3ttfb/go.mod
@ -0,0 +1,12 @@
+module git.deuxfleurs.fr/Deuxfleurs/mknet/benchmarks/s3ttfb
+
+go 1.16
+
+require (
+	github.com/aws/aws-sdk-go-v2 v1.16.16
+	github.com/aws/aws-sdk-go-v2/config v1.17.7
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11
+	github.com/google/uuid v1.1.1
+	github.com/minio/minio-go/v7 v7.0.16
+	golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd // indirect
+)
--- a/benchmarks/s3ttfb/go.sum
+++ b/benchmarks/s3ttfb/go.sum
@ -0,0 +1,123 @@
+github.com/aws/aws-sdk-go-v2 v1.16.16 h1:M1fj4FE2lB4NzRb9Y0xdWsn2P0+2UHVxwKyOa4YJNjk=
+github.com/aws/aws-sdk-go-v2 v1.16.16/go.mod h1:SwiyXi/1zTUZ6KIAmLK5V5ll8SiURNUYOqTerZPaF9k=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.8 h1:tcFliCWne+zOuUfKNRn8JdFBuWPDuISDH08wD2ULkhk=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.8/go.mod h1:JTnlBSot91steJeti4ryyu/tLd4Sk84O5W22L7O2EQU=
+github.com/aws/aws-sdk-go-v2/config v1.17.7 h1:odVM52tFHhpqZBKNjVW5h+Zt1tKHbhdTQRb+0WHrNtw=
+github.com/aws/aws-sdk-go-v2/config v1.17.7/go.mod h1:dN2gja/QXxFF15hQreyrqYhLBaQo1d9ZKe/v/uplQoI=
+github.com/aws/aws-sdk-go-v2/credentials v1.12.20 h1:9+ZhlDY7N9dPnUmf7CDfW9In4sW5Ff3bh7oy4DzS1IE=
+github.com/aws/aws-sdk-go-v2/credentials v1.12.20/go.mod h1:UKY5HyIux08bbNA7Blv4PcXQ8cTkGh7ghHMFklaviR4=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.17 h1:r08j4sbZu/RVi+BNxkBJwPMUYY3P8mgSDuKkZ/ZN1lE=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.17/go.mod h1:yIkQcCDYNsZfXpd5UX2Cy+sWA1jPgIhGTw9cOBzfVnQ=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.23 h1:s4g/wnzMf+qepSNgTvaQQHNxyMLKSawNhKCPNy++2xY=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.23/go.mod h1:2DFxAQ9pfIRy0imBCJv+vZ2X6RKxves6fbnEuSry6b4=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.17 h1:/K482T5A3623WJgWT8w1yRAFK4RzGzEl7y39yhtn9eA=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.17/go.mod h1:pRwaTYCJemADaqCbUAxltMoHKata7hmB5PjEXeu0kfg=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.3.24 h1:wj5Rwc05hvUSvKuOF29IYb9QrCLjU+rHAy/x/o0DK2c=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.3.24/go.mod h1:jULHjqqjDlbyTa7pfM7WICATnOv+iOhjletM3N0Xbu8=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.14 h1:ZSIPAkAsCCjYrhqfw2+lNzWDzxzHXEckFkTePL5RSWQ=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.14/go.mod h1:AyGgqiKv9ECM6IZeNQtdT8NnMvUb3/2wokeq2Fgryto=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.9 h1:Lh1AShsuIJTwMkoxVCAYPJgNG5H+eN6SmoUn8nOZ5wE=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.9/go.mod h1:a9j48l6yL5XINLHLcOKInjdvknN+vWqPBxqeIDw7ktw=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.18 h1:BBYoNQt2kUZUUK4bIPsKrCcjVPUMNsgQpNAwhznK/zo=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.18/go.mod h1:NS55eQ4YixUJPTC+INxi2/jCqe1y2Uw3rnh9wEOVJxY=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.17 h1:Jrd/oMh0PKQc6+BowB+pLEwLIgaQF29eYbe7E1Av9Ug=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.17/go.mod h1:4nYOrY41Lrbk2170/BGkcJKBhws9Pfn8MG3aGqjjeFI=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.17 h1:HfVVR1vItaG6le+Bpw6P4midjBDMKnjMyZnw9MXYUcE=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.17/go.mod h1:YqMdV+gEKCQ59NrB7rzrJdALeBIsYiVi8Inj3+KcqHI=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11 h1:3/gm/JTX9bX8CpzTgIlrtYpB3EVBDxyg/GY/QdcIEZw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11/go.mod h1:fmgDANqTUCxciViKl9hb/zD5LFbvPINFRgWhDbR+vZo=
+github.com/aws/aws-sdk-go-v2/service/sso v1.11.23 h1:pwvCchFUEnlceKIgPUouBJwK81aCkQ8UDMORfeFtW10=
+github.com/aws/aws-sdk-go-v2/service/sso v1.11.23/go.mod h1:/w0eg9IhFGjGyyncHIQrXtU8wvNsTJOP0R6PPj0wf80=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.5 h1:GUnZ62TevLqIoDyHeiWj2P7EqaosgakBKVvWriIdLQY=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.5/go.mod h1:csZuQY65DAdFBt1oIjO5hhBR49kQqop4+lcuCjf2arA=
+github.com/aws/aws-sdk-go-v2/service/sts v1.16.19 h1:9pPi0PsFNAGILFfPCk8Y0iyEBGc6lu6OQ97U7hmdesg=
+github.com/aws/aws-sdk-go-v2/service/sts v1.16.19/go.mod h1:h4J3oPZQbxLhzGnk+j9dfYHi5qIOVJ5kczZd658/ydM=
+github.com/aws/smithy-go v1.13.3 h1:l7LYxGuzK6/K+NzJ2mC+VvLUbae0sL3bXU//04MkmnA=
+github.com/aws/smithy-go v1.13.3/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
+github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
+github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
+github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
+github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
+github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
+github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
+github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
+github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
+github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
+github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
+github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
+github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
+github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
+github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
+github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
+github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
+github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
+github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
+github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
+github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
+github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
+github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
+github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
+golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
+golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e h1:fLOSk5Q00efkSvAm+4xcoXD+RRmLmmulPn5I3Y9F2EM=
+golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
+gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
--- a/benchmarks/s3ttfb/main.go
+++ b/benchmarks/s3ttfb/main.go
@ -0,0 +1,214 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"log"
+	"math/rand"
+	"net/http"
+	"net/http/httptrace"
+	"os"
+	"strconv"
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+
+	"github.com/aws/aws-sdk-go-v2/aws"
+	"github.com/aws/aws-sdk-go-v2/config"
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+)
+
+type PRNG struct {
+	rem int64
+}
+
+func (r *PRNG) Read(p []byte) (n int, err error) {
+	//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
+	if int64(len(p)) > r.rem {
+		p = p[:r.rem]
+	}
+
+	if int64(len(p)) > r.rem {
+		log.Fatal("LOGIC ERROR")
+	}
+
+	n, err = rand.Read(p)
+	if err != nil {
+		return
+	}
+	r.rem -= int64(n)
+	if r.rem <= 0 {
+		err = io.EOF
+		//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
+	}
+	return
+}
+
+func main() {
+	var err error
+	fmt.Printf("nanoseconds\n")
+
+	// Initial setup
+	_, isSSL := os.LookupEnv("SSL")
+	opts := minio.Options{
+		Creds:  credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
+		Secure: isSSL,
+	}
+
+	if region, ok := os.LookupEnv("AWS_REGION"); ok {
+		opts.Region = region
+	}
+
+	if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
+		opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
+	}
+
+	size := 1 * 1024 * 1024 // 1 MB
+	if env_size, ok := os.LookupEnv("SIZE"); ok {
+		size, err = strconv.Atoi(env_size)
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+	}
+
+	repeat := 60
+	if env_repeat, ok := os.LookupEnv("REPEAT"); ok {
+		repeat, err = strconv.Atoi(env_repeat)
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+	}
+
+	mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
+	if err != nil {
+		log.Fatal("failed connect", err)
+		return
+	}
+
+	// AWS SDK stuff
+	customResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
+        	return aws.Endpoint{
+            		PartitionID:   "aws",
+			URL:           "http://"+os.Getenv("ENDPOINT"),
+            		SigningRegion: region,
+        	}, nil
+	})
+
+
+	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithEndpointResolverWithOptions(customResolver))
+	if err != nil {
+		log.Fatalf("failed to load configuration, %v", err)
+	}
+
+	// Create an Amazon S3 service client
+	client := s3.NewFromConfig(cfg, func(opts *s3.Options) {
+		opts.UsePathStyle = true
+	})
+
+
+	// Create Bucket
+	buck := uuid.New().String()
+	err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
+	if err != nil {
+		log.Fatal(err)
+		return
+	}
+
+	// Send an object
+	prng := new(PRNG)
+	prng.rem = int64(size)
+
+
+	key := uuid.New().String()
+	_, err = mc.PutObject(
+		context.Background(),
+		buck,
+		key,
+		prng,
+		int64(size),
+		minio.PutObjectOptions{ContentType: "application/octet-stream"},
+	)
+	if err != nil {
+		log.Fatal("unable to upload our test object")
+		return
+	}
+
+	for i := 0; i < repeat; i++ {
+		// Now, try to fetch it *repeat* times (assume no caching)
+		var elapsed time.Duration
+
+		var start time.Time
+    		trace := &httptrace.ClientTrace{
+			GotFirstResponseByte: func() { elapsed = time.Since(start) },
+		}
+
+		start = time.Now()
+		/*object, err := mc.GetObject(httptrace.WithClientTrace(context.Background(), trace), buck, key, minio.GetObjectOptions{})
+		if err != nil {
+			log.Fatal(err)
+			return
+		}*/
+
+		out, err := client.GetObject(httptrace.WithClientTrace(context.Background(), trace), &s3.GetObjectInput {
+			Bucket: aws.String(buck),
+			Key: aws.String(key),
+		})
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+		object := out.Body
+
+
+		if _, err = io.Copy(ioutil.Discard, object) ; err != nil {
+			log.Fatal("failed getobject: ", err)
+			return
+		}
+
+		fmt.Printf("%v\n", elapsed.Nanoseconds())
+		//time.Sleep(10 * time.Second)
+	}
+}
+
+/*
+func timeGet(url string) {
+    req, _ := http.NewRequest("GET", url, nil)
+
+    var start, connect, dns, tlsHandshake time.Time
+
+    trace := &httptrace.ClientTrace{
+        DNSStart: func(dsi httptrace.DNSStartInfo) { dns = time.Now() },
+        DNSDone: func(ddi httptrace.DNSDoneInfo) {
+            fmt.Printf("DNS Done: %v\n", time.Since(dns))
+        },
+
+        TLSHandshakeStart: func() { tlsHandshake = time.Now() },
+        TLSHandshakeDone: func(cs tls.ConnectionState, err error) {
+            fmt.Printf("TLS Handshake: %v\n", time.Since(tlsHandshake))
+        },
+
+        ConnectStart: func(network, addr string) { connect = time.Now() },
+        ConnectDone: func(network, addr string, err error) {
+            fmt.Printf("Connect time: %v\n", time.Since(connect))
+        },
+
+        GotFirstResponseByte: func() {
+            fmt.Printf("Time from start to first byte: %v\n", time.Since(start))
+        },
+    }
+
+    req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
+    start = time.Now()
+    if _, err := http.DefaultTransport.RoundTrip(req); err != nil {
+        log.Fatal(err)
+    }
+    fmt.Printf("Total time: %v\n", time.Since(start))
+}
+*/
--- a/benchmarks/warp
+++ b/benchmarks/warp
@ -0,0 +1 @@
+Subproject commit df8f2cd7b7a8523755b4d3c04271a69571a455ca
--- a/example/deploy_garage.sh
+++ b/example/deploy_garage.sh
@ -1,61 +0,0 @@
-#!/bin/bash
-
-set -euo pipefail
-IFS=$'\n\t'
-
-GARAGE_PATH=garage
-STORAGE_PATH=/tmp/garage-testnet
-export RPC_SECRET=3e9abff5f9e480afbadb46a77b7a26fe0e404258f0dc3fd5386b0ba8e0ad2fba
-
-if [ -z "$ZONE" ]; then
-	NODE_STORAGE_PATH=${STORAGE_PATH}/${HOST}
-else
-	NODE_STORAGE_PATH=${STORAGE_PATH}/${ZONE}/${HOST}
-fi
-BOOTSTRAP_FILE=${STORAGE_PATH}/bootstrap_peer
-export GARAGE_CONFIG_FILE=${NODE_STORAGE_PATH}/garage.toml 
-
-
-mkdir -p ${NODE_STORAGE_PATH}
-cd ${NODE_STORAGE_PATH}
-rm ${BOOTSTRAP_FILE} 2>/dev/null || true
-
-cat > ${GARAGE_CONFIG_FILE} << EOF
-metadata_dir = "${NODE_STORAGE_PATH}/meta"
-data_dir = "${NODE_STORAGE_PATH}/data"
-
-replication_mode = "3"
-
-rpc_bind_addr = "[::]:3901"
-rpc_public_addr = "[${IP}]:3901"
-rpc_secret = "${RPC_SECRET}"
-
-bootstrap_peers=[]
-
-[s3_api]
-s3_region = "garage"
-api_bind_addr = "[::]:3900"
-root_domain = ".s3.garage"
-
-[s3_web]
-bind_addr = "[::]:3902"
-root_domain = ".web.garage"
-index = "index.html"
-EOF
-
-RUST_LOG=garage=debug ${GARAGE_PATH} server 2>> ${NODE_STORAGE_PATH}/logs & disown
-sleep 2
-
-CONFIG_NODE_FPATH=$(find /tmp/garage-testnet/ -maxdepth 3 -name garage.toml|head -n 1)
-
-SELF_ID=$(${GARAGE_PATH} node id 2>/dev/null)
-SHORT_ID=$(echo ${SELF_ID} | cut -c-64)
-
-${GARAGE_PATH} -c ${CONFIG_NODE_FPATH} node connect ${SELF_ID}
-${GARAGE_PATH} -c ${CONFIG_NODE_FPATH} layout assign ${SHORT_ID} -z ${ZONE:-unzonned-${HOST}} -c 1 -t ${HOST}
-
-if [ ${CONFIG_NODE_FPATH} == ${GARAGE_CONFIG_FILE} ]; then
-	sleep 2
-	${GARAGE_PATH} layout show
-	${GARAGE_PATH} layout apply --version 1
-fi
--- a/example/deploy_minio.py
+++ b/example/deploy_minio.py
@ -1,62 +0,0 @@
-#!/usr/bin/env python3
-import json, os, sys, time, pathlib, socket, shutil
-
-STORAGE_PATH = os.path.join(os.getcwd(), '.minio-testnet')
-HOSTS_PATH = os.path.join(STORAGE_PATH, 'hosts.txt')
-UNIX_SOCK = os.path.join(STORAGE_PATH, 'deploy.sock')
-DATA_PATH = lambda nid: os.path.join(STORAGE_PATH, 'data'+str(nid))
-
-def main():
-    if int(os.environ['ID']) == 1: leader()
-    else: follower()
-
-def leader():
-    shutil.rmtree(STORAGE_PATH, ignore_errors=True)
-    os.makedirs(STORAGE_PATH)
-    print(STORAGE_PATH)
-
-    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
-    sock.bind(UNIX_SOCK)
-    sock.listen()
-
-    n_serv = int(os.environ['SERVER_COUNT'])
-    fl = [ co for co, addr in [ sock.accept() for i in range(n_serv - 1) ]]
-
-    identities = [ json.loads(co.makefile().readline()) for co in fl ] + [ { "ip": os.environ['IP'], "path": make_data() } ]
-    print(f"ident: {identities}")
-    msg = f"{json.dumps(identities)}\n".encode()
-    [ co.send(msg) for co in fl ]
-
-    run_minio(identities)
-
-def follower():
-    co = None
-    while True:
-        time.sleep(1)
-        try:
-            sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
-            sock.connect(UNIX_SOCK)
-            co = sock.makefile()
-            break
-        except Exception as err:
-            print('conn failed, wait,', err)
-    my_identity = json.dumps({ "ip": os.environ['IP'], "path": make_data() })
-    sock.send(f"{my_identity}\n".encode())
-    identities = json.loads(co.readline())
-
-    run_minio(identities)
-
-def make_data():
-    data_path = DATA_PATH(os.environ['ID'])
-    os.makedirs(data_path)
-    return data_path
-
-def run_minio(identities):
-    cmd = f"minio server --console-address ':9001' --address ':9000'"
-    for ident in identities:
-        cmd += f" http://[{ident['ip']}]:9000{ident['path']}"
-    cmd += f" > {os.path.join(STORAGE_PATH, 'minio'+os.environ['ID']+'.log')} 2>&1"
-    print("launch: ", cmd)
-    os.system(cmd)
-
-__name__ == '__main__' and main()
--- a/31
+++ b/31
@ -87,19 +87,23 @@ class Bandwidth:
        

 class LinkInfo:
-    def __init__(self, bandwidth, latency, jitter = None, offset = None, **kwargs):
+    def __init__(self, bandwidth, latency, jitter = None, offset = None, limit = None, txqueuelen = None, **kwargs):
        self.bandwidth = Bandwidth(bandwidth)
        self.latency = Latency(latency, offset)
        self.jitter = Latency(jitter or 0)
+        self.limit = limit
+        self.txqueuelen = txqueuelen

    def __eq__(self, o):
        return (isinstance(o, LinkInfo) and 
            o.bandwidth == self.bandwidth and
            o.latency == self.latency and
-            o.jitter == self.jitter)
+            o.jitter == self.jitter and
+            o.limit == self.limit and
+            o.txqueuelen == self.txqueuelen)

    def __str__(self):
-        return f'LinkInfo{{bw: {self.bandwidth}, latency: {self.latency}, jitter: {self.jitter}}}'
+        return f'LinkInfo{{bw: {self.bandwidth}, latency: {self.latency}, jitter: {self.jitter}, limit: {self.limit}, txqueuelen: {self.txqueuelen}}}'

 class Server:
    def __init__(self, name, link):
@ -313,6 +317,7 @@ def runall(cmd):
    zones = parse(config).zones

    number = 1
+    wait_list = [ ]
    for zone in zones.values():
        if zone.is_zone():
            for server in zone.servers.values():
@ -323,7 +328,7 @@ def runall(cmd):
                env["ID"] = str(number)
                env["SERVER_COUNT"] = str(len(config['servers']))
                name = f'testnet-{zone.name}-{server.name}'
-                net.ns.run(name, cmd, env)
+                wait_list.append(net.ns.run(name, cmd, env))
                number +=1
        else:
            env = os.environ.copy()
@ -333,9 +338,12 @@ def runall(cmd):
            env["ID"] = str(number)
            env["SERVER_COUNT"] = str(len(config['servers']))
            name = f'testnet-{zone.name}-{zone.name}'
-            net.ns.run(name, cmd, env)
+            wait_list.append(net.ns.run(name, cmd, env))
            first = False
            number +=1
+    for p in wait_list:
+        p.wait()
+    print("runall done")

 def destroy():
    for ns in net.ns.list():
@ -343,22 +351,31 @@ def destroy():
    net.ns.forget("unconfined")
    os.remove(".current_state.yml")

+def scenario(config, cmd):
+    create(config)
+    runall(cmd)
+    destroy()
+
 if __name__ == "__main__":
    if len(sys.argv) < 2:
        progname = os.path.basename(sys.argv[0]) if len(sys.argv) > 0 else "mknet"
        print(f"""Usage:
-    {progname} create [config_path] # create a new network. config_path defailt to config.yml
+    {progname} scenario <config_path> <cmd> [args...] # all in one simulation: create a network, run the command on all nodes, destroy the network
+
+    {progname} create [config_path] # create a new network. config_path defailt to topo/with-vdsl.yml
    {progname} run-all <cmd> [args...] # run a command as each host. set the IP, NAME and ZONE environment variables
    {progname} run <name> [cmd [args...]] # run command in host named <name>. Use zonename:name if multiple zones hosts server with same name. If cmd is empty, run a shell
    {progname} destroy # destroy the current environment""")
        exit()
    cmd = sys.argv[1]
    if cmd == "create":
-        create(sys.argv[2] if len(sys.argv) > 2 else "config.yml")
+        create(sys.argv[2] if len(sys.argv) > 2 else "topo/with-vdsl.yml")
    elif cmd == "run":
        run(sys.argv[2], sys.argv[3:])
    elif cmd == "run-all":
        runall(sys.argv[2:])
+    elif cmd == "scenario":
+        scenario(sys.argv[2], sys.argv[3:])
    elif cmd == "destroy":
        destroy()    
    else:
--- a/net.py
+++ b/net.py
@ -37,7 +37,7 @@ class ns:
        run_netns("exec", name, "ip", "link", "set", "dev", "lo", "up")

    def run(name, cmd, env=None):
-        subprocess.Popen(_netns + ["exec", name] + cmd, env=env)
+        return subprocess.Popen(_netns + ["exec", name] + cmd, env=env)

 def create_bridge(name, namespace, ports=[]):
    run_netns("exec", namespace, "ip", "link", "add", "name", name, "type", "bridge")
@ -56,6 +56,9 @@ def create_veth(name1, ns1, name2, ns2, ip = None, subnet=0, link=None):
    run_netns("exec", ns2, "ip", "link", "set", "dev", name2, "up")
    
    if link:
+        if link.txqueuelen:
+            run_netns("exec", ns1, "ip", "link", "set", "dev", name1, "txqueuelen", str(link.txqueuelen))
+            run_netns("exec", ns2, "ip", "link", "set", "dev", name2, "txqueuelen", str(link.txqueuelen))
        tc(ns1, name1, link)
        tc(ns2, name2, link, True)

@ -66,4 +69,6 @@ def tc(namespace, name, link, invert=False):
        options += ["rate", str(link.bandwidth.down)]
    else:
        options += ["rate", str(link.bandwidth.up)]
+    if link.limit:
+        options += ["limit", str(link.limit)]
    run_netns("exec", namespace, "tc", "qdisc", "add", "dev", name, "root", "netem", *options)
--- a/prepare.py
+++ b/prepare.py
@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+from pathlib import Path
+from scenarios.fragments import shared
+import os, venv
+
+me = Path(os.path.dirname(__file__))
+print("--- git submodule ---")
+shared.exec("git submodule update --init")
+
+print("--- compile go benchmarks ---")
+os.chdir(me / "benchmarks" / "warp")
+shared.exec("go build")
+os.chdir(me / "benchmarks" / "s3concurrent")
+shared.exec("go build")
+os.chdir(me / "benchmarks" / "s3lat")
+shared.exec("go build")
+os.chdir(me / "benchmarks" / "s3ttfb")
+shared.exec("go build")
+os.chdir(me / "benchmarks" / "s3billion")
+shared.exec("go build")
+os.chdir(me)
+
+print("--- install python dependencies ---")
+venv.create(".venv", with_pip=True)
+shared.exec("""
+. .venv/bin/activate
+which python3
+python3 -m pip install .
+python3 -m pip install -r scenarios/requirements.txt
+""")
+
+print("--- download artifacts ---")
+from scenarios.fragments import flavor
+flavor.download()
+
+print(f"""done! now, run:
+> sudo -i
+> cd {os.getcwd()}
+> . ./.venv/bin/activate
+> ./mknet scenario ./topo/with-vdsl.yml ./scenarios/garage-s3lat garage-v0.8
+""")
--- a/scenarios/clean
+++ b/scenarios/clean
@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+import os
+from fragments import garage, shared, minio
+
+garage.destroy()
+minio.destroy()
+if len(shared.storage_path) > 8 and shared.id() == 1:
+    shared.exec(f"rm -r {shared.storage_path}")
+shared.log("clean done")
+
--- a/scenarios/fragments/.shared.py.swo
+++ b/scenarios/fragments/.shared.py.swo
--- a/scenarios/fragments/flavor.py
+++ b/scenarios/fragments/flavor.py
@ -0,0 +1,73 @@
+from pathlib import Path
+from . import shared
+from os.path import exists
+
+def grg_path(d):
+    for flav, desc in d.items():
+        if "path" in desc: continue
+        binary = f"garage-{desc['target']}-{desc['version']}"
+        desc['path'] = Path(shared.binary_path) / binary
+    return d
+
+def minio_path(d):
+    for flav, desc in d.items():
+        if "path" in desc: continue
+        desc['path'] = Path(shared.binary_path) / flav
+    return d
+
+garage = grg_path({
+    "garage-local": { "path": "./garage/target/release/garage" },
+    "garage-v0.7": { 
+        "version": "v0.7.3", 
+        "target": "x86_64-unknown-linux-musl", 
+    },
+    # no rpc optimization
+    "garage-v0.8-beta1": { 
+        "version": "89b8087ba81c508ba382aa6c9cb6bb3afa6a43c8", 
+        "target": "x86_64-unknown-linux-musl" 
+    },
+    # with rpc optimizations
+    # with fsync deactivated
+    "garage-v0.8-no-fsync": { 
+        "version": "v0.8.0-dangerous-no-fsync", 
+        "target": "x86_64-unknown-linux-musl" 
+    },
+    # with rpc optimizations
+    "garage-v0.8-beta2": {
+        "version": "af2b2f26b4bd9fcdeedf2cd03f9e1392e5781abc",
+        "target": "x86_64-unknown-linux-musl",
+    }
+})
+
+warp = {
+    "warp-fast": "mixed --obj.size 5M --objects 200 --duration=5m",
+    "warp-small-obj": "mixed --obj.size 256 --objects 200 --duration=5m",
+    "warp-default": "mixed"
+}
+
+minio = minio_path({
+    "minio-20220917": {
+        "version": "2022-09-17T00-09-45Z"
+    }
+})
+
+
+def download():
+    for flav, desc in garage.items():
+        if "version" not in desc: continue
+        if exists(desc['path']): continue
+
+        shared.exec(f"mkdir -p {shared.binary_path}")
+        shared.exec(f"wget https://garagehq.deuxfleurs.fr/_releases/{desc['version']}/{desc['target']}/garage -O {desc['path']}")
+        shared.exec(f"chmod +x {desc['path']}")
+        shared.exec(f"{desc['path']} --version")
+
+    for flav, desc in minio.items():
+        if "version" not in desc: continue
+        if exists(desc['path']): continue
+
+        shared.exec(f"mkdir -p {shared.binary_path}")
+        shared.exec(f"wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio.RELEASE.{desc['version']} -O {desc['path']}")
+        shared.exec(f"chmod +x {desc['path']}")
+        shared.exec(f"{desc['path']} --version")
+
--- a/scenarios/fragments/garage.py
+++ b/scenarios/fragments/garage.py
@ -0,0 +1,229 @@
+import glob, json, time
+from os.path import exists
+from os import environ as env
+from pathlib import Path
+from . import shared, flavor
+
+import garage_admin_sdk
+from garage_admin_sdk.api import nodes_api, layout_api, key_api
+from garage_admin_sdk.model.node_cluster_info import NodeClusterInfo
+from garage_admin_sdk.model.layout_version import LayoutVersion
+from garage_admin_sdk.model.add_key_request import AddKeyRequest
+from garage_admin_sdk.model.update_key_request import UpdateKeyRequest
+from garage_admin_sdk.model.update_key_request_allow import UpdateKeyRequestAllow
+
+storage_path = "./i/am/not/defined"
+rpc_secret = "3e9abff5f9e480afbadb46a77b7a26fe0e404258f0dc3fd5386b0ba8e0ad2fba"
+metrics = "cacce0b2de4bc2d9f5b5fdff551e01ac1496055aed248202d415398987e35f81"
+admin = "ae8cb40ea7368bbdbb6430af11cca7da833d3458a5f52086f4e805a570fb5c2a"
+key = None
+
+version = flavor.garage["garage-v0.7"]
+
+configuration = garage_admin_sdk.Configuration(
+    host = "http://localhost:3903/v0",
+    access_token = admin
+)
+api = garage_admin_sdk.ApiClient(configuration) 
+nodes = nodes_api.NodesApi(api)
+layout = layout_api.LayoutApi(api)
+keys = key_api.KeyApi(api)
+
+
+# Setup, launch on import
+if 'HOST' in env:
+    storage_path = Path(shared.storage_path) / "garage" / env['HOST']
+    if 'ZONE' in env and env['ZONE'] != "":
+        storage_path = Path(shared.storage_path) / "garage" / env['ZONE'] / env['HOST']
+    config = storage_path / "garage.toml"
+    env['GARAGE_CONFIG_FILE'] = str(config)
+
+def deploy_coord(gw=None, uroot={}, us3_api={}, us3_web={}, uadmin={}):
+    destroy()
+    shared.log("start daemon")
+    shared.exec(f"{version['path']} --version")
+    daemon(uroot,us3_api,us3_web,uadmin)
+    shared.log("discover nodes")
+    connect()
+    shared.log("build layout")
+    create_layout(gw=gw)
+    shared.log("create key")
+    create_key()
+    shared.log("ready")
+
+def deploy_follow(uroot={}, us3_api={}, us3_web={}, uadmin={}):
+    destroy()
+    shared.log("start daemon")
+    daemon(uroot,us3_api,us3_web,uadmin)
+    shared.log("wait for coord")
+    sync_on_key_up()
+    shared.log("ready")
+
+def to_toml(d): return "\n".join([ f"{k} = {v}" if type(v) is int else f"{k} = \"{v}\"" for k, v in d.items() ])
+def daemon(uroot={}, us3_api={}, us3_web={}, uadmin={}):
+    root = {
+        "metadata_dir": f"{storage_path}/meta",
+        "data_dir": f"{storage_path}/data",
+        "replication_mode": "3",
+        "rpc_bind_addr": "[::]:3901",
+        "rpc_public_addr": f"[{env['IP']}]:3901",
+        "rpc_secret": f"{rpc_secret}",
+    }
+    root.update(uroot)
+
+    s3_api = {
+        "s3_region": "garage",
+        "api_bind_addr": "[::]:3900",
+        "root_domain": ".s3.garage",
+    }
+    s3_api.update(us3_api)
+    
+    s3_web = {
+        "bind_addr": "[::]:3902",
+        "root_domain": ".web.garage",
+        "index": "index.html",
+    }
+    s3_web.update(us3_web)
+
+    sect_admin = {
+        "api_bind_addr": "0.0.0.0:3903",
+        "metrics_token": f"{metrics}",
+        "admin_token": f"{admin}",
+    }
+    sect_admin.update(uadmin)
+
+
+    shared.exec(f"mkdir -p {storage_path}")
+    with open(config, 'w+') as f:
+        f.write(f"""
+{to_toml(root)}
+bootstrap_peers = []
+
+[s3_api]
+{to_toml(s3_api)}
+
+[s3_web]
+{to_toml(s3_web)}
+
+[admin]
+{to_toml(sect_admin)}
+        """)
+
+    if shared.id() == 1:
+        shared.exec(f"{version['path']} --version")
+    shared.exec(f"{version['path']} server 2>> {storage_path}/logs.stderr 1>> {storage_path}/logs.stdout & echo $! > {storage_path}/daemon.pid")
+    time.sleep(1)
+
+    node_info = storage_path / "node_info"
+    node_id = shared.fn_retry(lambda: nodes.get_nodes().node)
+    with open(node_info, 'w+') as f:
+        f.write(json.dumps({
+            "node_addr": f"{node_id}@{env['IP']}:3901",
+            "node_id": node_id,
+            "zone": env['ZONE'] if 'ZONE' in env and env['ZONE'] != "" else env['HOST'],
+            "host": env['HOST'],
+        }))
+
+def destroy():
+    dpid = Path(storage_path) / "daemon.pid"
+    if exists(dpid):
+        try:
+            shared.exec(f"kill -9 $(cat {dpid})")
+        except:
+            pass
+        shared.exec(f"rm -f {dpid}")
+    if len(str(storage_path)) < 8: # arbitrary, stupid safe guard
+        print(storage_path)
+        raise Exception("You tried to clean a storage path that might be the root of your FS, panicking...")
+    shared.exec(f"rm -fr {storage_path}")
+
+# this function is ugly, sorry :s
+_cluster_info = None
+def cluster_info():
+    global _cluster_info
+    if _cluster_info is not None: return _cluster_info
+    shared.log("fetch cluster info")
+
+    while True:
+        node_files = glob.glob(f"{shared.storage_path}/**/node_info", recursive=True)
+        if len(node_files) >= shared.count(): break
+        shared.log(f"found {len(node_files)} over {shared.count()}, wait 1 sec.")
+        time.sleep(1)
+
+    _cluster_info = [ json.loads(Path(f).read_text()) for f in node_files ]
+    return _cluster_info
+
+
+def connect():
+    cinf = cluster_info()
+    shared.log("start connections...")
+    while True:
+        try:
+            ret = nodes.add_node([n['node_addr'] for n in cinf], _request_timeout=3)
+        except:
+            shared.log("not ready, retry in 1sec")
+            time.sleep(1)
+            continue
+
+        for st in ret:
+            if not st.success:
+                continue
+                #raise Exception("Node connect failed", ret)
+        break
+
+    shared.log("all nodes connected")
+
+def create_layout(gw=None):
+    if gw is None:
+        gw = []
+
+    v = layout.get_layout().version
+
+    cinf = cluster_info()
+    nlay = dict()
+    for n in cinf:
+        capa = 1
+        if n['host'] in gw:
+            capa = None
+
+        nlay[n['node_id']] = NodeClusterInfo(
+            zone = n['zone'],
+            capacity = capa,
+            tags = [ n['host'] ],
+        )
+    layout.add_layout(nlay)
+    layout.apply_layout(LayoutVersion(version=v+1))
+
+    shared.log(layout.get_layout())
+
+def create_key():
+    global key
+    kinfo = shared.fn_retry(lambda: keys.add_key(AddKeyRequest(name="mknet")))
+    allow_create = UpdateKeyRequestAllow(create_bucket=True)
+    keys.update_key(kinfo.access_key_id, UpdateKeyRequest(allow=allow_create))
+    key = kinfo
+
+
+def delete_key():
+    global key
+    keys.delete_key(key.access_key_id)
+    key = None
+
+def sync_on_key_up():
+    global key
+    while True:
+        try:
+            key = keys.search_key("mknet")
+            return key
+        except:
+            pass
+        time.sleep(1)
+
+def sync_on_key_down():
+    while True:
+        try:
+            keys.search_key("mknet")
+        except:
+            return
+        time.sleep(1)
+
--- a/scenarios/fragments/minio.py
+++ b/scenarios/fragments/minio.py
@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+import json, os, sys, time, pathlib, socket, shutil, urllib3
+import minio
+from os import environ as env
+from pathlib import Path
+from . import shared, flavor
+
+storage_path = "./i/am/not/defined"
+version = flavor.minio["minio-20220917"]
+unix_sock = str(Path(shared.storage_path) / "minio.sock")
+access_key = "minioadmin"
+secret_key = "minioadmin"
+client = minio.Minio(
+    f"[{env['IP']}]:9000",
+    access_key="minioadmin",
+    secret_key="minioadmin",
+    secure=False,
+    http_client=urllib3.PoolManager(
+        timeout=2,
+        retries=False,
+    )
+)
+
+if 'HOST' in env:
+    storage_path = str(Path(shared.storage_path) / "minio" / env['HOST'])
+    if 'ZONE' in env and env['ZONE'] != "":
+        storage_path = str(Path(shared.storage_path) / "minio" / env['ZONE'] / env['HOST'])
+
+stdout = Path(storage_path) / "minio.stdout"
+stderr = Path(storage_path) / "minio.stderr"
+pid = Path(storage_path) / "daemon.pid"
+
+def destroy():
+    if os.path.exists(pid):
+        try:
+            shared.exec(f"kill -9 `cat {pid}`")
+        except:
+            pass
+
+    if len(str(storage_path)) > 8:
+        shutil.rmtree(storage_path, ignore_errors=True)
+
+def deploy_coord():
+    destroy()
+    if os.path.exists(unix_sock):
+        os.unlink(unix_sock)
+
+    os.makedirs(storage_path)
+
+    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+    sock.bind(unix_sock)
+    sock.listen()
+
+    # Create sockets
+    fl = [ co for co, addr in [ sock.accept() for i in range(1, shared.count()) ]]
+
+    # Receive configurations, centralize them
+    me = [ { "ip": os.environ['IP'], "path": storage_path } ]
+    others = [ json.loads(co.makefile().readline()) for co in fl ]
+    identities = others + me
+    shared.log(f"ident: {identities}")
+
+    # Dispatch them
+    msg = f"{json.dumps(identities)}\n".encode()
+    [ co.send(msg) for co in fl ]
+
+    run_minio(identities)
+
+    while True:
+        try:
+            if client.bucket_exists("sync"): break
+            client.make_bucket("sync")
+            break
+        except Exception as e: 
+            shared.log("waiting for bootstrap...", e)
+            time.sleep(1)
+    shared.log("ready")
+
+def deploy_follow():
+    destroy()
+    os.makedirs(storage_path)
+
+    co = None
+    while True:
+        try:
+            sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+            sock.connect(unix_sock)
+            co = sock.makefile()
+            break
+        except Exception as err:
+            shared.log('conn failed, wait 1 sec, err is', err)
+            time.sleep(1)
+    
+    # send my identity
+    my_identity = json.dumps({ "ip": os.environ['IP'], "path": storage_path })
+    sock.send(f"{my_identity}\n".encode())
+
+    # get all
+    identities = json.loads(co.readline())
+
+    run_minio(identities)
+    sync_on_bucket_up()
+    shared.log("ready")
+
+def sync_on_bucket_up():
+    while True:
+        try:
+            if client.bucket_exists("sync"): break
+        except:
+            pass
+        shared.log("waiting for bucket 'sync'...")
+        time.sleep(1)
+
+def sync_on_bucket_down():
+    while True:
+        try:
+            if not client.bucket_exists("sync"): break
+            time.sleep(1)
+        except Exception as e:
+            shared.log("the cluster is probably already half shutdown, so errors are expected ->", e)
+            break
+
+def delete_sync_bucket():
+    client.remove_bucket("sync")
+
+def run_minio(identities):
+
+    # Required to prevent Minio error: "/tmp/mknet-store/minio/node1` is part of root drive, will not be used"
+    # https://github.com/minio/minio/issues/15720
+    env['CI'] = "true"
+    env['MINIO_CI_CD'] = "true"
+
+    cmd = f"{version['path']} server --console-address ':9001' --address ':9000'"
+    for ident in identities:
+        cmd += f" http://[{ident['ip']}]:9000{ident['path']}"
+    cmd += f" > {stdout} 2> {stderr}"
+    cmd += f" & echo $! > {pid}"
+
+    shared.log("launch: ", cmd)
+    os.system(cmd)
--- a/scenarios/fragments/s3billion.py
+++ b/scenarios/fragments/s3billion.py
@ -0,0 +1,24 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, minio, garage
+
+s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3billion/s3billion"
+
+def common():
+    out = Path(shared.storage_path) / "s3billion.csv"
+    shared.log(f"launching s3billion ({s3bin})")
+    shared.exec(f"{s3bin} > {out}")
+    shared.log(f"execution done, output written to {out}")
+
+def on_garage():
+    os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
+    os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
+    os.environ['ENDPOINT'] = f"[{os.environ['IP']}]:3900"
+    common()
+
+def on_minio():
+    os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
+    os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
+    os.environ['ENDPOINT'] = f"[{os.environ['IP']}]:9000"
+    common()
--- a/scenarios/fragments/s3concurrent.py
+++ b/scenarios/fragments/s3concurrent.py
@ -0,0 +1,16 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, garage
+
+s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3concurrent/s3concurrent"
+
+def on_garage():
+    os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
+    os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
+    os.environ['ENDPOINT'] = "localhost:3900"
+
+    out = Path(shared.storage_path) / "s3concurrent.csv"
+    shared.log(f"launching s3concurrent ({s3bin})")
+    shared.exec(f"{s3bin} > {out}")
+    shared.log(f"execution done, output written to {out}")
--- a/scenarios/fragments/s3lat.py
+++ b/scenarios/fragments/s3lat.py
@ -0,0 +1,24 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, minio, garage
+
+s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3lat/s3lat"
+
+def common():
+    out = Path(shared.storage_path) / "s3lat.csv"
+    shared.log(f"launching s3lat ({s3bin})")
+    shared.exec(f"{s3bin} > {out}")
+    shared.log(f"execution done, output written to {out}")
+
+def on_garage():
+    os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
+    os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
+    os.environ['ENDPOINT'] = "localhost:3900"
+    common()
+
+def on_minio():
+    os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
+    os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
+    os.environ['ENDPOINT'] = "localhost:9000"
+    common()
--- a/scenarios/fragments/s3ttfb.py
+++ b/scenarios/fragments/s3ttfb.py
@ -0,0 +1,26 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, minio, garage
+
+s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3ttfb/s3ttfb"
+
+def common():
+    out = Path(shared.storage_path) / "s3ttfb.csv"
+    shared.log(f"launching s3ttfb ({s3bin})")
+    shared.exec(f"{s3bin} > {out}")
+    shared.log(f"execution done, output written to {out}")
+
+def on_garage():
+    os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
+    os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
+    os.environ['ENDPOINT'] = "localhost:3900"
+    os.environ['AWS_REGION'] = "garage"
+    common()
+
+def on_minio():
+    os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
+    os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
+    os.environ['ENDPOINT'] = "localhost:9000"
+    os.environ['AWS_REGION'] = "us-east-1"
+    common()
--- a/scenarios/fragments/shared.py
+++ b/scenarios/fragments/shared.py
@ -0,0 +1,28 @@
+import os, time
+
+binary_path = "/tmp/mknet-bin"
+storage_path = "/tmp/mknet-store"
+
+def exec(s): 
+    if os.system(s) != 0:
+        raise Exception("Command terminated with an error")
+def exec_retry(s, cnt=16):
+    print(s)
+    for i in range(cnt):
+        time.sleep(i) # this is expected to sleep before running the command to reduce the noise
+        if os.system(s) == 0: return
+    raise Exception("Command terminated with an error too many times")
+def fn_retry(f, cnt=5):
+    for i in range(cnt):
+        try:
+            r = f()
+            return r
+        except Exception as e:
+            if i+1 == cnt: raise e
+            log(f"failed call, retry in {i} sec")
+            time.sleep(i)
+
+def id(): return int(os.environ['ID'])
+def count(): return int(os.environ['SERVER_COUNT'])
+def log(*args): print(f"[{id()}/{count()} - {os.environ['HOST']}]", *args)
+
--- a/scenarios/fragments/warp.py
+++ b/scenarios/fragments/warp.py
@ -0,0 +1,17 @@
+import os
+from os.path import exists
+from pathlib import Path
+from fragments import shared, garage, flavor, minio
+
+warp_bin = Path(os.path.dirname(__file__)) / "../../benchmarks/warp/warp"
+bench = flavor.warp["warp-fast"]
+
+def common(port, ak, sk):
+    out = Path(shared.storage_path) / "warp.csv"
+    shared.log(f"launching warp {warp_bin}")
+    shared.exec(f"{warp_bin} {bench} --host=[{os.environ['IP']}]:{port} --analyze.out={out} --access-key={ak} --secret-key={sk}")
+    shared.log(f"execution done")
+
+def on_garage(): common(3900, garage.key.access_key_id, garage.key.secret_access_key)
+def on_minio(): common(9000, minio.access_key, minio.secret_key)
+
--- a/scenarios/garage-manual
+++ b/scenarios/garage-manual
@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+from fragments import garage, s3lat, shared
+
+if shared.id() == 1:
+    garage.deploy_coord()
+else:
+    garage.deploy_follow()
--- a/scenarios/garage-s3billion
+++ b/scenarios/garage-s3billion
@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+from fragments import garage, s3billion, shared, flavor
+import sys, os
+
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+
+#os.environ['BATCH_COUNT'] = "2"
+shared.exec("ulimit -n 65535")
+conf_root = {
+    "db_engine": "lmdb",
+}
+
+if shared.id() == 1:
+    garage.deploy_coord(uroot=conf_root)
+    s3billion.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow(uroot=conf_root)
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/garage-s3concurrent
+++ b/scenarios/garage-s3concurrent
@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+from fragments import garage, s3concurrent, shared, flavor
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+
+if shared.id() == 1:
+    garage.deploy_coord()
+    s3concurrent.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow()
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/garage-s3lat
+++ b/scenarios/garage-s3lat
@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+from fragments import garage, s3lat, shared, flavor
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+
+if shared.id() == 1:
+    garage.deploy_coord()
+    s3lat.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow()
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/garage-s3ttfb
+++ b/scenarios/garage-s3ttfb
@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+from fragments import garage, s3ttfb, shared, flavor
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+
+if shared.id() == 1:
+    garage.deploy_coord(gw = [ 'node1' ])
+    s3ttfb.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow()
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/garage-warp
+++ b/scenarios/garage-warp
@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+from fragments import garage, warp, shared, flavor
+import sys
+
+db_engines = [ 'sled', 'lmdb', 'sqlite' ]
+conf_root = {}
+for fl in sys.argv[1:]:
+    if fl in flavor.garage:
+        garage.version = flavor.garage[fl]
+    elif fl in flavor.warp:
+        warp.bench = flavor.warp[fl]
+    elif fl in db_engines:
+        conf_root['db_engine'] = fl
+    elif 'sled_no_fsync' == fl:
+        conf_root['sled_flush_every_ms'] = 1000 * 60 * 10 # 10 minutes
+
+if shared.id() == 1:
+    garage.deploy_coord(uroot=conf_root)
+    warp.on_garage()
+    garage.delete_key()
+    garage.destroy()
+else:
+    garage.deploy_follow(uroot=conf_root)
+    garage.sync_on_key_down()
+    garage.destroy()
+shared.log("bye")
--- a/scenarios/minio-manual
+++ b/scenarios/minio-manual
@ -0,0 +1,12 @@
+#!/usr/bin/env python3
+from fragments import shared, flavor, minio
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+
+if shared.id() == 1:
+    minio.deploy_coord()
+else:
+    minio.deploy_follow()
--- a/scenarios/minio-s3billion
+++ b/scenarios/minio-s3billion
@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+from fragments import minio, s3billion, shared, flavor
+import sys, os
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+
+#os.environ['BATCH_COUNT'] = "2"
+shared.exec("ulimit -n 65535")
+
+if shared.id() == 1:
+    minio.deploy_coord()
+    s3billion.on_minio()
+    minio.delete_sync_bucket()
+    minio.destroy()
+else:
+    minio.deploy_follow()
+    minio.sync_on_bucket_down()
+    minio.destroy()
+
+shared.log("bye")
--- a/scenarios/minio-s3lat
+++ b/scenarios/minio-s3lat
@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+from fragments import minio, s3lat, shared, flavor
+import sys
+
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+
+if shared.id() == 1:
+    minio.deploy_coord()
+    s3lat.on_minio()
+    minio.delete_sync_bucket()
+    minio.destroy()
+else:
+    minio.deploy_follow()
+    minio.sync_on_bucket_down()
+    minio.destroy()
+shared.log("bye")
--- a/scenarios/minio-s3ttfb
+++ b/scenarios/minio-s3ttfb
@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+from fragments import minio, s3ttfb, shared, flavor
+import sys
+
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+
+if shared.id() == 1:
+    minio.deploy_coord()
+    s3ttfb.on_minio()
+    minio.delete_sync_bucket()
+    minio.destroy()
+else:
+    minio.deploy_follow()
+    minio.sync_on_bucket_down()
+    minio.destroy()
+shared.log("bye")
--- a/scenarios/minio-warp
+++ b/scenarios/minio-warp
@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+from fragments import minio, warp, shared, flavor
+import sys
+
+for fl in sys.argv[1:]:
+    if fl in flavor.minio:
+        minio.version = flavor.minio[fl]
+    if fl in flavor.warp:
+        warp.bench = flavor.warp[fl]
+
+if shared.id() == 1:
+    minio.deploy_coord()
+    warp.on_minio()
+    minio.delete_sync_bucket()
+    minio.destroy()
+else:
+    minio.deploy_follow()
+    minio.sync_on_bucket_down()
+    minio.destroy()
+shared.log("bye")
--- a/scenarios/requirements.txt
+++ b/scenarios/requirements.txt
@ -0,0 +1,2 @@
+git+https://git.deuxfleurs.fr/quentin/garage-admin-sdk@2b685b90be9aa848a9450dd582d1d322b25b5568#egg=garage-admin-sdk&subdirectory=python
+minio
--- a/setup.py
+++ b/setup.py
@ -3,4 +3,8 @@ setup(name='mknet',
      version='1.0',
      scripts=['mknet'],
      py_modules=['net'],
+      install_requires=[
+          'PyYAML',
+          'requests'
+      ],
      )
--- a/shell.nix
+++ b/shell.nix
@ -0,0 +1,12 @@
+{ pkgs ? import <nixpkgs> {} }:
+  pkgs.mkShell {
+    nativeBuildInputs = [ 
+      pkgs.python310
+      pkgs.python310Packages.pyyaml
+      pkgs.python310Packages.requests
+
+      # to test the pip setup
+      pkgs.python310Packages.pip
+      pkgs.python310Packages.setuptools
+    ];
+}
--- a/topo/100mbps.yml
+++ b/topo/100mbps.yml
@ -0,0 +1,25 @@
+links:
+  - &slow
+    bandwidth: 100M
+    latency: 500us
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/10mbps.yml
+++ b/topo/10mbps.yml
@ -0,0 +1,25 @@
+links:
+  - &slow
+    bandwidth: 10M
+    latency: 500us
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/1mbps.yml
+++ b/topo/1mbps.yml
@ -0,0 +1,27 @@
+links:
+  - &slow
+    bandwidth: 1M
+    latency: 500us
+    #txqueuelen: 10
+    limit: 10
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/50mbps.yml
+++ b/topo/50mbps.yml
@ -0,0 +1,25 @@
+links:
+  - &slow
+    bandwidth: 50M
+    latency: 500us
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/50ms.yml
+++ b/topo/50ms.yml
@ -0,0 +1,34 @@
+links:
+  - &fiber
+    bandwidth: 100M
+    latency: 50ms
+    jitter: 10ms
+
+zones:
+  - &dc1
+    name: dc1
+    internal: *fiber
+    external: *fiber
+
+servers:
+  - name: dc1s1
+    zone: *dc1
+  - name: dc1s2
+    zone: *dc1
+  - name: dc1s3
+    zone: *dc1
+  - name: dc1s4
+    zone: *dc1
+  - name: dc1s5
+    zone: *dc1
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *fiber
+
--- a/topo/5mbps.yml
+++ b/topo/5mbps.yml
@ -0,0 +1,28 @@
+links:
+  - &slow
+    bandwidth: 5M
+    latency: 500us
+    limit: 5
+  - &1000
+    bandwidth: 1000M
+    latency: 100us
+
+servers:
+  - name: node1
+    <<: *slow
+  - name: node2
+    <<: *slow
+  - name: node3
+    <<: *slow
+  - name: node4
+    <<: *slow
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/dc.yml
+++ b/topo/dc.yml
@ -0,0 +1,22 @@
+links:
+  - &1000
+    bandwidth: 1000M
+    latency: 1ms
+
+servers:
+  - name: node1
+    <<: *1000
+  - name: node2
+    <<: *1000
+  - name: node3
+    <<: *1000
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *1000
--- a/topo/multi-dc.yml
+++ b/topo/multi-dc.yml
@ -0,0 +1,46 @@
+links:
+  - &100
+    bandwidth: 100M
+    latency: 500us
+  - &wan
+    bandwidth: 100M
+    latency: 50ms
+    jitter: 10ms
+
+zones:
+  - &dc1
+    name: dc1
+    internal: *100
+    external: *wan
+  - &dc2
+    name: dc2
+    internal: *100
+    external: *wan
+  - &dc3
+    name: dc3
+    internal: *100
+    external: *wan
+
+servers:
+  - name: dc1s1
+    zone: *dc1
+  - name: dc1s2
+    zone: *dc1
+  - name: dc2s1
+    zone: *dc2
+  - name: dc2s2
+    zone: *dc2
+  - name: dc3s1
+    zone: *dc3
+  - name: dc3s2
+    zone: *dc3
+
+global:
+  subnet:
+    base: 'fc00:9a7a:9e::'
+    local: 64
+    zone: 16
+  latency-offset: 3ms
+  upstream:
+    ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
+    conn: *wan
--- a/topo/with-vdsl.yml
+++ b/topo/with-vdsl.yml
@ -5,16 +5,19 @@ links:
  - &1000
    bandwidth: 1000M
    latency: 100us
+    txqueuelen: 10000
  - &vdsl
    bandwidth:
      up: 3M
      down: 55M
    latency: 50ms
    jitter: 10ms
+    txqueuelen: 100
  - &fiber
    bandwidth: 400M
    latency: 10ms
    jitter: 3ms
+    txqueuelen: 1000

 zones:
  - &dc1
Author	SHA1	Message	Date
Quentin	db88b939c3	Merge pull request 'Fix venv sourcing' (#2 ) from baptiste/mknet:misc_fixes into main Reviewed-on: Deuxfleurs/mknet#2	2022-09-29 20:57:53 +02:00
Quentin	6d964937b3	Merge pull request 'Add result with 30M objects on Grid'5000' (#3 ) from baptiste/mknet:gros_30M into main Reviewed-on: Deuxfleurs/mknet#3	2022-09-29 20:53:21 +02:00
Baptiste Jonglez	c752ff7997	Add result with 30M objects on Grid'5000	2022-09-29 20:27:25 +02:00
Baptiste Jonglez	37040718ea	Document s3billion	2022-09-29 10:40:34 +02:00
Baptiste Jonglez	d1c659924a	prepare: fix venv sourcing when /bin/sh is Dash (e.g. on Debian)	2022-09-29 10:39:45 +02:00
Quentin	12a50ef2db	Remove all regressions	2022-09-28 14:59:42 +02:00
Quentin	1ae69c8e2a	To fit or not to fit	2022-09-28 14:59:18 +02:00
Quentin	7011ec431f	Add sled no-sync	2022-09-27 11:50:08 +02:00
Quentin	c78e4f8005	Details	2022-09-26 17:09:53 +02:00
Quentin	71d0d82361	Add s3billion minio graphs	2022-09-26 17:05:56 +02:00
Quentin	b0b4b162dc	Minio s3billion benchmark	2022-09-26 16:39:19 +02:00
Quentin	b4c1f58a3c	DB engine comparison	2022-09-24 18:09:18 +02:00
Quentin	c84dc6bd35	Better scales	2022-09-24 17:10:33 +02:00
Quentin	3fc014ad3c	Add unit	2022-09-24 17:09:17 +02:00
Quentin	c181eee22e	Fix plot	2022-09-24 17:07:39 +02:00
Quentin	4d61d4a72f	Refactor + fix typo in 1M+ objects bench	2022-09-24 17:06:43 +02:00
Quentin	a869e3614b	LMDB s3 million objects	2022-09-24 17:01:34 +02:00
Quentin	a744f72f15	Bench PutObject with many parallel small obj	2022-09-24 15:13:40 +02:00
Quentin	ac4f100da1	Add a first warp plot	2022-09-24 11:43:30 +02:00
Quentin	0dc16e5e80	Support minio warp	2022-09-23 23:20:10 +02:00
Quentin	60df0fb95d	Add black outline	2022-09-23 22:56:05 +02:00
Quentin	fa61df5e07	Add latency	2022-09-23 22:53:35 +02:00
Quentin	a40c123542	Add S3TTFB	2022-09-23 22:50:50 +02:00
Quentin	700025e479	Add plot	2022-09-23 19:15:58 +02:00
Quentin	5b460dd10f	Done some measurements for s3lat with Garage	2022-09-23 18:47:13 +02:00
Quentin	4a02f16489	Add support for minio-s3lat	2022-09-23 18:09:54 +02:00
Quentin	c9cbe5fc52	Add some support for minio	2022-09-23 17:55:45 +02:00
Quentin	0664442648	Add support for txqueuelen	2022-09-19 18:02:33 +02:00
Quentin	549c548e7c	Commit WIP	2022-09-19 14:16:08 +02:00
Quentin	bd6a125399	Create bandwidth bottleneck topo	2022-09-19 12:44:52 +02:00
Quentin	813b35e479	WIP s3concurrent + fix prepare.py	2022-09-19 12:41:51 +02:00
Quentin	1ea2244c8c	Update s3concurrent	2022-09-19 12:37:25 +02:00
Quentin	12764a020a	Fix prepare script	2022-09-16 16:19:09 +02:00
Quentin	b95db09b96	Create a prepare script	2022-09-16 15:43:58 +02:00
Quentin	4bec653fe9	Add warp	2022-09-15 18:04:37 +02:00
Quentin	18b8cb0f2b	update examples	2022-09-15 17:34:35 +02:00
Quentin	6588e878b2	Add flavors	2022-09-15 17:33:42 +02:00
Quentin	9652978433	Improve README	2022-09-15 17:12:21 +02:00
Quentin	62e4ed55a4	Implement scenario subcommand	2022-09-15 17:00:02 +02:00
Quentin	cfd3e500fe	run-all wait all processes before returning	2022-09-15 16:53:21 +02:00
Quentin	68cc7d635b	More doc	2022-09-14 18:35:09 +02:00
Quentin	8bebbd3181	Fix doc	2022-09-14 18:32:07 +02:00
Quentin	f0db8ba560	Scenario based benchmarks will be ready soon	2022-09-14 18:22:46 +02:00
Quentin	c75b239018	Update warp submodule	2022-09-14 18:04:03 +02:00
Quentin	c77d8dcfa7	Working s3lat with Garage	2022-09-14 18:01:44 +02:00
Quentin	e3409ce6b7	New way to deploy Garage	2022-09-14 17:25:16 +02:00
trinity-1686a	30643ca50d	add support for limiting queue len	2022-08-31 21:29:18 +02:00
				`@ -0,0 +1 @@`
				`Subproject commit 902555b9d5db594fdad927f51c2d8984d6e9f6a5`
				`@ -0,0 +1 @@`
				`Subproject commit df8f2cd7b7a8523755b4d3c04271a69571a455ca`