forked from Deuxfleurs/mknet
Compare commits
48 commits
improvemen
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| db88b939c3 | |||
| 6d964937b3 | |||
|
c752ff7997 | ||
|
|
37040718ea | ||
|
|
d1c659924a | ||
12a50ef2db
|
|||
|
1ae69c8e2a
|
|||
|
7011ec431f
|
|||
|
c78e4f8005
|
|||
|
71d0d82361
|
|||
|
b0b4b162dc
|
|||
|
b4c1f58a3c
|
|||
|
c84dc6bd35
|
|||
|
3fc014ad3c
|
|||
|
c181eee22e
|
|||
|
4d61d4a72f
|
|||
|
a869e3614b
|
|||
|
a744f72f15
|
|||
|
ac4f100da1
|
|||
|
0dc16e5e80
|
|||
|
60df0fb95d
|
|||
|
fa61df5e07
|
|||
|
a40c123542
|
|||
|
700025e479
|
|||
|
5b460dd10f
|
|||
|
4a02f16489
|
|||
|
c9cbe5fc52
|
|||
|
0664442648
|
|||
|
549c548e7c
|
|||
|
bd6a125399
|
|||
|
813b35e479
|
|||
|
1ea2244c8c
|
|||
|
12764a020a
|
|||
|
b95db09b96
|
|||
|
4bec653fe9
|
|||
|
18b8cb0f2b
|
|||
|
6588e878b2
|
|||
|
9652978433
|
|||
|
62e4ed55a4
|
|||
|
cfd3e500fe
|
|||
|
68cc7d635b
|
|||
|
8bebbd3181
|
|||
|
f0db8ba560
|
|||
|
c75b239018
|
|||
|
c77d8dcfa7
|
|||
|
e3409ce6b7
|
|||
| 30643ca50d | |||
| 5a8be6fc73 |
97 changed files with 8372 additions and 156 deletions
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -1,3 +1,6 @@
|
||||||
.current_state.yml
|
.current_state.yml
|
||||||
__pycache__
|
__pycache__
|
||||||
*.swp
|
*.swp
|
||||||
|
build
|
||||||
|
mknet.egg-info
|
||||||
|
.venv
|
||||||
|
|
|
||||||
6
.gitmodules
vendored
Normal file
6
.gitmodules
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
[submodule "benchmarks/warp"]
|
||||||
|
path = benchmarks/warp
|
||||||
|
url = https://github.com/minio/warp
|
||||||
|
[submodule "analysis/quentin"]
|
||||||
|
path = analysis/quentin
|
||||||
|
url = https://git.deuxfleurs.fr/quentin/benchmarks
|
||||||
84
README.md
84
README.md
|
|
@ -1,43 +1,77 @@
|
||||||
# mknet
|
# mknet
|
||||||
|
|
||||||
## Installation
|
mknet is a tool to simulate various network topologies
|
||||||
|
locally thanks to network namespaces and traffic control (tc).
|
||||||
|
|
||||||
|
## Prepare your environment
|
||||||
|
|
||||||
|
Get the repository:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo pip3 install git+https://git.deuxfleurs.fr/trinity-1686a/mknet
|
git clone https://git.deuxfleurs.fr/Deuxfleurs/mknet.git
|
||||||
|
cd mknet
|
||||||
```
|
```
|
||||||
|
|
||||||
## Usage
|
Run our configuration script:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo rm -rf /tmp/garage-testnet/ # always start by deleting previous run
|
./prepare.py
|
||||||
sudo mknet create ./config.yml
|
|
||||||
sudo mknet run-all example/deploy_garage.sh
|
|
||||||
sudo mknet run dc1:dc1s1 garage -c /tmp/garage-testnet/dc1/dc1s1/garage.toml status
|
|
||||||
sudo mknet destroy
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Instrumented daemons
|
Now, you are ready to launch an experiment:
|
||||||
|
|
||||||
If you want to use the scripts provided in the `example` folder,
|
|
||||||
you must add to your path some tools.
|
|
||||||
|
|
||||||
### Garage (`deploy_garage.sh`)
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# see versions on https://garagehq.deuxfleurs.fr/_releases.html
|
sudo -i
|
||||||
export GRG_ARCH=x86_64-unknown-linux-musl
|
source .venv/bin/activate
|
||||||
export GRG_VERSION=v0.5.0
|
./mknet scenario ./topo/50ms.yml ./scenarios/garage-s3lat garage-v0.8
|
||||||
|
|
||||||
sudo wget https://garagehq.deuxfleurs.fr/_releases/${GRG_VERSION}/${GRG_ARCH}/garage -O /usr/local/bin/garage
|
|
||||||
sudo chmod +x /usr/local/bin/garage
|
|
||||||
|
|
||||||
garage help
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Minio (`deploy_minio.py`)
|
If a script crash, you must manually destroy the topology:
|
||||||
|
|
||||||
```
|
```
|
||||||
sudo wget https://dl.min.io/server/minio/release/linux-amd64/minio -O /usr/local/bin/minio
|
./mknet destroy
|
||||||
sudo chmod +x /usr/local/bin/minio
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Topologies
|
||||||
|
|
||||||
|
All topologies:
|
||||||
|
- `./topo/dc.yml` - A 3 node topology connected with a 1Gbit/s link and a 1ms latency
|
||||||
|
- `./topo/with-vdsl.yml` - A topology mixing datacenters with fast internal connectivity and an isolated VDSL node
|
||||||
|
- `./topo/50ms.yml` - An artifical topology simulating nodes with high bandwidth but with a fixed 50ms latency, useful to quantify the impact of latency on a distributed software
|
||||||
|
- `./topo/multi-dc.yml` - Simulate 3 DC interconnected with 50ms latency WAN network and close to zero latency inside the DC
|
||||||
|
- `./topo/{1,5,10,50,100}mbps.yml` - Simulate 3 nodes interconnected with a bandwidth bottleneck.
|
||||||
|
|
||||||
|
Feel free to write new topologies!
|
||||||
|
|
||||||
|
## Scenarios
|
||||||
|
|
||||||
|
All scenarios:
|
||||||
|
- `./scenarios/garage-s3lat [garage-v0.7|garage-v0.8]` - Run s3lat on Garage
|
||||||
|
- `./scenarios/garage-concurrent [garage-v0.7|garage-v0.8]` - Run s3concurrent on Garage
|
||||||
|
- `./scenarios/garage-warp [garage-v0.7|garage-v0.8] [default|fast]` - Run warp on Garage. 2 flavors are available: fast and default.
|
||||||
|
- `./scenarios/garage-s3billion [garage-v0.7|garage-v0.8]` - Run s3billion on Garage
|
||||||
|
|
||||||
|
*Scenarios take optional flavors as input that modulate their behavioir. Passing them is not mandatory,
|
||||||
|
a default one will be selected for you.*
|
||||||
|
|
||||||
|
How to run them:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./mknet scenario <topology> <scenario> [flavors...]
|
||||||
|
./mknet scenario ./topo/50ms.yml ./scenarios/garage-s3lat garage-v0.8
|
||||||
|
```
|
||||||
|
|
||||||
|
How to write good scenarios:
|
||||||
|
- If a scenario can be run with multiple different parameters, write one scenario with multiple flavors
|
||||||
|
- If the logic ran is different, write a new scenario
|
||||||
|
- A scenario code must remain short and looks like a DSL, abstract the logic in the `fragments/` module
|
||||||
|
|
||||||
|
## Manual usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./mknet create ./topo/with-vdsl.yml
|
||||||
|
./mknet run-all ./scenarios/garage-manual
|
||||||
|
./mknet run dc1:dc1s1 /tmp/mknet-bin/garage* -c /tmp/mknet-store/garage/dc1/dc1s1/garage.toml status
|
||||||
|
./mknet run-all ./scenarios/clean
|
||||||
|
./mknet destroy
|
||||||
|
|
||||||
|
|
|
||||||
1
analysis/quentin
Submodule
1
analysis/quentin
Submodule
|
|
@ -0,0 +1 @@
|
||||||
|
Subproject commit 902555b9d5db594fdad927f51c2d8984d6e9f6a5
|
||||||
17
artifacts/2022-09-19-s3concurrent/garage-v0.7_10mbps.csv
Normal file
17
artifacts/2022-09-19-s3concurrent/garage-v0.7_10mbps.csv
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
sent,success,elapsed,elapsed_per_req
|
||||||
|
1,1,1733568406,1733568406
|
||||||
|
2,2,3717781951,1858890975
|
||||||
|
3,3,5554603016,1851534338
|
||||||
|
4,4,7243764558,1810941139
|
||||||
|
5,5,9088779023,1817755804
|
||||||
|
6,6,10853379229,1808896538
|
||||||
|
7,7,12687932044,1812561720
|
||||||
|
8,8,14457291732,1807161466
|
||||||
|
9,9,16271996844,1807999649
|
||||||
|
10,10,18066058997,1806605899
|
||||||
|
11,11,20370606012,1851873273
|
||||||
|
12,12,11605742376,967145198
|
||||||
|
13,13,13622546503,1047888192
|
||||||
|
14,14,16059774589,1147126756
|
||||||
|
15,15,20283463752,1352230916
|
||||||
|
16,16,23829428368,1489339273
|
||||||
|
17
artifacts/2022-09-19-s3concurrent/garage-v0.7_5mbps.csv
Normal file
17
artifacts/2022-09-19-s3concurrent/garage-v0.7_5mbps.csv
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
sent,success,elapsed,elapsed_per_req
|
||||||
|
1,1,3646698401,3646698401
|
||||||
|
2,2,7245492636,3622746318
|
||||||
|
3,3,10896710797,3632236932
|
||||||
|
4,4,14469160703,3617290175
|
||||||
|
5,4,18066381776,3613276355
|
||||||
|
6,3,19209830571,3201638428
|
||||||
|
7,0,186084824,26583546
|
||||||
|
8,0,183207903,22900987
|
||||||
|
9,0,199743038,22193670
|
||||||
|
10,0,195703069,19570306
|
||||||
|
11,0,198204447,18018586
|
||||||
|
12,0,187247329,15603944
|
||||||
|
13,0,193441556,14880119
|
||||||
|
14,0,192003016,13714501
|
||||||
|
15,0,140156357,9343757
|
||||||
|
16,0,186228134,11639258
|
||||||
|
17
artifacts/2022-09-19-s3concurrent/garage-v0.8_10mbps.csv
Normal file
17
artifacts/2022-09-19-s3concurrent/garage-v0.8_10mbps.csv
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
sent,success,elapsed,elapsed_per_req
|
||||||
|
1,1,1634632370,1634632370
|
||||||
|
2,2,3799299300,1899649650
|
||||||
|
3,3,5436258664,1812086221
|
||||||
|
4,4,7242028242,1810507060
|
||||||
|
5,5,9041569964,1808313992
|
||||||
|
6,6,10852330277,1808721712
|
||||||
|
7,7,12646683069,1806669009
|
||||||
|
8,8,14468867472,1808608434
|
||||||
|
9,9,16251724200,1805747133
|
||||||
|
10,10,18190271787,1819027178
|
||||||
|
11,11,19877017065,1807001551
|
||||||
|
12,12,20526867033,1710572252
|
||||||
|
13,13,24558838287,1889141406
|
||||||
|
14,14,25274586429,1805327602
|
||||||
|
15,15,27180202376,1812013491
|
||||||
|
16,16,29437555977,1839847248
|
||||||
|
17
artifacts/2022-09-19-s3concurrent/garage-v0.8_5mbps.csv
Normal file
17
artifacts/2022-09-19-s3concurrent/garage-v0.8_5mbps.csv
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
sent,success,elapsed,elapsed_per_req
|
||||||
|
1,1,3638584739,3638584739
|
||||||
|
2,2,7246358601,3623179300
|
||||||
|
3,3,11160920229,3720306743
|
||||||
|
4,4,14454330626,3613582656
|
||||||
|
5,5,18052975913,3610595182
|
||||||
|
6,6,21690341725,3615056954
|
||||||
|
7,7,25268069065,3609724152
|
||||||
|
8,8,29007558135,3625944766
|
||||||
|
9,9,32688763527,3632084836
|
||||||
|
10,10,36101205725,3610120572
|
||||||
|
11,11,28245933586,2567812144
|
||||||
|
12,12,43983751108,3665312592
|
||||||
|
13,13,53456719112,4112055316
|
||||||
|
14,14,54398690114,3885620722
|
||||||
|
15,15,54153860198,3610257346
|
||||||
|
16,16,57732527899,3608282993
|
||||||
|
501
artifacts/2022-09-23-s3lat/garage-v0.7.csv
Normal file
501
artifacts/2022-09-23-s3lat/garage-v0.7.csv
Normal file
|
|
@ -0,0 +1,501 @@
|
||||||
|
endpoint,nanoseconds
|
||||||
|
listbuckets,123446207
|
||||||
|
listbuckets,131497917
|
||||||
|
listbuckets,39054735
|
||||||
|
listbuckets,105410142
|
||||||
|
listbuckets,134197174
|
||||||
|
listbuckets,92373770
|
||||||
|
listbuckets,113362017
|
||||||
|
listbuckets,109195200
|
||||||
|
listbuckets,176558852
|
||||||
|
listbuckets,27177881
|
||||||
|
listbuckets,36252695
|
||||||
|
listbuckets,106228157
|
||||||
|
listbuckets,167364021
|
||||||
|
listbuckets,6904816
|
||||||
|
listbuckets,34264521
|
||||||
|
listbuckets,83810002
|
||||||
|
listbuckets,156429277
|
||||||
|
listbuckets,200458223
|
||||||
|
listbuckets,112529004
|
||||||
|
listbuckets,35129564
|
||||||
|
listbuckets,79573343
|
||||||
|
listbuckets,140150702
|
||||||
|
listbuckets,120555536
|
||||||
|
listbuckets,29181815
|
||||||
|
listbuckets,51198242
|
||||||
|
listbuckets,56133478
|
||||||
|
listbuckets,47130673
|
||||||
|
listbuckets,4870606
|
||||||
|
listbuckets,115299404
|
||||||
|
listbuckets,199760726
|
||||||
|
listbuckets,104304255
|
||||||
|
listbuckets,165317856
|
||||||
|
listbuckets,174747530
|
||||||
|
listbuckets,69139944
|
||||||
|
listbuckets,170345599
|
||||||
|
listbuckets,133286913
|
||||||
|
listbuckets,133428040
|
||||||
|
listbuckets,152317101
|
||||||
|
listbuckets,163419603
|
||||||
|
listbuckets,11954339
|
||||||
|
listbuckets,49001694
|
||||||
|
listbuckets,1909515
|
||||||
|
listbuckets,85317086
|
||||||
|
listbuckets,193461440
|
||||||
|
listbuckets,37110905
|
||||||
|
listbuckets,170866722
|
||||||
|
listbuckets,2820145
|
||||||
|
listbuckets,136496793
|
||||||
|
listbuckets,64132721
|
||||||
|
listbuckets,125251903
|
||||||
|
listbuckets,87449583
|
||||||
|
listbuckets,156290592
|
||||||
|
listbuckets,160402726
|
||||||
|
listbuckets,200273480
|
||||||
|
listbuckets,93021081
|
||||||
|
listbuckets,68304026
|
||||||
|
listbuckets,46970448
|
||||||
|
listbuckets,165308307
|
||||||
|
listbuckets,41980457
|
||||||
|
listbuckets,166480048
|
||||||
|
listbuckets,136400950
|
||||||
|
listbuckets,111337721
|
||||||
|
listbuckets,180455774
|
||||||
|
listbuckets,119142367
|
||||||
|
listbuckets,117091664
|
||||||
|
listbuckets,126145949
|
||||||
|
listbuckets,119181500
|
||||||
|
listbuckets,55147467
|
||||||
|
listbuckets,178508747
|
||||||
|
listbuckets,101172269
|
||||||
|
listbuckets,116156709
|
||||||
|
listbuckets,53168650
|
||||||
|
listbuckets,29044786
|
||||||
|
listbuckets,47899914
|
||||||
|
listbuckets,139056994
|
||||||
|
listbuckets,26962604
|
||||||
|
listbuckets,50780131
|
||||||
|
listbuckets,36960440
|
||||||
|
listbuckets,161531494
|
||||||
|
listbuckets,9947611
|
||||||
|
listbuckets,33134947
|
||||||
|
listbuckets,81109551
|
||||||
|
listbuckets,153415682
|
||||||
|
listbuckets,199163150
|
||||||
|
listbuckets,183053868
|
||||||
|
listbuckets,148142646
|
||||||
|
listbuckets,17965198
|
||||||
|
listbuckets,161258658
|
||||||
|
listbuckets,175383896
|
||||||
|
listbuckets,42076378
|
||||||
|
listbuckets,34970593
|
||||||
|
listbuckets,14042201
|
||||||
|
listbuckets,66182118
|
||||||
|
listbuckets,42936091
|
||||||
|
listbuckets,200730649
|
||||||
|
listbuckets,75462399
|
||||||
|
listbuckets,34967326
|
||||||
|
listbuckets,99414279
|
||||||
|
listbuckets,83041589
|
||||||
|
listbuckets,9915941
|
||||||
|
putobject,102659465
|
||||||
|
putobject,92835679
|
||||||
|
putobject,108731599
|
||||||
|
putobject,201811799
|
||||||
|
putobject,106443490
|
||||||
|
putobject,92567983
|
||||||
|
putobject,238871966
|
||||||
|
putobject,104158815
|
||||||
|
putobject,106243742
|
||||||
|
putobject,116316799
|
||||||
|
putobject,108757989
|
||||||
|
putobject,97366623
|
||||||
|
putobject,97898375
|
||||||
|
putobject,96967168
|
||||||
|
putobject,100256148
|
||||||
|
putobject,107432215
|
||||||
|
putobject,102248459
|
||||||
|
putobject,251441345
|
||||||
|
putobject,104946713
|
||||||
|
putobject,110238211
|
||||||
|
putobject,108091689
|
||||||
|
putobject,113202108
|
||||||
|
putobject,98765973
|
||||||
|
putobject,104565383
|
||||||
|
putobject,111568023
|
||||||
|
putobject,112197090
|
||||||
|
putobject,105911994
|
||||||
|
putobject,95484988
|
||||||
|
putobject,104482906
|
||||||
|
putobject,234879244
|
||||||
|
putobject,113432122
|
||||||
|
putobject,100713850
|
||||||
|
putobject,108868215
|
||||||
|
putobject,103499519
|
||||||
|
putobject,104655812
|
||||||
|
putobject,115506648
|
||||||
|
putobject,101683821
|
||||||
|
putobject,108092129
|
||||||
|
putobject,101964070
|
||||||
|
putobject,106886313
|
||||||
|
putobject,112577538
|
||||||
|
putobject,261098453
|
||||||
|
putobject,113249625
|
||||||
|
putobject,108966089
|
||||||
|
putobject,109411069
|
||||||
|
putobject,104522489
|
||||||
|
putobject,110678803
|
||||||
|
putobject,105536735
|
||||||
|
putobject,110833494
|
||||||
|
putobject,112977212
|
||||||
|
putobject,98197749
|
||||||
|
putobject,110378043
|
||||||
|
putobject,375687474
|
||||||
|
putobject,203462479
|
||||||
|
putobject,115639116
|
||||||
|
putobject,118053054
|
||||||
|
putobject,110270019
|
||||||
|
putobject,109764686
|
||||||
|
putobject,90674345
|
||||||
|
putobject,98625115
|
||||||
|
putobject,103500960
|
||||||
|
putobject,102995627
|
||||||
|
putobject,236288420
|
||||||
|
putobject,107760181
|
||||||
|
putobject,112772765
|
||||||
|
putobject,99998427
|
||||||
|
putobject,117912217
|
||||||
|
putobject,105863219
|
||||||
|
putobject,113652925
|
||||||
|
putobject,100828684
|
||||||
|
putobject,95361573
|
||||||
|
putobject,95500334
|
||||||
|
putobject,106068286
|
||||||
|
putobject,99928444
|
||||||
|
putobject,261266283
|
||||||
|
putobject,106924492
|
||||||
|
putobject,110718855
|
||||||
|
putobject,87597876
|
||||||
|
putobject,108996183
|
||||||
|
putobject,99171595
|
||||||
|
putobject,115447052
|
||||||
|
putobject,103467696
|
||||||
|
putobject,104448307
|
||||||
|
putobject,97229488
|
||||||
|
putobject,106477027
|
||||||
|
putobject,105427947
|
||||||
|
putobject,233050123
|
||||||
|
putobject,115135303
|
||||||
|
putobject,113237921
|
||||||
|
putobject,110612835
|
||||||
|
putobject,102891788
|
||||||
|
putobject,112480159
|
||||||
|
putobject,117117775
|
||||||
|
putobject,196493186
|
||||||
|
putobject,104291470
|
||||||
|
putobject,118422258
|
||||||
|
putobject,104951716
|
||||||
|
putobject,244946550
|
||||||
|
putobject,98641292
|
||||||
|
putobject,119142327
|
||||||
|
listobjects,227618086
|
||||||
|
listobjects,218373723
|
||||||
|
listobjects,220425637
|
||||||
|
listobjects,212659074
|
||||||
|
listobjects,194860492
|
||||||
|
listobjects,221131339
|
||||||
|
listobjects,227735226
|
||||||
|
listobjects,212887826
|
||||||
|
listobjects,202906841
|
||||||
|
listobjects,221091271
|
||||||
|
listobjects,216883738
|
||||||
|
listobjects,213001179
|
||||||
|
listobjects,217273734
|
||||||
|
listobjects,211956305
|
||||||
|
listobjects,213529174
|
||||||
|
listobjects,375065283
|
||||||
|
listobjects,218843356
|
||||||
|
listobjects,217068015
|
||||||
|
listobjects,223369781
|
||||||
|
listobjects,219801804
|
||||||
|
listobjects,225260921
|
||||||
|
listobjects,207088252
|
||||||
|
listobjects,229780152
|
||||||
|
listobjects,227569658
|
||||||
|
listobjects,211492577
|
||||||
|
listobjects,200421226
|
||||||
|
listobjects,224666076
|
||||||
|
listobjects,217078671
|
||||||
|
listobjects,220658029
|
||||||
|
listobjects,220137146
|
||||||
|
listobjects,205746397
|
||||||
|
listobjects,222356274
|
||||||
|
listobjects,210329960
|
||||||
|
listobjects,219004606
|
||||||
|
listobjects,226442588
|
||||||
|
listobjects,374038605
|
||||||
|
listobjects,221425877
|
||||||
|
listobjects,226949625
|
||||||
|
listobjects,198272437
|
||||||
|
listobjects,215884986
|
||||||
|
listobjects,210904573
|
||||||
|
listobjects,231082876
|
||||||
|
listobjects,220581992
|
||||||
|
listobjects,215368028
|
||||||
|
listobjects,204236634
|
||||||
|
listobjects,230323641
|
||||||
|
listobjects,221006783
|
||||||
|
listobjects,197521967
|
||||||
|
listobjects,217673459
|
||||||
|
listobjects,210402664
|
||||||
|
listobjects,222289473
|
||||||
|
listobjects,216105460
|
||||||
|
listobjects,233017185
|
||||||
|
listobjects,217884686
|
||||||
|
listobjects,216886613
|
||||||
|
listobjects,215456984
|
||||||
|
listobjects,342657645
|
||||||
|
listobjects,218310540
|
||||||
|
listobjects,346101165
|
||||||
|
listobjects,227114600
|
||||||
|
listobjects,207167945
|
||||||
|
listobjects,217958955
|
||||||
|
listobjects,256687056
|
||||||
|
listobjects,216946794
|
||||||
|
listobjects,211024233
|
||||||
|
listobjects,221148255
|
||||||
|
listobjects,214229333
|
||||||
|
listobjects,224119805
|
||||||
|
listobjects,217875907
|
||||||
|
listobjects,216024912
|
||||||
|
listobjects,220679148
|
||||||
|
listobjects,218154613
|
||||||
|
listobjects,222423060
|
||||||
|
listobjects,212511440
|
||||||
|
listobjects,215068766
|
||||||
|
listobjects,207965636
|
||||||
|
listobjects,374287848
|
||||||
|
listobjects,231411797
|
||||||
|
listobjects,344053841
|
||||||
|
listobjects,218949902
|
||||||
|
listobjects,208085171
|
||||||
|
listobjects,202097107
|
||||||
|
listobjects,220625984
|
||||||
|
listobjects,232280185
|
||||||
|
listobjects,214265437
|
||||||
|
listobjects,216862289
|
||||||
|
listobjects,219434755
|
||||||
|
listobjects,217021028
|
||||||
|
listobjects,214401512
|
||||||
|
listobjects,207317610
|
||||||
|
listobjects,215392944
|
||||||
|
listobjects,217703786
|
||||||
|
listobjects,215988628
|
||||||
|
listobjects,198853351
|
||||||
|
listobjects,222486834
|
||||||
|
listobjects,213094404
|
||||||
|
listobjects,211948650
|
||||||
|
listobjects,215660888
|
||||||
|
listobjects,222729552
|
||||||
|
listobjects,357486752
|
||||||
|
getobject,120789563
|
||||||
|
getobject,257364122
|
||||||
|
getobject,193090188
|
||||||
|
getobject,105639883
|
||||||
|
getobject,110039709
|
||||||
|
getobject,106107405
|
||||||
|
getobject,113226765
|
||||||
|
getobject,228216027
|
||||||
|
getobject,204992307
|
||||||
|
getobject,115070175
|
||||||
|
getobject,118444544
|
||||||
|
getobject,116666047
|
||||||
|
getobject,114052724
|
||||||
|
getobject,109807911
|
||||||
|
getobject,108123550
|
||||||
|
getobject,245998795
|
||||||
|
getobject,103858529
|
||||||
|
getobject,104810357
|
||||||
|
getobject,101301352
|
||||||
|
getobject,104881290
|
||||||
|
getobject,101871689
|
||||||
|
getobject,245400756
|
||||||
|
getobject,99811659
|
||||||
|
getobject,107963659
|
||||||
|
getobject,108133779
|
||||||
|
getobject,109433413
|
||||||
|
getobject,112918863
|
||||||
|
getobject,99263685
|
||||||
|
getobject,109925392
|
||||||
|
getobject,404106741
|
||||||
|
getobject,111264038
|
||||||
|
getobject,107867546
|
||||||
|
getobject,232065851
|
||||||
|
getobject,104837346
|
||||||
|
getobject,249346993
|
||||||
|
getobject,107660264
|
||||||
|
getobject,98665144
|
||||||
|
getobject,93135947
|
||||||
|
getobject,93305867
|
||||||
|
getobject,103816208
|
||||||
|
getobject,101191343
|
||||||
|
getobject,104046332
|
||||||
|
getobject,234664977
|
||||||
|
getobject,109126683
|
||||||
|
getobject,109256458
|
||||||
|
getobject,111475958
|
||||||
|
getobject,106229192
|
||||||
|
getobject,111367102
|
||||||
|
getobject,247160403
|
||||||
|
getobject,114725281
|
||||||
|
getobject,115208863
|
||||||
|
getobject,115752038
|
||||||
|
getobject,99434436
|
||||||
|
getobject,114499775
|
||||||
|
getobject,114262136
|
||||||
|
getobject,237897437
|
||||||
|
getobject,109039858
|
||||||
|
getobject,116366349
|
||||||
|
getobject,109513421
|
||||||
|
getobject,119350461
|
||||||
|
getobject,103568549
|
||||||
|
getobject,268923073
|
||||||
|
getobject,103935562
|
||||||
|
getobject,114438288
|
||||||
|
getobject,250696155
|
||||||
|
getobject,100985713
|
||||||
|
getobject,108943735
|
||||||
|
getobject,107398809
|
||||||
|
getobject,201770787
|
||||||
|
getobject,258949074
|
||||||
|
getobject,108722357
|
||||||
|
getobject,106948669
|
||||||
|
getobject,105753903
|
||||||
|
getobject,106842969
|
||||||
|
getobject,254243109
|
||||||
|
getobject,109977076
|
||||||
|
getobject,101122461
|
||||||
|
getobject,106152776
|
||||||
|
getobject,105146877
|
||||||
|
getobject,115352192
|
||||||
|
getobject,114154611
|
||||||
|
getobject,106352783
|
||||||
|
getobject,316675402
|
||||||
|
getobject,109678912
|
||||||
|
getobject,105611654
|
||||||
|
getobject,111231964
|
||||||
|
getobject,110622814
|
||||||
|
getobject,103520055
|
||||||
|
getobject,264080226
|
||||||
|
getobject,115777874
|
||||||
|
getobject,117219506
|
||||||
|
getobject,237658128
|
||||||
|
getobject,105861315
|
||||||
|
getobject,118189958
|
||||||
|
getobject,250123569
|
||||||
|
getobject,210303951
|
||||||
|
getobject,114146153
|
||||||
|
getobject,114062677
|
||||||
|
getobject,116419134
|
||||||
|
getobject,110233217
|
||||||
|
removeobject,230846328
|
||||||
|
removeobject,352354358
|
||||||
|
removeobject,213513979
|
||||||
|
removeobject,213021069
|
||||||
|
removeobject,201490202
|
||||||
|
removeobject,212111001
|
||||||
|
removeobject,215118297
|
||||||
|
removeobject,206781539
|
||||||
|
removeobject,490175308
|
||||||
|
removeobject,207408693
|
||||||
|
removeobject,201921334
|
||||||
|
removeobject,214565432
|
||||||
|
removeobject,223499977
|
||||||
|
removeobject,219187737
|
||||||
|
removeobject,353799701
|
||||||
|
removeobject,226319108
|
||||||
|
removeobject,204468762
|
||||||
|
removeobject,206638858
|
||||||
|
removeobject,227752042
|
||||||
|
removeobject,227770838
|
||||||
|
removeobject,368061761
|
||||||
|
removeobject,304793510
|
||||||
|
removeobject,231980092
|
||||||
|
removeobject,227557765
|
||||||
|
removeobject,216285686
|
||||||
|
removeobject,204581431
|
||||||
|
removeobject,332004595
|
||||||
|
removeobject,209905432
|
||||||
|
removeobject,208660772
|
||||||
|
removeobject,298520566
|
||||||
|
removeobject,221629301
|
||||||
|
removeobject,216850310
|
||||||
|
removeobject,346809979
|
||||||
|
removeobject,225626686
|
||||||
|
removeobject,335040322
|
||||||
|
removeobject,217416498
|
||||||
|
removeobject,221590626
|
||||||
|
removeobject,211254506
|
||||||
|
removeobject,218186009
|
||||||
|
removeobject,333605972
|
||||||
|
removeobject,214970419
|
||||||
|
removeobject,400660245
|
||||||
|
removeobject,226176422
|
||||||
|
removeobject,210009344
|
||||||
|
removeobject,214122397
|
||||||
|
removeobject,217046195
|
||||||
|
removeobject,348420624
|
||||||
|
removeobject,222496374
|
||||||
|
removeobject,200770472
|
||||||
|
removeobject,206765238
|
||||||
|
removeobject,212797886
|
||||||
|
removeobject,221853199
|
||||||
|
removeobject,366803902
|
||||||
|
removeobject,369766134
|
||||||
|
removeobject,254126638
|
||||||
|
removeobject,206946940
|
||||||
|
removeobject,214755369
|
||||||
|
removeobject,222743167
|
||||||
|
removeobject,214028196
|
||||||
|
removeobject,365328074
|
||||||
|
removeobject,222478296
|
||||||
|
removeobject,200478669
|
||||||
|
removeobject,228313711
|
||||||
|
removeobject,351395352
|
||||||
|
removeobject,347788382
|
||||||
|
removeobject,370554938
|
||||||
|
removeobject,220125163
|
||||||
|
removeobject,211178604
|
||||||
|
removeobject,478779432
|
||||||
|
removeobject,226303414
|
||||||
|
removeobject,212846913
|
||||||
|
removeobject,232756403
|
||||||
|
removeobject,352451452
|
||||||
|
removeobject,204864755
|
||||||
|
removeobject,386939750
|
||||||
|
removeobject,236333214
|
||||||
|
removeobject,213864532
|
||||||
|
removeobject,337968992
|
||||||
|
removeobject,208224425
|
||||||
|
removeobject,215012709
|
||||||
|
removeobject,220231159
|
||||||
|
removeobject,200163319
|
||||||
|
removeobject,201449085
|
||||||
|
removeobject,203033356
|
||||||
|
removeobject,362988387
|
||||||
|
removeobject,352665831
|
||||||
|
removeobject,255105686
|
||||||
|
removeobject,209154819
|
||||||
|
removeobject,218832617
|
||||||
|
removeobject,220790763
|
||||||
|
removeobject,345915165
|
||||||
|
removeobject,213304301
|
||||||
|
removeobject,218302224
|
||||||
|
removeobject,218254023
|
||||||
|
removeobject,204621450
|
||||||
|
removeobject,213964266
|
||||||
|
removeobject,383714183
|
||||||
|
removeobject,312511671
|
||||||
|
removeobject,207629778
|
||||||
|
removeobject,203927189
|
||||||
|
501
artifacts/2022-09-23-s3lat/garage-v0.8.csv
Normal file
501
artifacts/2022-09-23-s3lat/garage-v0.8.csv
Normal file
|
|
@ -0,0 +1,501 @@
|
||||||
|
endpoint,nanoseconds
|
||||||
|
listbuckets,142443772
|
||||||
|
listbuckets,78416939
|
||||||
|
listbuckets,58182021
|
||||||
|
listbuckets,131770558
|
||||||
|
listbuckets,23167382
|
||||||
|
listbuckets,100239233
|
||||||
|
listbuckets,115131496
|
||||||
|
listbuckets,104406526
|
||||||
|
listbuckets,12172321
|
||||||
|
listbuckets,196757641
|
||||||
|
listbuckets,154690693
|
||||||
|
listbuckets,200549327
|
||||||
|
listbuckets,97233951
|
||||||
|
listbuckets,157416927
|
||||||
|
listbuckets,91145569
|
||||||
|
listbuckets,19144199
|
||||||
|
listbuckets,64085612
|
||||||
|
listbuckets,113225008
|
||||||
|
listbuckets,166909251
|
||||||
|
listbuckets,34198118
|
||||||
|
listbuckets,174705394
|
||||||
|
listbuckets,87295494
|
||||||
|
listbuckets,57706664
|
||||||
|
listbuckets,140442963
|
||||||
|
listbuckets,14837772
|
||||||
|
listbuckets,135331146
|
||||||
|
listbuckets,111062526
|
||||||
|
listbuckets,192533228
|
||||||
|
listbuckets,78240866
|
||||||
|
listbuckets,49209288
|
||||||
|
listbuckets,139333059
|
||||||
|
listbuckets,159433712
|
||||||
|
listbuckets,168386408
|
||||||
|
listbuckets,54060994
|
||||||
|
listbuckets,182381749
|
||||||
|
listbuckets,107235324
|
||||||
|
listbuckets,19175648
|
||||||
|
listbuckets,79408539
|
||||||
|
listbuckets,102182087
|
||||||
|
listbuckets,138252068
|
||||||
|
listbuckets,200286419
|
||||||
|
listbuckets,171574163
|
||||||
|
listbuckets,139244520
|
||||||
|
listbuckets,162405069
|
||||||
|
listbuckets,191507551
|
||||||
|
listbuckets,156309385
|
||||||
|
listbuckets,45147822
|
||||||
|
listbuckets,123534893
|
||||||
|
listbuckets,114180389
|
||||||
|
listbuckets,200631309
|
||||||
|
listbuckets,183352440
|
||||||
|
listbuckets,42285029
|
||||||
|
listbuckets,185592518
|
||||||
|
listbuckets,166551994
|
||||||
|
listbuckets,63313864
|
||||||
|
listbuckets,51172191
|
||||||
|
listbuckets,162406210
|
||||||
|
listbuckets,51252382
|
||||||
|
listbuckets,5096417
|
||||||
|
listbuckets,177549418
|
||||||
|
listbuckets,21073149
|
||||||
|
listbuckets,98302685
|
||||||
|
listbuckets,173578013
|
||||||
|
listbuckets,91105261
|
||||||
|
listbuckets,105040220
|
||||||
|
listbuckets,112304298
|
||||||
|
listbuckets,65084455
|
||||||
|
listbuckets,181875861
|
||||||
|
listbuckets,142442935
|
||||||
|
listbuckets,131125617
|
||||||
|
listbuckets,76286586
|
||||||
|
listbuckets,24552325
|
||||||
|
listbuckets,121486988
|
||||||
|
listbuckets,186709964
|
||||||
|
listbuckets,5948226
|
||||||
|
listbuckets,161299340
|
||||||
|
listbuckets,40993441
|
||||||
|
listbuckets,66113978
|
||||||
|
listbuckets,42309585
|
||||||
|
listbuckets,124003416
|
||||||
|
listbuckets,97993931
|
||||||
|
listbuckets,31175113
|
||||||
|
listbuckets,98603492
|
||||||
|
listbuckets,21409874
|
||||||
|
listbuckets,194297813
|
||||||
|
listbuckets,50059832
|
||||||
|
listbuckets,148399545
|
||||||
|
listbuckets,48887920
|
||||||
|
listbuckets,152334411
|
||||||
|
listbuckets,57077936
|
||||||
|
listbuckets,26810977
|
||||||
|
listbuckets,194375880
|
||||||
|
listbuckets,138228951
|
||||||
|
listbuckets,188415922
|
||||||
|
listbuckets,59004763
|
||||||
|
listbuckets,28981072
|
||||||
|
listbuckets,41073893
|
||||||
|
listbuckets,29776223
|
||||||
|
listbuckets,140987798
|
||||||
|
listbuckets,96201298
|
||||||
|
putobject,96916217
|
||||||
|
putobject,204175953
|
||||||
|
putobject,116377273
|
||||||
|
putobject,200815462
|
||||||
|
putobject,189393367
|
||||||
|
putobject,201157286
|
||||||
|
putobject,209378110
|
||||||
|
putobject,200861878
|
||||||
|
putobject,127598269
|
||||||
|
putobject,198228669
|
||||||
|
putobject,206087830
|
||||||
|
putobject,190252106
|
||||||
|
putobject,213164125
|
||||||
|
putobject,113273726
|
||||||
|
putobject,184844831
|
||||||
|
putobject,217834778
|
||||||
|
putobject,213577904
|
||||||
|
putobject,187314812
|
||||||
|
putobject,197335012
|
||||||
|
putobject,191781450
|
||||||
|
putobject,245923315
|
||||||
|
putobject,249794209
|
||||||
|
putobject,154268417
|
||||||
|
putobject,204865983
|
||||||
|
putobject,200616863
|
||||||
|
putobject,218347254
|
||||||
|
putobject,225218951
|
||||||
|
putobject,204122038
|
||||||
|
putobject,187483799
|
||||||
|
putobject,224081164
|
||||||
|
putobject,313480753
|
||||||
|
putobject,207639555
|
||||||
|
putobject,231879901
|
||||||
|
putobject,213694483
|
||||||
|
putobject,221029143
|
||||||
|
putobject,210575405
|
||||||
|
putobject,203991790
|
||||||
|
putobject,97757552
|
||||||
|
putobject,202949604
|
||||||
|
putobject,187003632
|
||||||
|
putobject,192703458
|
||||||
|
putobject,198853855
|
||||||
|
putobject,187677763
|
||||||
|
putobject,217450238
|
||||||
|
putobject,201097498
|
||||||
|
putobject,204520778
|
||||||
|
putobject,209010932
|
||||||
|
putobject,103217905
|
||||||
|
putobject,208237561
|
||||||
|
putobject,212482783
|
||||||
|
putobject,223855484
|
||||||
|
putobject,215215920
|
||||||
|
putobject,196011159
|
||||||
|
putobject,207887130
|
||||||
|
putobject,203901248
|
||||||
|
putobject,195527114
|
||||||
|
putobject,194314145
|
||||||
|
putobject,118554688
|
||||||
|
putobject,174577871
|
||||||
|
putobject,182678497
|
||||||
|
putobject,214551635
|
||||||
|
putobject,193939918
|
||||||
|
putobject,210241591
|
||||||
|
putobject,218407872
|
||||||
|
putobject,212235655
|
||||||
|
putobject,191045716
|
||||||
|
putobject,199345769
|
||||||
|
putobject,212445320
|
||||||
|
putobject,213398017
|
||||||
|
putobject,196443131
|
||||||
|
putobject,111006983
|
||||||
|
putobject,206131544
|
||||||
|
putobject,227743928
|
||||||
|
putobject,205929843
|
||||||
|
putobject,210773624
|
||||||
|
putobject,201460969
|
||||||
|
putobject,178288219
|
||||||
|
putobject,194333228
|
||||||
|
putobject,200422249
|
||||||
|
putobject,211491788
|
||||||
|
putobject,277570466
|
||||||
|
putobject,194155773
|
||||||
|
putobject,202859287
|
||||||
|
putobject,210188078
|
||||||
|
putobject,200376501
|
||||||
|
putobject,222446189
|
||||||
|
putobject,216905202
|
||||||
|
putobject,203768673
|
||||||
|
putobject,211882053
|
||||||
|
putobject,199813369
|
||||||
|
putobject,202275734
|
||||||
|
putobject,225392157
|
||||||
|
putobject,108514537
|
||||||
|
putobject,183571858
|
||||||
|
putobject,220178028
|
||||||
|
putobject,244245555
|
||||||
|
putobject,246644462
|
||||||
|
putobject,202754498
|
||||||
|
putobject,125491965
|
||||||
|
putobject,198043126
|
||||||
|
listobjects,247336346
|
||||||
|
listobjects,222421821
|
||||||
|
listobjects,217409432
|
||||||
|
listobjects,213489945
|
||||||
|
listobjects,197741076
|
||||||
|
listobjects,204144700
|
||||||
|
listobjects,194520187
|
||||||
|
listobjects,207039462
|
||||||
|
listobjects,214175739
|
||||||
|
listobjects,222630443
|
||||||
|
listobjects,207519367
|
||||||
|
listobjects,204852995
|
||||||
|
listobjects,200389171
|
||||||
|
listobjects,223342436
|
||||||
|
listobjects,375412313
|
||||||
|
listobjects,197168061
|
||||||
|
listobjects,209902283
|
||||||
|
listobjects,208728457
|
||||||
|
listobjects,223192111
|
||||||
|
listobjects,197001637
|
||||||
|
listobjects,210692202
|
||||||
|
listobjects,210082361
|
||||||
|
listobjects,193242654
|
||||||
|
listobjects,190713450
|
||||||
|
listobjects,210137746
|
||||||
|
listobjects,210096607
|
||||||
|
listobjects,214155697
|
||||||
|
listobjects,203814873
|
||||||
|
listobjects,200252782
|
||||||
|
listobjects,198504662
|
||||||
|
listobjects,225529670
|
||||||
|
listobjects,219412867
|
||||||
|
listobjects,200873322
|
||||||
|
listobjects,366185537
|
||||||
|
listobjects,220358802
|
||||||
|
listobjects,203736154
|
||||||
|
listobjects,195030295
|
||||||
|
listobjects,206763616
|
||||||
|
listobjects,215264828
|
||||||
|
listobjects,196677593
|
||||||
|
listobjects,206296625
|
||||||
|
listobjects,203169834
|
||||||
|
listobjects,228228451
|
||||||
|
listobjects,201871694
|
||||||
|
listobjects,214160805
|
||||||
|
listobjects,221120508
|
||||||
|
listobjects,204784600
|
||||||
|
listobjects,215667356
|
||||||
|
listobjects,203478756
|
||||||
|
listobjects,193957098
|
||||||
|
listobjects,204028493
|
||||||
|
listobjects,200296721
|
||||||
|
listobjects,201366590
|
||||||
|
listobjects,217724938
|
||||||
|
listobjects,335372225
|
||||||
|
listobjects,212399058
|
||||||
|
listobjects,206802897
|
||||||
|
listobjects,203512327
|
||||||
|
listobjects,201467970
|
||||||
|
listobjects,207952516
|
||||||
|
listobjects,200169128
|
||||||
|
listobjects,193368365
|
||||||
|
listobjects,208873233
|
||||||
|
listobjects,206348419
|
||||||
|
listobjects,202684585
|
||||||
|
listobjects,255082511
|
||||||
|
listobjects,225277839
|
||||||
|
listobjects,216147871
|
||||||
|
listobjects,203396588
|
||||||
|
listobjects,215594185
|
||||||
|
listobjects,211893541
|
||||||
|
listobjects,202693570
|
||||||
|
listobjects,213428668
|
||||||
|
listobjects,345450849
|
||||||
|
listobjects,210503146
|
||||||
|
listobjects,206714788
|
||||||
|
listobjects,221670971
|
||||||
|
listobjects,209193505
|
||||||
|
listobjects,207268362
|
||||||
|
listobjects,217422133
|
||||||
|
listobjects,194262801
|
||||||
|
listobjects,214257460
|
||||||
|
listobjects,209999916
|
||||||
|
listobjects,200992698
|
||||||
|
listobjects,233030646
|
||||||
|
listobjects,197622419
|
||||||
|
listobjects,214531387
|
||||||
|
listobjects,207687472
|
||||||
|
listobjects,193650794
|
||||||
|
listobjects,216352563
|
||||||
|
listobjects,204419966
|
||||||
|
listobjects,220824902
|
||||||
|
listobjects,203824031
|
||||||
|
listobjects,345556923
|
||||||
|
listobjects,211586088
|
||||||
|
listobjects,221054525
|
||||||
|
listobjects,216966641
|
||||||
|
listobjects,212481105
|
||||||
|
listobjects,225077725
|
||||||
|
listobjects,206661945
|
||||||
|
getobject,106148474
|
||||||
|
getobject,103474999
|
||||||
|
getobject,110847021
|
||||||
|
getobject,89406401
|
||||||
|
getobject,107194067
|
||||||
|
getobject,103967969
|
||||||
|
getobject,99809582
|
||||||
|
getobject,218870171
|
||||||
|
getobject,116458559
|
||||||
|
getobject,116438822
|
||||||
|
getobject,114486708
|
||||||
|
getobject,98523284
|
||||||
|
getobject,107932449
|
||||||
|
getobject,96947000
|
||||||
|
getobject,111244400
|
||||||
|
getobject,107520933
|
||||||
|
getobject,96707408
|
||||||
|
getobject,98561777
|
||||||
|
getobject,100203555
|
||||||
|
getobject,102302806
|
||||||
|
getobject,238528526
|
||||||
|
getobject,271793475
|
||||||
|
getobject,103513430
|
||||||
|
getobject,110508380
|
||||||
|
getobject,88297618
|
||||||
|
getobject,93017464
|
||||||
|
getobject,92182439
|
||||||
|
getobject,100926740
|
||||||
|
getobject,90912953
|
||||||
|
getobject,89262659
|
||||||
|
getobject,106320567
|
||||||
|
getobject,114727932
|
||||||
|
getobject,93467784
|
||||||
|
getobject,233897134
|
||||||
|
getobject,99598463
|
||||||
|
getobject,100682658
|
||||||
|
getobject,196532397
|
||||||
|
getobject,114136005
|
||||||
|
getobject,107149842
|
||||||
|
getobject,102095284
|
||||||
|
getobject,101561076
|
||||||
|
getobject,110858641
|
||||||
|
getobject,112464019
|
||||||
|
getobject,89475711
|
||||||
|
getobject,104275969
|
||||||
|
getobject,98529866
|
||||||
|
getobject,241428685
|
||||||
|
getobject,101153106
|
||||||
|
getobject,119006084
|
||||||
|
getobject,89105802
|
||||||
|
getobject,95722098
|
||||||
|
getobject,267246924
|
||||||
|
getobject,106170312
|
||||||
|
getobject,106624860
|
||||||
|
getobject,101451128
|
||||||
|
getobject,114344067
|
||||||
|
getobject,98506251
|
||||||
|
getobject,85363560
|
||||||
|
getobject,109668504
|
||||||
|
getobject,97277512
|
||||||
|
getobject,217166462
|
||||||
|
getobject,95365244
|
||||||
|
getobject,105298136
|
||||||
|
getobject,107433406
|
||||||
|
getobject,91927105
|
||||||
|
getobject,103556189
|
||||||
|
getobject,101882261
|
||||||
|
getobject,97490353
|
||||||
|
getobject,109313364
|
||||||
|
getobject,107477889
|
||||||
|
getobject,105115091
|
||||||
|
getobject,100009677
|
||||||
|
getobject,96802735
|
||||||
|
getobject,243376778
|
||||||
|
getobject,116897654
|
||||||
|
getobject,246725949
|
||||||
|
getobject,104940522
|
||||||
|
getobject,104899223
|
||||||
|
getobject,99914467
|
||||||
|
getobject,101929660
|
||||||
|
getobject,96935645
|
||||||
|
getobject,107399321
|
||||||
|
getobject,94553040
|
||||||
|
getobject,110316235
|
||||||
|
getobject,112920198
|
||||||
|
getobject,100314041
|
||||||
|
getobject,239311246
|
||||||
|
getobject,114578006
|
||||||
|
getobject,106688650
|
||||||
|
getobject,89102575
|
||||||
|
getobject,82537327
|
||||||
|
getobject,102124717
|
||||||
|
getobject,107306736
|
||||||
|
getobject,113530840
|
||||||
|
getobject,102225458
|
||||||
|
getobject,105090914
|
||||||
|
getobject,109987193
|
||||||
|
getobject,107637409
|
||||||
|
getobject,99663933
|
||||||
|
getobject,252948739
|
||||||
|
removeobject,218637514
|
||||||
|
removeobject,459986423
|
||||||
|
removeobject,307487771
|
||||||
|
removeobject,425595066
|
||||||
|
removeobject,325293018
|
||||||
|
removeobject,326633357
|
||||||
|
removeobject,490052955
|
||||||
|
removeobject,252844239
|
||||||
|
removeobject,313024066
|
||||||
|
removeobject,301593436
|
||||||
|
removeobject,299154213
|
||||||
|
removeobject,396128610
|
||||||
|
removeobject,227674663
|
||||||
|
removeobject,439796598
|
||||||
|
removeobject,464313791
|
||||||
|
removeobject,433487342
|
||||||
|
removeobject,316319351
|
||||||
|
removeobject,324010444
|
||||||
|
removeobject,224091911
|
||||||
|
removeobject,259658834
|
||||||
|
removeobject,268735522
|
||||||
|
removeobject,466588289
|
||||||
|
removeobject,328431494
|
||||||
|
removeobject,456136096
|
||||||
|
removeobject,214305954
|
||||||
|
removeobject,349200623
|
||||||
|
removeobject,310132173
|
||||||
|
removeobject,433264807
|
||||||
|
removeobject,438724929
|
||||||
|
removeobject,333912094
|
||||||
|
removeobject,443030552
|
||||||
|
removeobject,338569189
|
||||||
|
removeobject,452852014
|
||||||
|
removeobject,450187315
|
||||||
|
removeobject,407122424
|
||||||
|
removeobject,371514674
|
||||||
|
removeobject,371585948
|
||||||
|
removeobject,327859889
|
||||||
|
removeobject,287348277
|
||||||
|
removeobject,319214105
|
||||||
|
removeobject,408538253
|
||||||
|
removeobject,315233492
|
||||||
|
removeobject,321187307
|
||||||
|
removeobject,308503342
|
||||||
|
removeobject,481438858
|
||||||
|
removeobject,464326677
|
||||||
|
removeobject,458208213
|
||||||
|
removeobject,334169040
|
||||||
|
removeobject,380953007
|
||||||
|
removeobject,306156310
|
||||||
|
removeobject,311735117
|
||||||
|
removeobject,304652713
|
||||||
|
removeobject,447482399
|
||||||
|
removeobject,464054872
|
||||||
|
removeobject,308287070
|
||||||
|
removeobject,366031357
|
||||||
|
removeobject,223529039
|
||||||
|
removeobject,305332454
|
||||||
|
removeobject,440680123
|
||||||
|
removeobject,283803681
|
||||||
|
removeobject,410491748
|
||||||
|
removeobject,223265289
|
||||||
|
removeobject,306268419
|
||||||
|
removeobject,347366809
|
||||||
|
removeobject,314563701
|
||||||
|
removeobject,302068212
|
||||||
|
removeobject,283520375
|
||||||
|
removeobject,377963268
|
||||||
|
removeobject,454052170
|
||||||
|
removeobject,298085627
|
||||||
|
removeobject,457265001
|
||||||
|
removeobject,233934260
|
||||||
|
removeobject,432713854
|
||||||
|
removeobject,315891533
|
||||||
|
removeobject,445005587
|
||||||
|
removeobject,321635590
|
||||||
|
removeobject,390314770
|
||||||
|
removeobject,230655210
|
||||||
|
removeobject,340190850
|
||||||
|
removeobject,305314423
|
||||||
|
removeobject,444872212
|
||||||
|
removeobject,223437549
|
||||||
|
removeobject,385529268
|
||||||
|
removeobject,280271669
|
||||||
|
removeobject,323868430
|
||||||
|
removeobject,307544418
|
||||||
|
removeobject,238040604
|
||||||
|
removeobject,310293395
|
||||||
|
removeobject,412163151
|
||||||
|
removeobject,367516076
|
||||||
|
removeobject,288358726
|
||||||
|
removeobject,303789421
|
||||||
|
removeobject,435084818
|
||||||
|
removeobject,411008890
|
||||||
|
removeobject,467616201
|
||||||
|
removeobject,219465170
|
||||||
|
removeobject,327976568
|
||||||
|
removeobject,332048651
|
||||||
|
removeobject,353130028
|
||||||
|
removeobject,329197320
|
||||||
|
501
artifacts/2022-09-23-s3lat/minio-20220917.csv
Normal file
501
artifacts/2022-09-23-s3lat/minio-20220917.csv
Normal file
|
|
@ -0,0 +1,501 @@
|
||||||
|
endpoint,nanoseconds
|
||||||
|
listbuckets,110526156
|
||||||
|
listbuckets,111223674
|
||||||
|
listbuckets,110350078
|
||||||
|
listbuckets,108763355
|
||||||
|
listbuckets,108151200
|
||||||
|
listbuckets,110049558
|
||||||
|
listbuckets,111261740
|
||||||
|
listbuckets,106870917
|
||||||
|
listbuckets,109675307
|
||||||
|
listbuckets,114928865
|
||||||
|
listbuckets,112976189
|
||||||
|
listbuckets,100144534
|
||||||
|
listbuckets,103672802
|
||||||
|
listbuckets,115743698
|
||||||
|
listbuckets,120312198
|
||||||
|
listbuckets,114315912
|
||||||
|
listbuckets,117664396
|
||||||
|
listbuckets,113756128
|
||||||
|
listbuckets,110819985
|
||||||
|
listbuckets,108017540
|
||||||
|
listbuckets,107455412
|
||||||
|
listbuckets,117667963
|
||||||
|
listbuckets,113112343
|
||||||
|
listbuckets,117163158
|
||||||
|
listbuckets,111710827
|
||||||
|
listbuckets,108587376
|
||||||
|
listbuckets,113677870
|
||||||
|
listbuckets,100432934
|
||||||
|
listbuckets,111720496
|
||||||
|
listbuckets,113415267
|
||||||
|
listbuckets,109700442
|
||||||
|
listbuckets,116870615
|
||||||
|
listbuckets,104193832
|
||||||
|
listbuckets,104911949
|
||||||
|
listbuckets,107990314
|
||||||
|
listbuckets,111855197
|
||||||
|
listbuckets,108076368
|
||||||
|
listbuckets,109569833
|
||||||
|
listbuckets,110641629
|
||||||
|
listbuckets,105085431
|
||||||
|
listbuckets,113240314
|
||||||
|
listbuckets,109189490
|
||||||
|
listbuckets,119818535
|
||||||
|
listbuckets,111259357
|
||||||
|
listbuckets,107390627
|
||||||
|
listbuckets,101319090
|
||||||
|
listbuckets,106351333
|
||||||
|
listbuckets,117073689
|
||||||
|
listbuckets,119127359
|
||||||
|
listbuckets,117266560
|
||||||
|
listbuckets,110077426
|
||||||
|
listbuckets,107452728
|
||||||
|
listbuckets,112721966
|
||||||
|
listbuckets,114883973
|
||||||
|
listbuckets,120073375
|
||||||
|
listbuckets,103458296
|
||||||
|
listbuckets,118342265
|
||||||
|
listbuckets,107996502
|
||||||
|
listbuckets,109767879
|
||||||
|
listbuckets,112580238
|
||||||
|
listbuckets,113636111
|
||||||
|
listbuckets,111267829
|
||||||
|
listbuckets,112798154
|
||||||
|
listbuckets,107265512
|
||||||
|
listbuckets,108961006
|
||||||
|
listbuckets,120928218
|
||||||
|
listbuckets,109095182
|
||||||
|
listbuckets,110399063
|
||||||
|
listbuckets,118328765
|
||||||
|
listbuckets,112884542
|
||||||
|
listbuckets,117913147
|
||||||
|
listbuckets,120238848
|
||||||
|
listbuckets,116074701
|
||||||
|
listbuckets,109990943
|
||||||
|
listbuckets,114816077
|
||||||
|
listbuckets,113207212
|
||||||
|
listbuckets,112350518
|
||||||
|
listbuckets,106189705
|
||||||
|
listbuckets,106836706
|
||||||
|
listbuckets,114519552
|
||||||
|
listbuckets,112357584
|
||||||
|
listbuckets,113073218
|
||||||
|
listbuckets,108251780
|
||||||
|
listbuckets,115340245
|
||||||
|
listbuckets,109100358
|
||||||
|
listbuckets,116337818
|
||||||
|
listbuckets,110881217
|
||||||
|
listbuckets,109903972
|
||||||
|
listbuckets,113243376
|
||||||
|
listbuckets,111667494
|
||||||
|
listbuckets,102401925
|
||||||
|
listbuckets,109793240
|
||||||
|
listbuckets,118365209
|
||||||
|
listbuckets,117380902
|
||||||
|
listbuckets,116149538
|
||||||
|
listbuckets,109060437
|
||||||
|
listbuckets,106900385
|
||||||
|
listbuckets,101213710
|
||||||
|
listbuckets,103102282
|
||||||
|
listbuckets,115980177
|
||||||
|
putobject,1181582068
|
||||||
|
putobject,1208319832
|
||||||
|
putobject,1139007207
|
||||||
|
putobject,1172748692
|
||||||
|
putobject,1213269319
|
||||||
|
putobject,1222148262
|
||||||
|
putobject,1160354410
|
||||||
|
putobject,1186840670
|
||||||
|
putobject,1178184375
|
||||||
|
putobject,1199772934
|
||||||
|
putobject,1243156199
|
||||||
|
putobject,1162462450
|
||||||
|
putobject,1160912226
|
||||||
|
putobject,1181903350
|
||||||
|
putobject,1178461918
|
||||||
|
putobject,1155923039
|
||||||
|
putobject,1198486430
|
||||||
|
putobject,1128172593
|
||||||
|
putobject,1161827240
|
||||||
|
putobject,1191934325
|
||||||
|
putobject,1241903028
|
||||||
|
putobject,1154793146
|
||||||
|
putobject,1182964786
|
||||||
|
putobject,1192949284
|
||||||
|
putobject,1211748830
|
||||||
|
putobject,1153969786
|
||||||
|
putobject,1173133011
|
||||||
|
putobject,1159925140
|
||||||
|
putobject,1151781019
|
||||||
|
putobject,1212786711
|
||||||
|
putobject,1136411443
|
||||||
|
putobject,1161607447
|
||||||
|
putobject,1194069897
|
||||||
|
putobject,1181340894
|
||||||
|
putobject,1154251172
|
||||||
|
putobject,1142482758
|
||||||
|
putobject,1184635609
|
||||||
|
putobject,1401096568
|
||||||
|
putobject,1193309386
|
||||||
|
putobject,1155633130
|
||||||
|
putobject,1185629277
|
||||||
|
putobject,1150317409
|
||||||
|
putobject,1189977499
|
||||||
|
putobject,1172978955
|
||||||
|
putobject,1167393215
|
||||||
|
putobject,1176612406
|
||||||
|
putobject,1220478477
|
||||||
|
putobject,1214146462
|
||||||
|
putobject,1201982436
|
||||||
|
putobject,1187568105
|
||||||
|
putobject,1233590518
|
||||||
|
putobject,1254380879
|
||||||
|
putobject,1195264797
|
||||||
|
putobject,1212347722
|
||||||
|
putobject,1265459581
|
||||||
|
putobject,1199507629
|
||||||
|
putobject,1213030838
|
||||||
|
putobject,1261491944
|
||||||
|
putobject,1175783260
|
||||||
|
putobject,1171776028
|
||||||
|
putobject,1175985781
|
||||||
|
putobject,1204880209
|
||||||
|
putobject,1185762634
|
||||||
|
putobject,1205618309
|
||||||
|
putobject,1156324684
|
||||||
|
putobject,1212986779
|
||||||
|
putobject,1153785732
|
||||||
|
putobject,1186241798
|
||||||
|
putobject,1169988181
|
||||||
|
putobject,1169236282
|
||||||
|
putobject,1159114879
|
||||||
|
putobject,1169866154
|
||||||
|
putobject,1185878817
|
||||||
|
putobject,1303357322
|
||||||
|
putobject,1198903604
|
||||||
|
putobject,1180614532
|
||||||
|
putobject,1198891504
|
||||||
|
putobject,1186606007
|
||||||
|
putobject,1178912791
|
||||||
|
putobject,1181214681
|
||||||
|
putobject,1148254776
|
||||||
|
putobject,1193521548
|
||||||
|
putobject,1172732700
|
||||||
|
putobject,1224354922
|
||||||
|
putobject,1189768028
|
||||||
|
putobject,1187610090
|
||||||
|
putobject,1158181354
|
||||||
|
putobject,1168240861
|
||||||
|
putobject,1184266557
|
||||||
|
putobject,1211834285
|
||||||
|
putobject,1133902391
|
||||||
|
putobject,1169623406
|
||||||
|
putobject,1153648781
|
||||||
|
putobject,1260717643
|
||||||
|
putobject,1213147555
|
||||||
|
putobject,1196178431
|
||||||
|
putobject,1193078015
|
||||||
|
putobject,1192743109
|
||||||
|
putobject,1231998132
|
||||||
|
putobject,1169290675
|
||||||
|
listobjects,445829270
|
||||||
|
listobjects,448358224
|
||||||
|
listobjects,234487499
|
||||||
|
listobjects,331279803
|
||||||
|
listobjects,239147274
|
||||||
|
listobjects,356161983
|
||||||
|
listobjects,231056963
|
||||||
|
listobjects,428733347
|
||||||
|
listobjects,234973895
|
||||||
|
listobjects,349647146
|
||||||
|
listobjects,238785967
|
||||||
|
listobjects,430581787
|
||||||
|
listobjects,325690302
|
||||||
|
listobjects,337483670
|
||||||
|
listobjects,335639646
|
||||||
|
listobjects,327396437
|
||||||
|
listobjects,230177176
|
||||||
|
listobjects,347284630
|
||||||
|
listobjects,230553012
|
||||||
|
listobjects,338821210
|
||||||
|
listobjects,343466400
|
||||||
|
listobjects,346533869
|
||||||
|
listobjects,311374291
|
||||||
|
listobjects,341162492
|
||||||
|
listobjects,235683323
|
||||||
|
listobjects,356305631
|
||||||
|
listobjects,234888137
|
||||||
|
listobjects,335916950
|
||||||
|
listobjects,312208053
|
||||||
|
listobjects,345090441
|
||||||
|
listobjects,449596366
|
||||||
|
listobjects,347544888
|
||||||
|
listobjects,361989101
|
||||||
|
listobjects,235502161
|
||||||
|
listobjects,232949965
|
||||||
|
listobjects,337218790
|
||||||
|
listobjects,447643000
|
||||||
|
listobjects,238259618
|
||||||
|
listobjects,235023776
|
||||||
|
listobjects,420231514
|
||||||
|
listobjects,341833697
|
||||||
|
listobjects,235388450
|
||||||
|
listobjects,232164722
|
||||||
|
listobjects,422949063
|
||||||
|
listobjects,501989025
|
||||||
|
listobjects,240957275
|
||||||
|
listobjects,337846216
|
||||||
|
listobjects,231434805
|
||||||
|
listobjects,430682948
|
||||||
|
listobjects,231812947
|
||||||
|
listobjects,465260992
|
||||||
|
listobjects,433379275
|
||||||
|
listobjects,236331095
|
||||||
|
listobjects,332051421
|
||||||
|
listobjects,237905130
|
||||||
|
listobjects,343574528
|
||||||
|
listobjects,242371130
|
||||||
|
listobjects,311241850
|
||||||
|
listobjects,239692903
|
||||||
|
listobjects,360069478
|
||||||
|
listobjects,317977063
|
||||||
|
listobjects,324532475
|
||||||
|
listobjects,246746304
|
||||||
|
listobjects,339939390
|
||||||
|
listobjects,238956624
|
||||||
|
listobjects,338334961
|
||||||
|
listobjects,236968116
|
||||||
|
listobjects,338529503
|
||||||
|
listobjects,235480073
|
||||||
|
listobjects,334150022
|
||||||
|
listobjects,234014534
|
||||||
|
listobjects,336955875
|
||||||
|
listobjects,245541026
|
||||||
|
listobjects,321342105
|
||||||
|
listobjects,241862995
|
||||||
|
listobjects,342284075
|
||||||
|
listobjects,223874385
|
||||||
|
listobjects,330173393
|
||||||
|
listobjects,230710360
|
||||||
|
listobjects,338012760
|
||||||
|
listobjects,243166839
|
||||||
|
listobjects,326244217
|
||||||
|
listobjects,217795705
|
||||||
|
listobjects,343998726
|
||||||
|
listobjects,234484396
|
||||||
|
listobjects,326419671
|
||||||
|
listobjects,228413611
|
||||||
|
listobjects,345526317
|
||||||
|
listobjects,242498473
|
||||||
|
listobjects,322335836
|
||||||
|
listobjects,325652509
|
||||||
|
listobjects,353395591
|
||||||
|
listobjects,241202612
|
||||||
|
listobjects,316928525
|
||||||
|
listobjects,415736209
|
||||||
|
listobjects,336640460
|
||||||
|
listobjects,235664816
|
||||||
|
listobjects,318470444
|
||||||
|
listobjects,233137178
|
||||||
|
listobjects,338995281
|
||||||
|
getobject,548766727
|
||||||
|
getobject,544293094
|
||||||
|
getobject,553996833
|
||||||
|
getobject,524118176
|
||||||
|
getobject,540256688
|
||||||
|
getobject,521985460
|
||||||
|
getobject,550010163
|
||||||
|
getobject,522839295
|
||||||
|
getobject,550206788
|
||||||
|
getobject,556115384
|
||||||
|
getobject,561480120
|
||||||
|
getobject,529867231
|
||||||
|
getobject,554654221
|
||||||
|
getobject,522713545
|
||||||
|
getobject,533201280
|
||||||
|
getobject,559755253
|
||||||
|
getobject,562719587
|
||||||
|
getobject,530367590
|
||||||
|
getobject,534634166
|
||||||
|
getobject,528476593
|
||||||
|
getobject,561330984
|
||||||
|
getobject,548154277
|
||||||
|
getobject,551287855
|
||||||
|
getobject,543519973
|
||||||
|
getobject,537798946
|
||||||
|
getobject,543821201
|
||||||
|
getobject,556706828
|
||||||
|
getobject,562779335
|
||||||
|
getobject,533088136
|
||||||
|
getobject,543496053
|
||||||
|
getobject,530090638
|
||||||
|
getobject,557966646
|
||||||
|
getobject,546652328
|
||||||
|
getobject,556049330
|
||||||
|
getobject,553087013
|
||||||
|
getobject,552319581
|
||||||
|
getobject,538039579
|
||||||
|
getobject,524047395
|
||||||
|
getobject,529321927
|
||||||
|
getobject,561628538
|
||||||
|
getobject,538975026
|
||||||
|
getobject,537678052
|
||||||
|
getobject,552261757
|
||||||
|
getobject,553552417
|
||||||
|
getobject,546538519
|
||||||
|
getobject,539953289
|
||||||
|
getobject,533392613
|
||||||
|
getobject,543443065
|
||||||
|
getobject,539024543
|
||||||
|
getobject,537067523
|
||||||
|
getobject,555793303
|
||||||
|
getobject,610983302
|
||||||
|
getobject,535123120
|
||||||
|
getobject,528049669
|
||||||
|
getobject,541554744
|
||||||
|
getobject,564902500
|
||||||
|
getobject,564639999
|
||||||
|
getobject,534295582
|
||||||
|
getobject,544951246
|
||||||
|
getobject,540628827
|
||||||
|
getobject,552900208
|
||||||
|
getobject,541614941
|
||||||
|
getobject,536825865
|
||||||
|
getobject,535642848
|
||||||
|
getobject,544302224
|
||||||
|
getobject,549999591
|
||||||
|
getobject,542030234
|
||||||
|
getobject,545602568
|
||||||
|
getobject,543278126
|
||||||
|
getobject,524720136
|
||||||
|
getobject,552230698
|
||||||
|
getobject,539300237
|
||||||
|
getobject,549856120
|
||||||
|
getobject,540587629
|
||||||
|
getobject,557880339
|
||||||
|
getobject,555153802
|
||||||
|
getobject,561113528
|
||||||
|
getobject,556108255
|
||||||
|
getobject,508374452
|
||||||
|
getobject,560044395
|
||||||
|
getobject,585922680
|
||||||
|
getobject,548465899
|
||||||
|
getobject,573308704
|
||||||
|
getobject,554195559
|
||||||
|
getobject,562441181
|
||||||
|
getobject,550048971
|
||||||
|
getobject,542032477
|
||||||
|
getobject,587508303
|
||||||
|
getobject,548308208
|
||||||
|
getobject,529401594
|
||||||
|
getobject,535337894
|
||||||
|
getobject,566348968
|
||||||
|
getobject,545058294
|
||||||
|
getobject,560623443
|
||||||
|
getobject,532580429
|
||||||
|
getobject,555774911
|
||||||
|
getobject,546448443
|
||||||
|
getobject,526984769
|
||||||
|
getobject,528538465
|
||||||
|
getobject,551758307
|
||||||
|
removeobject,1305475677
|
||||||
|
removeobject,1328695106
|
||||||
|
removeobject,1304086325
|
||||||
|
removeobject,1314629714
|
||||||
|
removeobject,1286472737
|
||||||
|
removeobject,1323794460
|
||||||
|
removeobject,1287026749
|
||||||
|
removeobject,1308989986
|
||||||
|
removeobject,1319191249
|
||||||
|
removeobject,1302459188
|
||||||
|
removeobject,1358027169
|
||||||
|
removeobject,1298833331
|
||||||
|
removeobject,1288044760
|
||||||
|
removeobject,1309860548
|
||||||
|
removeobject,1294431451
|
||||||
|
removeobject,1323518263
|
||||||
|
removeobject,1274537859
|
||||||
|
removeobject,1306292281
|
||||||
|
removeobject,1300283360
|
||||||
|
removeobject,1236080169
|
||||||
|
removeobject,1304088805
|
||||||
|
removeobject,1319360876
|
||||||
|
removeobject,1310348416
|
||||||
|
removeobject,1306714462
|
||||||
|
removeobject,1280353702
|
||||||
|
removeobject,1313350545
|
||||||
|
removeobject,1321192365
|
||||||
|
removeobject,1304475647
|
||||||
|
removeobject,1306586102
|
||||||
|
removeobject,1294987618
|
||||||
|
removeobject,1304176193
|
||||||
|
removeobject,1296005363
|
||||||
|
removeobject,1284994372
|
||||||
|
removeobject,1317600899
|
||||||
|
removeobject,1332743805
|
||||||
|
removeobject,1352245087
|
||||||
|
removeobject,1309279342
|
||||||
|
removeobject,1307620458
|
||||||
|
removeobject,1342551633
|
||||||
|
removeobject,1321581956
|
||||||
|
removeobject,1315162697
|
||||||
|
removeobject,1336302765
|
||||||
|
removeobject,1290969146
|
||||||
|
removeobject,1320968235
|
||||||
|
removeobject,1299056796
|
||||||
|
removeobject,1288002161
|
||||||
|
removeobject,1328125560
|
||||||
|
removeobject,1316703843
|
||||||
|
removeobject,1291268641
|
||||||
|
removeobject,1327821693
|
||||||
|
removeobject,1299867445
|
||||||
|
removeobject,1296882414
|
||||||
|
removeobject,1313288813
|
||||||
|
removeobject,1298520791
|
||||||
|
removeobject,1298185028
|
||||||
|
removeobject,1274221533
|
||||||
|
removeobject,1314194728
|
||||||
|
removeobject,1336907835
|
||||||
|
removeobject,1297413534
|
||||||
|
removeobject,1283613694
|
||||||
|
removeobject,1265657961
|
||||||
|
removeobject,1329591094
|
||||||
|
removeobject,1320682432
|
||||||
|
removeobject,1317595366
|
||||||
|
removeobject,1323562128
|
||||||
|
removeobject,1298197589
|
||||||
|
removeobject,1321950097
|
||||||
|
removeobject,1281669970
|
||||||
|
removeobject,1306586515
|
||||||
|
removeobject,1296748831
|
||||||
|
removeobject,1300070529
|
||||||
|
removeobject,1282255502
|
||||||
|
removeobject,1284141215
|
||||||
|
removeobject,1297879969
|
||||||
|
removeobject,1292837351
|
||||||
|
removeobject,1317095287
|
||||||
|
removeobject,1323647829
|
||||||
|
removeobject,1283782591
|
||||||
|
removeobject,1292866116
|
||||||
|
removeobject,1283574839
|
||||||
|
removeobject,1331580518
|
||||||
|
removeobject,1355362975
|
||||||
|
removeobject,1326405684
|
||||||
|
removeobject,1297225883
|
||||||
|
removeobject,1318610920
|
||||||
|
removeobject,1298833112
|
||||||
|
removeobject,1360833965
|
||||||
|
removeobject,1311610613
|
||||||
|
removeobject,1304567017
|
||||||
|
removeobject,1325168434
|
||||||
|
removeobject,1293403789
|
||||||
|
removeobject,1293213442
|
||||||
|
removeobject,1298548367
|
||||||
|
removeobject,1303459493
|
||||||
|
removeobject,1302979008
|
||||||
|
removeobject,1336663252
|
||||||
|
removeobject,1277657362
|
||||||
|
removeobject,1308832733
|
||||||
|
removeobject,1342049822
|
||||||
|
removeobject,1307255424
|
||||||
|
30
artifacts/2022-09-23-s3lat/plot.R
Normal file
30
artifacts/2022-09-23-s3lat/plot.R
Normal file
|
|
@ -0,0 +1,30 @@
|
||||||
|
library(tidyverse)
|
||||||
|
|
||||||
|
read_csv("garage-v0.7.csv") %>% add_column(daemon="garage 0.7.3") -> a
|
||||||
|
read_csv("garage-v0.8.csv") %>% add_column(daemon="garage 0.8.0 beta") -> b
|
||||||
|
read_csv("minio-20220917.csv") %>% add_column(daemon="minio RELEASE.2022-09-17") -> c
|
||||||
|
bind_rows(a,b,c) %>% group_by(daemon,endpoint) %>% summarise(
|
||||||
|
time_mean = mean(nanoseconds) / 1000 / 1000,
|
||||||
|
time_max = max(nanoseconds) / 1000 / 1000,
|
||||||
|
time_min = min(nanoseconds) / 1000 / 1000
|
||||||
|
) -> c
|
||||||
|
|
||||||
|
st <- "100 measurements, 5 nodes, 50ms RTT + 10ms jitter between nodes\nno contention: latency is due to intra-cluster communications\ncolored bar = mean latency, error bar = min and max latency"
|
||||||
|
|
||||||
|
|
||||||
|
ggplot(c, aes(x=endpoint,y=time_mean,fill=daemon,ymin=time_min,ymax=time_max)) +
|
||||||
|
geom_bar(stat="identity", position=position_dodge(),color="black") +
|
||||||
|
geom_errorbar(position=position_dodge(.9),width=.2) +
|
||||||
|
scale_y_continuous(expand=c(0,0))+
|
||||||
|
coord_flip() +
|
||||||
|
labs(
|
||||||
|
x="S3 Endpoint",
|
||||||
|
y="Request duration (ms)",
|
||||||
|
fill="Daemon",
|
||||||
|
caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
|
||||||
|
title="S3 endpoint latency in a simulated geo-distributed cluster",
|
||||||
|
subtitle=st) +
|
||||||
|
theme_classic() +
|
||||||
|
theme(legend.position = c(.8, .3))
|
||||||
|
|
||||||
|
ggsave("./plot.png", width=200, height=110, units="mm")
|
||||||
BIN
artifacts/2022-09-23-s3lat/plot.png
Normal file
BIN
artifacts/2022-09-23-s3lat/plot.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 144 KiB |
61
artifacts/2022-09-23-s3ttfb/garage-v0.7.csv
Normal file
61
artifacts/2022-09-23-s3ttfb/garage-v0.7.csv
Normal file
|
|
@ -0,0 +1,61 @@
|
||||||
|
nanoseconds
|
||||||
|
2043724295
|
||||||
|
1819672397
|
||||||
|
1856153241
|
||||||
|
1814937942
|
||||||
|
1818943177
|
||||||
|
1814245475
|
||||||
|
1828595614
|
||||||
|
1817263829
|
||||||
|
1857481054
|
||||||
|
1816538404
|
||||||
|
1816874169
|
||||||
|
1856305403
|
||||||
|
1857317361
|
||||||
|
1814517488
|
||||||
|
1817156893
|
||||||
|
1815832311
|
||||||
|
1857020274
|
||||||
|
1822556545
|
||||||
|
1816402439
|
||||||
|
1815664395
|
||||||
|
1818129711
|
||||||
|
1855229810
|
||||||
|
1817427508
|
||||||
|
1817253968
|
||||||
|
1855458740
|
||||||
|
1818412087
|
||||||
|
1816422102
|
||||||
|
1814017661
|
||||||
|
1821773596
|
||||||
|
1814924392
|
||||||
|
1819138108
|
||||||
|
1814140079
|
||||||
|
1814253338
|
||||||
|
1814717178
|
||||||
|
1823390172
|
||||||
|
1856664915
|
||||||
|
1817886566
|
||||||
|
1815441938
|
||||||
|
1815381667
|
||||||
|
1864383216
|
||||||
|
1815026715
|
||||||
|
1816346397
|
||||||
|
1814835227
|
||||||
|
1855973303
|
||||||
|
1814632968
|
||||||
|
1823675273
|
||||||
|
1817900846
|
||||||
|
1816282804
|
||||||
|
1817289580
|
||||||
|
1817288129
|
||||||
|
1860637032
|
||||||
|
1855146585
|
||||||
|
1816649086
|
||||||
|
1822524427
|
||||||
|
1814277922
|
||||||
|
1901037764
|
||||||
|
1818903030
|
||||||
|
1815404774
|
||||||
|
1815446844
|
||||||
|
1815223628
|
||||||
|
61
artifacts/2022-09-23-s3ttfb/garage-v0.8.csv
Normal file
61
artifacts/2022-09-23-s3ttfb/garage-v0.8.csv
Normal file
|
|
@ -0,0 +1,61 @@
|
||||||
|
nanoseconds
|
||||||
|
376764319
|
||||||
|
268350943
|
||||||
|
310303481
|
||||||
|
266580460
|
||||||
|
267172958
|
||||||
|
46771899
|
||||||
|
271623330
|
||||||
|
270463024
|
||||||
|
47401936
|
||||||
|
45384570
|
||||||
|
129505789
|
||||||
|
46244307
|
||||||
|
270594700
|
||||||
|
267908944
|
||||||
|
310163759
|
||||||
|
271096844
|
||||||
|
268930686
|
||||||
|
309894956
|
||||||
|
273312715
|
||||||
|
269766963
|
||||||
|
267947351
|
||||||
|
270443270
|
||||||
|
266292594
|
||||||
|
215426567
|
||||||
|
383550730
|
||||||
|
84593093
|
||||||
|
380970185
|
||||||
|
419898767
|
||||||
|
381574216
|
||||||
|
422289246
|
||||||
|
380535266
|
||||||
|
268204508
|
||||||
|
46449693
|
||||||
|
270408730
|
||||||
|
268572254
|
||||||
|
308981862
|
||||||
|
266303861
|
||||||
|
46092770
|
||||||
|
308073958
|
||||||
|
383696451
|
||||||
|
381725197
|
||||||
|
379008890
|
||||||
|
379902614
|
||||||
|
382568588
|
||||||
|
45370580
|
||||||
|
83526768
|
||||||
|
381230253
|
||||||
|
267980416
|
||||||
|
269187238
|
||||||
|
84449786
|
||||||
|
267524276
|
||||||
|
46169934
|
||||||
|
43509278
|
||||||
|
308776528
|
||||||
|
270626080
|
||||||
|
271203190
|
||||||
|
216167767
|
||||||
|
239794865
|
||||||
|
46230173
|
||||||
|
267821837
|
||||||
|
61
artifacts/2022-09-23-s3ttfb/minio-20220917.csv
Normal file
61
artifacts/2022-09-23-s3ttfb/minio-20220917.csv
Normal file
|
|
@ -0,0 +1,61 @@
|
||||||
|
nanoseconds
|
||||||
|
941737601
|
||||||
|
970059899
|
||||||
|
956848578
|
||||||
|
981034132
|
||||||
|
968456906
|
||||||
|
986209313
|
||||||
|
947371347
|
||||||
|
961890275
|
||||||
|
951011001
|
||||||
|
947766957
|
||||||
|
946381130
|
||||||
|
1172968280
|
||||||
|
944672409
|
||||||
|
946216350
|
||||||
|
960072933
|
||||||
|
947966145
|
||||||
|
947998544
|
||||||
|
957730811
|
||||||
|
960898554
|
||||||
|
950611253
|
||||||
|
953195848
|
||||||
|
974654976
|
||||||
|
982578891
|
||||||
|
957419979
|
||||||
|
947448876
|
||||||
|
945953214
|
||||||
|
936687543
|
||||||
|
950116533
|
||||||
|
979971727
|
||||||
|
941427574
|
||||||
|
931898843
|
||||||
|
928494073
|
||||||
|
927924296
|
||||||
|
940995728
|
||||||
|
927965457
|
||||||
|
928098595
|
||||||
|
933759012
|
||||||
|
929902652
|
||||||
|
953753466
|
||||||
|
931448442
|
||||||
|
934503899
|
||||||
|
933297864
|
||||||
|
934775226
|
||||||
|
936615695
|
||||||
|
950945991
|
||||||
|
935054323
|
||||||
|
1162136321
|
||||||
|
940502403
|
||||||
|
1158490674
|
||||||
|
945337099
|
||||||
|
948016443
|
||||||
|
939863143
|
||||||
|
938112935
|
||||||
|
933271354
|
||||||
|
946325434
|
||||||
|
954355353
|
||||||
|
948997521
|
||||||
|
1124820589
|
||||||
|
989829976
|
||||||
|
938639125
|
||||||
|
25
artifacts/2022-09-23-s3ttfb/plot.R
Normal file
25
artifacts/2022-09-23-s3ttfb/plot.R
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
library(tidyverse)
|
||||||
|
|
||||||
|
|
||||||
|
read_csv("garage-v0.7.csv") %>% add_column(daemon="garage 0.7.3") -> a
|
||||||
|
read_csv("garage-v0.8.csv") %>% add_column(daemon="garage 0.8.0 beta") -> b
|
||||||
|
read_csv("minio-20220917.csv") %>% add_column(daemon="minio RELEASE.2022-09-17") -> c
|
||||||
|
bind_rows(a,b,c) %>% mutate(ms = nanoseconds / 1000 / 1000) -> d
|
||||||
|
|
||||||
|
ggplot(d, aes(y = ms, group = daemon, fill=daemon)) +
|
||||||
|
#geom_violin(trim = TRUE, width=2) +
|
||||||
|
geom_histogram(binwidth=50, color="black") +
|
||||||
|
scale_y_continuous(expand=c(0,0))+
|
||||||
|
scale_x_continuous(expand=c(0,0))+
|
||||||
|
coord_flip() +
|
||||||
|
labs(
|
||||||
|
y="TTFB (ms)",
|
||||||
|
x="Count",
|
||||||
|
fill="Daemon",
|
||||||
|
caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
|
||||||
|
title="TTFB (Time To First Byte) on GetObject over a slow network (5 Mbps, 500 µs)",
|
||||||
|
subtitle="A 1MB file is uploaded and then fetched 60 times.\nExcept for Minio, the queried node does not store any data (gateway) to force net. communications.") +
|
||||||
|
theme_classic() +
|
||||||
|
theme(legend.position = c(.15, .8))
|
||||||
|
|
||||||
|
ggsave("./plot.png", width=200, height=110, units="mm")
|
||||||
BIN
artifacts/2022-09-23-s3ttfb/plot.png
Normal file
BIN
artifacts/2022-09-23-s3ttfb/plot.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 128 KiB |
BIN
artifacts/2022-09-23-warp/dc-garage-v0.7.zst
Normal file
BIN
artifacts/2022-09-23-warp/dc-garage-v0.7.zst
Normal file
Binary file not shown.
21
artifacts/2022-09-23-warp/plot.R
Normal file
21
artifacts/2022-09-23-warp/plot.R
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
library(tidyverse)
|
||||||
|
|
||||||
|
|
||||||
|
read_csv("summary.csv")-> s
|
||||||
|
|
||||||
|
hum_names <- as_labeller(c(`obj_per_sec` = "Objects/sec", `mbyte_per_sec` = "MByte/sec"))
|
||||||
|
|
||||||
|
|
||||||
|
ggplot(s, aes(x=value,y=config)) +
|
||||||
|
geom_bar(stat="identity", fill="#00BFC4", color="black") +
|
||||||
|
facet_wrap(~measurement,dir="v", scales = "free", labeller = hum_names) +
|
||||||
|
scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
|
||||||
|
labs(
|
||||||
|
y="Daemon",
|
||||||
|
x="Metric",
|
||||||
|
caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
|
||||||
|
title="\"minio/warp\" benchmark, \"cluster total\" result",
|
||||||
|
subtitle="Ran on a local machine (Ryzen 5 1400, 16GB RAM, SSD) with mknet\nDC topology (3 nodes, 1GB/s, 1ms lat)\nwarp in mixed mode, 5min bench, 5MB objects, initialized with 200 objects") +
|
||||||
|
theme_classic()
|
||||||
|
|
||||||
|
ggsave("./plot.png", width=200, height=120, units="mm")
|
||||||
BIN
artifacts/2022-09-23-warp/plot.png
Normal file
BIN
artifacts/2022-09-23-warp/plot.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 189 KiB |
11
artifacts/2022-09-23-warp/summary.csv
Normal file
11
artifacts/2022-09-23-warp/summary.csv
Normal file
|
|
@ -0,0 +1,11 @@
|
||||||
|
config,measurement,value
|
||||||
|
garage v0.7,mbyte_per_sec,8.60
|
||||||
|
garage v0.7,obj_per_sec,3.02
|
||||||
|
garage v0.8-beta1,mbyte_per_sec,12.41
|
||||||
|
garage v0.8-beta1,obj_per_sec,4.36
|
||||||
|
minio 20220917,mbyte_per_sec,20.22
|
||||||
|
minio 20220917,obj_per_sec,7.11
|
||||||
|
garage v0.8-beta2,mbyte_per_sec,14.56
|
||||||
|
garage v0.8-beta2,obj_per_sec,5.19
|
||||||
|
garage v0.8-no-fsync,mbyte_per_sec,17.31
|
||||||
|
garage v0.8-no-fsync,obj_per_sec,6.11
|
||||||
|
BIN
artifacts/2022-09-24-s3billion/garage-regression.png
Normal file
BIN
artifacts/2022-09-24-s3billion/garage-regression.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 285 KiB |
129
artifacts/2022-09-24-s3billion/garage-v0.8-beta2-lmdb.csv
Normal file
129
artifacts/2022-09-24-s3billion/garage-v0.8-beta2-lmdb.csv
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
total_objects,batch_dur_nanoseconds
|
||||||
|
0,2018191689
|
||||||
|
8192,1819596313
|
||||||
|
16384,2102802010
|
||||||
|
24576,2473935158
|
||||||
|
32768,4510296173
|
||||||
|
40960,6613958698
|
||||||
|
49152,7977651423
|
||||||
|
57344,13671435594
|
||||||
|
65536,8060203586
|
||||||
|
73728,3795495684
|
||||||
|
81920,5794535775
|
||||||
|
90112,5543820710
|
||||||
|
98304,4519985809
|
||||||
|
106496,5110068206
|
||||||
|
114688,6708672963
|
||||||
|
122880,7857479433
|
||||||
|
131072,20642163540
|
||||||
|
139264,5635269044
|
||||||
|
147456,9351347336
|
||||||
|
155648,5666222073
|
||||||
|
163840,6501995906
|
||||||
|
172032,5293598878
|
||||||
|
180224,9350079270
|
||||||
|
188416,13303962641
|
||||||
|
196608,17428285824
|
||||||
|
204800,11733919058
|
||||||
|
212992,8372494836
|
||||||
|
221184,26574754854
|
||||||
|
229376,6314078661
|
||||||
|
237568,9683594309
|
||||||
|
245760,21850576414
|
||||||
|
253952,8991130243
|
||||||
|
262144,6567887673
|
||||||
|
270336,9454781018
|
||||||
|
278528,25552173846
|
||||||
|
286720,9580711949
|
||||||
|
294912,10257632597
|
||||||
|
303104,7478191638
|
||||||
|
311296,10501501215
|
||||||
|
319488,3816776322
|
||||||
|
327680,26795462810
|
||||||
|
335872,9396360089
|
||||||
|
344064,11208895014
|
||||||
|
352256,21126662513
|
||||||
|
360448,14198029001
|
||||||
|
368640,5093580346
|
||||||
|
376832,21004203835
|
||||||
|
385024,14433396301
|
||||||
|
393216,8933953570
|
||||||
|
401408,5461923067
|
||||||
|
409600,30844223516
|
||||||
|
417792,8315342751
|
||||||
|
425984,6166609079
|
||||||
|
434176,29359253375
|
||||||
|
442368,7642009005
|
||||||
|
450560,7036982475
|
||||||
|
458752,27168386971
|
||||||
|
466944,10015226557
|
||||||
|
475136,9245612551
|
||||||
|
483328,24224252348
|
||||||
|
491520,13827842768
|
||||||
|
499712,9702435577
|
||||||
|
507904,26083366912
|
||||||
|
516096,7829114232
|
||||||
|
524288,10792095771
|
||||||
|
532480,28058503484
|
||||||
|
540672,9984663485
|
||||||
|
548864,10714515140
|
||||||
|
557056,28472749292
|
||||||
|
565248,10832278691
|
||||||
|
573440,7594452219
|
||||||
|
581632,23858923965
|
||||||
|
589824,38909249537
|
||||||
|
598016,15112735865
|
||||||
|
606208,9644628857
|
||||||
|
614400,28367134578
|
||||||
|
622592,35180699079
|
||||||
|
630784,14207640136
|
||||||
|
638976,13236547125
|
||||||
|
647168,37982462879
|
||||||
|
655360,8448278050
|
||||||
|
663552,11632520113
|
||||||
|
671744,22661429490
|
||||||
|
679936,29261192164
|
||||||
|
688128,11982484321
|
||||||
|
696320,30381827880
|
||||||
|
704512,23530964587
|
||||||
|
712704,8219614193
|
||||||
|
720896,34002514929
|
||||||
|
729088,21279203762
|
||||||
|
737280,6880671212
|
||||||
|
745472,10884265763
|
||||||
|
753664,41999879903
|
||||||
|
761856,13465319479
|
||||||
|
770048,10792229369
|
||||||
|
778240,29004881357
|
||||||
|
786432,24372150617
|
||||||
|
794624,13327280319
|
||||||
|
802816,18478244110
|
||||||
|
811008,48319469581
|
||||||
|
819200,14174987827
|
||||||
|
827392,14368184404
|
||||||
|
835584,56647745264
|
||||||
|
843776,9880099642
|
||||||
|
851968,12808097118
|
||||||
|
860160,28380242726
|
||||||
|
868352,44970657202
|
||||||
|
876544,18733551704
|
||||||
|
884736,9340561806
|
||||||
|
892928,25246663514
|
||||||
|
901120,46462124425
|
||||||
|
909312,12856639771
|
||||||
|
917504,9028224897
|
||||||
|
925696,38580954489
|
||||||
|
933888,48184562190
|
||||||
|
942080,11204051215
|
||||||
|
950272,19754687230
|
||||||
|
958464,36364975788
|
||||||
|
966656,40759996520
|
||||||
|
974848,10562565477
|
||||||
|
983040,13905331718
|
||||||
|
991232,54140254529
|
||||||
|
999424,46643736817
|
||||||
|
1007616,8362351299
|
||||||
|
1015808,13514305332
|
||||||
|
1024000,39389278472
|
||||||
|
1032192,40328458712
|
||||||
|
1040384,15958365846
|
||||||
|
BIN
artifacts/2022-09-24-s3billion/garage.png
Normal file
BIN
artifacts/2022-09-24-s3billion/garage.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 232 KiB |
129
artifacts/2022-09-24-s3billion/minio.csv
Normal file
129
artifacts/2022-09-24-s3billion/minio.csv
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
total_objects,batch_dur_nanoseconds
|
||||||
|
0,35604165299
|
||||||
|
8192,30027091125
|
||||||
|
16384,36769805630
|
||||||
|
24576,44154399690
|
||||||
|
32768,36841962377
|
||||||
|
40960,52989512753
|
||||||
|
49152,52536807234
|
||||||
|
57344,53537348443
|
||||||
|
65536,53276055435
|
||||||
|
73728,55539526464
|
||||||
|
81920,54013225676
|
||||||
|
90112,68561851906
|
||||||
|
98304,69298493247
|
||||||
|
106496,114291303648
|
||||||
|
114688,78507905949
|
||||||
|
122880,126223986043
|
||||||
|
131072,121999273534
|
||||||
|
139264,153563457891
|
||||||
|
147456,151032625349
|
||||||
|
155648,149855834712
|
||||||
|
163840,153190455466
|
||||||
|
172032,150844647704
|
||||||
|
180224,156256328509
|
||||||
|
188416,151849908862
|
||||||
|
196608,154337246115
|
||||||
|
204800,154008292236
|
||||||
|
212992,160001140992
|
||||||
|
221184,154134240373
|
||||||
|
229376,147943023290
|
||||||
|
237568,150989793433
|
||||||
|
245760,161064597934
|
||||||
|
253952,150058583272
|
||||||
|
262144,158784561065
|
||||||
|
270336,150652453675
|
||||||
|
278528,108917197085
|
||||||
|
286720,158143657766
|
||||||
|
294912,151991188276
|
||||||
|
303104,150904775153
|
||||||
|
311296,152538005225
|
||||||
|
319488,158677889479
|
||||||
|
327680,151948293662
|
||||||
|
335872,131622851465
|
||||||
|
344064,120774952190
|
||||||
|
352256,160866374832
|
||||||
|
360448,157913387369
|
||||||
|
368640,157965094573
|
||||||
|
376832,155015634674
|
||||||
|
385024,152010190282
|
||||||
|
393216,151353578675
|
||||||
|
401408,152816629041
|
||||||
|
409600,100685222932
|
||||||
|
417792,151049307826
|
||||||
|
425984,152092080473
|
||||||
|
434176,156352511800
|
||||||
|
442368,156617304416
|
||||||
|
450560,156604667758
|
||||||
|
458752,163954135953
|
||||||
|
466944,102070359622
|
||||||
|
475136,155481160507
|
||||||
|
483328,157268905807
|
||||||
|
491520,162446380203
|
||||||
|
499712,159234723683
|
||||||
|
507904,141918580163
|
||||||
|
516096,133297515392
|
||||||
|
524288,92684666697
|
||||||
|
532480,153688292459
|
||||||
|
540672,154151441890
|
||||||
|
548864,157288990588
|
||||||
|
557056,159081065544
|
||||||
|
565248,153244088615
|
||||||
|
573440,161424543833
|
||||||
|
581632,174752406147
|
||||||
|
589824,155541074973
|
||||||
|
598016,159471780365
|
||||||
|
606208,158247043033
|
||||||
|
614400,157224045247
|
||||||
|
622592,160337074095
|
||||||
|
630784,157993307652
|
||||||
|
638976,156821062432
|
||||||
|
647168,158852305038
|
||||||
|
655360,161012982349
|
||||||
|
663552,156762111328
|
||||||
|
671744,159782719621
|
||||||
|
679936,156694048459
|
||||||
|
688128,175969992462
|
||||||
|
696320,159626684812
|
||||||
|
704512,156680995342
|
||||||
|
712704,155624836949
|
||||||
|
720896,161583816024
|
||||||
|
729088,156832016477
|
||||||
|
737280,160834148361
|
||||||
|
745472,157761321600
|
||||||
|
753664,157553638650
|
||||||
|
761856,176048273498
|
||||||
|
770048,156697917513
|
||||||
|
778240,162791069016
|
||||||
|
786432,156496526158
|
||||||
|
794624,161388816617
|
||||||
|
802816,159569344379
|
||||||
|
811008,159393281615
|
||||||
|
819200,158389581978
|
||||||
|
827392,164002225318
|
||||||
|
835584,162722200475
|
||||||
|
843776,160146392865
|
||||||
|
851968,164551137708
|
||||||
|
860160,177034374419
|
||||||
|
868352,161672813995
|
||||||
|
876544,169063897017
|
||||||
|
884736,160829838695
|
||||||
|
892928,157613804708
|
||||||
|
901120,162687585988
|
||||||
|
909312,166612325773
|
||||||
|
917504,160649709927
|
||||||
|
925696,163214320096
|
||||||
|
933888,178447566516
|
||||||
|
942080,155968188136
|
||||||
|
950272,161252816115
|
||||||
|
958464,159781104139
|
||||||
|
966656,163854372812
|
||||||
|
974848,157176626359
|
||||||
|
983040,160325050560
|
||||||
|
991232,165218492441
|
||||||
|
999424,160172384289
|
||||||
|
1007616,159316385255
|
||||||
|
1015808,178044565059
|
||||||
|
1024000,164817520071
|
||||||
|
1032192,159428724027
|
||||||
|
1040384,159973237521
|
||||||
|
BIN
artifacts/2022-09-24-s3billion/minio.png
Normal file
BIN
artifacts/2022-09-24-s3billion/minio.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 235 KiB |
65
artifacts/2022-09-24-s3billion/plot.R
Normal file
65
artifacts/2022-09-24-s3billion/plot.R
Normal file
|
|
@ -0,0 +1,65 @@
|
||||||
|
library(tidyverse)
|
||||||
|
library(ggpmisc)
|
||||||
|
|
||||||
|
|
||||||
|
read_csv("garage-v0.8-beta2-lmdb.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000) %>% filter(total_objects != 0) -> s
|
||||||
|
|
||||||
|
reg1 <- lm(s$batch_dur_sec~s$total_objects)
|
||||||
|
reg2 <- lm(s$batch_dur_sec ~ log(s$total_objects))
|
||||||
|
|
||||||
|
f1 <- y~log(x)
|
||||||
|
f2 <- y~x
|
||||||
|
|
||||||
|
ggplot(s, aes(x=total_objects, y=batch_dur_sec)) +
|
||||||
|
geom_point() +
|
||||||
|
#geom_smooth(method="lm",formula=f1, se = FALSE, color="red") +
|
||||||
|
#geom_smooth(method="lm",formula=f2, se = FALSE, color="blue") +
|
||||||
|
#stat_poly_eq(formula = f1, label.y = 0.9, color = "red", aes(label=paste(..eq.label..,..rr.label..,..adj.rr.label..,..AIC.label..,..BIC.label.., sep = "~~~"))) +
|
||||||
|
#stat_poly_eq(formula = f2, label.y = 0.8, color="blue",aes(label=paste(..eq.label..,..rr.label..,..adj.rr.label..,..AIC.label..,..BIC.label.., sep = "~~~"))) +
|
||||||
|
#geom_smooth(method = "gam", se = FALSE) +
|
||||||
|
scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
|
||||||
|
scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
|
||||||
|
coord_cartesian(ylim=c(0,60)) +
|
||||||
|
labs(
|
||||||
|
y="Time (in sec) spent sending a batch (8192 objects)",
|
||||||
|
x="Total number of objects stored in the cluster",
|
||||||
|
caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
|
||||||
|
title="Storing 1M+ files in Garage! Impact of existing data on cluster interactiveness",
|
||||||
|
subtitle="Daemon: Garage v0.8 beta 2 with LMDB as db_engine\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
|
||||||
|
theme_classic()
|
||||||
|
ggsave("./garage.png", width=200, height=120, units="mm")
|
||||||
|
#ggsave("./garage-regression.png", width=200, height=120, units="mm")
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
read_csv("minio.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000 ) -> s2
|
||||||
|
ggplot(s2, aes(x=total_objects, y=batch_dur_sec)) +
|
||||||
|
geom_point() +
|
||||||
|
#geom_smooth(method = "gam", se = FALSE) +
|
||||||
|
scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
|
||||||
|
scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
|
||||||
|
labs(
|
||||||
|
y="Time (in sec) spent sending a batch (8192 objects)",
|
||||||
|
x="Total number of objects stored in the cluster",
|
||||||
|
caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
|
||||||
|
title="Storing 1M+ files in Minio! Impact of existing data on cluster interactiveness",
|
||||||
|
subtitle="Daemon: Minio RELEASE 20220917\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
|
||||||
|
theme_classic()
|
||||||
|
ggsave("./minio.png", width=200, height=120, units="mm")
|
||||||
|
|
||||||
|
bind_rows(s %>% add_column(daemon="garage 0.8 beta2 lmdb"), s2 %>% add_column(daemon="minio RELEASE 20220917")) -> sc
|
||||||
|
ggplot(sc, aes(x=total_objects, y=batch_dur_sec, group=daemon, color=daemon)) +
|
||||||
|
geom_point() +
|
||||||
|
#geom_smooth(method = "gam", se = FALSE) +
|
||||||
|
scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
|
||||||
|
scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
|
||||||
|
labs(
|
||||||
|
y="Time (in sec) spent sending a batch (8192 objects)",
|
||||||
|
x="Total number of objects stored in the cluster",
|
||||||
|
fill="Daemon",
|
||||||
|
caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
|
||||||
|
title="Storing 1M+ files in Garage and Minio! Impact of existing data on cluster interactiveness",
|
||||||
|
subtitle="Daemon: Garage v0.8 beta 2 with LMDB as db_engine, Minio RELEASE 20220917\nBenchmark: 128 batch. 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).") +
|
||||||
|
theme_classic() +
|
||||||
|
theme(legend.position = c(.85, .55))
|
||||||
|
ggsave("./plot.png", width=200, height=120, units="mm")
|
||||||
BIN
artifacts/2022-09-24-s3billion/plot.png
Normal file
BIN
artifacts/2022-09-24-s3billion/plot.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 295 KiB |
15
artifacts/2022-09-24-warp-db-engine/lmdb.log
Normal file
15
artifacts/2022-09-24-warp-db-engine/lmdb.log
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
Mixed operations.
|
||||||
|
Operation: DELETE, 10%, Concurrency: 20, Ran 4m59s.
|
||||||
|
* Throughput: 266.79 obj/s
|
||||||
|
|
||||||
|
Operation: GET, 45%, Concurrency: 20, Ran 4m59s.
|
||||||
|
* Throughput: 0.29 MiB/s, 1200.51 obj/s
|
||||||
|
|
||||||
|
Operation: PUT, 15%, Concurrency: 20, Ran 4m59s.
|
||||||
|
* Throughput: 0.10 MiB/s, 400.20 obj/s
|
||||||
|
|
||||||
|
Operation: STAT, 30%, Concurrency: 20, Ran 4m59s.
|
||||||
|
* Throughput: 800.32 obj/s
|
||||||
|
|
||||||
|
Cluster Total: 0.39 MiB/s, 2667.83 obj/s over 5m0s.
|
||||||
|
Aggregated data saved to /tmp/mknet-store/warp.csv
|
||||||
17
artifacts/2022-09-24-warp-db-engine/plot.R
Normal file
17
artifacts/2022-09-24-warp-db-engine/plot.R
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
library(tidyverse)
|
||||||
|
|
||||||
|
read_csv("summary.csv")-> s
|
||||||
|
ggplot(s, aes(x=db_engine,y=rate)) +
|
||||||
|
scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10))+
|
||||||
|
geom_bar(stat='identity',fill="#00BFC4", color="black") +
|
||||||
|
coord_flip() +
|
||||||
|
facet_wrap(~operation, scales = "free", dir="v") +
|
||||||
|
labs(
|
||||||
|
x="Metadata Engine",
|
||||||
|
y="Object/sec",
|
||||||
|
caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
|
||||||
|
title="Comparison of Garage's metadata engines with \"minio/warp\"",
|
||||||
|
subtitle="Daemon: Garage v0.8 no-fsync to avoid being impacted by block manager\nBenchmark: warp, mixed mode, 5min bench, 256B objects, initialized with 200 objects.\nEnvironment: mknet (Ryzen 5 1400, 16GB RAM, SSD). DC topo (3 nodes, 1Gb/s, 1ms latency).",
|
||||||
|
) +
|
||||||
|
theme_classic()
|
||||||
|
ggsave("./plot.png", width=200, height=120, units="mm")
|
||||||
BIN
artifacts/2022-09-24-warp-db-engine/plot.png
Normal file
BIN
artifacts/2022-09-24-warp-db-engine/plot.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 177 KiB |
13
artifacts/2022-09-24-warp-db-engine/sled-nosync.log
Normal file
13
artifacts/2022-09-24-warp-db-engine/sled-nosync.log
Normal file
|
|
@ -0,0 +1,13 @@
|
||||||
|
Operation: DELETE, 10%, Concurrency: 20, Ran 3m42s.
|
||||||
|
* Throughput: 161.98 obj/s
|
||||||
|
|
||||||
|
Operation: GET, 45%, Concurrency: 20, Ran 3m42s.
|
||||||
|
* Throughput: 0.18 MiB/s, 729.07 obj/s
|
||||||
|
|
||||||
|
Operation: PUT, 15%, Concurrency: 20, Ran 3m42s.
|
||||||
|
* Throughput: 0.06 MiB/s, 242.99 obj/s
|
||||||
|
|
||||||
|
Operation: STAT, 30%, Concurrency: 20, Ran 3m42s.
|
||||||
|
* Throughput: 486.09 obj/s
|
||||||
|
|
||||||
|
Cluster Total: 0.24 MiB/s, 1620.14 obj/s over 3m42s.
|
||||||
15
artifacts/2022-09-24-warp-db-engine/sled.log
Normal file
15
artifacts/2022-09-24-warp-db-engine/sled.log
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
Mixed operations.
|
||||||
|
Operation: DELETE, 10%, Concurrency: 20, Ran 3m49s.
|
||||||
|
* Throughput: 129.83 obj/s
|
||||||
|
|
||||||
|
Operation: GET, 45%, Concurrency: 20, Ran 3m49s.
|
||||||
|
* Throughput: 0.14 MiB/s, 584.32 obj/s
|
||||||
|
|
||||||
|
Operation: PUT, 15%, Concurrency: 20, Ran 3m49s.
|
||||||
|
* Throughput: 0.05 MiB/s, 194.81 obj/s
|
||||||
|
|
||||||
|
Operation: STAT, 30%, Concurrency: 20, Ran 3m49s.
|
||||||
|
* Throughput: 389.52 obj/s
|
||||||
|
|
||||||
|
Cluster Total: 0.19 MiB/s, 1298.49 obj/s over 3m49s.
|
||||||
|
Aggregated data saved to /tmp/mknet-store/warp.csv
|
||||||
15
artifacts/2022-09-24-warp-db-engine/sqlite.log
Normal file
15
artifacts/2022-09-24-warp-db-engine/sqlite.log
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
Mixed operations.
|
||||||
|
Operation: DELETE, 10%, Concurrency: 20, Ran 4m55s.
|
||||||
|
* Throughput: 1.29 obj/s
|
||||||
|
|
||||||
|
Operation: GET, 45%, Concurrency: 20, Ran 4m55s.
|
||||||
|
* Throughput: 0.00 MiB/s, 5.75 obj/s
|
||||||
|
|
||||||
|
Operation: PUT, 15%, Concurrency: 20, Ran 4m55s.
|
||||||
|
* Throughput: 0.00 MiB/s, 1.93 obj/s
|
||||||
|
|
||||||
|
Operation: STAT, 30%, Concurrency: 20, Ran 4m56s.
|
||||||
|
* Throughput: 3.86 obj/s
|
||||||
|
|
||||||
|
Cluster Total: 0.00 MiB/s, 12.83 obj/s over 4m55s.
|
||||||
|
Aggregated data saved to /tmp/mknet-store/warp.csv
|
||||||
17
artifacts/2022-09-24-warp-db-engine/summary.csv
Normal file
17
artifacts/2022-09-24-warp-db-engine/summary.csv
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
db_engine,operation,rate
|
||||||
|
lmdb,delete,266.79
|
||||||
|
lmdb,get,1200.51
|
||||||
|
lmdb,put,400
|
||||||
|
lmdb,stat,800
|
||||||
|
sled,delete,129.83
|
||||||
|
sled,get,584.32
|
||||||
|
sled,put,194.81
|
||||||
|
sled,stat,389.52
|
||||||
|
sqlite,delete,1.29
|
||||||
|
sqlite,get,5.75
|
||||||
|
sqlite,put,1.93
|
||||||
|
sqlite,stat,3.86
|
||||||
|
sled-nosync,delete,161.98
|
||||||
|
sled-nosync,get,729.07
|
||||||
|
sled-nosync,put,242.99
|
||||||
|
sled-nosync,stat,486.09
|
||||||
|
BIN
artifacts/2022-09-29-s3billion/garage-0.8-gros.png
Normal file
BIN
artifacts/2022-09-29-s3billion/garage-0.8-gros.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 459 KiB |
27
artifacts/2022-09-29-s3billion/plot.R
Normal file
27
artifacts/2022-09-29-s3billion/plot.R
Normal file
|
|
@ -0,0 +1,27 @@
|
||||||
|
library(tidyverse)
|
||||||
|
library(ggpmisc)
|
||||||
|
library(scales)
|
||||||
|
|
||||||
|
|
||||||
|
read_csv("s3billion-dc-gros-32threads-256batchsize-128000batchcount-v0.8-fix-time-30M.csv") %>% mutate(batch_dur_sec = batch_dur_nanoseconds / 1000 / 1000 / 1000) %>% filter(total_objects != 0) -> s
|
||||||
|
|
||||||
|
reg <- lm(s$batch_dur_sec~s$total_objects)
|
||||||
|
|
||||||
|
f <- y~x
|
||||||
|
|
||||||
|
ggplot(s, aes(x=total_objects, y=batch_dur_sec)) +
|
||||||
|
geom_point(shape = "bullet", alpha=0.4) +
|
||||||
|
geom_smooth(method="lm", formula=f, se = FALSE, color="blue") +
|
||||||
|
scale_x_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10), labels = unit_format(unit = "M", scale = 1e-6)) +
|
||||||
|
scale_y_continuous(expand=c(0,0), breaks = scales::pretty_breaks(n = 10)) +
|
||||||
|
coord_cartesian(ylim=c(0,3)) +
|
||||||
|
labs(
|
||||||
|
y="Time (in sec) spent sending a batch (8192 objects)",
|
||||||
|
x="Total number of objects stored in the cluster",
|
||||||
|
caption="Get the code to reproduce this graph at https://git.deuxfleurs.fr/Deuxfleurs/mknet",
|
||||||
|
title="Storing 30M small objects in Garage!",
|
||||||
|
subtitle="Daemon: Garage v0.8.0-rc1 (+ #398 pull request) with LMDB as db_engine\nBenchmark: 8192 objects/batch. 32 threads/batch. 256 objects/thread. 16-byte/objects.\nEnvironment: mknet on Grid'5000 (gros) https://www.grid5000.fr/w/Nancy:Hardware#gros (SATA SSD).\nDC topo (3 logical nodes, 1Gb/s, 1ms latency).") +
|
||||||
|
theme_classic()
|
||||||
|
ggsave("./garage-0.8-gros.png", width=200, height=120, units="mm")
|
||||||
|
#ggsave("./garage-regression.png", width=200, height=120, units="mm")
|
||||||
|
|
||||||
File diff suppressed because it is too large
Load diff
1
benchmarks/s3billion/.gitignore
vendored
Normal file
1
benchmarks/s3billion/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
s3billion
|
||||||
22
benchmarks/s3billion/README.md
Normal file
22
benchmarks/s3billion/README.md
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
# s3lat
|
||||||
|
|
||||||
|
## installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go get git.deuxfleurs.fr/quentin/s3lat@latest
|
||||||
|
```
|
||||||
|
|
||||||
|
## usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export ENDPOINT=[fc00:9a7a:9e::1]:9000
|
||||||
|
export AWS_ACCESS_KEY_ID=minioadmin
|
||||||
|
export AWS_SECRET_ACCESS_KEY=minioadmin
|
||||||
|
|
||||||
|
s3lat
|
||||||
|
```
|
||||||
|
|
||||||
|
## see also
|
||||||
|
|
||||||
|
- https://git.deuxfleurs.fr/quentin/benchmarks
|
||||||
|
- https://git.deuxfleurs.fr/trinity-1686a/mknet
|
||||||
8
benchmarks/s3billion/go.mod
Normal file
8
benchmarks/s3billion/go.mod
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
module git.deuxfleurs.fr/Deuxfleurs/mknet/benchmarks/s3billion
|
||||||
|
|
||||||
|
go 1.16
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/google/uuid v1.1.1
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16
|
||||||
|
)
|
||||||
78
benchmarks/s3billion/go.sum
Normal file
78
benchmarks/s3billion/go.sum
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
|
||||||
|
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
|
||||||
|
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||||
|
github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
|
||||||
|
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
|
||||||
|
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
|
||||||
|
github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
|
||||||
|
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
|
||||||
|
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
|
||||||
|
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
|
||||||
|
github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
|
||||||
|
github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
|
||||||
|
github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
|
||||||
|
github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
|
||||||
|
github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
|
||||||
|
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||||
|
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||||
|
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||||
|
github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
|
||||||
|
github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
|
||||||
|
github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
|
||||||
|
github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
|
||||||
|
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
|
||||||
|
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
|
||||||
|
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
|
||||||
|
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
|
||||||
|
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
|
||||||
|
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
|
||||||
|
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
|
||||||
|
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
|
||||||
|
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
|
||||||
|
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
|
||||||
|
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
|
||||||
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
|
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
|
||||||
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
|
github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
|
||||||
|
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
|
||||||
|
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||||
|
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||||
|
golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
|
||||||
|
golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
|
||||||
|
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||||
|
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||||
|
golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
|
||||||
|
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
|
||||||
|
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||||
|
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae h1:Ih9Yo4hSPImZOpfGuA4bR/ORKTAbhZo2AbWNRCnevdo=
|
||||||
|
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
|
||||||
|
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||||
|
golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
|
||||||
|
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||||
|
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||||
|
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||||
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
|
||||||
|
gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
|
||||||
|
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||||
|
gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
|
||||||
|
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||||
176
benchmarks/s3billion/main.go
Normal file
176
benchmarks/s3billion/main.go
Normal file
|
|
@ -0,0 +1,176 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/tls"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"math/rand"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"strconv"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/minio/minio-go/v7"
|
||||||
|
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||||
|
)
|
||||||
|
|
||||||
|
type PRNG struct {
|
||||||
|
rem int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *PRNG) Read(p []byte) (n int, err error) {
|
||||||
|
//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
|
||||||
|
if int64(len(p)) > r.rem {
|
||||||
|
p = p[:r.rem]
|
||||||
|
}
|
||||||
|
|
||||||
|
if int64(len(p)) > r.rem {
|
||||||
|
log.Fatal("LOGIC ERROR")
|
||||||
|
}
|
||||||
|
|
||||||
|
n, err = rand.Read(p)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.rem -= int64(n)
|
||||||
|
if r.rem <= 0 {
|
||||||
|
err = io.EOF
|
||||||
|
//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
func putObj(mc *minio.Client, buck string, size int64) error {
|
||||||
|
prng := new(PRNG)
|
||||||
|
prng.rem = size
|
||||||
|
|
||||||
|
key := uuid.New().String()
|
||||||
|
|
||||||
|
_, err := mc.PutObject(
|
||||||
|
context.Background(),
|
||||||
|
buck,
|
||||||
|
key,
|
||||||
|
prng,
|
||||||
|
size,
|
||||||
|
minio.PutObjectOptions{ContentType: "application/octet-stream"},
|
||||||
|
)
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
fmt.Printf("total_objects,batch_dur_nanoseconds\n")
|
||||||
|
|
||||||
|
minio.MaxRetry = 1
|
||||||
|
|
||||||
|
_, isSSL := os.LookupEnv("SSL")
|
||||||
|
opts := minio.Options{
|
||||||
|
Creds: credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
|
||||||
|
Secure: isSSL,
|
||||||
|
}
|
||||||
|
|
||||||
|
if region, ok := os.LookupEnv("AWS_REGION"); ok {
|
||||||
|
opts.Region = region
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
|
||||||
|
opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
|
||||||
|
}
|
||||||
|
|
||||||
|
mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal("failed connect", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
thread := 32
|
||||||
|
if env_thread, ok := os.LookupEnv("THREAD"); ok {
|
||||||
|
tmp, err := strconv.Atoi(env_thread)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("invalid value for THREAD: %v\n", env_thread)
|
||||||
|
}
|
||||||
|
thread = tmp
|
||||||
|
}
|
||||||
|
|
||||||
|
batch_size := 256
|
||||||
|
if env_batch_size, ok := os.LookupEnv("BATCH_SIZE"); ok {
|
||||||
|
tmp, err := strconv.Atoi(env_batch_size)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("invalid value for BATCH_SIZE: %v\n", env_batch_size)
|
||||||
|
}
|
||||||
|
batch_size = tmp
|
||||||
|
}
|
||||||
|
|
||||||
|
batch_count := 128
|
||||||
|
if env_batch_count, ok := os.LookupEnv("BATCH_COUNT"); ok {
|
||||||
|
tmp, err := strconv.Atoi(env_batch_count)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("invalid value for BATCH_COUNT: %v\n", env_batch_count)
|
||||||
|
}
|
||||||
|
batch_count = tmp
|
||||||
|
}
|
||||||
|
|
||||||
|
obj_size := 16
|
||||||
|
if env_obj_size, ok := os.LookupEnv("OBJ_SIZE"); ok {
|
||||||
|
tmp, err := strconv.Atoi(env_obj_size)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("invalid value for OBJ_SIZE: %v\n", env_obj_size)
|
||||||
|
}
|
||||||
|
obj_size = tmp
|
||||||
|
}
|
||||||
|
|
||||||
|
total_obj := thread * batch_size * batch_count
|
||||||
|
total_size := total_obj * obj_size
|
||||||
|
log.Printf("if bench succeed, %v objects (%v bytes) will be created\n", total_obj, total_size)
|
||||||
|
|
||||||
|
// Create Bucket
|
||||||
|
buck := uuid.New().String()
|
||||||
|
err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
log.Printf("created bucket %s\n", buck)
|
||||||
|
|
||||||
|
// Start sending...
|
||||||
|
for bc := 0; bc < batch_count; bc++ {
|
||||||
|
log.Printf("batch %d/%d - start\n", bc+1, batch_count)
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
syn := make(chan error)
|
||||||
|
|
||||||
|
for tc := 0; tc < thread; tc++ {
|
||||||
|
go func() {
|
||||||
|
for bs := 0; bs < batch_size; bs++ {
|
||||||
|
err := putObj(mc, buck, int64(obj_size))
|
||||||
|
if err != nil {
|
||||||
|
syn <- err
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
syn <- nil
|
||||||
|
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
log.Printf("batch %d/%d - all threads started\n", bc+1, batch_count)
|
||||||
|
|
||||||
|
errCount := 0
|
||||||
|
for tc := 0; tc < thread; tc++ {
|
||||||
|
cerr := <-syn
|
||||||
|
if cerr != nil {
|
||||||
|
errCount += 1
|
||||||
|
log.Printf("thread %d/%d failed with %s\n", tc, thread, cerr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if errCount > 0 {
|
||||||
|
log.Fatal("Too many errors, exiting...")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
fmt.Printf("%d,%v\n", bc * thread * batch_size, elapsed.Nanoseconds())
|
||||||
|
log.Printf("batch %d/%d - all threads returned\n", bc+1, batch_count)
|
||||||
|
}
|
||||||
|
}
|
||||||
1
benchmarks/s3concurrent/.gitignore
vendored
Normal file
1
benchmarks/s3concurrent/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
s3concurrent
|
||||||
24
benchmarks/s3concurrent/go.mod
Normal file
24
benchmarks/s3concurrent/go.mod
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
module git.deuxfleurs.fr/quentin/s3concurrent
|
||||||
|
|
||||||
|
go 1.18
|
||||||
|
|
||||||
|
require github.com/minio/minio-go/v7 v7.0.34
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/dustin/go-humanize v1.0.0 // indirect
|
||||||
|
github.com/google/uuid v1.3.0 // indirect
|
||||||
|
github.com/json-iterator/go v1.1.12 // indirect
|
||||||
|
github.com/klauspost/compress v1.15.9 // indirect
|
||||||
|
github.com/klauspost/cpuid/v2 v2.1.0 // indirect
|
||||||
|
github.com/minio/md5-simd v1.1.2 // indirect
|
||||||
|
github.com/minio/sha256-simd v1.0.0 // indirect
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
||||||
|
github.com/modern-go/reflect2 v1.0.2 // indirect
|
||||||
|
github.com/rs/xid v1.4.0 // indirect
|
||||||
|
github.com/sirupsen/logrus v1.9.0 // indirect
|
||||||
|
golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa // indirect
|
||||||
|
golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
|
||||||
|
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
|
||||||
|
golang.org/x/text v0.3.7 // indirect
|
||||||
|
gopkg.in/ini.v1 v1.66.6 // indirect
|
||||||
|
)
|
||||||
52
benchmarks/s3concurrent/go.sum
Normal file
52
benchmarks/s3concurrent/go.sum
Normal file
|
|
@ -0,0 +1,52 @@
|
||||||
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
|
||||||
|
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
|
||||||
|
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||||
|
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
|
||||||
|
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
|
||||||
|
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
|
||||||
|
github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY=
|
||||||
|
github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
|
||||||
|
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||||
|
github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||||
|
github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0=
|
||||||
|
github.com/klauspost/cpuid/v2 v2.1.0/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY=
|
||||||
|
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||||
|
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||||
|
github.com/minio/minio-go/v7 v7.0.34 h1:JMfS5fudx1mN6V2MMNyCJ7UMrjEzZzIvMgfkWc1Vnjk=
|
||||||
|
github.com/minio/minio-go/v7 v7.0.34/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw=
|
||||||
|
github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g=
|
||||||
|
github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
|
||||||
|
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY=
|
||||||
|
github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
|
||||||
|
github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0=
|
||||||
|
github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
|
||||||
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
|
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
|
||||||
|
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
|
golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c=
|
||||||
|
golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
|
||||||
|
golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
|
||||||
|
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
||||||
|
golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
|
||||||
|
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
|
||||||
|
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||||
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI=
|
||||||
|
gopkg.in/ini.v1 v1.66.6/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
|
||||||
|
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
|
||||||
|
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
132
benchmarks/s3concurrent/main.go
Normal file
132
benchmarks/s3concurrent/main.go
Normal file
|
|
@ -0,0 +1,132 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/tls"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"math/rand"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/minio/minio-go/v7"
|
||||||
|
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||||
|
)
|
||||||
|
|
||||||
|
func buildMc() (*minio.Client, error) {
|
||||||
|
_, isSSL := os.LookupEnv("SSL")
|
||||||
|
opts := minio.Options{
|
||||||
|
Creds: credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
|
||||||
|
Secure: isSSL,
|
||||||
|
}
|
||||||
|
|
||||||
|
if region, ok := os.LookupEnv("REGION"); ok {
|
||||||
|
opts.Region = region
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
|
||||||
|
opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
|
||||||
|
}
|
||||||
|
|
||||||
|
mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
|
||||||
|
return mc, err
|
||||||
|
}
|
||||||
|
|
||||||
|
type PRNG struct {
|
||||||
|
rem int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *PRNG) Read(p []byte) (n int, err error) {
|
||||||
|
//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
|
||||||
|
if int64(len(p)) > r.rem {
|
||||||
|
p = p[:r.rem]
|
||||||
|
}
|
||||||
|
|
||||||
|
if int64(len(p)) > r.rem {
|
||||||
|
log.Fatal("LOGIC ERROR")
|
||||||
|
}
|
||||||
|
|
||||||
|
n, err = rand.Read(p)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.rem -= int64(n)
|
||||||
|
if r.rem <= 0 {
|
||||||
|
err = io.EOF
|
||||||
|
//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
func putObj(buck string, size int64) error {
|
||||||
|
mc, err := buildMc()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
prng := new(PRNG)
|
||||||
|
prng.rem = size
|
||||||
|
|
||||||
|
key := uuid.New().String()
|
||||||
|
|
||||||
|
_, err = mc.PutObject(
|
||||||
|
context.Background(),
|
||||||
|
buck,
|
||||||
|
key,
|
||||||
|
prng,
|
||||||
|
size,
|
||||||
|
minio.PutObjectOptions{ContentType: "application/octet-stream"},
|
||||||
|
)
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
minio.MaxRetry = 1
|
||||||
|
mc, err := buildMc()
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal("failed connect", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create Bucket
|
||||||
|
buck := uuid.New().String()
|
||||||
|
err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
log.Printf("created bucket %s\n", buck)
|
||||||
|
|
||||||
|
fmt.Println("sent,success,elapsed,elapsed_per_req")
|
||||||
|
|
||||||
|
// Send to bucket
|
||||||
|
for i := 1; i <= 16; i++ {
|
||||||
|
log.Printf("start concurrent loop with %d coroutines\n", i)
|
||||||
|
syn := make(chan error)
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
for j := 1; j <= i; j++ {
|
||||||
|
go func() {
|
||||||
|
syn <- putObj(buck, 1024*1024)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
errCount := 0
|
||||||
|
for j := 1; j <= i; j++ {
|
||||||
|
cerr := <-syn
|
||||||
|
if cerr != nil {
|
||||||
|
errCount += 1
|
||||||
|
log.Printf("%d/%d failed with %s\n", j, i, cerr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
fmt.Printf("%d,%d,%v,%v\n", i, i - errCount, elapsed.Nanoseconds(), elapsed.Nanoseconds() / int64(i))
|
||||||
|
log.Printf("done, %d coroutines returned\n", i)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Println("bench done")
|
||||||
|
}
|
||||||
1
benchmarks/s3lat/.gitignore
vendored
Normal file
1
benchmarks/s3lat/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
s3lat
|
||||||
22
benchmarks/s3lat/README.md
Normal file
22
benchmarks/s3lat/README.md
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
# s3lat
|
||||||
|
|
||||||
|
## installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go get git.deuxfleurs.fr/quentin/s3lat@latest
|
||||||
|
```
|
||||||
|
|
||||||
|
## usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export ENDPOINT=[fc00:9a7a:9e::1]:9000
|
||||||
|
export AWS_ACCESS_KEY_ID=minioadmin
|
||||||
|
export AWS_SECRET_ACCESS_KEY=minioadmin
|
||||||
|
|
||||||
|
s3lat
|
||||||
|
```
|
||||||
|
|
||||||
|
## see also
|
||||||
|
|
||||||
|
- https://git.deuxfleurs.fr/quentin/benchmarks
|
||||||
|
- https://git.deuxfleurs.fr/trinity-1686a/mknet
|
||||||
8
benchmarks/s3lat/go.mod
Normal file
8
benchmarks/s3lat/go.mod
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
module git.deuxfleurs.fr/quentin/s3lat
|
||||||
|
|
||||||
|
go 1.16
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/google/uuid v1.1.1
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16
|
||||||
|
)
|
||||||
78
benchmarks/s3lat/go.sum
Normal file
78
benchmarks/s3lat/go.sum
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
|
||||||
|
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
|
||||||
|
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||||
|
github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
|
||||||
|
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
|
||||||
|
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
|
||||||
|
github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
|
||||||
|
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
|
||||||
|
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
|
||||||
|
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
|
||||||
|
github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
|
||||||
|
github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
|
||||||
|
github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
|
||||||
|
github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
|
||||||
|
github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
|
||||||
|
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||||
|
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||||
|
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||||
|
github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
|
||||||
|
github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
|
||||||
|
github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
|
||||||
|
github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
|
||||||
|
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
|
||||||
|
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
|
||||||
|
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
|
||||||
|
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
|
||||||
|
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
|
||||||
|
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
|
||||||
|
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
|
||||||
|
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
|
||||||
|
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
|
||||||
|
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
|
||||||
|
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
|
||||||
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
|
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
|
||||||
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
|
github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
|
||||||
|
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
|
||||||
|
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||||
|
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||||
|
golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
|
||||||
|
golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
|
||||||
|
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||||
|
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||||
|
golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
|
||||||
|
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
|
||||||
|
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||||
|
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae h1:Ih9Yo4hSPImZOpfGuA4bR/ORKTAbhZo2AbWNRCnevdo=
|
||||||
|
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
|
||||||
|
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||||
|
golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
|
||||||
|
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||||
|
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||||
|
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||||
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
|
||||||
|
gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
|
||||||
|
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||||
|
gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
|
||||||
|
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||||
135
benchmarks/s3lat/main.go
Normal file
135
benchmarks/s3lat/main.go
Normal file
|
|
@ -0,0 +1,135 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
"io"
|
||||||
|
"io/ioutil"
|
||||||
|
"strings"
|
||||||
|
"net/http"
|
||||||
|
"crypto/tls"
|
||||||
|
"strconv"
|
||||||
|
|
||||||
|
"github.com/minio/minio-go/v7"
|
||||||
|
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
fmt.Printf("endpoint,nanoseconds\n")
|
||||||
|
|
||||||
|
// Initial setup
|
||||||
|
_, isSSL := os.LookupEnv("SSL");
|
||||||
|
opts := minio.Options {
|
||||||
|
Creds: credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
|
||||||
|
Secure: isSSL,
|
||||||
|
}
|
||||||
|
|
||||||
|
if region, ok := os.LookupEnv("REGION"); ok {
|
||||||
|
opts.Region = region
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
|
||||||
|
opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
|
||||||
|
}
|
||||||
|
|
||||||
|
mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal("failed connect", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create Bucket
|
||||||
|
buck := uuid.New().String()
|
||||||
|
err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{ })
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// List Buckets
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
start := time.Now()
|
||||||
|
_, err := mc.ListBuckets(context.Background())
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal("failed listbucket: ", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Printf("listbuckets,%v\n", elapsed.Nanoseconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
// PutObject
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
istr := strconv.Itoa(i)
|
||||||
|
content := istr + " hello world " + istr
|
||||||
|
start := time.Now()
|
||||||
|
_, err := mc.PutObject(context.Background(), buck, "element"+istr, strings.NewReader(content), int64(len(content)), minio.PutObjectOptions{ContentType:"application/octet-stream"})
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal("failed putObject: ",err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Printf("putobject,%v\n", elapsed.Nanoseconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListObject
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
objectCh := mc.ListObjects(ctx, buck, minio.ListObjectsOptions{
|
||||||
|
Recursive: true,
|
||||||
|
})
|
||||||
|
for object := range objectCh {
|
||||||
|
if object.Err != nil {
|
||||||
|
log.Fatal(object.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
fmt.Printf("listobjects,%v\n", elapsed.Nanoseconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetObject
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
istr := strconv.Itoa(i)
|
||||||
|
start := time.Now()
|
||||||
|
object, err := mc.GetObject(context.Background(), buck, "element"+istr, minio.GetObjectOptions{})
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, err = io.Copy(ioutil.Discard, object) ; err != nil {
|
||||||
|
log.Fatal("failed getobject: ", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
fmt.Printf("getobject,%v\n", elapsed.Nanoseconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
// RemoveObject
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
istr := strconv.Itoa(i)
|
||||||
|
start := time.Now()
|
||||||
|
err = mc.RemoveObject(context.Background(), buck, "element"+istr, minio.RemoveObjectOptions{})
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Printf("removeobject,%v\n", elapsed.Nanoseconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
// RemoveBucket
|
||||||
|
err = mc.RemoveBucket(context.Background(), buck)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
1
benchmarks/s3ttfb/.gitignore
vendored
Normal file
1
benchmarks/s3ttfb/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
s3ttfb
|
||||||
12
benchmarks/s3ttfb/go.mod
Normal file
12
benchmarks/s3ttfb/go.mod
Normal file
|
|
@ -0,0 +1,12 @@
|
||||||
|
module git.deuxfleurs.fr/Deuxfleurs/mknet/benchmarks/s3ttfb
|
||||||
|
|
||||||
|
go 1.16
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/aws/aws-sdk-go-v2 v1.16.16
|
||||||
|
github.com/aws/aws-sdk-go-v2/config v1.17.7
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11
|
||||||
|
github.com/google/uuid v1.1.1
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16
|
||||||
|
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd // indirect
|
||||||
|
)
|
||||||
123
benchmarks/s3ttfb/go.sum
Normal file
123
benchmarks/s3ttfb/go.sum
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
github.com/aws/aws-sdk-go-v2 v1.16.16 h1:M1fj4FE2lB4NzRb9Y0xdWsn2P0+2UHVxwKyOa4YJNjk=
|
||||||
|
github.com/aws/aws-sdk-go-v2 v1.16.16/go.mod h1:SwiyXi/1zTUZ6KIAmLK5V5ll8SiURNUYOqTerZPaF9k=
|
||||||
|
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.8 h1:tcFliCWne+zOuUfKNRn8JdFBuWPDuISDH08wD2ULkhk=
|
||||||
|
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.8/go.mod h1:JTnlBSot91steJeti4ryyu/tLd4Sk84O5W22L7O2EQU=
|
||||||
|
github.com/aws/aws-sdk-go-v2/config v1.17.7 h1:odVM52tFHhpqZBKNjVW5h+Zt1tKHbhdTQRb+0WHrNtw=
|
||||||
|
github.com/aws/aws-sdk-go-v2/config v1.17.7/go.mod h1:dN2gja/QXxFF15hQreyrqYhLBaQo1d9ZKe/v/uplQoI=
|
||||||
|
github.com/aws/aws-sdk-go-v2/credentials v1.12.20 h1:9+ZhlDY7N9dPnUmf7CDfW9In4sW5Ff3bh7oy4DzS1IE=
|
||||||
|
github.com/aws/aws-sdk-go-v2/credentials v1.12.20/go.mod h1:UKY5HyIux08bbNA7Blv4PcXQ8cTkGh7ghHMFklaviR4=
|
||||||
|
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.17 h1:r08j4sbZu/RVi+BNxkBJwPMUYY3P8mgSDuKkZ/ZN1lE=
|
||||||
|
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.17/go.mod h1:yIkQcCDYNsZfXpd5UX2Cy+sWA1jPgIhGTw9cOBzfVnQ=
|
||||||
|
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.23 h1:s4g/wnzMf+qepSNgTvaQQHNxyMLKSawNhKCPNy++2xY=
|
||||||
|
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.23/go.mod h1:2DFxAQ9pfIRy0imBCJv+vZ2X6RKxves6fbnEuSry6b4=
|
||||||
|
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.17 h1:/K482T5A3623WJgWT8w1yRAFK4RzGzEl7y39yhtn9eA=
|
||||||
|
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.17/go.mod h1:pRwaTYCJemADaqCbUAxltMoHKata7hmB5PjEXeu0kfg=
|
||||||
|
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.24 h1:wj5Rwc05hvUSvKuOF29IYb9QrCLjU+rHAy/x/o0DK2c=
|
||||||
|
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.24/go.mod h1:jULHjqqjDlbyTa7pfM7WICATnOv+iOhjletM3N0Xbu8=
|
||||||
|
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.14 h1:ZSIPAkAsCCjYrhqfw2+lNzWDzxzHXEckFkTePL5RSWQ=
|
||||||
|
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.14/go.mod h1:AyGgqiKv9ECM6IZeNQtdT8NnMvUb3/2wokeq2Fgryto=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.9 h1:Lh1AShsuIJTwMkoxVCAYPJgNG5H+eN6SmoUn8nOZ5wE=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.9/go.mod h1:a9j48l6yL5XINLHLcOKInjdvknN+vWqPBxqeIDw7ktw=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.18 h1:BBYoNQt2kUZUUK4bIPsKrCcjVPUMNsgQpNAwhznK/zo=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.18/go.mod h1:NS55eQ4YixUJPTC+INxi2/jCqe1y2Uw3rnh9wEOVJxY=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.17 h1:Jrd/oMh0PKQc6+BowB+pLEwLIgaQF29eYbe7E1Av9Ug=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.17/go.mod h1:4nYOrY41Lrbk2170/BGkcJKBhws9Pfn8MG3aGqjjeFI=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.17 h1:HfVVR1vItaG6le+Bpw6P4midjBDMKnjMyZnw9MXYUcE=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.17/go.mod h1:YqMdV+gEKCQ59NrB7rzrJdALeBIsYiVi8Inj3+KcqHI=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11 h1:3/gm/JTX9bX8CpzTgIlrtYpB3EVBDxyg/GY/QdcIEZw=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/s3 v1.27.11/go.mod h1:fmgDANqTUCxciViKl9hb/zD5LFbvPINFRgWhDbR+vZo=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/sso v1.11.23 h1:pwvCchFUEnlceKIgPUouBJwK81aCkQ8UDMORfeFtW10=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/sso v1.11.23/go.mod h1:/w0eg9IhFGjGyyncHIQrXtU8wvNsTJOP0R6PPj0wf80=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.5 h1:GUnZ62TevLqIoDyHeiWj2P7EqaosgakBKVvWriIdLQY=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.5/go.mod h1:csZuQY65DAdFBt1oIjO5hhBR49kQqop4+lcuCjf2arA=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/sts v1.16.19 h1:9pPi0PsFNAGILFfPCk8Y0iyEBGc6lu6OQ97U7hmdesg=
|
||||||
|
github.com/aws/aws-sdk-go-v2/service/sts v1.16.19/go.mod h1:h4J3oPZQbxLhzGnk+j9dfYHi5qIOVJ5kczZd658/ydM=
|
||||||
|
github.com/aws/smithy-go v1.13.3 h1:l7LYxGuzK6/K+NzJ2mC+VvLUbae0sL3bXU//04MkmnA=
|
||||||
|
github.com/aws/smithy-go v1.13.3/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
|
||||||
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
|
||||||
|
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
|
||||||
|
github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
|
||||||
|
github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||||
|
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||||
|
github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
|
||||||
|
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
|
||||||
|
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
|
||||||
|
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
|
||||||
|
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
|
||||||
|
github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
|
||||||
|
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
|
||||||
|
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
|
||||||
|
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
|
||||||
|
github.com/klauspost/compress v1.13.5 h1:9O69jUPDcsT9fEm74W92rZL9FQY7rCdaXVneq+yyzl4=
|
||||||
|
github.com/klauspost/compress v1.13.5/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
|
||||||
|
github.com/klauspost/cpuid v1.2.3/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
|
||||||
|
github.com/klauspost/cpuid v1.3.1 h1:5JNjFYYQrZeKRJ0734q51WCEEn2huer72Dc7K+R/b6s=
|
||||||
|
github.com/klauspost/cpuid v1.3.1/go.mod h1:bYW4mA6ZgKPob1/Dlai2LviZJO7KGI3uoWLd42rAQw4=
|
||||||
|
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||||
|
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||||
|
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||||
|
github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4=
|
||||||
|
github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw=
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4=
|
||||||
|
github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g=
|
||||||
|
github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
|
||||||
|
github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
|
||||||
|
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
|
||||||
|
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||||
|
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
|
||||||
|
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
|
||||||
|
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
|
||||||
|
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
|
||||||
|
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
|
||||||
|
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
|
||||||
|
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
|
||||||
|
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
|
||||||
|
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
|
||||||
|
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
|
||||||
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
|
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
|
||||||
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
|
github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
|
||||||
|
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
|
||||||
|
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||||
|
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||||
|
golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f h1:aZp0e2vLN4MToVqnjNEYEtrEA8RH8U8FN1CU7JgqsPU=
|
||||||
|
golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
|
||||||
|
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||||
|
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||||
|
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
|
||||||
|
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
|
||||||
|
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
|
||||||
|
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||||
|
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e h1:fLOSk5Q00efkSvAm+4xcoXD+RRmLmmulPn5I3Y9F2EM=
|
||||||
|
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
|
||||||
|
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||||
|
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||||
|
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||||
|
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
|
||||||
|
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||||
|
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||||
|
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||||
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww=
|
||||||
|
gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
|
||||||
|
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||||
|
gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
|
||||||
|
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||||
214
benchmarks/s3ttfb/main.go
Normal file
214
benchmarks/s3ttfb/main.go
Normal file
|
|
@ -0,0 +1,214 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/tls"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"io/ioutil"
|
||||||
|
"log"
|
||||||
|
"math/rand"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptrace"
|
||||||
|
"os"
|
||||||
|
"strconv"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/minio/minio-go/v7"
|
||||||
|
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/config"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
)
|
||||||
|
|
||||||
|
type PRNG struct {
|
||||||
|
rem int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *PRNG) Read(p []byte) (n int, err error) {
|
||||||
|
//log.Printf("rem=%d, buf=%d\n", r.rem, len(p))
|
||||||
|
if int64(len(p)) > r.rem {
|
||||||
|
p = p[:r.rem]
|
||||||
|
}
|
||||||
|
|
||||||
|
if int64(len(p)) > r.rem {
|
||||||
|
log.Fatal("LOGIC ERROR")
|
||||||
|
}
|
||||||
|
|
||||||
|
n, err = rand.Read(p)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.rem -= int64(n)
|
||||||
|
if r.rem <= 0 {
|
||||||
|
err = io.EOF
|
||||||
|
//log.Printf("PRNG file has been fully read. rem=%d,n=%d,err=%s\n", r.rem, n, err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
var err error
|
||||||
|
fmt.Printf("nanoseconds\n")
|
||||||
|
|
||||||
|
// Initial setup
|
||||||
|
_, isSSL := os.LookupEnv("SSL")
|
||||||
|
opts := minio.Options{
|
||||||
|
Creds: credentials.NewStaticV4(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
|
||||||
|
Secure: isSSL,
|
||||||
|
}
|
||||||
|
|
||||||
|
if region, ok := os.LookupEnv("AWS_REGION"); ok {
|
||||||
|
opts.Region = region
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, ok := os.LookupEnv("SSL_INSECURE"); ok {
|
||||||
|
opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
|
||||||
|
}
|
||||||
|
|
||||||
|
size := 1 * 1024 * 1024 // 1 MB
|
||||||
|
if env_size, ok := os.LookupEnv("SIZE"); ok {
|
||||||
|
size, err = strconv.Atoi(env_size)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
repeat := 60
|
||||||
|
if env_repeat, ok := os.LookupEnv("REPEAT"); ok {
|
||||||
|
repeat, err = strconv.Atoi(env_repeat)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
mc, err := minio.New(os.Getenv("ENDPOINT"), &opts)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal("failed connect", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// AWS SDK stuff
|
||||||
|
customResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
|
||||||
|
return aws.Endpoint{
|
||||||
|
PartitionID: "aws",
|
||||||
|
URL: "http://"+os.Getenv("ENDPOINT"),
|
||||||
|
SigningRegion: region,
|
||||||
|
}, nil
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithEndpointResolverWithOptions(customResolver))
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("failed to load configuration, %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create an Amazon S3 service client
|
||||||
|
client := s3.NewFromConfig(cfg, func(opts *s3.Options) {
|
||||||
|
opts.UsePathStyle = true
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
// Create Bucket
|
||||||
|
buck := uuid.New().String()
|
||||||
|
err = mc.MakeBucket(context.Background(), buck, minio.MakeBucketOptions{})
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Send an object
|
||||||
|
prng := new(PRNG)
|
||||||
|
prng.rem = int64(size)
|
||||||
|
|
||||||
|
|
||||||
|
key := uuid.New().String()
|
||||||
|
_, err = mc.PutObject(
|
||||||
|
context.Background(),
|
||||||
|
buck,
|
||||||
|
key,
|
||||||
|
prng,
|
||||||
|
int64(size),
|
||||||
|
minio.PutObjectOptions{ContentType: "application/octet-stream"},
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal("unable to upload our test object")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
for i := 0; i < repeat; i++ {
|
||||||
|
// Now, try to fetch it *repeat* times (assume no caching)
|
||||||
|
var elapsed time.Duration
|
||||||
|
|
||||||
|
var start time.Time
|
||||||
|
trace := &httptrace.ClientTrace{
|
||||||
|
GotFirstResponseByte: func() { elapsed = time.Since(start) },
|
||||||
|
}
|
||||||
|
|
||||||
|
start = time.Now()
|
||||||
|
/*object, err := mc.GetObject(httptrace.WithClientTrace(context.Background(), trace), buck, key, minio.GetObjectOptions{})
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}*/
|
||||||
|
|
||||||
|
out, err := client.GetObject(httptrace.WithClientTrace(context.Background(), trace), &s3.GetObjectInput {
|
||||||
|
Bucket: aws.String(buck),
|
||||||
|
Key: aws.String(key),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
object := out.Body
|
||||||
|
|
||||||
|
|
||||||
|
if _, err = io.Copy(ioutil.Discard, object) ; err != nil {
|
||||||
|
log.Fatal("failed getobject: ", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf("%v\n", elapsed.Nanoseconds())
|
||||||
|
//time.Sleep(10 * time.Second)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
func timeGet(url string) {
|
||||||
|
req, _ := http.NewRequest("GET", url, nil)
|
||||||
|
|
||||||
|
var start, connect, dns, tlsHandshake time.Time
|
||||||
|
|
||||||
|
trace := &httptrace.ClientTrace{
|
||||||
|
DNSStart: func(dsi httptrace.DNSStartInfo) { dns = time.Now() },
|
||||||
|
DNSDone: func(ddi httptrace.DNSDoneInfo) {
|
||||||
|
fmt.Printf("DNS Done: %v\n", time.Since(dns))
|
||||||
|
},
|
||||||
|
|
||||||
|
TLSHandshakeStart: func() { tlsHandshake = time.Now() },
|
||||||
|
TLSHandshakeDone: func(cs tls.ConnectionState, err error) {
|
||||||
|
fmt.Printf("TLS Handshake: %v\n", time.Since(tlsHandshake))
|
||||||
|
},
|
||||||
|
|
||||||
|
ConnectStart: func(network, addr string) { connect = time.Now() },
|
||||||
|
ConnectDone: func(network, addr string, err error) {
|
||||||
|
fmt.Printf("Connect time: %v\n", time.Since(connect))
|
||||||
|
},
|
||||||
|
|
||||||
|
GotFirstResponseByte: func() {
|
||||||
|
fmt.Printf("Time from start to first byte: %v\n", time.Since(start))
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
|
||||||
|
start = time.Now()
|
||||||
|
if _, err := http.DefaultTransport.RoundTrip(req); err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
fmt.Printf("Total time: %v\n", time.Since(start))
|
||||||
|
}
|
||||||
|
*/
|
||||||
1
benchmarks/warp
Submodule
1
benchmarks/warp
Submodule
|
|
@ -0,0 +1 @@
|
||||||
|
Subproject commit df8f2cd7b7a8523755b4d3c04271a69571a455ca
|
||||||
|
|
@ -1,61 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
set -euo pipefail
|
|
||||||
IFS=$'\n\t'
|
|
||||||
|
|
||||||
GARAGE_PATH=garage
|
|
||||||
STORAGE_PATH=/tmp/garage-testnet
|
|
||||||
export RPC_SECRET=3e9abff5f9e480afbadb46a77b7a26fe0e404258f0dc3fd5386b0ba8e0ad2fba
|
|
||||||
|
|
||||||
if [ -z "$ZONE" ]; then
|
|
||||||
NODE_STORAGE_PATH=${STORAGE_PATH}/${HOST}
|
|
||||||
else
|
|
||||||
NODE_STORAGE_PATH=${STORAGE_PATH}/${ZONE}/${HOST}
|
|
||||||
fi
|
|
||||||
BOOTSTRAP_FILE=${STORAGE_PATH}/bootstrap_peer
|
|
||||||
export GARAGE_CONFIG_FILE=${NODE_STORAGE_PATH}/garage.toml
|
|
||||||
|
|
||||||
|
|
||||||
mkdir -p ${NODE_STORAGE_PATH}
|
|
||||||
cd ${NODE_STORAGE_PATH}
|
|
||||||
rm ${BOOTSTRAP_FILE} 2>/dev/null || true
|
|
||||||
|
|
||||||
cat > ${GARAGE_CONFIG_FILE} << EOF
|
|
||||||
metadata_dir = "${NODE_STORAGE_PATH}/meta"
|
|
||||||
data_dir = "${NODE_STORAGE_PATH}/data"
|
|
||||||
|
|
||||||
replication_mode = "3"
|
|
||||||
|
|
||||||
rpc_bind_addr = "[::]:3901"
|
|
||||||
rpc_public_addr = "[${IP}]:3901"
|
|
||||||
rpc_secret = "${RPC_SECRET}"
|
|
||||||
|
|
||||||
bootstrap_peers=[]
|
|
||||||
|
|
||||||
[s3_api]
|
|
||||||
s3_region = "garage"
|
|
||||||
api_bind_addr = "[::]:3900"
|
|
||||||
root_domain = ".s3.garage"
|
|
||||||
|
|
||||||
[s3_web]
|
|
||||||
bind_addr = "[::]:3902"
|
|
||||||
root_domain = ".web.garage"
|
|
||||||
index = "index.html"
|
|
||||||
EOF
|
|
||||||
|
|
||||||
RUST_LOG=garage=debug ${GARAGE_PATH} server 2>> ${NODE_STORAGE_PATH}/logs & disown
|
|
||||||
sleep 2
|
|
||||||
|
|
||||||
CONFIG_NODE_FPATH=$(find /tmp/garage-testnet/ -maxdepth 3 -name garage.toml|head -n 1)
|
|
||||||
|
|
||||||
SELF_ID=$(${GARAGE_PATH} node id 2>/dev/null)
|
|
||||||
SHORT_ID=$(echo ${SELF_ID} | cut -c-64)
|
|
||||||
|
|
||||||
${GARAGE_PATH} -c ${CONFIG_NODE_FPATH} node connect ${SELF_ID}
|
|
||||||
${GARAGE_PATH} -c ${CONFIG_NODE_FPATH} layout assign ${SHORT_ID} -z ${ZONE:-unzonned-${HOST}} -c 1 -t ${HOST}
|
|
||||||
|
|
||||||
if [ ${CONFIG_NODE_FPATH} == ${GARAGE_CONFIG_FILE} ]; then
|
|
||||||
sleep 2
|
|
||||||
${GARAGE_PATH} layout show
|
|
||||||
${GARAGE_PATH} layout apply --version 1
|
|
||||||
fi
|
|
||||||
|
|
@ -1,62 +0,0 @@
|
||||||
#!/usr/bin/env python3
|
|
||||||
import json, os, sys, time, pathlib, socket, shutil
|
|
||||||
|
|
||||||
STORAGE_PATH = os.path.join(os.getcwd(), '.minio-testnet')
|
|
||||||
HOSTS_PATH = os.path.join(STORAGE_PATH, 'hosts.txt')
|
|
||||||
UNIX_SOCK = os.path.join(STORAGE_PATH, 'deploy.sock')
|
|
||||||
DATA_PATH = lambda nid: os.path.join(STORAGE_PATH, 'data'+str(nid))
|
|
||||||
|
|
||||||
def main():
|
|
||||||
if int(os.environ['ID']) == 1: leader()
|
|
||||||
else: follower()
|
|
||||||
|
|
||||||
def leader():
|
|
||||||
shutil.rmtree(STORAGE_PATH, ignore_errors=True)
|
|
||||||
os.makedirs(STORAGE_PATH)
|
|
||||||
print(STORAGE_PATH)
|
|
||||||
|
|
||||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
|
||||||
sock.bind(UNIX_SOCK)
|
|
||||||
sock.listen()
|
|
||||||
|
|
||||||
n_serv = int(os.environ['SERVER_COUNT'])
|
|
||||||
fl = [ co for co, addr in [ sock.accept() for i in range(n_serv - 1) ]]
|
|
||||||
|
|
||||||
identities = [ json.loads(co.makefile().readline()) for co in fl ] + [ { "ip": os.environ['IP'], "path": make_data() } ]
|
|
||||||
print(f"ident: {identities}")
|
|
||||||
msg = f"{json.dumps(identities)}\n".encode()
|
|
||||||
[ co.send(msg) for co in fl ]
|
|
||||||
|
|
||||||
run_minio(identities)
|
|
||||||
|
|
||||||
def follower():
|
|
||||||
co = None
|
|
||||||
while True:
|
|
||||||
time.sleep(1)
|
|
||||||
try:
|
|
||||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
|
||||||
sock.connect(UNIX_SOCK)
|
|
||||||
co = sock.makefile()
|
|
||||||
break
|
|
||||||
except Exception as err:
|
|
||||||
print('conn failed, wait,', err)
|
|
||||||
my_identity = json.dumps({ "ip": os.environ['IP'], "path": make_data() })
|
|
||||||
sock.send(f"{my_identity}\n".encode())
|
|
||||||
identities = json.loads(co.readline())
|
|
||||||
|
|
||||||
run_minio(identities)
|
|
||||||
|
|
||||||
def make_data():
|
|
||||||
data_path = DATA_PATH(os.environ['ID'])
|
|
||||||
os.makedirs(data_path)
|
|
||||||
return data_path
|
|
||||||
|
|
||||||
def run_minio(identities):
|
|
||||||
cmd = f"minio server --console-address ':9001' --address ':9000'"
|
|
||||||
for ident in identities:
|
|
||||||
cmd += f" http://[{ident['ip']}]:9000{ident['path']}"
|
|
||||||
cmd += f" > {os.path.join(STORAGE_PATH, 'minio'+os.environ['ID']+'.log')} 2>&1"
|
|
||||||
print("launch: ", cmd)
|
|
||||||
os.system(cmd)
|
|
||||||
|
|
||||||
__name__ == '__main__' and main()
|
|
||||||
31
mknet
31
mknet
|
|
@ -87,19 +87,23 @@ class Bandwidth:
|
||||||
|
|
||||||
|
|
||||||
class LinkInfo:
|
class LinkInfo:
|
||||||
def __init__(self, bandwidth, latency, jitter = None, offset = None, **kwargs):
|
def __init__(self, bandwidth, latency, jitter = None, offset = None, limit = None, txqueuelen = None, **kwargs):
|
||||||
self.bandwidth = Bandwidth(bandwidth)
|
self.bandwidth = Bandwidth(bandwidth)
|
||||||
self.latency = Latency(latency, offset)
|
self.latency = Latency(latency, offset)
|
||||||
self.jitter = Latency(jitter or 0)
|
self.jitter = Latency(jitter or 0)
|
||||||
|
self.limit = limit
|
||||||
|
self.txqueuelen = txqueuelen
|
||||||
|
|
||||||
def __eq__(self, o):
|
def __eq__(self, o):
|
||||||
return (isinstance(o, LinkInfo) and
|
return (isinstance(o, LinkInfo) and
|
||||||
o.bandwidth == self.bandwidth and
|
o.bandwidth == self.bandwidth and
|
||||||
o.latency == self.latency and
|
o.latency == self.latency and
|
||||||
o.jitter == self.jitter)
|
o.jitter == self.jitter and
|
||||||
|
o.limit == self.limit and
|
||||||
|
o.txqueuelen == self.txqueuelen)
|
||||||
|
|
||||||
def __str__(self):
|
def __str__(self):
|
||||||
return f'LinkInfo{{bw: {self.bandwidth}, latency: {self.latency}, jitter: {self.jitter}}}'
|
return f'LinkInfo{{bw: {self.bandwidth}, latency: {self.latency}, jitter: {self.jitter}, limit: {self.limit}, txqueuelen: {self.txqueuelen}}}'
|
||||||
|
|
||||||
class Server:
|
class Server:
|
||||||
def __init__(self, name, link):
|
def __init__(self, name, link):
|
||||||
|
|
@ -313,6 +317,7 @@ def runall(cmd):
|
||||||
zones = parse(config).zones
|
zones = parse(config).zones
|
||||||
|
|
||||||
number = 1
|
number = 1
|
||||||
|
wait_list = [ ]
|
||||||
for zone in zones.values():
|
for zone in zones.values():
|
||||||
if zone.is_zone():
|
if zone.is_zone():
|
||||||
for server in zone.servers.values():
|
for server in zone.servers.values():
|
||||||
|
|
@ -323,7 +328,7 @@ def runall(cmd):
|
||||||
env["ID"] = str(number)
|
env["ID"] = str(number)
|
||||||
env["SERVER_COUNT"] = str(len(config['servers']))
|
env["SERVER_COUNT"] = str(len(config['servers']))
|
||||||
name = f'testnet-{zone.name}-{server.name}'
|
name = f'testnet-{zone.name}-{server.name}'
|
||||||
net.ns.run(name, cmd, env)
|
wait_list.append(net.ns.run(name, cmd, env))
|
||||||
number +=1
|
number +=1
|
||||||
else:
|
else:
|
||||||
env = os.environ.copy()
|
env = os.environ.copy()
|
||||||
|
|
@ -333,9 +338,12 @@ def runall(cmd):
|
||||||
env["ID"] = str(number)
|
env["ID"] = str(number)
|
||||||
env["SERVER_COUNT"] = str(len(config['servers']))
|
env["SERVER_COUNT"] = str(len(config['servers']))
|
||||||
name = f'testnet-{zone.name}-{zone.name}'
|
name = f'testnet-{zone.name}-{zone.name}'
|
||||||
net.ns.run(name, cmd, env)
|
wait_list.append(net.ns.run(name, cmd, env))
|
||||||
first = False
|
first = False
|
||||||
number +=1
|
number +=1
|
||||||
|
for p in wait_list:
|
||||||
|
p.wait()
|
||||||
|
print("runall done")
|
||||||
|
|
||||||
def destroy():
|
def destroy():
|
||||||
for ns in net.ns.list():
|
for ns in net.ns.list():
|
||||||
|
|
@ -343,22 +351,31 @@ def destroy():
|
||||||
net.ns.forget("unconfined")
|
net.ns.forget("unconfined")
|
||||||
os.remove(".current_state.yml")
|
os.remove(".current_state.yml")
|
||||||
|
|
||||||
|
def scenario(config, cmd):
|
||||||
|
create(config)
|
||||||
|
runall(cmd)
|
||||||
|
destroy()
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
if len(sys.argv) < 2:
|
if len(sys.argv) < 2:
|
||||||
progname = os.path.basename(sys.argv[0]) if len(sys.argv) > 0 else "mknet"
|
progname = os.path.basename(sys.argv[0]) if len(sys.argv) > 0 else "mknet"
|
||||||
print(f"""Usage:
|
print(f"""Usage:
|
||||||
{progname} create [config_path] # create a new network. config_path defailt to config.yml
|
{progname} scenario <config_path> <cmd> [args...] # all in one simulation: create a network, run the command on all nodes, destroy the network
|
||||||
|
|
||||||
|
{progname} create [config_path] # create a new network. config_path defailt to topo/with-vdsl.yml
|
||||||
{progname} run-all <cmd> [args...] # run a command as each host. set the IP, NAME and ZONE environment variables
|
{progname} run-all <cmd> [args...] # run a command as each host. set the IP, NAME and ZONE environment variables
|
||||||
{progname} run <name> [cmd [args...]] # run command in host named <name>. Use zonename:name if multiple zones hosts server with same name. If cmd is empty, run a shell
|
{progname} run <name> [cmd [args...]] # run command in host named <name>. Use zonename:name if multiple zones hosts server with same name. If cmd is empty, run a shell
|
||||||
{progname} destroy # destroy the current environment""")
|
{progname} destroy # destroy the current environment""")
|
||||||
exit()
|
exit()
|
||||||
cmd = sys.argv[1]
|
cmd = sys.argv[1]
|
||||||
if cmd == "create":
|
if cmd == "create":
|
||||||
create(sys.argv[2] if len(sys.argv) > 2 else "config.yml")
|
create(sys.argv[2] if len(sys.argv) > 2 else "topo/with-vdsl.yml")
|
||||||
elif cmd == "run":
|
elif cmd == "run":
|
||||||
run(sys.argv[2], sys.argv[3:])
|
run(sys.argv[2], sys.argv[3:])
|
||||||
elif cmd == "run-all":
|
elif cmd == "run-all":
|
||||||
runall(sys.argv[2:])
|
runall(sys.argv[2:])
|
||||||
|
elif cmd == "scenario":
|
||||||
|
scenario(sys.argv[2], sys.argv[3:])
|
||||||
elif cmd == "destroy":
|
elif cmd == "destroy":
|
||||||
destroy()
|
destroy()
|
||||||
else:
|
else:
|
||||||
|
|
|
||||||
7
net.py
7
net.py
|
|
@ -37,7 +37,7 @@ class ns:
|
||||||
run_netns("exec", name, "ip", "link", "set", "dev", "lo", "up")
|
run_netns("exec", name, "ip", "link", "set", "dev", "lo", "up")
|
||||||
|
|
||||||
def run(name, cmd, env=None):
|
def run(name, cmd, env=None):
|
||||||
subprocess.Popen(_netns + ["exec", name] + cmd, env=env)
|
return subprocess.Popen(_netns + ["exec", name] + cmd, env=env)
|
||||||
|
|
||||||
def create_bridge(name, namespace, ports=[]):
|
def create_bridge(name, namespace, ports=[]):
|
||||||
run_netns("exec", namespace, "ip", "link", "add", "name", name, "type", "bridge")
|
run_netns("exec", namespace, "ip", "link", "add", "name", name, "type", "bridge")
|
||||||
|
|
@ -56,6 +56,9 @@ def create_veth(name1, ns1, name2, ns2, ip = None, subnet=0, link=None):
|
||||||
run_netns("exec", ns2, "ip", "link", "set", "dev", name2, "up")
|
run_netns("exec", ns2, "ip", "link", "set", "dev", name2, "up")
|
||||||
|
|
||||||
if link:
|
if link:
|
||||||
|
if link.txqueuelen:
|
||||||
|
run_netns("exec", ns1, "ip", "link", "set", "dev", name1, "txqueuelen", str(link.txqueuelen))
|
||||||
|
run_netns("exec", ns2, "ip", "link", "set", "dev", name2, "txqueuelen", str(link.txqueuelen))
|
||||||
tc(ns1, name1, link)
|
tc(ns1, name1, link)
|
||||||
tc(ns2, name2, link, True)
|
tc(ns2, name2, link, True)
|
||||||
|
|
||||||
|
|
@ -66,4 +69,6 @@ def tc(namespace, name, link, invert=False):
|
||||||
options += ["rate", str(link.bandwidth.down)]
|
options += ["rate", str(link.bandwidth.down)]
|
||||||
else:
|
else:
|
||||||
options += ["rate", str(link.bandwidth.up)]
|
options += ["rate", str(link.bandwidth.up)]
|
||||||
|
if link.limit:
|
||||||
|
options += ["limit", str(link.limit)]
|
||||||
run_netns("exec", namespace, "tc", "qdisc", "add", "dev", name, "root", "netem", *options)
|
run_netns("exec", namespace, "tc", "qdisc", "add", "dev", name, "root", "netem", *options)
|
||||||
|
|
|
||||||
41
prepare.py
Executable file
41
prepare.py
Executable file
|
|
@ -0,0 +1,41 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from pathlib import Path
|
||||||
|
from scenarios.fragments import shared
|
||||||
|
import os, venv
|
||||||
|
|
||||||
|
me = Path(os.path.dirname(__file__))
|
||||||
|
print("--- git submodule ---")
|
||||||
|
shared.exec("git submodule update --init")
|
||||||
|
|
||||||
|
print("--- compile go benchmarks ---")
|
||||||
|
os.chdir(me / "benchmarks" / "warp")
|
||||||
|
shared.exec("go build")
|
||||||
|
os.chdir(me / "benchmarks" / "s3concurrent")
|
||||||
|
shared.exec("go build")
|
||||||
|
os.chdir(me / "benchmarks" / "s3lat")
|
||||||
|
shared.exec("go build")
|
||||||
|
os.chdir(me / "benchmarks" / "s3ttfb")
|
||||||
|
shared.exec("go build")
|
||||||
|
os.chdir(me / "benchmarks" / "s3billion")
|
||||||
|
shared.exec("go build")
|
||||||
|
os.chdir(me)
|
||||||
|
|
||||||
|
print("--- install python dependencies ---")
|
||||||
|
venv.create(".venv", with_pip=True)
|
||||||
|
shared.exec("""
|
||||||
|
. .venv/bin/activate
|
||||||
|
which python3
|
||||||
|
python3 -m pip install .
|
||||||
|
python3 -m pip install -r scenarios/requirements.txt
|
||||||
|
""")
|
||||||
|
|
||||||
|
print("--- download artifacts ---")
|
||||||
|
from scenarios.fragments import flavor
|
||||||
|
flavor.download()
|
||||||
|
|
||||||
|
print(f"""done! now, run:
|
||||||
|
> sudo -i
|
||||||
|
> cd {os.getcwd()}
|
||||||
|
> . ./.venv/bin/activate
|
||||||
|
> ./mknet scenario ./topo/with-vdsl.yml ./scenarios/garage-s3lat garage-v0.8
|
||||||
|
""")
|
||||||
11
scenarios/clean
Executable file
11
scenarios/clean
Executable file
|
|
@ -0,0 +1,11 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import os
|
||||||
|
from fragments import garage, shared, minio
|
||||||
|
|
||||||
|
garage.destroy()
|
||||||
|
minio.destroy()
|
||||||
|
if len(shared.storage_path) > 8 and shared.id() == 1:
|
||||||
|
shared.exec(f"rm -r {shared.storage_path}")
|
||||||
|
shared.log("clean done")
|
||||||
|
|
||||||
BIN
scenarios/fragments/.shared.py.swo
Normal file
BIN
scenarios/fragments/.shared.py.swo
Normal file
Binary file not shown.
73
scenarios/fragments/flavor.py
Normal file
73
scenarios/fragments/flavor.py
Normal file
|
|
@ -0,0 +1,73 @@
|
||||||
|
from pathlib import Path
|
||||||
|
from . import shared
|
||||||
|
from os.path import exists
|
||||||
|
|
||||||
|
def grg_path(d):
|
||||||
|
for flav, desc in d.items():
|
||||||
|
if "path" in desc: continue
|
||||||
|
binary = f"garage-{desc['target']}-{desc['version']}"
|
||||||
|
desc['path'] = Path(shared.binary_path) / binary
|
||||||
|
return d
|
||||||
|
|
||||||
|
def minio_path(d):
|
||||||
|
for flav, desc in d.items():
|
||||||
|
if "path" in desc: continue
|
||||||
|
desc['path'] = Path(shared.binary_path) / flav
|
||||||
|
return d
|
||||||
|
|
||||||
|
garage = grg_path({
|
||||||
|
"garage-local": { "path": "./garage/target/release/garage" },
|
||||||
|
"garage-v0.7": {
|
||||||
|
"version": "v0.7.3",
|
||||||
|
"target": "x86_64-unknown-linux-musl",
|
||||||
|
},
|
||||||
|
# no rpc optimization
|
||||||
|
"garage-v0.8-beta1": {
|
||||||
|
"version": "89b8087ba81c508ba382aa6c9cb6bb3afa6a43c8",
|
||||||
|
"target": "x86_64-unknown-linux-musl"
|
||||||
|
},
|
||||||
|
# with rpc optimizations
|
||||||
|
# with fsync deactivated
|
||||||
|
"garage-v0.8-no-fsync": {
|
||||||
|
"version": "v0.8.0-dangerous-no-fsync",
|
||||||
|
"target": "x86_64-unknown-linux-musl"
|
||||||
|
},
|
||||||
|
# with rpc optimizations
|
||||||
|
"garage-v0.8-beta2": {
|
||||||
|
"version": "af2b2f26b4bd9fcdeedf2cd03f9e1392e5781abc",
|
||||||
|
"target": "x86_64-unknown-linux-musl",
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
warp = {
|
||||||
|
"warp-fast": "mixed --obj.size 5M --objects 200 --duration=5m",
|
||||||
|
"warp-small-obj": "mixed --obj.size 256 --objects 200 --duration=5m",
|
||||||
|
"warp-default": "mixed"
|
||||||
|
}
|
||||||
|
|
||||||
|
minio = minio_path({
|
||||||
|
"minio-20220917": {
|
||||||
|
"version": "2022-09-17T00-09-45Z"
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def download():
|
||||||
|
for flav, desc in garage.items():
|
||||||
|
if "version" not in desc: continue
|
||||||
|
if exists(desc['path']): continue
|
||||||
|
|
||||||
|
shared.exec(f"mkdir -p {shared.binary_path}")
|
||||||
|
shared.exec(f"wget https://garagehq.deuxfleurs.fr/_releases/{desc['version']}/{desc['target']}/garage -O {desc['path']}")
|
||||||
|
shared.exec(f"chmod +x {desc['path']}")
|
||||||
|
shared.exec(f"{desc['path']} --version")
|
||||||
|
|
||||||
|
for flav, desc in minio.items():
|
||||||
|
if "version" not in desc: continue
|
||||||
|
if exists(desc['path']): continue
|
||||||
|
|
||||||
|
shared.exec(f"mkdir -p {shared.binary_path}")
|
||||||
|
shared.exec(f"wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio.RELEASE.{desc['version']} -O {desc['path']}")
|
||||||
|
shared.exec(f"chmod +x {desc['path']}")
|
||||||
|
shared.exec(f"{desc['path']} --version")
|
||||||
|
|
||||||
229
scenarios/fragments/garage.py
Normal file
229
scenarios/fragments/garage.py
Normal file
|
|
@ -0,0 +1,229 @@
|
||||||
|
import glob, json, time
|
||||||
|
from os.path import exists
|
||||||
|
from os import environ as env
|
||||||
|
from pathlib import Path
|
||||||
|
from . import shared, flavor
|
||||||
|
|
||||||
|
import garage_admin_sdk
|
||||||
|
from garage_admin_sdk.api import nodes_api, layout_api, key_api
|
||||||
|
from garage_admin_sdk.model.node_cluster_info import NodeClusterInfo
|
||||||
|
from garage_admin_sdk.model.layout_version import LayoutVersion
|
||||||
|
from garage_admin_sdk.model.add_key_request import AddKeyRequest
|
||||||
|
from garage_admin_sdk.model.update_key_request import UpdateKeyRequest
|
||||||
|
from garage_admin_sdk.model.update_key_request_allow import UpdateKeyRequestAllow
|
||||||
|
|
||||||
|
storage_path = "./i/am/not/defined"
|
||||||
|
rpc_secret = "3e9abff5f9e480afbadb46a77b7a26fe0e404258f0dc3fd5386b0ba8e0ad2fba"
|
||||||
|
metrics = "cacce0b2de4bc2d9f5b5fdff551e01ac1496055aed248202d415398987e35f81"
|
||||||
|
admin = "ae8cb40ea7368bbdbb6430af11cca7da833d3458a5f52086f4e805a570fb5c2a"
|
||||||
|
key = None
|
||||||
|
|
||||||
|
version = flavor.garage["garage-v0.7"]
|
||||||
|
|
||||||
|
configuration = garage_admin_sdk.Configuration(
|
||||||
|
host = "http://localhost:3903/v0",
|
||||||
|
access_token = admin
|
||||||
|
)
|
||||||
|
api = garage_admin_sdk.ApiClient(configuration)
|
||||||
|
nodes = nodes_api.NodesApi(api)
|
||||||
|
layout = layout_api.LayoutApi(api)
|
||||||
|
keys = key_api.KeyApi(api)
|
||||||
|
|
||||||
|
|
||||||
|
# Setup, launch on import
|
||||||
|
if 'HOST' in env:
|
||||||
|
storage_path = Path(shared.storage_path) / "garage" / env['HOST']
|
||||||
|
if 'ZONE' in env and env['ZONE'] != "":
|
||||||
|
storage_path = Path(shared.storage_path) / "garage" / env['ZONE'] / env['HOST']
|
||||||
|
config = storage_path / "garage.toml"
|
||||||
|
env['GARAGE_CONFIG_FILE'] = str(config)
|
||||||
|
|
||||||
|
def deploy_coord(gw=None, uroot={}, us3_api={}, us3_web={}, uadmin={}):
|
||||||
|
destroy()
|
||||||
|
shared.log("start daemon")
|
||||||
|
shared.exec(f"{version['path']} --version")
|
||||||
|
daemon(uroot,us3_api,us3_web,uadmin)
|
||||||
|
shared.log("discover nodes")
|
||||||
|
connect()
|
||||||
|
shared.log("build layout")
|
||||||
|
create_layout(gw=gw)
|
||||||
|
shared.log("create key")
|
||||||
|
create_key()
|
||||||
|
shared.log("ready")
|
||||||
|
|
||||||
|
def deploy_follow(uroot={}, us3_api={}, us3_web={}, uadmin={}):
|
||||||
|
destroy()
|
||||||
|
shared.log("start daemon")
|
||||||
|
daemon(uroot,us3_api,us3_web,uadmin)
|
||||||
|
shared.log("wait for coord")
|
||||||
|
sync_on_key_up()
|
||||||
|
shared.log("ready")
|
||||||
|
|
||||||
|
def to_toml(d): return "\n".join([ f"{k} = {v}" if type(v) is int else f"{k} = \"{v}\"" for k, v in d.items() ])
|
||||||
|
def daemon(uroot={}, us3_api={}, us3_web={}, uadmin={}):
|
||||||
|
root = {
|
||||||
|
"metadata_dir": f"{storage_path}/meta",
|
||||||
|
"data_dir": f"{storage_path}/data",
|
||||||
|
"replication_mode": "3",
|
||||||
|
"rpc_bind_addr": "[::]:3901",
|
||||||
|
"rpc_public_addr": f"[{env['IP']}]:3901",
|
||||||
|
"rpc_secret": f"{rpc_secret}",
|
||||||
|
}
|
||||||
|
root.update(uroot)
|
||||||
|
|
||||||
|
s3_api = {
|
||||||
|
"s3_region": "garage",
|
||||||
|
"api_bind_addr": "[::]:3900",
|
||||||
|
"root_domain": ".s3.garage",
|
||||||
|
}
|
||||||
|
s3_api.update(us3_api)
|
||||||
|
|
||||||
|
s3_web = {
|
||||||
|
"bind_addr": "[::]:3902",
|
||||||
|
"root_domain": ".web.garage",
|
||||||
|
"index": "index.html",
|
||||||
|
}
|
||||||
|
s3_web.update(us3_web)
|
||||||
|
|
||||||
|
sect_admin = {
|
||||||
|
"api_bind_addr": "0.0.0.0:3903",
|
||||||
|
"metrics_token": f"{metrics}",
|
||||||
|
"admin_token": f"{admin}",
|
||||||
|
}
|
||||||
|
sect_admin.update(uadmin)
|
||||||
|
|
||||||
|
|
||||||
|
shared.exec(f"mkdir -p {storage_path}")
|
||||||
|
with open(config, 'w+') as f:
|
||||||
|
f.write(f"""
|
||||||
|
{to_toml(root)}
|
||||||
|
bootstrap_peers = []
|
||||||
|
|
||||||
|
[s3_api]
|
||||||
|
{to_toml(s3_api)}
|
||||||
|
|
||||||
|
[s3_web]
|
||||||
|
{to_toml(s3_web)}
|
||||||
|
|
||||||
|
[admin]
|
||||||
|
{to_toml(sect_admin)}
|
||||||
|
""")
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
shared.exec(f"{version['path']} --version")
|
||||||
|
shared.exec(f"{version['path']} server 2>> {storage_path}/logs.stderr 1>> {storage_path}/logs.stdout & echo $! > {storage_path}/daemon.pid")
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
node_info = storage_path / "node_info"
|
||||||
|
node_id = shared.fn_retry(lambda: nodes.get_nodes().node)
|
||||||
|
with open(node_info, 'w+') as f:
|
||||||
|
f.write(json.dumps({
|
||||||
|
"node_addr": f"{node_id}@{env['IP']}:3901",
|
||||||
|
"node_id": node_id,
|
||||||
|
"zone": env['ZONE'] if 'ZONE' in env and env['ZONE'] != "" else env['HOST'],
|
||||||
|
"host": env['HOST'],
|
||||||
|
}))
|
||||||
|
|
||||||
|
def destroy():
|
||||||
|
dpid = Path(storage_path) / "daemon.pid"
|
||||||
|
if exists(dpid):
|
||||||
|
try:
|
||||||
|
shared.exec(f"kill -9 $(cat {dpid})")
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
shared.exec(f"rm -f {dpid}")
|
||||||
|
if len(str(storage_path)) < 8: # arbitrary, stupid safe guard
|
||||||
|
print(storage_path)
|
||||||
|
raise Exception("You tried to clean a storage path that might be the root of your FS, panicking...")
|
||||||
|
shared.exec(f"rm -fr {storage_path}")
|
||||||
|
|
||||||
|
# this function is ugly, sorry :s
|
||||||
|
_cluster_info = None
|
||||||
|
def cluster_info():
|
||||||
|
global _cluster_info
|
||||||
|
if _cluster_info is not None: return _cluster_info
|
||||||
|
shared.log("fetch cluster info")
|
||||||
|
|
||||||
|
while True:
|
||||||
|
node_files = glob.glob(f"{shared.storage_path}/**/node_info", recursive=True)
|
||||||
|
if len(node_files) >= shared.count(): break
|
||||||
|
shared.log(f"found {len(node_files)} over {shared.count()}, wait 1 sec.")
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
_cluster_info = [ json.loads(Path(f).read_text()) for f in node_files ]
|
||||||
|
return _cluster_info
|
||||||
|
|
||||||
|
|
||||||
|
def connect():
|
||||||
|
cinf = cluster_info()
|
||||||
|
shared.log("start connections...")
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
ret = nodes.add_node([n['node_addr'] for n in cinf], _request_timeout=3)
|
||||||
|
except:
|
||||||
|
shared.log("not ready, retry in 1sec")
|
||||||
|
time.sleep(1)
|
||||||
|
continue
|
||||||
|
|
||||||
|
for st in ret:
|
||||||
|
if not st.success:
|
||||||
|
continue
|
||||||
|
#raise Exception("Node connect failed", ret)
|
||||||
|
break
|
||||||
|
|
||||||
|
shared.log("all nodes connected")
|
||||||
|
|
||||||
|
def create_layout(gw=None):
|
||||||
|
if gw is None:
|
||||||
|
gw = []
|
||||||
|
|
||||||
|
v = layout.get_layout().version
|
||||||
|
|
||||||
|
cinf = cluster_info()
|
||||||
|
nlay = dict()
|
||||||
|
for n in cinf:
|
||||||
|
capa = 1
|
||||||
|
if n['host'] in gw:
|
||||||
|
capa = None
|
||||||
|
|
||||||
|
nlay[n['node_id']] = NodeClusterInfo(
|
||||||
|
zone = n['zone'],
|
||||||
|
capacity = capa,
|
||||||
|
tags = [ n['host'] ],
|
||||||
|
)
|
||||||
|
layout.add_layout(nlay)
|
||||||
|
layout.apply_layout(LayoutVersion(version=v+1))
|
||||||
|
|
||||||
|
shared.log(layout.get_layout())
|
||||||
|
|
||||||
|
def create_key():
|
||||||
|
global key
|
||||||
|
kinfo = shared.fn_retry(lambda: keys.add_key(AddKeyRequest(name="mknet")))
|
||||||
|
allow_create = UpdateKeyRequestAllow(create_bucket=True)
|
||||||
|
keys.update_key(kinfo.access_key_id, UpdateKeyRequest(allow=allow_create))
|
||||||
|
key = kinfo
|
||||||
|
|
||||||
|
|
||||||
|
def delete_key():
|
||||||
|
global key
|
||||||
|
keys.delete_key(key.access_key_id)
|
||||||
|
key = None
|
||||||
|
|
||||||
|
def sync_on_key_up():
|
||||||
|
global key
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
key = keys.search_key("mknet")
|
||||||
|
return key
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
def sync_on_key_down():
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
keys.search_key("mknet")
|
||||||
|
except:
|
||||||
|
return
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
140
scenarios/fragments/minio.py
Normal file
140
scenarios/fragments/minio.py
Normal file
|
|
@ -0,0 +1,140 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
import json, os, sys, time, pathlib, socket, shutil, urllib3
|
||||||
|
import minio
|
||||||
|
from os import environ as env
|
||||||
|
from pathlib import Path
|
||||||
|
from . import shared, flavor
|
||||||
|
|
||||||
|
storage_path = "./i/am/not/defined"
|
||||||
|
version = flavor.minio["minio-20220917"]
|
||||||
|
unix_sock = str(Path(shared.storage_path) / "minio.sock")
|
||||||
|
access_key = "minioadmin"
|
||||||
|
secret_key = "minioadmin"
|
||||||
|
client = minio.Minio(
|
||||||
|
f"[{env['IP']}]:9000",
|
||||||
|
access_key="minioadmin",
|
||||||
|
secret_key="minioadmin",
|
||||||
|
secure=False,
|
||||||
|
http_client=urllib3.PoolManager(
|
||||||
|
timeout=2,
|
||||||
|
retries=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if 'HOST' in env:
|
||||||
|
storage_path = str(Path(shared.storage_path) / "minio" / env['HOST'])
|
||||||
|
if 'ZONE' in env and env['ZONE'] != "":
|
||||||
|
storage_path = str(Path(shared.storage_path) / "minio" / env['ZONE'] / env['HOST'])
|
||||||
|
|
||||||
|
stdout = Path(storage_path) / "minio.stdout"
|
||||||
|
stderr = Path(storage_path) / "minio.stderr"
|
||||||
|
pid = Path(storage_path) / "daemon.pid"
|
||||||
|
|
||||||
|
def destroy():
|
||||||
|
if os.path.exists(pid):
|
||||||
|
try:
|
||||||
|
shared.exec(f"kill -9 `cat {pid}`")
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if len(str(storage_path)) > 8:
|
||||||
|
shutil.rmtree(storage_path, ignore_errors=True)
|
||||||
|
|
||||||
|
def deploy_coord():
|
||||||
|
destroy()
|
||||||
|
if os.path.exists(unix_sock):
|
||||||
|
os.unlink(unix_sock)
|
||||||
|
|
||||||
|
os.makedirs(storage_path)
|
||||||
|
|
||||||
|
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||||
|
sock.bind(unix_sock)
|
||||||
|
sock.listen()
|
||||||
|
|
||||||
|
# Create sockets
|
||||||
|
fl = [ co for co, addr in [ sock.accept() for i in range(1, shared.count()) ]]
|
||||||
|
|
||||||
|
# Receive configurations, centralize them
|
||||||
|
me = [ { "ip": os.environ['IP'], "path": storage_path } ]
|
||||||
|
others = [ json.loads(co.makefile().readline()) for co in fl ]
|
||||||
|
identities = others + me
|
||||||
|
shared.log(f"ident: {identities}")
|
||||||
|
|
||||||
|
# Dispatch them
|
||||||
|
msg = f"{json.dumps(identities)}\n".encode()
|
||||||
|
[ co.send(msg) for co in fl ]
|
||||||
|
|
||||||
|
run_minio(identities)
|
||||||
|
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
if client.bucket_exists("sync"): break
|
||||||
|
client.make_bucket("sync")
|
||||||
|
break
|
||||||
|
except Exception as e:
|
||||||
|
shared.log("waiting for bootstrap...", e)
|
||||||
|
time.sleep(1)
|
||||||
|
shared.log("ready")
|
||||||
|
|
||||||
|
def deploy_follow():
|
||||||
|
destroy()
|
||||||
|
os.makedirs(storage_path)
|
||||||
|
|
||||||
|
co = None
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||||
|
sock.connect(unix_sock)
|
||||||
|
co = sock.makefile()
|
||||||
|
break
|
||||||
|
except Exception as err:
|
||||||
|
shared.log('conn failed, wait 1 sec, err is', err)
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
# send my identity
|
||||||
|
my_identity = json.dumps({ "ip": os.environ['IP'], "path": storage_path })
|
||||||
|
sock.send(f"{my_identity}\n".encode())
|
||||||
|
|
||||||
|
# get all
|
||||||
|
identities = json.loads(co.readline())
|
||||||
|
|
||||||
|
run_minio(identities)
|
||||||
|
sync_on_bucket_up()
|
||||||
|
shared.log("ready")
|
||||||
|
|
||||||
|
def sync_on_bucket_up():
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
if client.bucket_exists("sync"): break
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
shared.log("waiting for bucket 'sync'...")
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
def sync_on_bucket_down():
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
if not client.bucket_exists("sync"): break
|
||||||
|
time.sleep(1)
|
||||||
|
except Exception as e:
|
||||||
|
shared.log("the cluster is probably already half shutdown, so errors are expected ->", e)
|
||||||
|
break
|
||||||
|
|
||||||
|
def delete_sync_bucket():
|
||||||
|
client.remove_bucket("sync")
|
||||||
|
|
||||||
|
def run_minio(identities):
|
||||||
|
|
||||||
|
# Required to prevent Minio error: "/tmp/mknet-store/minio/node1` is part of root drive, will not be used"
|
||||||
|
# https://github.com/minio/minio/issues/15720
|
||||||
|
env['CI'] = "true"
|
||||||
|
env['MINIO_CI_CD'] = "true"
|
||||||
|
|
||||||
|
cmd = f"{version['path']} server --console-address ':9001' --address ':9000'"
|
||||||
|
for ident in identities:
|
||||||
|
cmd += f" http://[{ident['ip']}]:9000{ident['path']}"
|
||||||
|
cmd += f" > {stdout} 2> {stderr}"
|
||||||
|
cmd += f" & echo $! > {pid}"
|
||||||
|
|
||||||
|
shared.log("launch: ", cmd)
|
||||||
|
os.system(cmd)
|
||||||
24
scenarios/fragments/s3billion.py
Normal file
24
scenarios/fragments/s3billion.py
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
import os
|
||||||
|
from os.path import exists
|
||||||
|
from pathlib import Path
|
||||||
|
from fragments import shared, minio, garage
|
||||||
|
|
||||||
|
s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3billion/s3billion"
|
||||||
|
|
||||||
|
def common():
|
||||||
|
out = Path(shared.storage_path) / "s3billion.csv"
|
||||||
|
shared.log(f"launching s3billion ({s3bin})")
|
||||||
|
shared.exec(f"{s3bin} > {out}")
|
||||||
|
shared.log(f"execution done, output written to {out}")
|
||||||
|
|
||||||
|
def on_garage():
|
||||||
|
os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
|
||||||
|
os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
|
||||||
|
os.environ['ENDPOINT'] = f"[{os.environ['IP']}]:3900"
|
||||||
|
common()
|
||||||
|
|
||||||
|
def on_minio():
|
||||||
|
os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
|
||||||
|
os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
|
||||||
|
os.environ['ENDPOINT'] = f"[{os.environ['IP']}]:9000"
|
||||||
|
common()
|
||||||
16
scenarios/fragments/s3concurrent.py
Normal file
16
scenarios/fragments/s3concurrent.py
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
import os
|
||||||
|
from os.path import exists
|
||||||
|
from pathlib import Path
|
||||||
|
from fragments import shared, garage
|
||||||
|
|
||||||
|
s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3concurrent/s3concurrent"
|
||||||
|
|
||||||
|
def on_garage():
|
||||||
|
os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
|
||||||
|
os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
|
||||||
|
os.environ['ENDPOINT'] = "localhost:3900"
|
||||||
|
|
||||||
|
out = Path(shared.storage_path) / "s3concurrent.csv"
|
||||||
|
shared.log(f"launching s3concurrent ({s3bin})")
|
||||||
|
shared.exec(f"{s3bin} > {out}")
|
||||||
|
shared.log(f"execution done, output written to {out}")
|
||||||
24
scenarios/fragments/s3lat.py
Normal file
24
scenarios/fragments/s3lat.py
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
import os
|
||||||
|
from os.path import exists
|
||||||
|
from pathlib import Path
|
||||||
|
from fragments import shared, minio, garage
|
||||||
|
|
||||||
|
s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3lat/s3lat"
|
||||||
|
|
||||||
|
def common():
|
||||||
|
out = Path(shared.storage_path) / "s3lat.csv"
|
||||||
|
shared.log(f"launching s3lat ({s3bin})")
|
||||||
|
shared.exec(f"{s3bin} > {out}")
|
||||||
|
shared.log(f"execution done, output written to {out}")
|
||||||
|
|
||||||
|
def on_garage():
|
||||||
|
os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
|
||||||
|
os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
|
||||||
|
os.environ['ENDPOINT'] = "localhost:3900"
|
||||||
|
common()
|
||||||
|
|
||||||
|
def on_minio():
|
||||||
|
os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
|
||||||
|
os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
|
||||||
|
os.environ['ENDPOINT'] = "localhost:9000"
|
||||||
|
common()
|
||||||
26
scenarios/fragments/s3ttfb.py
Normal file
26
scenarios/fragments/s3ttfb.py
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
import os
|
||||||
|
from os.path import exists
|
||||||
|
from pathlib import Path
|
||||||
|
from fragments import shared, minio, garage
|
||||||
|
|
||||||
|
s3bin = Path(os.path.dirname(__file__)) / "../../benchmarks/s3ttfb/s3ttfb"
|
||||||
|
|
||||||
|
def common():
|
||||||
|
out = Path(shared.storage_path) / "s3ttfb.csv"
|
||||||
|
shared.log(f"launching s3ttfb ({s3bin})")
|
||||||
|
shared.exec(f"{s3bin} > {out}")
|
||||||
|
shared.log(f"execution done, output written to {out}")
|
||||||
|
|
||||||
|
def on_garage():
|
||||||
|
os.environ['AWS_ACCESS_KEY_ID'] = garage.key.access_key_id
|
||||||
|
os.environ['AWS_SECRET_ACCESS_KEY'] = garage.key.secret_access_key
|
||||||
|
os.environ['ENDPOINT'] = "localhost:3900"
|
||||||
|
os.environ['AWS_REGION'] = "garage"
|
||||||
|
common()
|
||||||
|
|
||||||
|
def on_minio():
|
||||||
|
os.environ['AWS_ACCESS_KEY_ID'] = minio.access_key
|
||||||
|
os.environ['AWS_SECRET_ACCESS_KEY'] = minio.secret_key
|
||||||
|
os.environ['ENDPOINT'] = "localhost:9000"
|
||||||
|
os.environ['AWS_REGION'] = "us-east-1"
|
||||||
|
common()
|
||||||
28
scenarios/fragments/shared.py
Normal file
28
scenarios/fragments/shared.py
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
import os, time
|
||||||
|
|
||||||
|
binary_path = "/tmp/mknet-bin"
|
||||||
|
storage_path = "/tmp/mknet-store"
|
||||||
|
|
||||||
|
def exec(s):
|
||||||
|
if os.system(s) != 0:
|
||||||
|
raise Exception("Command terminated with an error")
|
||||||
|
def exec_retry(s, cnt=16):
|
||||||
|
print(s)
|
||||||
|
for i in range(cnt):
|
||||||
|
time.sleep(i) # this is expected to sleep before running the command to reduce the noise
|
||||||
|
if os.system(s) == 0: return
|
||||||
|
raise Exception("Command terminated with an error too many times")
|
||||||
|
def fn_retry(f, cnt=5):
|
||||||
|
for i in range(cnt):
|
||||||
|
try:
|
||||||
|
r = f()
|
||||||
|
return r
|
||||||
|
except Exception as e:
|
||||||
|
if i+1 == cnt: raise e
|
||||||
|
log(f"failed call, retry in {i} sec")
|
||||||
|
time.sleep(i)
|
||||||
|
|
||||||
|
def id(): return int(os.environ['ID'])
|
||||||
|
def count(): return int(os.environ['SERVER_COUNT'])
|
||||||
|
def log(*args): print(f"[{id()}/{count()} - {os.environ['HOST']}]", *args)
|
||||||
|
|
||||||
17
scenarios/fragments/warp.py
Normal file
17
scenarios/fragments/warp.py
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
import os
|
||||||
|
from os.path import exists
|
||||||
|
from pathlib import Path
|
||||||
|
from fragments import shared, garage, flavor, minio
|
||||||
|
|
||||||
|
warp_bin = Path(os.path.dirname(__file__)) / "../../benchmarks/warp/warp"
|
||||||
|
bench = flavor.warp["warp-fast"]
|
||||||
|
|
||||||
|
def common(port, ak, sk):
|
||||||
|
out = Path(shared.storage_path) / "warp.csv"
|
||||||
|
shared.log(f"launching warp {warp_bin}")
|
||||||
|
shared.exec(f"{warp_bin} {bench} --host=[{os.environ['IP']}]:{port} --analyze.out={out} --access-key={ak} --secret-key={sk}")
|
||||||
|
shared.log(f"execution done")
|
||||||
|
|
||||||
|
def on_garage(): common(3900, garage.key.access_key_id, garage.key.secret_access_key)
|
||||||
|
def on_minio(): common(9000, minio.access_key, minio.secret_key)
|
||||||
|
|
||||||
7
scenarios/garage-manual
Executable file
7
scenarios/garage-manual
Executable file
|
|
@ -0,0 +1,7 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import garage, s3lat, shared
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
garage.deploy_coord()
|
||||||
|
else:
|
||||||
|
garage.deploy_follow()
|
||||||
24
scenarios/garage-s3billion
Executable file
24
scenarios/garage-s3billion
Executable file
|
|
@ -0,0 +1,24 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import garage, s3billion, shared, flavor
|
||||||
|
import sys, os
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.garage:
|
||||||
|
garage.version = flavor.garage[fl]
|
||||||
|
|
||||||
|
#os.environ['BATCH_COUNT'] = "2"
|
||||||
|
shared.exec("ulimit -n 65535")
|
||||||
|
conf_root = {
|
||||||
|
"db_engine": "lmdb",
|
||||||
|
}
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
garage.deploy_coord(uroot=conf_root)
|
||||||
|
s3billion.on_garage()
|
||||||
|
garage.delete_key()
|
||||||
|
garage.destroy()
|
||||||
|
else:
|
||||||
|
garage.deploy_follow(uroot=conf_root)
|
||||||
|
garage.sync_on_key_down()
|
||||||
|
garage.destroy()
|
||||||
|
shared.log("bye")
|
||||||
18
scenarios/garage-s3concurrent
Executable file
18
scenarios/garage-s3concurrent
Executable file
|
|
@ -0,0 +1,18 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import garage, s3concurrent, shared, flavor
|
||||||
|
import sys
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.garage:
|
||||||
|
garage.version = flavor.garage[fl]
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
garage.deploy_coord()
|
||||||
|
s3concurrent.on_garage()
|
||||||
|
garage.delete_key()
|
||||||
|
garage.destroy()
|
||||||
|
else:
|
||||||
|
garage.deploy_follow()
|
||||||
|
garage.sync_on_key_down()
|
||||||
|
garage.destroy()
|
||||||
|
shared.log("bye")
|
||||||
18
scenarios/garage-s3lat
Executable file
18
scenarios/garage-s3lat
Executable file
|
|
@ -0,0 +1,18 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import garage, s3lat, shared, flavor
|
||||||
|
import sys
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.garage:
|
||||||
|
garage.version = flavor.garage[fl]
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
garage.deploy_coord()
|
||||||
|
s3lat.on_garage()
|
||||||
|
garage.delete_key()
|
||||||
|
garage.destroy()
|
||||||
|
else:
|
||||||
|
garage.deploy_follow()
|
||||||
|
garage.sync_on_key_down()
|
||||||
|
garage.destroy()
|
||||||
|
shared.log("bye")
|
||||||
18
scenarios/garage-s3ttfb
Executable file
18
scenarios/garage-s3ttfb
Executable file
|
|
@ -0,0 +1,18 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import garage, s3ttfb, shared, flavor
|
||||||
|
import sys
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.garage:
|
||||||
|
garage.version = flavor.garage[fl]
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
garage.deploy_coord(gw = [ 'node1' ])
|
||||||
|
s3ttfb.on_garage()
|
||||||
|
garage.delete_key()
|
||||||
|
garage.destroy()
|
||||||
|
else:
|
||||||
|
garage.deploy_follow()
|
||||||
|
garage.sync_on_key_down()
|
||||||
|
garage.destroy()
|
||||||
|
shared.log("bye")
|
||||||
26
scenarios/garage-warp
Executable file
26
scenarios/garage-warp
Executable file
|
|
@ -0,0 +1,26 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import garage, warp, shared, flavor
|
||||||
|
import sys
|
||||||
|
|
||||||
|
db_engines = [ 'sled', 'lmdb', 'sqlite' ]
|
||||||
|
conf_root = {}
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.garage:
|
||||||
|
garage.version = flavor.garage[fl]
|
||||||
|
elif fl in flavor.warp:
|
||||||
|
warp.bench = flavor.warp[fl]
|
||||||
|
elif fl in db_engines:
|
||||||
|
conf_root['db_engine'] = fl
|
||||||
|
elif 'sled_no_fsync' == fl:
|
||||||
|
conf_root['sled_flush_every_ms'] = 1000 * 60 * 10 # 10 minutes
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
garage.deploy_coord(uroot=conf_root)
|
||||||
|
warp.on_garage()
|
||||||
|
garage.delete_key()
|
||||||
|
garage.destroy()
|
||||||
|
else:
|
||||||
|
garage.deploy_follow(uroot=conf_root)
|
||||||
|
garage.sync_on_key_down()
|
||||||
|
garage.destroy()
|
||||||
|
shared.log("bye")
|
||||||
12
scenarios/minio-manual
Executable file
12
scenarios/minio-manual
Executable file
|
|
@ -0,0 +1,12 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import shared, flavor, minio
|
||||||
|
import sys
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.minio:
|
||||||
|
minio.version = flavor.minio[fl]
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
minio.deploy_coord()
|
||||||
|
else:
|
||||||
|
minio.deploy_follow()
|
||||||
22
scenarios/minio-s3billion
Executable file
22
scenarios/minio-s3billion
Executable file
|
|
@ -0,0 +1,22 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import minio, s3billion, shared, flavor
|
||||||
|
import sys, os
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.minio:
|
||||||
|
minio.version = flavor.minio[fl]
|
||||||
|
|
||||||
|
#os.environ['BATCH_COUNT'] = "2"
|
||||||
|
shared.exec("ulimit -n 65535")
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
minio.deploy_coord()
|
||||||
|
s3billion.on_minio()
|
||||||
|
minio.delete_sync_bucket()
|
||||||
|
minio.destroy()
|
||||||
|
else:
|
||||||
|
minio.deploy_follow()
|
||||||
|
minio.sync_on_bucket_down()
|
||||||
|
minio.destroy()
|
||||||
|
|
||||||
|
shared.log("bye")
|
||||||
19
scenarios/minio-s3lat
Executable file
19
scenarios/minio-s3lat
Executable file
|
|
@ -0,0 +1,19 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import minio, s3lat, shared, flavor
|
||||||
|
import sys
|
||||||
|
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.minio:
|
||||||
|
minio.version = flavor.minio[fl]
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
minio.deploy_coord()
|
||||||
|
s3lat.on_minio()
|
||||||
|
minio.delete_sync_bucket()
|
||||||
|
minio.destroy()
|
||||||
|
else:
|
||||||
|
minio.deploy_follow()
|
||||||
|
minio.sync_on_bucket_down()
|
||||||
|
minio.destroy()
|
||||||
|
shared.log("bye")
|
||||||
19
scenarios/minio-s3ttfb
Executable file
19
scenarios/minio-s3ttfb
Executable file
|
|
@ -0,0 +1,19 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import minio, s3ttfb, shared, flavor
|
||||||
|
import sys
|
||||||
|
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.minio:
|
||||||
|
minio.version = flavor.minio[fl]
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
minio.deploy_coord()
|
||||||
|
s3ttfb.on_minio()
|
||||||
|
minio.delete_sync_bucket()
|
||||||
|
minio.destroy()
|
||||||
|
else:
|
||||||
|
minio.deploy_follow()
|
||||||
|
minio.sync_on_bucket_down()
|
||||||
|
minio.destroy()
|
||||||
|
shared.log("bye")
|
||||||
20
scenarios/minio-warp
Executable file
20
scenarios/minio-warp
Executable file
|
|
@ -0,0 +1,20 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from fragments import minio, warp, shared, flavor
|
||||||
|
import sys
|
||||||
|
|
||||||
|
for fl in sys.argv[1:]:
|
||||||
|
if fl in flavor.minio:
|
||||||
|
minio.version = flavor.minio[fl]
|
||||||
|
if fl in flavor.warp:
|
||||||
|
warp.bench = flavor.warp[fl]
|
||||||
|
|
||||||
|
if shared.id() == 1:
|
||||||
|
minio.deploy_coord()
|
||||||
|
warp.on_minio()
|
||||||
|
minio.delete_sync_bucket()
|
||||||
|
minio.destroy()
|
||||||
|
else:
|
||||||
|
minio.deploy_follow()
|
||||||
|
minio.sync_on_bucket_down()
|
||||||
|
minio.destroy()
|
||||||
|
shared.log("bye")
|
||||||
2
scenarios/requirements.txt
Normal file
2
scenarios/requirements.txt
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
git+https://git.deuxfleurs.fr/quentin/garage-admin-sdk@2b685b90be9aa848a9450dd582d1d322b25b5568#egg=garage-admin-sdk&subdirectory=python
|
||||||
|
minio
|
||||||
4
setup.py
4
setup.py
|
|
@ -3,4 +3,8 @@ setup(name='mknet',
|
||||||
version='1.0',
|
version='1.0',
|
||||||
scripts=['mknet'],
|
scripts=['mknet'],
|
||||||
py_modules=['net'],
|
py_modules=['net'],
|
||||||
|
install_requires=[
|
||||||
|
'PyYAML',
|
||||||
|
'requests'
|
||||||
|
],
|
||||||
)
|
)
|
||||||
|
|
|
||||||
12
shell.nix
Normal file
12
shell.nix
Normal file
|
|
@ -0,0 +1,12 @@
|
||||||
|
{ pkgs ? import <nixpkgs> {} }:
|
||||||
|
pkgs.mkShell {
|
||||||
|
nativeBuildInputs = [
|
||||||
|
pkgs.python310
|
||||||
|
pkgs.python310Packages.pyyaml
|
||||||
|
pkgs.python310Packages.requests
|
||||||
|
|
||||||
|
# to test the pip setup
|
||||||
|
pkgs.python310Packages.pip
|
||||||
|
pkgs.python310Packages.setuptools
|
||||||
|
];
|
||||||
|
}
|
||||||
25
topo/100mbps.yml
Normal file
25
topo/100mbps.yml
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
links:
|
||||||
|
- &slow
|
||||||
|
bandwidth: 100M
|
||||||
|
latency: 500us
|
||||||
|
- &1000
|
||||||
|
bandwidth: 1000M
|
||||||
|
latency: 100us
|
||||||
|
|
||||||
|
servers:
|
||||||
|
- name: node1
|
||||||
|
<<: *slow
|
||||||
|
- name: node2
|
||||||
|
<<: *slow
|
||||||
|
- name: node3
|
||||||
|
<<: *slow
|
||||||
|
|
||||||
|
global:
|
||||||
|
subnet:
|
||||||
|
base: 'fc00:9a7a:9e::'
|
||||||
|
local: 64
|
||||||
|
zone: 16
|
||||||
|
latency-offset: 3ms
|
||||||
|
upstream:
|
||||||
|
ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
|
||||||
|
conn: *1000
|
||||||
25
topo/10mbps.yml
Normal file
25
topo/10mbps.yml
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
links:
|
||||||
|
- &slow
|
||||||
|
bandwidth: 10M
|
||||||
|
latency: 500us
|
||||||
|
- &1000
|
||||||
|
bandwidth: 1000M
|
||||||
|
latency: 100us
|
||||||
|
|
||||||
|
servers:
|
||||||
|
- name: node1
|
||||||
|
<<: *slow
|
||||||
|
- name: node2
|
||||||
|
<<: *slow
|
||||||
|
- name: node3
|
||||||
|
<<: *slow
|
||||||
|
|
||||||
|
global:
|
||||||
|
subnet:
|
||||||
|
base: 'fc00:9a7a:9e::'
|
||||||
|
local: 64
|
||||||
|
zone: 16
|
||||||
|
latency-offset: 3ms
|
||||||
|
upstream:
|
||||||
|
ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
|
||||||
|
conn: *1000
|
||||||
27
topo/1mbps.yml
Normal file
27
topo/1mbps.yml
Normal file
|
|
@ -0,0 +1,27 @@
|
||||||
|
links:
|
||||||
|
- &slow
|
||||||
|
bandwidth: 1M
|
||||||
|
latency: 500us
|
||||||
|
#txqueuelen: 10
|
||||||
|
limit: 10
|
||||||
|
- &1000
|
||||||
|
bandwidth: 1000M
|
||||||
|
latency: 100us
|
||||||
|
|
||||||
|
servers:
|
||||||
|
- name: node1
|
||||||
|
<<: *slow
|
||||||
|
- name: node2
|
||||||
|
<<: *slow
|
||||||
|
- name: node3
|
||||||
|
<<: *slow
|
||||||
|
|
||||||
|
global:
|
||||||
|
subnet:
|
||||||
|
base: 'fc00:9a7a:9e::'
|
||||||
|
local: 64
|
||||||
|
zone: 16
|
||||||
|
latency-offset: 3ms
|
||||||
|
upstream:
|
||||||
|
ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
|
||||||
|
conn: *1000
|
||||||
25
topo/50mbps.yml
Normal file
25
topo/50mbps.yml
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
links:
|
||||||
|
- &slow
|
||||||
|
bandwidth: 50M
|
||||||
|
latency: 500us
|
||||||
|
- &1000
|
||||||
|
bandwidth: 1000M
|
||||||
|
latency: 100us
|
||||||
|
|
||||||
|
servers:
|
||||||
|
- name: node1
|
||||||
|
<<: *slow
|
||||||
|
- name: node2
|
||||||
|
<<: *slow
|
||||||
|
- name: node3
|
||||||
|
<<: *slow
|
||||||
|
|
||||||
|
global:
|
||||||
|
subnet:
|
||||||
|
base: 'fc00:9a7a:9e::'
|
||||||
|
local: 64
|
||||||
|
zone: 16
|
||||||
|
latency-offset: 3ms
|
||||||
|
upstream:
|
||||||
|
ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
|
||||||
|
conn: *1000
|
||||||
34
topo/50ms.yml
Normal file
34
topo/50ms.yml
Normal file
|
|
@ -0,0 +1,34 @@
|
||||||
|
links:
|
||||||
|
- &fiber
|
||||||
|
bandwidth: 100M
|
||||||
|
latency: 50ms
|
||||||
|
jitter: 10ms
|
||||||
|
|
||||||
|
zones:
|
||||||
|
- &dc1
|
||||||
|
name: dc1
|
||||||
|
internal: *fiber
|
||||||
|
external: *fiber
|
||||||
|
|
||||||
|
servers:
|
||||||
|
- name: dc1s1
|
||||||
|
zone: *dc1
|
||||||
|
- name: dc1s2
|
||||||
|
zone: *dc1
|
||||||
|
- name: dc1s3
|
||||||
|
zone: *dc1
|
||||||
|
- name: dc1s4
|
||||||
|
zone: *dc1
|
||||||
|
- name: dc1s5
|
||||||
|
zone: *dc1
|
||||||
|
|
||||||
|
global:
|
||||||
|
subnet:
|
||||||
|
base: 'fc00:9a7a:9e::'
|
||||||
|
local: 64
|
||||||
|
zone: 16
|
||||||
|
latency-offset: 3ms
|
||||||
|
upstream:
|
||||||
|
ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
|
||||||
|
conn: *fiber
|
||||||
|
|
||||||
28
topo/5mbps.yml
Normal file
28
topo/5mbps.yml
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
links:
|
||||||
|
- &slow
|
||||||
|
bandwidth: 5M
|
||||||
|
latency: 500us
|
||||||
|
limit: 5
|
||||||
|
- &1000
|
||||||
|
bandwidth: 1000M
|
||||||
|
latency: 100us
|
||||||
|
|
||||||
|
servers:
|
||||||
|
- name: node1
|
||||||
|
<<: *slow
|
||||||
|
- name: node2
|
||||||
|
<<: *slow
|
||||||
|
- name: node3
|
||||||
|
<<: *slow
|
||||||
|
- name: node4
|
||||||
|
<<: *slow
|
||||||
|
|
||||||
|
global:
|
||||||
|
subnet:
|
||||||
|
base: 'fc00:9a7a:9e::'
|
||||||
|
local: 64
|
||||||
|
zone: 16
|
||||||
|
latency-offset: 3ms
|
||||||
|
upstream:
|
||||||
|
ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
|
||||||
|
conn: *1000
|
||||||
22
topo/dc.yml
Normal file
22
topo/dc.yml
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
links:
|
||||||
|
- &1000
|
||||||
|
bandwidth: 1000M
|
||||||
|
latency: 1ms
|
||||||
|
|
||||||
|
servers:
|
||||||
|
- name: node1
|
||||||
|
<<: *1000
|
||||||
|
- name: node2
|
||||||
|
<<: *1000
|
||||||
|
- name: node3
|
||||||
|
<<: *1000
|
||||||
|
|
||||||
|
global:
|
||||||
|
subnet:
|
||||||
|
base: 'fc00:9a7a:9e::'
|
||||||
|
local: 64
|
||||||
|
zone: 16
|
||||||
|
latency-offset: 3ms
|
||||||
|
upstream:
|
||||||
|
ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
|
||||||
|
conn: *1000
|
||||||
46
topo/multi-dc.yml
Normal file
46
topo/multi-dc.yml
Normal file
|
|
@ -0,0 +1,46 @@
|
||||||
|
links:
|
||||||
|
- &100
|
||||||
|
bandwidth: 100M
|
||||||
|
latency: 500us
|
||||||
|
- &wan
|
||||||
|
bandwidth: 100M
|
||||||
|
latency: 50ms
|
||||||
|
jitter: 10ms
|
||||||
|
|
||||||
|
zones:
|
||||||
|
- &dc1
|
||||||
|
name: dc1
|
||||||
|
internal: *100
|
||||||
|
external: *wan
|
||||||
|
- &dc2
|
||||||
|
name: dc2
|
||||||
|
internal: *100
|
||||||
|
external: *wan
|
||||||
|
- &dc3
|
||||||
|
name: dc3
|
||||||
|
internal: *100
|
||||||
|
external: *wan
|
||||||
|
|
||||||
|
servers:
|
||||||
|
- name: dc1s1
|
||||||
|
zone: *dc1
|
||||||
|
- name: dc1s2
|
||||||
|
zone: *dc1
|
||||||
|
- name: dc2s1
|
||||||
|
zone: *dc2
|
||||||
|
- name: dc2s2
|
||||||
|
zone: *dc2
|
||||||
|
- name: dc3s1
|
||||||
|
zone: *dc3
|
||||||
|
- name: dc3s2
|
||||||
|
zone: *dc3
|
||||||
|
|
||||||
|
global:
|
||||||
|
subnet:
|
||||||
|
base: 'fc00:9a7a:9e::'
|
||||||
|
local: 64
|
||||||
|
zone: 16
|
||||||
|
latency-offset: 3ms
|
||||||
|
upstream:
|
||||||
|
ip: fc00:9a7a:9e:ffff:ffff:ffff:ffff:ffff
|
||||||
|
conn: *wan
|
||||||
|
|
@ -5,16 +5,19 @@ links:
|
||||||
- &1000
|
- &1000
|
||||||
bandwidth: 1000M
|
bandwidth: 1000M
|
||||||
latency: 100us
|
latency: 100us
|
||||||
|
txqueuelen: 10000
|
||||||
- &vdsl
|
- &vdsl
|
||||||
bandwidth:
|
bandwidth:
|
||||||
up: 3M
|
up: 3M
|
||||||
down: 55M
|
down: 55M
|
||||||
latency: 50ms
|
latency: 50ms
|
||||||
jitter: 10ms
|
jitter: 10ms
|
||||||
|
txqueuelen: 100
|
||||||
- &fiber
|
- &fiber
|
||||||
bandwidth: 400M
|
bandwidth: 400M
|
||||||
latency: 10ms
|
latency: 10ms
|
||||||
jitter: 3ms
|
jitter: 3ms
|
||||||
|
txqueuelen: 1000
|
||||||
|
|
||||||
zones:
|
zones:
|
||||||
- &dc1
|
- &dc1
|
||||||
Loading…
Reference in a new issue