forked from Deuxfleurs/garage
Quentin
b4592a00fe
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: Deuxfleurs/garage#171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>
91 lines
2.7 KiB
Bash
Executable file
91 lines
2.7 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
|
|
set -e
|
|
|
|
SCRIPT_FOLDER="`dirname \"$0\"`"
|
|
REPO_FOLDER="${SCRIPT_FOLDER}/../"
|
|
GARAGE_DEBUG="${REPO_FOLDER}/target/debug/"
|
|
GARAGE_RELEASE="${REPO_FOLDER}/target/release/"
|
|
NIX_RELEASE="${REPO_FOLDER}/result/bin/"
|
|
PATH="${GARAGE_DEBUG}:${GARAGE_RELEASE}:${NIX_RELEASE}:$PATH"
|
|
FANCYCOLORS=("41m" "42m" "44m" "45m" "100m" "104m")
|
|
|
|
export RUST_BACKTRACE=1
|
|
export RUST_LOG=garage=info,garage_api=debug
|
|
MAIN_LABEL="\e[${FANCYCOLORS[0]}[main]\e[49m"
|
|
|
|
WHICH_GARAGE=$(which garage || exit 1)
|
|
echo -en "${MAIN_LABEL} Found garage at: ${WHICH_GARAGE}\n"
|
|
|
|
NETWORK_SECRET="$(openssl rand -hex 32)"
|
|
|
|
|
|
# <<<<<<<<< BEGIN FOR LOOP ON NODES
|
|
for count in $(seq 1 3); do
|
|
CONF_PATH="/tmp/config.$count.toml"
|
|
LABEL="\e[${FANCYCOLORS[$count]}[$count]\e[49m"
|
|
|
|
cat > $CONF_PATH <<EOF
|
|
block_size = 1048576 # objects are split in blocks of maximum this number of bytes
|
|
metadata_dir = "/tmp/garage-meta-$count"
|
|
data_dir = "/tmp/garage-data-$count"
|
|
rpc_bind_addr = "0.0.0.0:$((3900+$count))" # the port other Garage nodes will use to talk to this node
|
|
rpc_public_addr = "127.0.0.1:$((3900+$count))"
|
|
bootstrap_peers = []
|
|
replication_mode = "3"
|
|
rpc_secret = "$NETWORK_SECRET"
|
|
|
|
[s3_api]
|
|
api_bind_addr = "0.0.0.0:$((3910+$count))" # the S3 API port, HTTP without TLS. Add a reverse proxy for the TLS part.
|
|
s3_region = "garage" # set this to anything. S3 API calls will fail if they are not made against the region set here.
|
|
|
|
[s3_web]
|
|
bind_addr = "0.0.0.0:$((3920+$count))"
|
|
root_domain = ".garage.tld"
|
|
index = "index.html"
|
|
EOF
|
|
|
|
echo -en "$LABEL configuration written to $CONF_PATH\n"
|
|
|
|
(garage -c /tmp/config.$count.toml server 2>&1|while read r; do echo -en "$LABEL $r\n"; done) &
|
|
done
|
|
# >>>>>>>>>>>>>>>> END FOR LOOP ON NODES
|
|
|
|
if [ -z "$SKIP_HTTPS" ]; then
|
|
echo -en "$LABEL Starting dummy HTTPS reverse proxy\n"
|
|
mkdir -p /tmp/garagessl
|
|
openssl req \
|
|
-new \
|
|
-x509 \
|
|
-keyout /tmp/garagessl/test.key \
|
|
-out /tmp/garagessl/test.crt \
|
|
-nodes \
|
|
-subj "/C=XX/ST=XX/L=XX/O=XX/OU=XX/CN=localhost/emailAddress=X@X.XX" \
|
|
-addext "subjectAltName = DNS:localhost, IP:127.0.0.1"
|
|
cat /tmp/garagessl/test.key /tmp/garagessl/test.crt > /tmp/garagessl/test.pem
|
|
socat openssl-listen:4443,reuseaddr,fork,cert=/tmp/garagessl/test.pem,verify=0 tcp4-connect:localhost:3911 &
|
|
fi
|
|
|
|
sleep 3
|
|
# Establish connections between nodes
|
|
for count in $(seq 1 3); do
|
|
NODE=$(garage -c /tmp/config.$count.toml node id -q)
|
|
for count2 in $(seq 1 3); do
|
|
garage -c /tmp/config.$count2.toml node connect $NODE
|
|
done
|
|
done
|
|
|
|
RETRY=120
|
|
until garage -c /tmp/config.1.toml status 2>&1|grep -q HEALTHY ; do
|
|
(( RETRY-- ))
|
|
if (( RETRY <= 0 )); then
|
|
echo -en "${MAIN_LABEL} Garage did not start"
|
|
exit 1
|
|
fi
|
|
echo -en "${MAIN_LABEL} cluster starting...\n"
|
|
sleep 1
|
|
done
|
|
|
|
echo -en "${MAIN_LABEL} cluster started\n"
|
|
|
|
wait
|