slow deletion? #594
Labels
No labels
action
check-aws
action
discussion-needed
action
for-external-contributors
action
for-newcomers
action
more-info-needed
action
need-funding
action
triage-required
kind
correctness
kind
ideas
kind
improvement
kind
performance
kind
testing
kind
usability
kind
wrong-behavior
prio
critical
prio
low
scope
admin-api
scope
background-healing
scope
build
scope
documentation
scope
k8s
scope
layout
scope
metadata
scope
ops
scope
rpc
scope
s3-api
scope
security
scope
telemetry
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: Deuxfleurs/garage#594
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
hello,
we currently deleted a ton of objects inside garage cluster that is inside k8s without any resource constraints. the cluster is not even close to be satisified:
and these are the nodes:
so we still have some headroom.
the disks are NVMe drives in a software raid 10, so and have a way higher iops rate, however after deleting like 1 million or more objects (we use wal-g and had a misconfiguration that it did not cleanup old backups, so we had like 1 1/2 years of wal files) it only deletes like a few each second, these are the busy worker stats:
and these are some of the logs:
it deletes, but really really slowly. is there any way to speed this up? i'm not sure if it is even a good idea to change the config now.
does a restart do any harm to the deletion process? it's not that big of a deal that it deletes that slowly, but something is probably not correct here
Start by setting the resync worker count to 4 and resync tranquility to zero.
To make this run faster overall, we might need to change some parameters which are currently compile time constants, like the maximum number of resync workers or the batch sizes for internal data transfers. I can point you to the lines of code to change if you are interested in trying this.
These are quite large volumes of data which I don't think we ever experienced running garage with, so it's not necessarily expected that things will go smoothly. Please keep us updated on how it goes.
Btw, to answer your question, no, restarting garage will not impact the deletion process
btw. what is also a little bit strange is that the queue did raise, besides that we did not change a thing:
I will change the values, however I'm not sure if I'm coming to that today, I will give an update tomorrow.
well under normal circumstances that wouldn't be the amount of data that I wanted to work with aswell, was just a wal-g misconfiguration...
might be a good idea.
it's still running, but I can just let it keep going, will probably take a while:
somehow it did not reclaim that much storage space, yet
actually it does not look that everything is deleted:
this is now after a few days.
statscall gives me like:however garage is still at like 1.1 tb of storage usage:
we use kubernetes with local storage, so I'm not sure what to do now?
rebuilding garage ? with better parameter?
I would say the biggest problem is not the fact, that there are not enough worker or anything, it is more or less a bigger problem that it will only does work every 10 seconds and not clearing the queue faster.
in the current speed it will probably years to cleanup the whole cluster.
What is most worrying to me is the size of your block_ref resync queue, 12M items will definitely take a while to clear unless you increase the batch size on this line (unfortunately this is not tunable right now). This queue has to be processed before the block resync worker can do stuff, I'm not sure exactly of what's in your cluster but that might be a reason why blocks don't get deleted fast enough.
The other source of possible delay is the resync tranquility parameter which you can check with
garage worker get -a resync-tranquility. If it is larger than zero, Garage introduces some delays between block resyncs to avoid saturating the network. This is a safety measure that is usefull during standard operation but in your case you just want the resync queue to clear as fast as possible, so you should do agarage worker set -a resync-tranquility 0.Concerning the size taken by Garage in your local storage, be mindfull that Garage reports available space. So if 1.1T is your used space, Garage reporting 680GB free space for 1.9TB total is consistent.
actually the resync-tranquility is already zero:
not sure if it needs a restart tough. (the worker were updated on the fly)
actually that is the stuff that I deleted. So it's not new data or anything. I stopped wal-g completly till I can resolve this problem.
that is fine, garage took 1.1T right now while the rest is used by other stuff. the disk is not partitioned so instead we have a raid 10 software raideded NVMe drive that is not at it's i/o limit (not even close at the limit and in the night the only thing that needs i/o is basically garage deletions...) (it's not the best idea to put multiple things on the same disk(-array) but we still have plenty i/o left and we will probably keep that until we get to our next roadblock)
the 1.1T should go down since basically we were at like ~1.3T and I deleted ~800G, but as you can see it's now a week ago and did not really delete that much. only ~150g so it's painfull slow.
and as said resync-tranq is at zero and it still only does deletions every around 10s or something like that, I only get new log entries every 10-20 seconds.
today I restarted the nodes, but it did more harm.
it now hangs when I try to run a command and the logs say something like:
the crd of the pod has the correct ip address tough
EDIT: Looks like after an hour or so the commands started to work again
i tried to compile garage on my own, however whatever I did the binary turned out to be quite big > 500 even with a low amount of features, if I used bundled-libs, did I do something wrong?
in the top level
Cargo.toml, you can comment these linesto get smaller binaries
To get smaller binaries, you should build Garage with LTO, panic=abort and strip the resulting binary.
Instead of modifying
Cargo.toml, you can also set the following environment variables:You can also link garage dynamically with system-provided libsodium, sqlite and zstd instead of bundling vendored libraries.
(BTW, that’s how is the garage package in Alpine Linux built. The resulting size is 14.24 MiB.)
I tried compiling it without the debug and that made it smaller.
However the docker image did not work, but than I compiled it via:
however for some reason it than failed by correctly applying the k8s discovery:
maybe there is a problem when using a scratch docker image. somehow the dns stuff starts failing than, hm...
Maybe I should use the musl target:
x86_64-unknown-linux-musl(but that resulted in build failures...) I need to look more into itI remade the whole cluster