Deuxfleurs/garage
42
39
Fork 56
Code Issues 75 Pull requests 8 Releases 29 Wiki Activity

Add semaphore to limit RAM used by buffered outgoing requests #143

Merged
lx merged 1 commit from request-buffer-semaphore into main 2021-11-04 08:30:12 +00:00
Conversation 1 Commits 1 Files changed 3 +34 -7
lx commented 2021-11-03 16:02:32 +00:00
Owner
Copy link

Currently Garage OOM's very often, probably because lots of requests get buffered in netapp client connections. This patch tries to avoid this by limiting the nubmer of outgoing requests that are buffered with a semaphore.

Status: making this PR so that I can make a build. Then I will test and try to get experimental data.

Currently Garage OOM's very often, probably because lots of requests get buffered in netapp client connections. This patch tries to avoid this by limiting the nubmer of outgoing requests that are buffered with a semaphore. Status: making this PR so that I can make a build. Then I will test and try to get experimental data.
lx force-pushed request-buffer-semaphore from 2c5336ba13 to face98681a 2021-11-03 16:22:14 +00:00 Compare
lx force-pushed request-buffer-semaphore from face98681a to 8b871284ab 2021-11-03 16:39:34 +00:00 Compare
lx changed target branch from main to cleanups 2021-11-03 16:51:03 +00:00
lx commented 2021-11-03 17:00:54 +00:00
Author
Owner
Copy link

Observation: it seems to work wonderfully. Garage now stays constantly under 400MB of RAM used in a scenario where it caused an OOM before. No performance degradation: data is still injected at maximum rate and network is still saturated. We could probably set the semaphore size to 200MB, wouldn't be a big problem.

EDIT: semaphore size set to 200MB

Observation: it seems to work wonderfully. Garage now stays constantly under 400MB of RAM used in a scenario where it caused an OOM before. No performance degradation: data is still injected at maximum rate and network is still saturated. We could probably set the semaphore size to 200MB, wouldn't be a big problem. EDIT: semaphore size set to 200MB
lx changed target branch from cleanups to main 2021-11-03 17:02:21 +00:00
lx force-pushed request-buffer-semaphore from 8b871284ab to 6f13d083ab 2021-11-03 17:03:06 +00:00 Compare
lx changed title from WIP: Add semaphore to limit RAM used by buffered outgoing requests to Add semaphore to limit RAM used by buffered outgoing requests 2021-11-04 08:30:08 +00:00
lx merged commit 6f13d083ab into main 2021-11-04 08:30:12 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
AdminAPI

Issues concerning the administration API

  
Bug

   
Check AWS

The behaviour should be checked against AWS S3

  
CI

Issues with Nix, Drone, Rust Compiler, etc.

  
Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  
Critical

   
Documentation

   
Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
Improvement

   
Low priority

For issues that we don't want to spend time on until made absolutely necessary

  
Newcomer

A bug that should be newcomer friendly.

  
Performance

   
S3 Compatibility

   
Testing

   
Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No labels
AdminAPI
Bug
Check AWS
CI
Correctness
Critical
Documentation
Ideas
Improvement
Low priority
Newcomer
Performance
S3 Compatibility
Testing
Usability
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#143
No description provided.
Delete branch "request-buffer-semaphore"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?