Deuxfleurs
/
garage
41
37
Fork 55
Code Issues 70 Pull Requests 6 Releases 29 Wiki Activity

Fix unbounded buffering when one node has slower network #792

Merged
lx merged 2 commits from fix-buffering into main 2024-03-28 12:40:28 +00:00
Conversation 1 Commits 2 Files Changed 11 +228 -58
lx commented 2024-03-27 15:07:43 +00:00
Owner
Copy Link

  • Add mechanism to limit the number of blocks buffered in RAM as they await being sent to storage nodes. Backpressure is applied to the clients when the limit is reached. The size of this buffer can be controled using the config parameter block_ram_buffer_max

  • Add metric block_ram_buffer_free_kb to visualize memory still available for buffers (when it drops to zero, backpressure is applied)

  • Change the network scheduling algorithm to send requests one after the other, instead of least-attained-service that was equivalent to a round-robin and would cause all blocks to be buffered simultaneously

Related: #788

- [x] Add mechanism to limit the number of blocks buffered in RAM as they await being sent to storage nodes. Backpressure is applied to the clients when the limit is reached. The size of this buffer can be controled using the config parameter `block_ram_buffer_max` - [x] Add metric `block_ram_buffer_free_kb` to visualize memory still available for buffers (when it drops to zero, backpressure is applied) - [x] Change the network scheduling algorithm to send requests one after the other, instead of least-attained-service that was equivalent to a round-robin and would cause all blocks to be buffered simultaneously Related: #788
lx added 2 commits 2024-03-27 15:07:43 +00:00
ci/woodpecker/push/debug Pipeline failed Details
ci/woodpecker/pr/debug Pipeline was successful Details
9613f9495e
[fix-buffering] change request sending strategy and fix priorities 
remove LAS, priorize new requests but otherwise just do standard queuing
lx changed title from Fix buffering issues (fix #788) to Fix unbounded buffering when one node has slower network (fix #788) 2024-03-27 15:09:19 +00:00
lx force-pushed fix-buffering from 9613f9495e to 85f580cbde 2024-03-27 15:23:01 +00:00 Compare
lx commented 2024-03-28 12:02:36 +00:00
Author
Owner
Copy Link

Confirmed that this avoids unbounded memory usage growth inducing OOM kill on a test deployment when sending a single 8GB file to an S3 API server on localhost, which had to send to two other storage nodes, one a bit slower than the other.

With the default value of block_ram_buffer_free_kb (256MB), the RAM usage of Garage as measured by RssAnon is stable under 300MB during the transfer, and the overall speed of the transfer is correctly capped to the speed of the slowest internet link between nodes. Without the patch the API server received data faster than it could send back to storage nodes and RAM usage grew to 1GB, 2GB, 3GB, ... and provoked an OOM kill of the daemon.

Confirmed that this avoids unbounded memory usage growth inducing OOM kill on a test deployment when sending a single 8GB file to an S3 API server on localhost, which had to send to two other storage nodes, one a bit slower than the other. With the default value of `block_ram_buffer_free_kb` (256MB), the RAM usage of Garage as measured by RssAnon is stable under 300MB during the transfer, and the overall speed of the transfer is correctly capped to the speed of the slowest internet link between nodes. Without the patch the API server received data faster than it could send back to storage nodes and RAM usage grew to 1GB, 2GB, 3GB, ... and provoked an OOM kill of the daemon.
lx changed title from Fix unbounded buffering when one node has slower network (fix #788) to Fix unbounded buffering when one node has slower network 2024-03-28 12:05:05 +00:00
lx changed title from Fix unbounded buffering when one node has slower network to Fix unbounded buffering when one node has slower network (fix #788) 2024-03-28 12:37:30 +00:00
lx changed title from Fix unbounded buffering when one node has slower network (fix #788) to Fix unbounded buffering when one node has slower network 2024-03-28 12:39:41 +00:00
lx merged commit ecf641d88c into main 2024-03-28 12:40:28 +00:00
lx deleted branch fix-buffering 2024-03-28 12:40:28 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
AdminAPI

Issues concerning the administration API

  
Bug

   
Check AWS

The behaviour should be checked against AWS S3

  
CI

Issues with Nix, Drone, Rust Compiler, etc.

  
Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  
Critical

   
Documentation

   
Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
Improvement

   
Low priority

For issues that we don't want to spend time on until made absolutely necessary

  
Newcomer

A bug that should be newcomer friendly.

  
Performance

   
S3 Compatibility

   
Testing

   
Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label
AdminAPI
Bug
Check AWS
CI
Correctness
Critical
Documentation
Ideas
Improvement
Low priority
Newcomer
Performance
S3 Compatibility
Testing
Usability
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#792
No description provided.
Delete Branch "fix-buffering"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?