diff --git a/content/blog/2024-ram-usage-encryption-s3/copy-move.png b/content/blog/2024-ram-usage-encryption-s3/copy-move.png new file mode 100644 index 0000000..aaae369 Binary files /dev/null and b/content/blog/2024-ram-usage-encryption-s3/copy-move.png differ diff --git a/content/blog/2024-ram-usage-encryption-s3/index.md b/content/blog/2024-ram-usage-encryption-s3/index.md index f149b88..505a2ff 100644 --- a/content/blog/2024-ram-usage-encryption-s3/index.md +++ b/content/blog/2024-ram-usage-encryption-s3/index.md @@ -51,6 +51,8 @@ Below I plotted the empirical distribution for both my dataset and my personal i ![ECDF mailbox](ecdf_mbx.svg) +*[Get the 100 emails dataset](https://git.deuxfleurs.fr/Deuxfleurs/aerogramme/src/commit/0b20d726bbc75e0dfd2ba1900ca5ea697645a8f1/tests/emails/aero100.mbox.zstd) - [Get the CSV used to plot this graph](https://git.deuxfleurs.fr/Deuxfleurs/aerogramme/src/branch/perf/cpu-ram-bottleneck/tests/emails/mailbox_email_sizes.csv)* + We see that the curves are close together and follow the same pattern: most emails are between 1kB and 100kB, and then we have a long tail (up to 20MB in my inbox, up to 6MB in the dataset). It's not that surprising: on many places on the Internet, the limit on emails is set to 25MB. Overall I am quite satisfied by this simple dataset, even if having one or two bigger emails could make it even more representative of my real inbox... @@ -65,6 +67,8 @@ The following bar plot depicts the command distribution per command name; top is ![Commands](command-run.svg) +*[Get the IMAP command log](https://git.deuxfleurs.fr/Deuxfleurs/aerogramme/src/branch/perf/cpu-ram-bottleneck/tests/emails/imap_commands_dataset.log) - [Get the CSV used to plot this graph](https://git.deuxfleurs.fr/Deuxfleurs/aerogramme/src/branch/perf/cpu-ram-bottleneck/tests/emails/imap_commands_summary.csv)* + First, we can handle separately some commands: LOGIN, CAPABILITY, ENABLE, SELECT, EXAMINE, CLOSE, UNSELECT, LOGOUT as they are part of a **connection workflow**. We do not plan on studying them directly as they will be used in all other tests. @@ -104,6 +108,99 @@ UID SEARCH BEFORE 2024-02-09 ``` --> + + +In the following, I will keep these 3 categories: **writing**, **notification**, and **query** to evaluate Aerogramme's ressource usage +based on command patterns observed on real IMAP commands and the provided dataset. + +--- + + +## Write Commands + +We start by the write commands as it will enable us to fill the mailboxes for the following evaluations. + +I inserted the full dataset (100 emails) to 16 accounts (in other words, in the end, the server handles 1 600 emails) with APPEND. +*[Get the Python script](https://git.deuxfleurs.fr/Deuxfleurs/aerogramme/src/branch/main/tests/instrumentation/mbox-to-imap.py)* + +### Filling a mailbox + +![Append Custom Build](01-append-tokio-console-musl.png) + + +First, I observed this *scary* linear memory increase. It seems we are not releasing some memory, +and that's an issue! I quickly suspected tokio-console of being the culprit. +A quick search lead me to an issue entitled [Continuous memory leak with console_subscriber #184](https://github.com/tokio-rs/console/issues/184) +that confirmed my intuition. +Instead of waiting for an hour or trying to tweak the retention time, I built Aerogramme without tokio console. + +*So in a first approach, we observed the impact of tokio console instead of our code! Still, we want to +have performances as predictable as possible.* + +![Append Cargo Release](02-append-glibc.png) + +Which got us to this second pattern: a stable but high memory usage compared to previous run. +It appears I built the binary with `cargo release`, which creates a binary that dynamically link to the GNU libc. +The previous binary was built with our custom Nix toolchain that statically link musl libc to our binary. +In the process, we changed the allocator: it seems the GNU libc allocator allocates bigger chunks at once. + +*It would be wrong to conclude the musl libc allocator is more efficient: allocating and unallocating +memory on the kernel side is costly, and thus it might be better for the allocator to keep some kernel allocated memory +for future memory allocations that will not require system calls. This is another example of why this benchmark is wrong: we observe +the memory allocated by the allocator, not the memory used by program itself.* + +For the next graph, I removed tokio-console and built Aerogramme with a static musl libc. + +![Append Custom Build](03-append-musl.png) + +The observed patterns match way better what I was expecting. +We observe 16 spikes of memory allocation, around 50MB, followed by a 25MB memory usage. +In the end, we drop to ~18MB. + +In this scenario, we can say that a user needs between 32MB of RAM and 7MB. + +In the previous runs, we were doing the inserts sequentially. But in the real world, multiple users interact with the server +at the same time. In the next run, we run the same test but in parrallel. + +![Append Parallel](04-append-parallel.png) + +We see 2 spikes: a short one at the beggining, and a longer one at the end. +The first spike is probably due to the argon2 decoding, a key derivation function +that is purposedly built to be expensive in term of RAM and CPU. +The second spike is due to the fact that big emails (multiple MB) are at the end of the dataset, +and they are stored fully in RAM before being sent. However, our biggest email weighs 6MB, +and we are running 16 threads, so we should expect around a memory usage that is around 100MB, +not 400MB. This difference would be a good starting point for an investigation: we might +copy a same email multiple times in RAM. + +It seems that in this first test that Aerogramme is particularly sensitive to 1) login commands due to argon2 and 2) large emails. + +### Copy and Move + +You might need to organize your folders, copying or moving your email across your mailboxes. +COPY is a standard IMAP command, MOVE is an extension. +I will focus on a brutal test: copying 1k emails from the INBOX to Sent, then moving these 1k emails to Archive. +Below is the graph depicting Aerogramme resource usage during this test. + +![Copy and move](copy-move.png) + +Memory usage remains stable and low (below 25MB), but the operations are CPU intensive (close to 100% for 40 seconds). +Both COPY and MOVE depict the same pattern: indeed, as emails are considered immutable, Aerogramme only handle pointers in both cases +and do not really copy their content. + +Real world clients would probably not send such brutal commands, but would do it progressively, either one by one, or with small batches, +to keep the UI responsive. + +While CPU optimizations could probably be imagined, I find this behavior satisfying, especially as memory remains stable and low. + +### Setting flags + +Setting flags (Seen, Deleted, Answered, NonJunk, etc.) is done through the STORE command. +Our run will be made in 3 parts: 1) putting one flag on one email, 2) putting 16 flags on one email, and 3) putting one flag on 1k emails. +The result is depicted in the graph below. + +![Store flags](store.png) + -In the following, I will keep these 3 categories: **writing**, **notification**, and **query** to evaluate Aerogramme's ressource usage -based on command patterns observed on real IMAP commands. +The first and last spike are due respectively to the LOGIN/SELECT and CLOSE/LOGOUT commands. +We thus have 3 CPU spikes, one for each command, memory remains stable. +The last command is bar far the most expensive, and indeed, it has to generate 1k events in our event log and rebuild many things in the index. +However, there is no reason for the 2nd command to be less expensive than the first one except from the fact it reuses some ressources / cache entries +from the first request. ---- - - -## Write Commands - -I inserted the full dataset (100 emails) to 16 accounts (the server now handles 1 600 emails then). -*[See the script](https://git.deuxfleurs.fr/Deuxfleurs/aerogramme/src/branch/main/tests/instrumentation/mbox-to-imap.py)* - -`APPEND` - -![Append Custom Build](01-append-tokio-console-musl.png) - - -First, I observed this *scary* linear memory increase. It seems we are not releasing some memory, -and that's an issue! I quickly suspected tokio-console of being the culprit. -A quick search lead me to an issue entitled [Continuous memory leak with console_subscriber #184](https://github.com/tokio-rs/console/issues/184) -that confirmed my intuition. -Instead of waiting for an hour or trying to tweak the retention time, I tried a build without tokio console. - -*So in a first approach, we observed the impact of tokio console instead of our code! Still, we want to -have performances as predictable as possible.* - -![Append Cargo Release](02-append-glibc.png) - -Which got us to this second pattern: a stable but high memory usage compared to previous run. -It appears I built the binary with `cargo release`, which creates a binary that dynamically link to the GNU libc. -While the previous binary was made with our custom Nix toolchain that statically compiles the Musl libc. -In the process, we changed the allocator: it seems the GNU libc allocator allocates bigger chunks at once. - -*It would be wrong to conclude the musl libc allocator is more efficient: allocating and unallocating -memory on the kernel side is costly, and thus it might be better for the allocator to keep some kernel allocated memory -for future memory allocations that will not require system calls. This is another example of why this benchmark is wrong: we observe -the memory allocated by the allocator, not the memory used by program itself.* - -For the next graph, I removed tokio-console but built Aerogramme with static musl libc. - -![Append Custom Build](03-append-musl.png) - -We observe 16 spikes of memory allocation, around 50MB, followed by a 25MB memory usage. In the end, -we drop to ~18MB. We do not try to analyze the spike for now. However, we can assume the 25MB memory usage accounts for the base memory consumption -plus the index of the user's mailbox. Once the last user logged out, memory drops to 18MB. -In this scenario, a user accounts for around 7MB. - -*We will see later that some other use cases lead to a lower per-user RAM consumption. -An hypothesis: we are doing some requests on S3 with the aws-sdk library that is intended to be configured once -per process, and handles internally the threading logic. In our case, we instantiate it once per user, -tweaking its configuration might help. Again, we are not observing - only - our code!* - -In the previous runs, we were doing the inserts sequentially. But in the real world, multiple users interact with the server -at the same time. In the next run, we run the same test but in parrallel. - -![Append Parallel](04-append-parallel.png) - -We see 2 spikes: a short one at the beggining, and a longer one at the end. +Interacting with the index is really efficient in term of memory. Generating many changes +lead to high CPU (and possibly lot of IO), but from our dataset we observe most changes are done on one or two emails +and never on all the mailbox. +Interacting with flags should not be an issue for Aerogramme in the near future. ## Notification Commands -`NOOP` & `CHECK` +Notification commands are expected to be run regularly in background by clients. +They are particularly sensitive as they are correlated to your number of users, +independently of the number of emails they receive. I split them in 2 parts: +the ones that are intermittent, and like HTTP, closes the connection after being run, +and the ones that are continuous, where the socket is maintained open forever. -*TODO* +### The cost of a refresh -`STATUS` +NOOP, CHECK, STATUS are commands that trigger a refresh of the IMAP +view, and are part of the "intermittent" commands. In some ways, the SELECT and/or EXAMINE +commands could also be interpreted as a notification command: a client that is configured +to poll a mailbox every 15 minutes will not use the NOOP, running EXAMIME will be enough. -*TODO* +In our case, all these commands are similar in the sense that they load or refresh the in-memory index +of the targeted mailbox. To illustrate my point, I will run SELECT, NOOP, CHECK, and STATUS on another mailbox in a row. -`IDLE` +![Refresh plot](refresh.png) + +The first CPU spike is LOGIN/SELECT, the second is NOOP, the third CHECK, the last one STATUS. +CPU spikes are short, memory usage is stable. + +Refresh commands should not be an issue for Aerogramme in the near future. + +### Continuously connected clients + +IDLE (and NOTIFY that is currently not implemented in Aerogramme) are commands +that maintain a socket opened. These commands are sensitive, as while many protocols +are one shot, and then your users spread their requests over time, with these commands, +all your users are continuously connected. + +In the graph below, we plot the resource usage of 16 users that log into the system, +select inbox, and switch to IDLE, then, one by one, they receive an email and are notified. ![Idle Parallel](05-idle-parallel.png) diff --git a/content/blog/2024-ram-usage-encryption-s3/refresh.png b/content/blog/2024-ram-usage-encryption-s3/refresh.png new file mode 100644 index 0000000..cb5d66e Binary files /dev/null and b/content/blog/2024-ram-usage-encryption-s3/refresh.png differ diff --git a/content/blog/2024-ram-usage-encryption-s3/store.png b/content/blog/2024-ram-usage-encryption-s3/store.png new file mode 100644 index 0000000..940aeec Binary files /dev/null and b/content/blog/2024-ram-usage-encryption-s3/store.png differ