In the previous blog post, we asked ourselves [Does Aerogramme use too much memory?](@/blog/2024-ram-usage-encryption-s3/index.md).
From the discussion, we surfaced it was not acceptable that some specific queries (FETCH FULL or SEARCH TEXT) where loading the
full mailbox in memory. It's concerning as the mail server will be used by multiple users and as a limited amount of resources,
so we don't want a user allocating all the memory. In other words, we want to have a per-user resource usage that remain as stable as possible.
These ideas are developed more in depth in the Amazon article [Reliability & Constant Work](https://aws.amazon.com/fr/builders-library/reliability-and-constant-work/).
For both FETCH and SEARCH, the changes are identical. Before, the command was executed in ~5 seconds, allocated up to 300MB of RAM, and used up to 150% of CPU.
After, the command took ~30 to get executed, allocated up to 400MB of RAM, used up to 40MB of RAM, and used up to 80% of CPU sporadically.
and it adds fairness between users. Indeed, a user can't monopolize all the ressources of the servers (CPU, I/O, etc.) anymore, and thus multiple users
requests are thus intertwined by the server (we assume the number of users is greatly superior to the number of cores). And again, it leads to better predictability, as per-user requests completion will be less impacted by other requests.
Another concern is the RAM consumption with the IDLE feature.
The real cause is that we retain in-memory the full user profile (mailbox data, IO connectors, etc.): we should instead
keep only the minimum data to be waken-up. That's the ideal fix, the final solution, that would take lots of time to design and implement.
This fix is not necessary *now*, instead we can simply try to optimize the size of a full user profile in memory.
On this aspect, the [aws-sdk-s3](https://crates.io/crates/aws-sdk-s3) crate has [the following note](https://docs.rs/aws-sdk-s3/1.16.0/aws_sdk_s3/client/index.html):
> Client construction is expensive due to connection thread pool initialization, and should be done once at application start-up.
Digging deeper in the crate dependencies, we learn from the [aws-smithy-runtime](https://crates.io/crates/aws-smithy-runtime) crate, [we can read](https://docs.rs/aws-smithy-runtime/1.1.7/aws_smithy_runtime/client/http/hyper_014/struct.HyperClientBuilder.html):
> [Constructing] a Hyper client with the default TLS implementation (rustls) [...] can be useful when you want to share a Hyper connector between multiple generated Smithy clients.
First, the spikes are more spaced because I am typing the command by hands, not because Aerogramme is slower!
Another non relevant artifact is the end of the memory plot: memory is not released on the left plot as I cut the recording before typing the LOGOUT command.
What's interesting is the memory usage range: on the left, it's ~20MB, on the right it's ~10MB.
By sharing the HTTP Client, we thus use twice less memory per-user, down to around ~650kB/user for Aerogramme 0.2.2
Back to our 1k users, we get 650MB of RAM, 6.5GB for 10k, and thus 65GB for 100k. So *in theory*, it seems OK,
but our sample and our methodology is too dubious to confirm that *in practice* such memory usage will be observed.
In the end, I think for now IDLE RAM usage in Aerogramme is acceptable, and thus we can move
on other aspects without fearing that IDLE will make the software unusable.
K9 beta (6.714) does not support some values marked as obsolete in the `authentication` field: `plain` is not supported anymore, `password-cleartext` must be used instead.
Content-Type is important also, if a wrong one is sent, content is silently ignored by some clients.
*Broken LITERAL+ (reported by Maxime)* - It was not possible to copy more than one email at once to an Aerogramme mailbox.
It was due to the fact we were using an old version of imap-flow that was not correctly supporting LITERAL+.
Upgrading imap-flow to the latest version fixed the problem.
*Broken IDLE (reported by Maxime)* - After updating imap-flow, we started noticing some timeouts in Thunderbird due to IDLE bugs.
When IDLE was implemented in Aerogramme, the code was not ready in imap-flow,
and thus I used some hacks. But upgrading the library broke my hacks for the best: now
imap-flow supports IDLE out of the box, and thus Aerogramme code is now cleaner and more maintainable.
Some others quality of life feedbacks not reported here have been made by MrFlos & Nicolas, thanks to all the people that took part in this debugging adventure.
