117 lines
7.9 KiB
Markdown
117 lines
7.9 KiB
Markdown
+++
|
|
title="Aerogramme 0.2.2: predictability & user testing"
|
|
date=2024-02-22
|
|
+++
|
|
|
|
*Let's review how Aerogramme performances became more predictable, why it's important, and showcase how user testing helped surfacing bugs.*
|
|
|
|
<!-- more -->
|
|
|
|
---
|
|
|
|
This minor version of Aerogramme put the focus on 2 aspects of the software: predictable performances & collecting user feedbacks.
|
|
In the following, I describe both aspect in details.
|
|
|
|
## Improving predictability
|
|
|
|
In the previous blog post, we asked ourselves [Does Aerogramme use too much memory?](@/blog/2024-ram-usage-encryption-s3/index.md).
|
|
From the discussion, we surfaced it was not acceptable that some specific queries (FETCH FULL or SEARCH TEXT) where loading the
|
|
full mailbox in memory. It's concerning as the mail server will be used by multiple users and as a limited amount of resources,
|
|
so we don't want a user allocating all the memory. In other words, we want to have a per-user resource usage that remain as stable as possible.
|
|
These ideas are developed more in depth in the Amazon article [Reliability & Constant Work](https://aws.amazon.com/fr/builders-library/reliability-and-constant-work/).
|
|
|
|
As part of the conclusion, we identified that streaming emails content would solve our problem.
|
|
In practice, I rewrote the relevant part of the code to return a `futures::stream::Stream` instead of a `Vec`.
|
|
Then, for both requests, I re-run the same benchmark to have a before/after comparison for both commands.
|
|
Next, the first pictures is for FETCH FULL, the second one for SEARCH TEXT.
|
|
|
|
|
|
|
|
|
|
|
|
For both FETCH and SEARCH, the changes are identical. Before, the command was executed in ~5 seconds, allocated up to 300MB of RAM, and used up to 150% of CPU.
|
|
After, the command took ~30 to get executed, allocated up to 400MB of RAM, used up to 40MB of RAM, and used up to 80% of CPU sporadically.
|
|
|
|
Here is why it's a positive thing: now the memory consumption of Aerogramme is capped approximately by the biggest email accepted
|
|
by your system (25MB, multiplied by a small constant).
|
|
|
|
It has also other benefits: it prevents the file descriptor and other network ressource exhaustion,
|
|
and it adds fairness between users. Indeed, a user can't monopolize all the ressources of the servers (CPU, I/O, etc.) anymore, and thus multiple users
|
|
requests are thus intertwined by the server (we assume the number of users is greatly superior to the number of cores). And again, it leads to better predictability, as per-user requests completion will be less impacted by other requests.
|
|
|
|
Another concern is the RAM consumption with the IDLE feature.
|
|
The real cause is that we retain in-memory the full user profile (mailbox data, IO connectors, etc.): we should instead
|
|
keep only the minimum data to be waken-up. That's the ideal fix, the final solution, that would take lots of time to design and implement.
|
|
This fix is not necessary *now*, instead we can simply try to optimize the size of a full user profile in memory.
|
|
On this aspect, the [aws-sdk-s3](https://crates.io/crates/aws-sdk-s3) crate has [the following note](https://docs.rs/aws-sdk-s3/1.16.0/aws_sdk_s3/client/index.html):
|
|
|
|
> Client construction is expensive due to connection thread pool initialization, and should be done once at application start-up.
|
|
|
|
Digging deeper in the crate dependencies, we learn from the [aws-smithy-runtime](https://crates.io/crates/aws-smithy-runtime) crate, [we can read](https://docs.rs/aws-smithy-runtime/1.1.7/aws_smithy_runtime/client/http/hyper_014/struct.HyperClientBuilder.html):
|
|
|
|
> [Constructing] a Hyper client with the default TLS implementation (rustls) [...] can be useful when you want to share a Hyper connector between multiple generated Smithy clients.
|
|
|
|
It seems to be exactly what we want to do, but to be effective, we must do the same thing for K2V.
|
|
After implementing these features, I got the following plot:
|
|
|
|
|
|
|
|
First, the spikes are more spaced because I am typing the command by hands, not because Aerogramme is slower!
|
|
Another non relevant artifact is the end of the memory plot: memory is not released on the left plot as I cut the recording before typing the LOGOUT command.
|
|
What's interesting is the memory usage range: on the left, it's ~20MB, on the right it's ~10MB.
|
|
By sharing the HTTP Client, we thus use twice less memory per-user, down to around ~650kB/user for Aerogramme 0.2.2
|
|
Back to our 1k users, we get 650MB of RAM, 6.5GB for 10k, and thus 65GB for 100k. So *in theory*, it seems OK,
|
|
but our sample and our methodology is too dubious to confirm that *in practice* such memory usage will be observed.
|
|
|
|
In the end, I think for now IDLE RAM usage in Aerogramme is acceptable, and thus we can move
|
|
on other aspects without fearing that IDLE will make the software unusable.
|
|
|
|
|
|
## Users feedbacks
|
|
|
|
*Dovecot AUTH continuation inlining* - When a username + password is short,
|
|
the Dovecot SASL Auth protocol allows the client (here Postfix)
|
|
to send the base64 inlined, without having to wait for the continuation.
|
|
It was not supported by Aerogramme and was preventing some users from authenticating.
|
|
|
|
*Pipelining limits (reported by Nicolas)* - Pipeling limit set to 3.
|
|
Avoiding DoS resources. But failing some honest clients like Mutt.
|
|
Bumped to 64, will be watched in the next months.
|
|
|
|
*SASL Auth subtleties (reported by Nicolas)* - Authorization can be empty, or can be set to the same value as Authentication.
|
|
Second case not handled but required by Fair Email (thx Nicolas)
|
|
|
|
*Thunderbird Autodiscovery issues (reported by LX & Nicolas)* - K9 stable does not support the `%EMAILLOCALPART%` placeholder.
|
|
K9 beta (6.714) does not support some values marked as obsolete in the `authentication` field: `plain` is not supported anymore, `password-cleartext` must be used instead.
|
|
Content-Type is important also, if a wrong one is sent, content is silently ignored by some clients.
|
|
Today autodiscovery is not part of Aerogramme, but due to these issues, and given that generating these files
|
|
on the fly could improve compatibility (email is passed a query parameter), so we could directly put the right username
|
|
instead of relying on placeholders that are not well supported.
|
|
|
|
*Broken LITERAL+ (reported by Maxime)* - It was not possible to copy more than one email at once to an Aerogramme mailbox.
|
|
It was due to the fact we were using an old version of imap-flow that was not correctly supporting LITERAL+.
|
|
Upgrading imap-flow to the latest version fixed the problem.
|
|
|
|
*Broken IDLE (reported by Maxime)* - After updating imap-flow, we started noticing some timeouts in Thunderbird due to IDLE bugs.
|
|
When IDLE was implemented in Aerogramme, the code was not ready in imap-flow,
|
|
and thus I used some hacks. But upgrading the library broke my hacks for the best: now
|
|
imap-flow supports IDLE out of the box, and thus Aerogramme code is now cleaner and more maintainable.
|
|
|
|
Some others quality of life feedbacks not reported here have been made by MrFlos & Nicolas, thanks to all the people that took part in this debugging adventure.
|
|
|
|
## Conclusion: download and test this new version!
|
|
|
|
Do not get me wrong: Aerogramme is still not ready for prime time.
|
|
But by operating it for real, we start understanding better how it behaves, what are the rough edges, what we want to improve, what we need, etc.
|
|
|
|
If you are interested in being an Aerogramme early adopter, it might be the good time to setup a test cluster, try Aerogramme,
|
|
share your use cases, in order to shape its future and be ready to go in production when its public beta will be announced.
|
|
|
|
|
|
```bash
|
|
docker run --net host registry.deuxfleurs.org/aerogramme:0.2.2
|
|
```
|
|
|
|
*It will launch Aerogramme in development mode. Username is `alice`, password is `hunter2`, email is `alice@example.tld`.*
|
|
|
|
[Download](/download/) - [Changelog](https://git.deuxfleurs.fr/Deuxfleurs/aerogramme/releases/tag/0.2.2) - [Getting started](/documentation/)
|