eml-codec/README.md

122 lines
5.6 KiB
Markdown
Raw Normal View History

2023-06-08 19:59:41 +00:00
# imf-codec
2023-06-12 14:05:06 +00:00
**Work in progress, do not use in production**
2023-06-17 09:43:54 +00:00
**This is currently only a decoder (parser), encoding is not supported.**
2023-06-12 14:05:06 +00:00
2023-06-17 09:43:54 +00:00
## Goals
2023-06-12 14:05:06 +00:00
2023-07-03 15:05:17 +00:00
- Maintainability - modifying the code does not create regression and is possible for someone exterior to the project. Keep cyclomatic complexity low.
- Composability - build your own parser by picking the relevant passes, avoid work that is not needed.
- Compatibility - always try to parse something, do not panic or return an error.
2023-07-04 12:37:53 +00:00
- Exhaustivity - serve as a common project to encode knowledge about emails (existing mime types, existing headers, etc.).
2023-06-17 09:43:54 +00:00
## Non goals
2023-07-03 15:05:17 +00:00
- Parsing optimization that would make more complicated to understand the logic.
- Optimization for a specific use case, to the detriment of other use cases.
2023-06-17 09:43:54 +00:00
- Pipelining/streaming/buffering as the parser can arbitrarily backtrack + our result contains reference to the whole buffer, imf-codec must keep the whole buffer in memory. Avoiding the sequential approach would certainly speed-up a little bit the parsing, but it's too much work to implement currently.
- Zerocopy. It might be implementable in the future, but to quickly bootstrap this project, I avoided it for now.
## Missing / known bugs
Current known limitations/bugs:
- Resent Header Fields are not implemented
- Return-Path/Received headers might be hard to use as their order is important, and it's currently lost in the final datastructure.
- Datetime parsing of invalid date might return `None` instead of falling back to the `bad_body` field
2023-07-03 15:05:17 +00:00
- Comments contained in the email headers are dropped during parsing
- No support is provided for message/external-body (read data from local computer) and message/partial (aggregate multiple fragmented emails) as they seem obsolete and dangerous to implement.
2023-06-17 09:43:54 +00:00
## Design
2023-06-13 07:03:51 +00:00
2023-07-03 15:05:17 +00:00
Multipass design: each pass is in charge of a specific work.
*Having multiple pass does not necessarily lead to abyssmal performances.
For example, the [Chez Scheme compiler](https://legacy.cs.indiana.edu/~dyb/pubs/commercial-nanopass.pdf)
pioneered the "Nanopass" concept and showcases excellent performances.*
Currently, you can use the following passes:
- `segment.rs` - Extract the header section by finding the `CRLFCRLF` token.
- `guess_charset.rs` - Find the header section encoding (should be ASCII or UTF8 but some corpus contains ISO-8859-1 headers)
- `extract_fields.rs` - Extract the headers line by lines, taking into account Foldable White Space.
- `field_lazy.rs` - Try to recognize the header fields (`From`, `To`, `Date`, etc.) but do not parse their value.
- `field_eager.rs` - Parse the value of each known header fields.
- `header_section.rs` - Aggregate the various fields in a single structure.
2023-06-17 09:43:54 +00:00
## Testing strategy
2023-06-19 14:09:11 +00:00
imf-codec aims to be as much tested as possible against reald
### Unit testing: parser combinator independently (done)
### Selected full emails (expected)
### Existing datasets
**Enron 500k** - Took 20 minutes to parse ~517k emails and check that
RFC5322 headers (From, To, Cc, etc.) are correctly parsed.
From this list, we had to exclude ~50 emails on which
the From/To/Cc fields were simply completely wrong, but while
some fields failed to parse, the parser did not crash and
parsed the other fields of the email correctly.
2023-06-19 14:10:02 +00:00
Run it on your machine:
```bash
cargo test -- --ignored --nocapture enron500k
```
2023-06-19 14:09:11 +00:00
Planned: jpbush, my inbox, etc.
### Fuzzing (expected)
### Across reference IMAP servers (dovevot, cyrus) (expected)
2023-06-17 09:43:54 +00:00
## Development status
Early development. Not ready.
Do not use it in production or any software at all.
2023-06-19 09:22:51 +00:00
Todo:
2023-06-19 14:09:11 +00:00
- [X] test over the enron dataset
2023-06-22 13:05:52 +00:00
- [X] convert to multipass parser
2023-06-22 13:11:11 +00:00
- [X] fix warnings, put examples, refactor the code
2023-06-19 09:22:51 +00:00
- [ ] implement mime part 3 (encoded headers)
- [ ] implement mime part 1 (new headers)
- [ ] review part 2 (media types) and part 4 (registration procedure) but might be out of scope
- [ ] implement some targeted testing as part of mime part 5
- [ ] implement fuzzing through cargo fuzz
- [ ] test over other datasets (jpbush, ml, my inbox)
- [ ] backport to aerogramme
2023-07-03 16:25:51 +00:00
## Targeted RFC and IANA references
2023-06-13 07:03:51 +00:00
2023-06-19 10:15:05 +00:00
| 🚩 | # | Name |
|----|---|------|
| 🟩 |822 | ARPA INTERNET TEXT MESSAGES|
2023-06-19 10:15:45 +00:00
| 🟩 |2822 | Internet Message Format (2001) |
2023-06-19 10:15:05 +00:00
| 🟩 |5322 | Internet Message Format (2008) |
| 🔴 |2045 | ↳ Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies |
| 🔴 |2046 | ↳ Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types |
2023-07-14 08:44:03 +00:00
| 🟩 |2047 | ↳ MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text |
2023-06-19 10:15:05 +00:00
| 🔴 |2048 | ↳ Multipurpose Internet Mail Extensions (MIME) Part Four: Registration Procedures |
| 🔴 |2049 | ↳ Multipurpose Internet Mail Extensions (MIME) Part Five: Conformance Criteria and Examples |
2023-07-04 13:50:45 +00:00
| | | **Headers extensions** |
| 🔴 |2183 | ↳ Communicating Presentation Information in Internet Messages: The Content-Disposition Header Field |
| 🟩 |6532 | ↳ Internationalized Email Headers |
| 🔴 |9228 | ↳ Delivered-To Email Header Field |
| | | **MIME extensions** |
| 🔴 |1847 | ↳ Security Multiparts for MIME: Multipart/Signed and Multipart/Encrypted |
| 🔴 |2387 | ↳ The MIME Multipart/Related Content-type |
| 🔴 |3462 | ↳ The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages |
| 🔴 |3798 | ↳ Message Disposition Notification |
| 🔴 |6838 | ↳ Media Type Specifications and Registration Procedures |
2023-06-17 09:43:54 +00:00
2023-07-03 16:25:51 +00:00
IANA references :
- (tbd) MIME subtypes
- [IANA character sets](https://www.iana.org/assignments/character-sets/character-sets.xhtml)
2023-06-17 09:43:54 +00:00
## Alternatives
`stalwartlab/mail_parser`