[New-bugs-announce] [issue34138] RFC 6855 issue

[Sam Varshavchik]

New submission from Sam Varshavchik <sam.varshavchik at gmail.com>:

Greetings. I am in the process of implementing RFC 6855 in Courier-IMAP. A Google search for IMAP clients that implement RFC 6855 led me to https://bugs.python.org/issue21800 and looking over the code that was added to imaplib, to support RFC 6855, a few things stood out. I checked, and the changes introduces in 21800 still appear to be unchanged in https://github.com/python/cpython/blob/master/Lib/imaplib.py 

Issue 21800 modified sub append(), that implements the IMAP APPEND command, thusly:

-        self.literal = MapCRLF.sub(CRLF, message)
+        literal = MapCRLF.sub(CRLF, message)
+        if self.utf8_enabled:
+            literal = b'UTF8 (' + literal + b')'
+        self.literal = literal

"literal" here appears to be the contents of the message with CRLF line ending. But section 4 of https://tools.ietf.org/html/rfc6855.html states:

  The ABNF for the "APPEND" data extension and "CATENATE" extension 
  follows:

        utf8-literal   = "UTF8" SP "(" literal8 ")"

        literal8       = <Defined in RFC 4466>

        append-data    =/ utf8-literal

        cat-part       =/ utf8-literal

As indicated above, "literal8" comes from RFC 4466, which also defines "append-data". RFC 4466 additionally states:

   In addition, the non-terminal "literal8" defined in [BINARY] got
   extended to allow for non-synchronizing literals if both [BINARY] and
   [LITERAL+] extensions are supported by the server.

I'll come back to this revealing paragraph in a moment, but, as stated, "literal8" actually comes from [BINARY] which is RFC 3516, which specifies the following:

   append         =/  "APPEND" SP mailbox [SP flag-list]
                      [SP date-time] SP literal8

   fetch-att      =/  "BINARY" [".PEEK"] section-binary [partial]
                      / "BINARY.SIZE" section-binary

   literal8       =   "~{" number "}" CRLF *OCTET
                      ; <number> represents the number of OCTETs
                      ; in the response string.

An exhaustive search of imaplib.py seems to indicate that this pesky tilde is in hiding. And the wrong thing seems to be quoted as the actual literal. Anyway, back to the RFCs: combine all of the above together, spin it in a blender, and you get the following result:

Supposing that the message being appended consists of a single header line "Subject: test", and a blank line, a sample command of what actually goes out the wire (based on the above, and other parts of these, and related RFCs):

APPEND INBOX NIL NIL UTF8 (~{17}<CR><LF>Subject: test<CR><LF><CR><LF>)<CR><LF>

I haven't tested imaplib against Courier-IMAP in this respect, but it doesn't seem like this is going to be results.

But wait, there's more!

"literal8" is a synchronizing literal, like "literal" from RFC 3501, which specifies:

                                              ...In the case of
   literals transmitted from client to server, the client MUST wait
   to receive a command continuation request (described later in
   this document) before sending the octet data (and the remainder
   of the command).

The LITERAL+ IMAP extension, that was mentioned in the excerpt from RFC 4466 that I cited above, introduced non-synchronizing literals:

   The protocol receiver of an IMAP4 server must check the end of every
   received line for an open brace ('{') followed by an octet count, a
   plus ('+'), and a close brace ('}') immediately preceeding the CRLF.
   If it finds this sequence, it is the octet count of a non-
   synchronizing literal and the server MUST treat the specified number
   of following octets and the following line as part of the same
   command.

Otherwise, after the closing brace and the <CR><LF> the IMAP client must wait for the continuation response from the server.

So, to summarize:

1) RFC 4466, combined with RFC 6855 an IMAP UTF-8 client talking to an IMAP UTF-8 server can send the following, on the wire, if the server supports LITERAL+:

APPEND INBOX NIL NIL UTF8 (~{17+}<CR><LF>Subject: test<CR><LF><CR><LF>)<CR><LF>

2) But, if the server did not advertise LITERAL+, the IMAP client is required to send only:

APPEND INBOX NIL NIL UTF8 (~{17}<CR><LF>

Then wait for the continuation response from the server, then send the rest of the command.

IMAP specifications have been painful to read, for the 20+ years I've been reading them. Historically there's been a lot of interoperability problems between IMAP clients and servers. I lay the blame squarely on the horrible specs, but that's off-topic. Suffice to say, nothing of that sort has been observed for POP3 and SMTP, and I think there's a very good reason for that.

----------
components: email
messages: 321819
nosy: Sam Varshavchik, barry, r.david.murray
priority: normal
severity: normal
status: open
title: RFC 6855 issue
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34138>
_______________________________________