gmail and the infamous "530 5.7.0 Must issue a STARTTLS command first"

[Paul Bibbings]

There is a lot of discussion from many sources on the net about setting
up gmail smtp in gnus and the "530 5.7.0 Must issue a STARTTLS command
first" error. What I haven't been able to find is a solution that works
(in Emacs 22.3.1 with gnus v5.11). Has anyone been able to find a
workaround for this that uses either starttls or gnutls-cli?

I've done some debugging and the problem is, as the error messages
suggest, that gnus is trying to send an AUTH command /before/ it has
issued a STARTTLS. The error can be reproduced at the command line with
something like:

/usr/bin $gnutls-cli --crlf -s -p 587 smtp.gmail.com
Resolving 'smtp.gmail.com'...
Connecting to '74.125.77.109:587'...

- Simple Client Mode:

220 mx.google.com ESMTP 14sm14646818ewy.15
ehlo test
250-mx.google.com at your service, [86.15.147.8]
250-SIZE 35651584
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250 PIPELINING
auth plain AHBhdWwuYmliYmluZ3NAZ21haWwuY29tAHdoaXRlc294
530 5.7.0 Must issue a STARTTLS command first. 14sm14646818ewy.15

I've traced the beginning of the problem down to the code around line
719 in mail/smtpmail.el:

(if (and do-starttls
       (smtpmail-find-credentials smtpmail-starttls-credentials host port)
       (member 'starttls supported-extensions)                  // line 719
       (numberp (process-id process)))
    (progn
       (smtpmail-send-command process (format "STARTTLS"))
    (if (or (null (car (setq response-code (smtpmail-read-response process))))
       (not (integerp (car response-code)))
              (>= (car response-code) 400))
                          (throw 'done nil))
                      (starttls-negotiate process)
                      (setq do-starttls nil))
                  (setq do-ehlo nil))))

Here, (member !startls supported-extensions) is failing, not because
gmail doesn't require a STARTTLS here, but because, apparently, one of
the other extensions "isn't supported." Consequently, the sending of
STARTTLS is skipped and the whole thing fails. I haven't gone into that
much detail to remember which it is that is failing, and I have to admit
that I don't know Lisp at all. Does anyone here know enough to be able
to dig a little deeper, and perhaps offer a solution?

Perhaps better still, does anyone have an alternative or add-on
implementation that uses openssl directly, rather than through starttls?
openssl has a very different command line invocation to both starttls
and gnutls-cli, but if it could be hooked in in their place, it handles
the STARTLS directly, so the above code wouldn't fail for not being able
to issue it in this case. Call it a challenge, but I would like to get
gmail smtp working with one of these (starttls or gnutls-cli), even
though I am aware and have tested many other alternatives - mostly
without success, or at least with issues of their own.

Regards

Paul Bibbings