From mark at msapiro.net  Tue Aug  7 12:37:53 2018
From: mark at msapiro.net (Mark Sapiro)
Date: Tue, 07 Aug 2018 16:37:53 -0000
Subject: [Bug 1785854] [NEW] TypeError from confirm CGI.
Message-ID: <153365987313.17868.5140007562982999428.malonedeb@soybean.canonical.com>

Public bug reported:

We have seen some massive, robotic subscribes and confirmations. In rare
cases a TypeError can be thrown. This occurs because in processing, the
CGI script retrieves the pending entry more than once. If the initial
retrieval returns None, an appropriate message is returned, but if two
clients confirm the same token at the same time, both initial retrievals
may succeed, but then subsequent processing locks the list and retrieves
the data again. The first client to get there processes the confirmation
which removes the token. Then when the list is unlocked, the second
client gets None for the pending data and throws TypeError when
retrieving things from None.

** Affects: mailman
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1785854

Title:
  TypeError from confirm CGI.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1785854/+subscriptions

From 1785854 at bugs.launchpad.net  Tue Aug  7 22:52:59 2018
From: 1785854 at bugs.launchpad.net (Launchpad Bug Tracker)
Date: Wed, 08 Aug 2018 02:52:59 -0000
Subject: [Bug 1785854] Re: TypeError from confirm CGI.
References: <153365987313.17868.5140007562982999428.malonedeb@soybean.canonical.com>
Message-ID: <153369678373.28671.6724006925832738218.launchpad@ackee.canonical.com>

** Branch linked: lp:mailman/2.1

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1785854

Title:
  TypeError from confirm CGI.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1785854/+subscriptions

From mark at msapiro.net  Tue Aug  7 22:53:35 2018
From: mark at msapiro.net (Mark Sapiro)
Date: Wed, 08 Aug 2018 02:53:35 -0000
Subject: [Bug 1785854] Re: TypeError from confirm CGI.
References: <153365987313.17868.5140007562982999428.malonedeb@soybean.canonical.com>
Message-ID: <153369681596.28538.4482844313772124149.launchpad@chaenomeles.canonical.com>

** Changed in: mailman
   Importance: Undecided => Low

** Changed in: mailman
       Status: New => Fix Committed

** Changed in: mailman
    Milestone: None => 2.1.30

** Changed in: mailman
     Assignee: (unassigned) => Mark Sapiro (msapiro)

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1785854

Title:
  TypeError from confirm CGI.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1785854/+subscriptions

From mark at msapiro.net  Tue Aug 14 19:08:07 2018
From: mark at msapiro.net (Mark Sapiro)
Date: Tue, 14 Aug 2018 23:08:07 -0000
Subject: [Bug 1787054] [NEW] Attempts to get organizational domain data fail
Message-ID: <153428808802.28081.10052819751065626008.malonedeb@chaenomeles.canonical.com>

Public bug reported:

As part of DMARC mitigation processing, Mailman looks up the DMARC
policy for the From: domain. If it doesn't find a DMARC policy it
attempts to look up the policy for the "organizational domain"
corresponding to the From: domain if the organizational domain is
different. To determine the organizational domain it uses information
from the list at https://publicsuffix.org/list/public_suffix_list.dat.

Recent changes at publicsuffix.org are causing Mailman's attempt to
retrieve the list to fail with

urllib2.URLError: <urlopen error [Errno 1] _ssl.c:510:
error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake
failure>

This failure has been observed with Python 2.7.6 but not with Python
2.7.12. There are changes 2.7.9 which affect the underlying ssl module,
and I think retrieval of this URL via urllib2, urllib or the Python
requests module will all fail with Python < 2.7.9.

The effect of this issue other than writing an error log entry for every
failed retrieval is that in some cases, the organizational domain will
not be properly found. If the TLD is .com, .net, .gov, .edu, etc. There
will be no issue, but if for example the From: domain is
some.sub.domain.school.k12.ca.us and that domain doesn't publish a DMARC
policy, we should look up the policy for the school.k12.ca.us
organizational domain, but instead we will look up ca.us.

This will probably be more of an issue with non-US lists than with US
lists, and it is not known how significant the issue is.

At present, the only known workaround is to upgrade the underlying
Python.

** Affects: mailman
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1787054

Title:
  Attempts to get organizational domain data fail

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1787054/+subscriptions

From mark at msapiro.net  Tue Aug 14 23:55:45 2018
From: mark at msapiro.net (Mark Sapiro)
Date: Wed, 15 Aug 2018 03:55:45 -0000
Subject: [Bug 377840] Re: enhancement: allow fullname to be required
References: <20090518043236.25827.58629.malonedeb@potassium.ubuntu.com>
Message-ID: <153430534647.27869.5887304924868108870.launchpad@chaenomeles.canonical.com>

** Changed in: mailman
   Importance: Undecided => Wishlist

** Changed in: mailman
       Status: New => Won't Fix

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/377840

Title:
  enhancement: allow fullname to be required

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/377840/+subscriptions

From 1787690 at bugs.launchpad.net  Fri Aug 17 22:46:23 2018
From: 1787690 at bugs.launchpad.net (Jeremy Gregson)
Date: Sat, 18 Aug 2018 02:46:23 -0000
Subject: [Bug 1787690] [NEW] Mailman rejects multipart content from
 Gmail/android
Message-ID: <153456038309.5317.654473709447762611.malonedeb@chaenomeles.canonical.com>

Public bug reported:

With a mailman list set to apply content filtering, and with the default settings to pass_mime_types of text/html, multipart/mixed and multipart/alternative, mail sent from anyone using the Gmail app on an android device is bounced.  The same user can send the same message from the native Mail client on an Android device, or from an Apple device.
The problem seems to be that the Gmail app includes a header in the form 
Content-Type: multipart/mixed; boundary="===============8130305584729002058=="
...with no carriage return after the semi-colon.

The only way I've been able to get a list to accept mail from that
client app (which is very popular) is to remove pass_mime_types
altogether.

** Affects: mailman
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1787690

Title:
  Mailman rejects multipart content from Gmail/android

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1787690/+subscriptions

From j+launchpad.net at jkt.im  Mon Aug 20 16:11:55 2018
From: j+launchpad.net at jkt.im (Jonathan Tullett)
Date: Mon, 20 Aug 2018 20:11:55 -0000
Subject: [Bug 1778363] Re: exception (utf8 codec) in pipermail
References: <152979038533.27033.5958097208292576716.malonedeb@gac.canonical.com>
Message-ID: <153479591592.3943.13537868396499544890.malone@gac.canonical.com>

I am experiencing the same issue migrating from 2.1.16 (on Ubuntu 14.04)
to 2.1.26 (18.04).

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1778363

Title:
  exception (utf8 codec) in pipermail

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1778363/+subscriptions

From j+launchpad.net at jkt.im  Mon Aug 20 16:16:03 2018
From: j+launchpad.net at jkt.im (Jonathan Tullett)
Date: Mon, 20 Aug 2018 20:16:03 -0000
Subject: [Bug 1778363] Re: exception (utf8 codec) in pipermail
References: <152979038533.27033.5958097208292576716.malonedeb@gac.canonical.com>
Message-ID: <153479616321.17397.3790436783848840738.malone@wampee.canonical.com>

As this was a blocker for the migration, I rsynced the /usr/lib/mailman
directory from the 2.1.16 installation onto the new machine and the
bin/arch/wipe worked again.

Nothing I looked at on the author or decoded author looked out of place.

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1778363

Title:
  exception (utf8 codec) in pipermail

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1778363/+subscriptions

From mark at msapiro.net  Sun Aug 26 20:16:05 2018
From: mark at msapiro.net (Mark Sapiro)
Date: Mon, 27 Aug 2018 00:16:05 -0000
Subject: [Bug 1787690] Re: Mailman rejects multipart content from Gmail/android
References: <153456038309.5317.654473709447762611.malonedeb@chaenomeles.canonical.com>
Message-ID: <153532896615.5870.14132525474575637085.malone@gac.canonical.com>

Possibly the message does not have a text/html part. Have you tried the
following in pass_mime_types:

multipart
text/html
text/plain

The fact that the Content-Type: header is not folded with a CR and
whitespace following the semicolon is not an issue.

If adding text/plain to pass_mime_types doesn't fix this, please post a
complete raw message. You can elide the actual MIME part contents, but
we need to see all the Content-Type: headers and part boundaries.

** Changed in: mailman
       Status: New => Incomplete

** Changed in: mailman
     Assignee: (unassigned) => Mark Sapiro (msapiro)

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1787690

Title:
  Mailman rejects multipart content from Gmail/android

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1787690/+subscriptions

From 1787690 at bugs.launchpad.net  Mon Aug 27 00:24:58 2018
From: 1787690 at bugs.launchpad.net (Jeremy Gregson)
Date: Mon, 27 Aug 2018 04:24:58 -0000
Subject: [Bug 1787690] Re: Mailman rejects multipart content from Gmail/android
References: <153456038309.5317.654473709447762611.malonedeb@chaenomeles.canonical.com>
Message-ID: <153534389860.26014.7314298013821470758.malone@wampee.canonical.com>

Hi Mark -

First of all, thanks very much for getting back to me.  Secondly,
there's something I realised I didn't mention in the bug report, which
may count as a 'free pass' for you to handball the problem back - I'm
running Mailman 2.1.20 on Ubuntu 16, not Mailman 3.  I've put off
upgrading because I understand the move to v3 is a major architectural
change, and I have a lot of legacy lists and users to worry about.

Anyway - in the course of experimenting with the content filter
settings, I tried a number of options.  Finding a tip online from
someone else who identified different subtypes of multipart, I tried
adding multipart (without any subtype) to pass_mime_types.  When that
didn't work, I tried multipart* to see if there was such a thing as a
wildcard (I think I tried multipart% and multipart/* as well), but I
didn't try adding text/plain.

So, I've just done that - no dice.  When I send to the list from the
Gmail app on an Android phone with the pass_mime_types field completely
clear, mail gets to the list.  With the mime type set to your three
options, nothing gets through and nothing appears in the vette log -
which is consistent with the message having been so gutted by the
filtering process that there was nothing left to pass or reject.

I've attached an example of a message that gets through to the list with
pass_mime_types cleared, and is lost with the filtering turned on.  In
case the act of attaching it loses anything, the headers as received at
the list were:

Return-Path: <selenetest-bounces at lochac.sca.org>
X-Original-To: nico at jeremygregson.com
Delivered-To: nico at jeremygregson.com
Received: by mail.jeremygregson.com (Postfix, from userid 500)
	id 52C2F4021C; Mon, 27 Aug 2018 13:47:18 +1000 (AEST)
Received: from mail.lochac.sca.org (vmx20482.hosting24.com.au [163.53.249.53])
	by mail.jeremygregson.com (Postfix) with ESMTP id 36B0440215
	for <nico at jeremygregson.com>; Mon, 27 Aug 2018 13:47:18 +1000 (AEST)
Received: by mail.lochac.sca.org (Postfix, from userid 500)
	id E453E481FD5; Mon, 27 Aug 2018 13:48:12 +1000 (AEST)
X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on
	vmx20482.hosting24.com.au
X-Spam-Level: 
X-Spam-Status: No, score=-0.0 required=3.2 tests=BAYES_00,HTML_MESSAGE,
	MAILING_LIST_MULTI,MISSING_MIMEOLE,RDNS_DYNAMIC,SPF_HELO_PASS,URIBL_BLOCKED
	autolearn=no autolearn_force=no version=3.4.1
Received: from mail.lochac.sca.org (localhost [IPv6:::1])
	by mail.lochac.sca.org (Postfix) with ESMTP id 8B2F4481F89
	for <nico at jeremygregson.com>; Mon, 27 Aug 2018 13:48:12 +1000 (AEST)
X-Original-To: selenetest at lochac.sca.org
Delivered-To: selenetest at lochac.sca.org
Received: by mail.lochac.sca.org (Postfix, from userid 500)  id 9BA8F481FD5; Mon, 27 Aug 2018 13:48:10 +1000 (AEST)
Received: from mail.jeremygregson.com (203-217-61-26.perm.iinet.net.au
 [203.217.61.26])
 by mail.lochac.sca.org (Postfix) with ESMTP id 36E9A481F89  for <selenetest at lochac.sca.org>; Mon, 27 Aug 2018 13:48:10 +1000 (AEST)
Received: by mail.jeremygregson.com (Postfix, from userid 500)  id E964A4021C; Mon, 27 Aug 2018 13:47:13 +1000 (AEST)
Received: from [10.183.142.121] (unknown [1.144.228.246])  by mail.jeremygregson.com (Postfix) with ESMTPSA id 87B3940215  for <selenetest at lochac.sca.org>; Mon, 27 Aug 2018 13:47:13 +1000 (AEST)
Date: Mon, 27 Aug 2018 13:48:04 +1000
Message-ID: <40d9ef29-01e7-4e04-bc82-8abe1884594c at email.android.com>
X-Android-Message-ID: <40d9ef29-01e7-4e04-bc82-8abe1884594c at email.android.com>
To: selenetest at lochac.sca.org
Importance: Normal
X-Priority: 3
X-MSMail-Priority: Normal
MIME-Version: 1.0
Subject: [Selenetest] Test subject
X-BeenThere: selenetest at lochac.sca.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: <selenetest.lochac.sca.org>
List-Unsubscribe: <https://lochac.sca.org/mailman/options/selenetest>,
 <mailto:selenetest-request at lochac.sca.org?subject=unsubscribe>
List-Archive: <http://lochac.sca.org/pipermail/selenetest/>
List-Post: <mailto:selenetest at lochac.sca.org>
List-Help: <mailto:selenetest-request at lochac.sca.org?subject=help>
List-Subscribe: <https://lochac.sca.org/mailman/listinfo/selenetest>,
 <mailto:selenetest-request at lochac.sca.org?subject=subscribe>
From: Nico via Selenetest <selenetest at lochac.sca.org>
Reply-To: Nico <nico at jeremygregson.com>
Content-Type: multipart/mixed; boundary="===============8988888446188673456=="
Errors-To: selenetest-bounces at lochac.sca.org
Sender: "Selenetest" <selenetest-bounces at lochac.sca.org>

Any further suggestions will be appreciated.
Cheers,
Jeremy

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1787690

Title:
  Mailman rejects multipart content from Gmail/android

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1787690/+subscriptions

From 1787690 at bugs.launchpad.net  Mon Aug 27 00:26:27 2018
From: 1787690 at bugs.launchpad.net (Jeremy Gregson)
Date: Mon, 27 Aug 2018 04:26:27 -0000
Subject: [Bug 1787690] Re: Mailman rejects multipart content from Gmail/android
References: <153456038309.5317.654473709447762611.malonedeb@chaenomeles.canonical.com>
Message-ID: <153534398718.12235.17311785703797635372.malone@chaenomeles.canonical.com>

** Attachment added: "Message blocked/reduced to nothing by content filter"
   https://bugs.launchpad.net/mailman/+bug/1787690/+attachment/5181125/+files/Selenetest%20Test%20subject.msg

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1787690

Title:
  Mailman rejects multipart content from Gmail/android

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1787690/+subscriptions

From mark at msapiro.net  Tue Aug 28 15:17:10 2018
From: mark at msapiro.net (Mark Sapiro)
Date: Tue, 28 Aug 2018 19:17:10 -0000
Subject: [Bug 1778363] Re: exception (utf8 codec) in pipermail
References: <152979038533.27033.5958097208292576716.malonedeb@gac.canonical.com>
Message-ID: <153548383016.24235.2243604591928395954.malone@wampee.canonical.com>

Are you saying that given the same input mbox file that 'bin/arch
--wipe' throws the UnicodeDecodeError with Mailman 2.1.26 but not with
2.1.16? If so, that's strange as there are no changes between 2.1.16 and
2.1.26 in pipermail.py in the area of 'fixAuthor', and there don't seem
to be any Debian patches in this area either.

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1778363

Title:
  exception (utf8 codec) in pipermail

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1778363/+subscriptions

From mark at msapiro.net  Tue Aug 28 19:15:26 2018
From: mark at msapiro.net (Mark Sapiro)
Date: Tue, 28 Aug 2018 23:15:26 -0000
Subject: [Bug 1787690] Re: Mailman rejects multipart content from Gmail/android
References: <153456038309.5317.654473709447762611.malonedeb@chaenomeles.canonical.com>
Message-ID: <153549812695.20853.7723823612181263822.malone@chaenomeles.canonical.com>

This is the correct tracker for Mailman 2.1 issues.

The headers at
https://bugs.launchpad.net/mailman/+bug/1787690/comments/2 are only the
message headers and only say that the message's Conternt-Type: is
multipart/mixed. If I send a message from Gmail on my android phone it
has the following MIME structure.

multipart/alternative
    text/plain
        the plain text alternative
    text/html
        the html alternative

I suspect the message you are looking at from the list has structure

multipart/mixed
    multipart/alternative
        text/plain
            the plain text alternative
        text/html
            the html alternative
    text/plain
        the list footer

I can download the
https://bugs.launchpad.net/mailman/+bug/1787690/+attachment/5181125/+files/Selenetest%20Test%20subject.msg
attachment, but I can't open it with anything that renders it properly.
'file' tells me "Composite Document File V2 Document, Cannot read
section info".

Note that you can't use wildcards in pass_mime_types, but you can
specify a main type without a subtype. I.e., this

multipart
text

in pass_mime_types will accept any text/* elemental parts within any
multipart/* parts.

Also note that Content filtering -> filter_action controls what is done
with a message from which content filtering removes everything and there
should always be a Message discarded or Message rejected message in the
vette log in any case.

Is there anything in Mailman's error log?

If you have problem posting a raw message, just send a message from your
android gmail app with Subject: Bug 1787690 to mark at msapiro.net.

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1787690

Title:
  Mailman rejects multipart content from Gmail/android

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1787690/+subscriptions