Re:Large posts getting duplicated
As a continuation of my SmartList problem with duplicate resent posts with large bodies (around 4.5K) on one of my lists. I have included the headers from one such article. The first is how the message was received by my box, before any processing by SmartList. The next two are the duplicated re-sends by Smartlist. Notice the message refernces are the same and the archive number is the same, but the outgoing ID's are different. This is consistantly happening (have had three in the last two days). I am unable to duplicate it with a test list though (even copied all rc and processing files to the test list), but the test list only has one subscriber where as the list in question has over 300. This is with procmail and SmartList v 3.15.1 and I am using the default choplist to distribute the messages. Here is the sendmail setup from rc.init: minnames = 32 # minimum number of names per call mindiffnames = 8 # minimum for maxnames-minnames maxnames = 64 # maximum number of names per call maxsplits = 0 # maximum number of parts to split dist in maxsize = 200000 # maximal disk space to be taken up per mail maxconcur = 20 # maximum number of concurrent sendmail calls alt_sendmail="\ choplist $minnames $mindiffnames $maxnames $maxsplits $maxsize $maxconcur dist" minnames mindiffnames maxnames maxsplits maxsize maxconcur #alt_sendmail # uncomment if you'd prefer sendmail # to handle the $listdist expansion sendmailOPT="-oem -odb -oi -om -f$listreq" sendmailOPTp=-oep sendmailOPTq=-odq sendmailQflush="$SENDMAIL -q" I am at a total loss, any idea where to look next? Would a verbose log indicate if choplist was at fault? Thanks for any help... Doug ----- as received by my box--------
From user@telocity.com Sun Nov 11 22:21:06 2001 Received: from fxst.harley.com (dsl-64-192-236-53.telocity.com [64.192.236.53]) by zoostation.cns.ohiou.edu (8.11.2/8.11.2) with ESMTP id fAC3L5s25514 for <xl-list@sportster.org>; Sun, 11 Nov 2001 22:21:05 -0500 Received: from telocity.com (fxst.harley.com [192.168.1.250]) by fxst.harley.com (SGI-8.9.3/8.9.3) with ESMTP id WAA39381 for <xl-list@sportster.org>; Sun, 11 Nov 2001 22:13:52 -0500 (EST) Message-ID: <3BEF3E6E.6070807@telocity.com> Date: Sun, 11 Nov 2001 22:13:50 -0500 From: "user" <user@telocity.com> User-Agent: Mozilla/5.0 (X11; U; IRIX64 IP28; en-US; rv:0.9.5) Gecko/20011022 X-Accept-Language: en-us MIME-Version: 1.0 To: xl-list@sportster.org Subject: and web site References: <200111102100.fAAL01U09447@zoostation.cns.ohiou.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit
------ first resent message ----- Return-Path: <xl-list-request@sportster.org> Received: (from lists@localhost) by zoostation.cns.ohiou.edu (8.11.2/8.11.2) id fAC3LAn25596; Sun, 11 Nov 2001 22:21:10 -0500 Resent-Date: Sun, 11 Nov 2001 22:21:06 -0500 Message-ID: <3BEF3E6E.6070807@telocity.com> Date: Sun, 11 Nov 2001 22:13:50 -0500 From: "user" <user@telocity.com> User-Agent: Mozilla/5.0 (X11; U; IRIX64 IP28; en-US; rv:0.9.5) Gecko/20011022 X-Accept-Language: en-us MIME-Version: 1.0 To: xl-list@sportster.org Subject: and web site References: <200111102100.fAAL01U09447@zoostation.cns.ohiou.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Resent-Message-ID: <8elbdD.A.5OG.iA077@zoostation.cns.ohiou.edu> Resent-From: xl-list@sportster.org X-Mailing-List: <xl-list@sportster.org> archive/latest/103311 X-Loop: xl-list@sportster.org List-Post: <mailto:xl-list@sportster.org> List-Help: <mailto:xl-list-request@sportster.org?subject=help> List-Subscribe: <mailto:xl-list-request@sportster.org?subject=subscribe> List-Unsubscribe: <mailto:xl-list-request@sportster.org?subject=unsubscribe> Precedence: list Resent-Sender: xl-list-request@sportster.org ------ second resent message ----- Return-Path: <xl-list-request@sportster.org> Received: (from lists@localhost) by zoostation.cns.ohiou.edu (8.11.2/8.11.2) id fAC3L7g25542; Sun, 11 Nov 2001 22:21:07 -0500 Resent-Date: Sun, 11 Nov 2001 22:21:07 -0500 Message-ID: <3BEF3E6E.6070807@telocity.com> Date: Sun, 11 Nov 2001 22:13:50 -0500 From: "user" <user@telocity.com> User-Agent: Mozilla/5.0 (X11; U; IRIX64 IP28; en-US; rv:0.9.5) Gecko/20011022 X-Accept-Language: en-us MIME-Version: 1.0 To: xl-list@sportster.org Subject: and web site References: <200111102100.fAAL01U09447@zoostation.cns.ohiou.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Resent-Message-ID: <8elbdD.A.5OG.iA077@zoostation.cns.ohiou.edu> Resent-From: xl-list@sportster.org X-Mailing-List: <xl-list@sportster.org> archive/latest/103311 X-Loop: xl-list@sportster.org List-Post: <mailto:xl-list@sportster.org> List-Help: <mailto:xl-list-request@sportster.org?subject=help> List-Subscribe: <mailto:xl-list-request@sportster.org?subject=subscribe> List-Unsubscribe: <mailto:xl-list-request@sportster.org?subject=unsubscribe> Precedence: list Resent-Sender: xl-list-request@sportster.org
Did Smartlist put these headers in, or Doug, did you add them to the original post? My mailing list messages don't have them, and I'd like them too. Anne Doug Mansfield wrote:
------ first resent message ----- . . . List-Post: <mailto:xl-list@sportster.org> List-Help: <mailto:xl-list-request@sportster.org?subject=help> List-Subscribe: <mailto:xl-list-request@sportster.org?subject=subscribe> List-Unsubscribe: <mailto:xl-list-request@sportster.org?subject=unsubscribe>
--On Tuesday, November 13, 2001 10:10 AM -0500 Anne Judge <anne.judge@alum.mit.edu> wrote:
Did Smartlist put these headers in, or Doug, did you add them to the original post? My mailing list messages don't have them, and I'd like them too.
Doug Mansfield wrote:
List-Post: <mailto:xl-list@sportster.org> List-Help: <mailto:xl-list-request@sportster.org?subject=help> List-Subscribe: <mailto:xl-list-request@sportster.org?subject=subscribe> List-Unsubscribe: <mailto:xl-list-request@sportster.org?subject=unsubscribe>
Anne, yes, SmartList places those headers there. That was an addition to 3.15? -- Doug Mansfield
On Tue, 13 Nov 2001, Doug Mansfield wrote:
I am at a total loss, any idea where to look next? Would a verbose log indicate if choplist was at fault?
A verbose log would at least tell you whether choplist is being invoked multiple times, and exactly how it is being invoked. I looked back over your previous posts on this topic, and I can't tell whether everyone on the list gets two copies when this happens, or whether only a subset of the list gets two copies. The latter would almost certainly point to a problem with choplist. That it seems to be size-related could also indicate some kind of race condition. You said it only happens sometimes with messages >4.5K; does it happen more often as the message size increases? Verbose logging might suppress the problem entirely in that case, or it might make it worse. It's mildly interesting that you received first the copy with the later (by 1 sec) resent-date, but that could have happened for any number of mail transport reasons and may be a red herring. Good luck ...
Just to close this subject, I finally figured out what was happening to cause duplicated posts of large messages. I use a hacked up version of the Perl LWGate GCI to publish my list to a web page. One of these pages has a "Most Recent" listing of articles. I use rc.local.s20 to add messages to this mailbox and then run a script to delete the oldest message, keeping the count at 100. (you can see it at http://www.sportster.org/cgi-bin/lwgate.cgi/XL-LIST/archives%5Ccurrent/curr ent/subject/ if you are interested). I have to create a copy of the message, since I actually deliver it to a mailbox. For some reason, on larger messages (>4.5KB), my script was retruning an error exitcode. Therefore, procmail though the message had not been delivered and there was then two copies of it which continued through the rest of the SmartList processing. I had to add and "i" to the recipe so that the exitcode was ignored. -- Doug Mansfield
participants (3)
-
Anne Judge -
Bart Schaefer -
Doug Mansfield