Kitz Forum

Internet => Web Hosting & Web Design => Topic started by: d2d4j on February 26, 2019, 07:53:39 PM

Title: SA issue on email
Post by: d2d4j on February 26, 2019, 07:53:39 PM
Hi

I hope I have posted in correct area. Please move as needed

I am just letting those who use our platforms that we have received 2 reports of an issue with email

Support have identified this to SA, which was updated this morning and after just speaking with them, I said I would look when I am back later tonight (still have a couple of hours driving a head of me) or tomorrow morning

In the meantime, we have decided to turn off SA to allow email flows through

The issue is the new SA appears to be scoring very high, so most email was not been accepted.

This decision will mean some spam been allowed through (note Antivirus scanning is unaffected)

Please accept my apologies for this brief issue

Many thanks

John
Title: Re: SA issue on email
Post by: Ronski on February 26, 2019, 08:00:31 PM
Hi John,

I'm guessing SA is Spam Assasin. I've not noticed any issues, perhaps a facebook email in my spam but one could argue that belongs there.

Thanks for letting us know.
Title: Re: SA issue on email
Post by: atkinsong on February 26, 2019, 10:01:38 PM
Hi John. As you know, I was responsible for one of the problem reports this afternoon, so thank you very much for this update.
Title: Re: SA issue on email
Post by: vic0239 on February 26, 2019, 10:07:36 PM
... and I the second. Still not receiving some email and concerned that earlier rejections are lost!
Title: Re: SA issue on email
Post by: jelv on February 26, 2019, 11:27:12 PM
I've just (@23:00:07) received an email marked as spam with a score of 25.4. The email was from my NAS box to notify me of the availability of an App update. It's an email I receive fairly regularly and has never been marked as spam before. The headers show it has gone through Spam Assassin 3.4.2 (2018-09-13).

John, I'll PM you the full headers.
Title: Re: SA issue on email
Post by: jelv on February 26, 2019, 11:29:58 PM
Just spotted this in the headers in the X-Spam-Report section:

Code: [Select]
24 AWL AWL: Adjusted score from AWL reputation of From: address
Could somebody explain please?
Title: Re: SA issue on email
Post by: d2d4j on February 26, 2019, 11:43:40 PM
Hi

Sorry just arrived back but tired

@jelv - many thanks and if you look at header just received, you will see the figures do not add up

The earlier intervention appears to be working and I will have a think overnight and look at it tommorow refreshed sorry if thatís alright

@vic, I will pull of the log and let you know

It is a little strange though, as all appears to work lovely but the scores are way to high

I have also just emailed IW but I do know they are behind uk time, so will see tommorow

Once again I apologise for any inconvenience this has caused and we have acted very quickly from the 2 reports received this afternoon

Once fully resolved I will update the thread.

Many thanks

John
Title: Re: SA issue on email
Post by: jelv on February 26, 2019, 11:45:54 PM
Just seen another set of stupid headers for a Facebook notification email:

Code: [Select]
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on haveworx.co.uk
X-Spam-Flag: YES
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=305.9 required=7.0 tests=AWL,DKIMWL_WL_HIGH,
DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HTML_FONT_LOW_CONTRAST,
HTML_MESSAGE,RDNS_NONE,SPF_HELO_PASS,UNPARSEABLE_RELAY,
USER_IN_DEF_DKIM_WL autolearn=disabled version=3.4.2
X-Spam-Report:
* -0.0 SPF_HELO_PASS SPF: HELO matches SPF record
* -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM
*      white-list
*  0.0 HTML_MESSAGE BODY: HTML included in message
*  0.0 HTML_FONT_LOW_CONTRAST BODY: HTML font color similar or
*      identical to background
* -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
*      author's domain
* -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
*  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
*       valid
*  1.0 RDNS_NONE Delivered to internal network by a host with no rDNS
*  0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay
*      lines
* -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender
*  312 AWL AWL: Adjusted score from AWL reputation of From: address
Title: Re: SA issue on email
Post by: d2d4j on February 26, 2019, 11:47:22 PM
Hi jelv

Sorry soo tired so just realised you asked what awl is re SA

AWL Authorised white list and should not score that high

Many thanks

John
Title: Re: SA issue on email
Post by: jelv on February 27, 2019, 08:07:08 AM
In the meantime, we have decided to turn off SA to allow email flows through

SA still appears to be running as I've had further false positive emails overnight. One of these is an emails that I get every day from the overnight daily virus scan on my NAS box.

There is a very significant difference in the headers on the good email I received yesterday and the one I received this morning. (Apart from the date/time, the body of the emails is identical).

Yesterdays good email:
Quote
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on haveworx.co.uk
X-Spam-Level:
X-Spam-Status: No, score=0.9 required=7.0 tests=ALL_TRUSTED,HTML_MESSAGE,
   HTML_MIME_NO_HTML_TAG,MIME_HTML_ONLY,MISSING_MID autolearn=disabled
   version=3.4.2

Today's false positive:
Quote
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on haveworx.co.uk
X-Spam-Flag: YES
X-Spam-Level: *************
X-Spam-Status: Yes, score=13.1 required=7.0 tests=ALL_TRUSTED,AWL,HTML_MESSAGE,   <-------------
   HTML_MIME_NO_HTML_TAG,MIME_HTML_ONLY,MISSING_MID autolearn=disabled
   version=3.4.2
X-Spam-Report:
   * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
   *  0.0 HTML_MESSAGE BODY: HTML included in message
   *  1.1 MIME_HTML_ONLY BODY: Message only has text/html MIME parts
   *  0.1 MISSING_MID Missing Message-Id: header
   *  0.6 HTML_MIME_NO_HTML_TAG HTML-only message, but there is no HTML
   *      tag
   *   12 AWL AWL: Adjusted score from AWL reputation of From: address

The update seems to have turned on AutoWhiteList and it is the high score that is giving that is causing the false positive.
Title: Re: SA issue on email
Post by: jelv on February 27, 2019, 08:16:34 AM
I've now compared headers on emails that are not being marked as spam and am consistently seeing the same difference. AWL was not in the string of tests before the update and it is now and they are now getting much higher scores (although not enough to be marked as spam).
Title: Re: SA issue on email
Post by: Ronski on February 27, 2019, 08:19:51 AM
I emptied my spam folder last night, this morning I have three emails in there that shouldn't be, one from my nas, one from Facebook and another.

At least I know to check in there.
Title: Re: SA issue on email
Post by: jelv on February 27, 2019, 08:20:16 AM
Just found this:

Quote from: https://wiki.apache.org/spamassassin/AutoWhitelist
The Auto-WhiteList
In March 2014 (rev 1579980), SpamAssassin (https://wiki.apache.org/spamassassin/SpamAssassin) introduced the TxRep (https://wiki.apache.org/spamassassin/TxRep) plugin. The new plugin enhances the functionality of AWL, and works around some of its shortcomings.
Title: Re: SA issue on email
Post by: d2d4j on February 27, 2019, 08:28:41 AM
Hi

@jelv and @ronski - many thanks

Yes I know AWL is appearing to score high mostly and AWL has been around for many years, even in the earlier version 3.3.2.

I am looking into this and the control panel has gone into a maintenance mode

I believe we had the earlier SA about right and the new SA, which upgraded 3.3.2 to 3.4.2 should not have changed the rules but it appears something is not right

If I cannot see why/whatís gone wrong I will open a ticket with IW, which I did email IW last night, but not heard back as yet

Many thanks

John
Title: Re: SA issue on email
Post by: jelv on February 27, 2019, 08:36:01 AM
John,

According to https://wiki.apache.org/spamassassin/TxRep, TxRep replaces AWL so shouldn't AWL be turned off (as it was before the upgrade)?
Title: Re: SA issue on email
Post by: d2d4j on February 27, 2019, 09:15:34 AM
Hi jelv

Many thanks

AWL was never turned off sorry. It is used in conjunction with baysian which is turned off and the default score for AWL should be 0.5

It is almost like all areas are turned on but shows as turned off for AWL to score at those figures.

Also, please remember this is an IW package so will have adapted.

I know IW are going to drop qmail in favour of dovecot and we can change on this platform, so I am pondering if the newer SA is geared more to dovecot then Qmail

Itís just a thought and I could be entirely wrong sorry.

I have a ticket open with IW, so hopefully all will be revealed later today

I do apologise for this inconvenience to all users and it is a rarity

Many thanks

John
Title: Re: SA issue on email
Post by: vic0239 on February 27, 2019, 09:35:18 AM
@vic, I will pull of the log and let you know
John, it's ok, I have found the marked emails. I hadn't mapped the spam folder correctly and, not having any spam emails usually, it has only just apered in the folder list.
Title: Re: SA issue on email
Post by: d2d4j on February 27, 2019, 10:04:50 AM
Hi

@vic - many thanks

I think I may have found the issue but it needs testing and watching for about 30 minutes

It is hard though, as we do not see the emails, only the logs which show tests and scores

If anyone is able to do a quick test it would help

Many thanks

John
Title: Re: SA issue on email
Post by: vic0239 on February 27, 2019, 10:08:47 AM
I've just done a test from my Outlook account which has come through as normal.
Title: Re: SA issue on email
Post by: d2d4j on February 27, 2019, 10:15:43 AM
Hi Vic

Many thanks

I removed outlook from whitelist so thatís looking promising

I am looking at logs and the figures are looking normal I think but time will tell

Many thanks

John
Title: Re: SA issue on email
Post by: jelv on February 27, 2019, 11:13:16 AM
I'm still seeing false positives.

This email came in at 10:32:

Code: [Select]
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on haveworx.co.uk
X-Spam-Flag: YES
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=137.0 required=7.0 tests=AWL,DKIMWL_WL_HIGH,
DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HTML_FONT_LOW_CONTRAST,
HTML_IMAGE_RATIO_08,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RDNS_NONE,
SPF_HELO_PASS autolearn=disabled version=3.4.2
X-Spam-Report:
* -0.0 SPF_HELO_PASS SPF: HELO matches SPF record
* -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/,
*       no trust
*      [87.253.236.19 listed in list.dnswl.org]
*  0.0 HTML_IMAGE_RATIO_08 BODY: HTML has a low ratio of text to image
*       area
*  0.0 HTML_MESSAGE BODY: HTML included in message
*  0.0 HTML_FONT_LOW_CONTRAST BODY: HTML font color similar or
*      identical to background
* -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
*      author's domain
* -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
*  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
*       valid
*  1.3 RDNS_NONE Delivered to internal network by a host with no rDNS
* -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender
*  136 AWL AWL: Adjusted score from AWL reputation of From: address
Received: (qmail 8281 invoked by uid 108); 27 Feb 2019 10:32:07 +0000
Title: Re: SA issue on email
Post by: d2d4j on February 27, 2019, 11:21:22 AM
Hi jelv

Many thanks

Sorry it was seconds after that time I applied some changes

Please if possible could you check as all whitelisted have been cleared

Many thanks

John
Title: Re: SA issue on email
Post by: jelv on February 27, 2019, 12:19:38 PM
Looking better on more recent emails!
Title: Re: SA issue on email
Post by: vic0239 on February 27, 2019, 12:27:12 PM
Yes, looking back to normal now. Thanks John.
Title: Re: SA issue on email
Post by: atkinsong on February 27, 2019, 12:28:42 PM
Hi John

Those emails from thepixiepit that I reported problems with yesterday are arriving ok, but I suspect that is just because they are whitelisted. On Sunday they had a spam score of 3.6. The latest one at 12.02 today has a spam score of 96.1.
Title: Re: SA issue on email
Post by: d2d4j on February 27, 2019, 12:57:53 PM
Hi

Many thanks everyone

I will be making a few more changes but only to lower initial threshold which we set to max yesterday.

We will keep checking/monitoring for a few days though as it does appear to have fully stabilized now

I do apologise once more but at least you know there is support and I am sorry, I was just to tired last night so had to wait until this morning.

I will close any tickets opened at billing platform but any issues, you can reopen ticket at any time or open a new ticket. Please leave feedback as itís feedback for level 2 support. I guess my feedback would be a minus figure sorry

There is one further thing I would like to ask, which refers to spam - please could you post after a couple of days the amount of spam which has been allowed through. Ideally showing SA score as this is one area we do not see

@atkinson - many thanks and yes, we left that domain whitelisted as we can see the SA score is high, but for other SA scores other then AWL

Many thanks

John