Kitz Forum

Announcements => Site Announcements => Topic started by: kitz on May 26, 2018, 12:42:45 PM

Title: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on May 26, 2018, 12:42:45 PM
Notification to members that emails to any of the following domains are being bounced -  Yet again!

@live.com
@outlook.com
@hotmail.com

This is affecting the registration confirmation for new forum members and notifications for subscribed topics & personal messages.
Yet again we appear to be an innocent victim of Microsoft's 'Smart Filter' despite following all their rules.
The domain is clean and definitely not on any blacklists - See MX Toolbox Blacklist check.

------------

This is the third time this has happened.  Both times in the past Microsoft admitted it was an error on their part and we should not have been blacklisted and agreed to mitigate the domain.   
We are not the only reputable site this is happening to and its becoming an irksome routine for many other websites too.

See here (https://forum.kitz.co.uk/index.php/topic,20791.0.html) and here (https://forum.kitz.co.uk/index.php/topic,20409.0.html)

TBF I do not have time today to spend the hours that it takes chasing MS and am so fed up of their ridiculously broken system that I am considering blacklisting all their email addresses if I can find a way which doesn't affect existing members accounts.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Chrysalis on May 26, 2018, 01:23:18 PM
yeah just ban freemail addresses, Microsoft emails system also has been broken for years on their algorithms, same with outlook in office as well but at least on that the filters can be disabled
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: d2d4j on May 26, 2018, 06:18:58 PM
Hi kitz

Ohh dear

My best advice to you would be not to block Microsoft. It is very well used by many and blocking Microsoft would only hurt kitz, not Microsoft

As this is the third time, I would simply reply back to the very last email received confirming you should have been whitelisted, and explain that it has happened again, and ask politely to have it corrected ASAP please. Also quote your original 2 ticket numbers.

This should save a lot of time of going through the same hoops to be whitelisted as they will be able to read the previous tickets.

I would consider it may have happened due to Microsoft changes for GDPR perhaps

I am sorry if I wrong but good luck, and my offer still stands if you ever need the use of it.

Many thanks

John
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 10, 2018, 10:57:43 AM
Confirmation that the block has been removed this morning.
Quote
Conditionally mitigated

Our investigation has determined that the above IP(s) qualify for conditional mitigation.

Mitigation may take 24 - 48 hours to replicate completely throughout our system.

Like I say, fed up that I have to jump through hoops every few months.    I can block people from these domains from signing up for new accounts, but that would also impact on existing members.   If I can find a way of doing so then I damn well will.

My webhosts also had this to say

Quote
Lately they have been denying messages from legitimate servers and they provide no information as to why this occurs.
Microsoft have no automatic way of providing details for these cases.

What a stupid system whereby each time I have to start from scratch again and prove that:  I am not on any RBL blacklists, that I don't have a bad mail reputation and get my hosts to confirm there is no suspicious  behaviour or spam messages sent from my server  >:( >:(
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: burakkucat on June 10, 2018, 03:58:32 PM
<Sigh!>  :wall:
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Weaver on June 10, 2018, 05:29:26 PM
Regarding emails sent out, it is good that there an SPF record (for example) set up on the kitz.co.uk domain - I assume that is the email From: domain?

Is there anything we could suggest to help boost Kitz' mail sender reputation further ?

I am also wondering if anyone is impersonating you by sending fake emails to MS (for example) with spoofed From: addresses that match your domains.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 10, 2018, 06:33:24 PM
Regarding emails sent out, it is good that there an SPF record (for example) set up on the kitz.co.uk domain - I assume that is the email From: domain?

Yes - I've done absolutely everything by the book.   There is no reason at all way they should do this.
btw they say they're not blocking on the domain, but on the server IP so that makes it even weirder

It concerns me that this time they have responded with "conditionally mitigated" whereas in the past they just said "mitigated".   Mind you fat lot of good that was when I still get blocked every few months. 

Quote
Is there anything we could suggest to help boost Kitz' mail sender reputation further ?

Those members who have MS mail services not just straight sending PM notifications to the recycle bin and marking them as good is about all I can think.

The other alternative reading between the lines of the full email from them is that someone marked mail from this IP as spam.  I have no idea why a forum member would deliberately mark notifications as junk mail..  but thats getting into the realms of paranoia so I'm not going there.. and going to assume (hope) that was standard script.

Bear in mind I am not alone in this problem, since it began I have had several emails from people who I've never heard of before asking my advice how I managed MS to mitigate my server because they are having the same problems and finding it extremely difficult to get their own server unblocked.
MS even blacklisted TalkTalk mail and a couple of other prominent UK websites a few months ago so it shows you how pathetic their system is. 

Quote
I am also wondering if anyone is impersonating you by sending fake emails to MS (for example) with spoofed From: addresses that match your domains.

I think over about the past six months I've seen bounce messages for just 2 or 3 spoofed from the domain name..  certainly no more than half a dozen.   Because I have set up SPF & DKIM.... and the fact they blacklist the IP (not the domain) would suggest this is not the reason.
As discussed in one of the threads on one of the previous occasions, they base it on behaviour of recipients of mail from me.    So someone getting lots of PM and thread notifications who send them straight to recycle bin negatively impacts me.

Thus I feel I have a valid reason to refuse any forum signups from MS mail.   My personal mail (but on a different account & domain name) is also sent from the same server, so it means that I also can't mail any of my personal contacts with MS mail either because the block is IP based.

atm I could block signups from MS mail..  but the side effect is that it would ban members already registered with MS mail accounts.  :(
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Chrysalis on June 10, 2018, 10:35:25 PM
Weaver I have seen it myself on servers, valid DKIM, SPF, RDNS, Helo, headers etc.  Clean IP,domain on blacklists but blocked by hotmail.  Its Microsoft's systems been unreliable.

In regards to blocking freemail, I am registered on a lot of forums and other free sites, I would say roughly 1/2 block freemail, and some of these are quite large communities.

If you considering blocking MS kitz, then perhaps give a month notice for people to update their contact email address.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Weaver on June 11, 2018, 03:22:14 AM
Hotmail admins are just hurting themselves as well as their customers by this lack of clue.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: j0hn on June 11, 2018, 09:18:41 AM
I don't think anyone mentioned blocking freemail... that would kill the forum in a heartbeat.
Gmail is freemail.

I don't think I've ever attempted to register for a site that has blanket banned all free mail providers.

Actually I've only ever seen MS provided email addresses being blocked from signups, with the main reason given that the verification email goes straight to spam.

I haven't paid for email since I signed up for gmail beta many years ago.
My Demon email subscription was allowed to expire shortly after.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 11, 2018, 10:04:47 AM
Quote
I don't think anyone mentioned blocking freemail.

No just MS from registrations.   I think a lot of other forums do this now because of how they seem over zealous with marking mail as spam when its not.
I do have a block on a couple of others such as mail.ru which are frequently used by spambots.   

Problem is currently if I block, then it will impact live users such as skyeci and michty.   There is an SMF mod which will block just new registrants, but unfortunately its not currently supported for the version of SMF that I use and fails on install.

I don't have any problems with gmail.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Ronski on June 11, 2018, 10:15:12 AM
Is there many users with MS email addresses ?

If there isn't many it may be worth PM/Emailing to see if they could change them.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: michty_me on June 11, 2018, 10:32:59 AM
Just a heads up, I received a flurry of email notifications today to my hotmail account after not receiving any for some time.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: j0hn on June 11, 2018, 11:42:33 AM
Just a heads up, I received a flurry of email notifications today to my hotmail account after not receiving any for some time.

You didn't automatically delete them without reading them did you?
The suspicion is that's what's flagging the site email as spam.

MS only unblocked kitz today that's why you haven't been receiving any emails.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Chrysalis on June 11, 2018, 01:08:22 PM
I don't think anyone mentioned blocking freemail... that would kill the forum in a heartbeat.
Gmail is freemail.

I don't think I've ever attempted to register for a site that has blanket banned all free mail providers.

Actually I've only ever seen MS provided email addresses being blocked from signups, with the main reason given that the verification email goes straight to spam.

I haven't paid for email since I signed up for gmail beta many years ago.
My Demon email subscription was allowed to expire shortly after.

no kitz is only going to block hotmail I think, however there is definitely sites that block gmail and yahoo also and they not dead.

But in this case I think just blocking hotmail is adequate.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: michty_me on June 11, 2018, 01:26:17 PM
You didn't automatically delete them without reading them did you?
The suspicion is that's what's flagging the site email as spam.

MS only unblocked kitz today that's why you haven't been receiving any emails.

No I always open them and to read the content before clicking the link.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: 4candles on June 11, 2018, 01:34:19 PM
Notification to members that emails to any of the following domains are being bounced -  Yet again!

@live.com
@outlook.com
@hotmail.com
Does this apply to live.co.uk and hotmail.co.uk too?
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 12, 2018, 12:35:06 AM
Is there many users with MS email addresses ?

If there isn't many it may be worth PM/Emailing to see if they could change them.

Not sure tbh,  would possibly need to query the userbase.   Even so there may be several whose accounts are seldom used.
There's been a few hundred over the past week or so, so I havent looked at them all, but a very quick glance show that 2 or 3 currently active members are most affected.

Does this apply to live.co.uk and hotmail.co.uk too?

I'm afraid so.   For example one of the emails I have rec'd notification of fail from is

Quote
  m*********7@hotmail.co.uk
    host eur.olc.protection.outlook.com [104.47.6.33]
    SMTP error from remote mail server after pipelined MAIL FROM:<*****kitz.co.uk> SIZE=3688:
    550 5.7.1 Unfortunately, messages from [185.24.98.37] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3140). ]

 
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 12, 2018, 01:03:55 AM
no kitz is only going to block hotmail I think, however there is definitely sites that block gmail and yahoo also and they not dead.

But in this case I think just blocking hotmail is adequate.

I'm not quite sure yet how I am going to handle it.   It won't just be hotmail it will also include outlook and live as they all use the same system.

I can't just do a block or the side effect would be the likes of michty and skyeci would end up banned and it requires something more customised.   
As mentioned above there is an SMF mod, but it's been unsupported for quite a while and doesnt work with later versions of SMF2.    Someone has tried to do an update which requires a manual amendment but that also fails.   
The core code has changed quite a bit and the 'find'  & 'replace' wont work as there is something more major going on with some else/ifs in the core code.
I will have attempt some code, but will also be asking for help on the SMF forums before I apply because touching core code is something I don't like doing.

I have other priorities going on too atm.   Some of you know that my hand has got a lot worse and typing becomes more and more difficult... as well as struggling with FM.   I am due to go in for an op very soon to try regain control of my fingers, but nerve transposition recovery is slow (12-18m) and I will be in a full arm cast for a few weeks so no time scale right now.   
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: banger on June 12, 2018, 04:18:30 AM
I don't know if this is helpful Kitz but have you tried Speech to text? Not even sure what OS you use.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Weaver on June 12, 2018, 04:23:05 AM
@kitz I have used speech recognition on my ipad when my hands have been hurting me.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Chrysalis on June 14, 2018, 03:29:29 PM
Kitz of course, when I say hotmail, I mean all hotmail related branding, so MSN, outlook, hotmail, live etc.  They all the same service just different branding frontend's.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: j0hn on June 14, 2018, 05:17:52 PM
no kitz is only going to block hotmail I think, however there is definitely sites that block gmail and yahoo also and they not dead.

But in this case I think just blocking hotmail is adequate.
Been using Gmail since Beta and never come across a site that won't allow Gmail addresses for sign-up.

I've come across dozens that don't allow hotmail and other MS email accounts due to problems with activation emails not being received or going straight to spam.

I wouldn't put the 2 in the same category.

I'd be interested to know the % of users on kitz with Gmail addresses. I'd guess it's pretty high.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 14, 2018, 06:22:00 PM
Kitz of course, when I say hotmail, I mean all hotmail related branding, so MSN, outlook, hotmail, live etc.  They all the same service just different branding frontend's.

Yep all MS mail :/

Quote
I'd be interested to know the % of users on kitz with Gmail addresses. I'd guess it's pretty high.

Don't know without running some sort of customised query on the database, but I would imagine like you say it will be quite high.   I can't recall ever having had a problem with gmail nor can I remember gmail particularly being used for spam - unlike hotmail & ru mail.   
Hard for me to check on that as most spam type accounts got deleted a while ago during a clean out.   Those with sharp eyes may have spotted a decrease in the number of registered users.  Judging from the db and SMF numbers well over 5k 'members' have been deleted.   
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Chrysalis on June 15, 2018, 07:55:14 AM
This mod seems to only affect registration so wouldnt ban existing users, but is very dated from 2009, may no longer be relevant for current version of SMF.

https://custom.simplemachines.org/mods/index.php?mod=1493
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 15, 2018, 08:12:30 AM
Thats the mod I meant when I said

As mentioned above there is an SMF mod, but it's been unsupported for quite a while and doesnt work with later versions of SMF2.    Someone has tried to do an update which requires a manual amendment but that also fails.   
The core code has changed quite a bit and the 'find'  & 'replace' wont work as there is something more major going on with some else/ifs in the core code.

Reply #138 on that SMF thread about the mod is mine
Tried to install this and Im getting the following errors. 

Is there any update please.  I so want to install this mod and stop people registering with hot-mail etc accounts.

I have since looked at it and I will probably be on my own trying to see if I can get it working with the current version.   It's no longer a one line change and I need to spend some time looking at the code properly to see if I can figure it out because its now smack bang in between an elseif and no longer a straight forward find and replace to attempt to even do a manual update nvm the fact the package failed.

This is a link to the image I posted in that mod thread and what happens why I try to install
https://www.simplemachines.org/community/index.php?action=dlattach;topic=274317.0;attach=256061;image

It's sickening that it doesnt work as it is exactly what I need :'(
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Chrysalis on June 15, 2018, 09:03:34 AM
can you post the image here kitz? I need to be registered on there to view it.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: d2d4j on June 15, 2018, 09:27:38 AM
Hi kitz

Iím sure you have tried already sorry, but in case not, have you tried that package in compatibility mode with smfv1.

You maybe lucky

If it helps, I can give you a test account and one click install will install smf in less then a minute is a default install. Then you could test/change at will without any risk to kitz, which should give you the chance to see what needs changing first on a standard default smf

Many thanks

John
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: jelv on June 15, 2018, 09:58:40 AM
There's people on the SMF forums that are assuming that the reason we want M$ emails blocked is because spammers are registering on our forums using those addresses - I've just posted a clarification.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: jelv on June 15, 2018, 11:17:01 AM
I'm taking a look at the mod. I can see why it failed and it may not be difficult to get it working (I've downloaded the source for 2.0.1 and can see what has changed).
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 15, 2018, 11:24:47 AM
I think the person above me posted an update, but that failed too.    I had a look at the code and saw that my Subs-Members.php had a totally different set of code rather than just one symbol in the one line.     I was going to download a fresh Subs-Menu and compare to mine to see if it was perhaps another mod (such as Stop Spammer) which had inserted the extra code.    I never got that far though.    As you will see it was earlier this year I started looking at it, but then left it as MS mitigated me.    It's only with it happening again recently that I need to revisit it, but just not had chance yet. 
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 15, 2018, 11:27:06 AM
If it helps, this is where it fails.


Find
Code: [Select]
// !!! Separate the sprintf?
if (empty($regOptions['email']) || preg_match('~^[0-9A-Za-z=_+\-/][0-9A-Za-z=_\'+\-/\.]*@[\w\-]+(\.[\w\-]+)*(\.[\w]{2,6})$~', $regOptions['email']) === 0 || strlen($regOptions['email']) > 255)
$reg_errors[] = array('done', sprintf($txt['valid_email_needed'], $smcFunc['htmlspecialchars']($regOptions['username'])));



Code: (Replace)
Code: [Select]
// Lets restrict some email providers
   if (!empty($modSettings['restricted_provider']))
   {
      $restricted_provider = explode(",", preg_replace("/(\@[^a-zA-Z0-9,])/", "", $modSettings['restricted_provider']));

      foreach ($restricted_provider as $key => $value)
         if (empty($value))
            unset($restricted_provider[$key]);
   }
   else
      $restricted_provider = array();



   if (!empty($modSettings['enable_restrict_EmailProvider']) && !empty($modSettings['restricted_provider']))
   {
      foreach($restricted_provider as $provider)
      {       
         preg_match('/' . $provider . '+/i', $_POST['email'], $matches);
   
         if(count($matches) > 0)
            fatal_error(sprintf($txt['restricted'], $regOptions['username']), false);
      }
   }
   
    if (!empty($modSettings['accepted_provider']))
   {
      $accepted_provider = explode(",", preg_replace("/(\@[^a-zA-Z0-9,])/", "", $modSettings['accepted_provider']));

      foreach ($accepted_provider as $akey => $avalue)
         if (empty($avalue))
            unset($accepted_provider[$akey]);
   }
   else
      $accepted_provider = array(); 
   
    if (!empty($modSettings['enable_restrict_EmailProvider']) && !empty($modSettings['accepted_provider']))
   {
      foreach($accepted_provider as $aprovider)
      {       
         preg_match('/' . $aprovider . '+/i', $_POST['email'], $matches);

if(count($accepted_provider) == 1)
{
if(count($matches) == 0)
fatal_error(sprintf($txt['restricted'], $regOptions['username']), false);
}
else
{
if(count($matches) == 0 && !(each($accepted_provider)))
fatal_error(sprintf($txt['restricted'], $regOptions['username']), false);
}
      }
   }
   // !!! Separate the sprintf?
   elseif (empty($regOptions['email']) || preg_match('~^[0-9A-Za-z=_+\-/][0-9A-Za-z=_\'+\-/\.]*@[\w\-]+(\.[\w\-]+)*(\.[\w]{2,6})$~', $regOptions['email']) === 0 || strlen($regOptions['email']) > 255)
$reg_errors[] = array('done', sprintf($txt['valid_email_needed'], $smcFunc['htmlspecialchars']($regOptions['username'])));


Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 15, 2018, 11:30:32 AM
This is my Subs-Members.php


Code: [Select]
// !!! Separate the sprintf?
if (empty($regOptions['email']) || filter_var($regOptions['email'], FILTER_VALIDATE_EMAIL) === false || strlen($regOptions['email']) > 255)
$reg_errors[] = array('lang', 'profile_error_bad_email');

if (!empty($regOptions['check_reserved_name']) && isReservedName($regOptions['username'], 0, false))
{
if ($regOptions['password'] == 'chocolate cake')
$reg_errors[] = array('done', 'Sorry, I don\'t take bribes... you\'ll need to come up with a different name.');
$reg_errors[] = array('done', '(' . htmlspecialchars($regOptions['username']) . ') ' . $txt['name_in_use']);
}

Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: Chrysalis on June 15, 2018, 11:58:28 AM
I will wait to see what Jelv and others come up with before I attempt anything.

But yeah having test accounts is great for things like this so no experimentation done on a live site then.
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: jelv on June 15, 2018, 12:08:45 PM
I've got a modified version that works on a vanilla SMF installation!
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: jelv on June 15, 2018, 12:24:15 PM
PM sent to Kitz with link to modified forum for her to try and my modified package.
John
Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: kitz on June 15, 2018, 12:26:50 PM
Well done you!  :thumbs:


I'm not sure if you've seen reply #133 (https://www.simplemachines.org/community/index.php?topic=274317.msg3906211#msg3906211) and not sure if it applies

Quote
Please note if you use this fix you MUST add another string to the code after the elseif... or users can double-register names/emails.

Replace elseif with:

I was also going to look at reply #134 (https://www.simplemachines.org/community/index.php?topic=274317.msg3918030#msg3918030)

Quote
To prevent the use of a restricted email provider when members change their email address while modifying their profile make the following change to ./Sources/Profile-Modify.php

I have to go out in a few mins, but if you could kindly email me the files and I will test later.



--------------
ETA
Warning - while you were typing a new reply has been posted. You may wish to review your post.

Just seen the further message and the PM.   
Sorry but I really do need to go out now as I have an appt, but will get back to you asap.



Title: Re: Forum mail problem for those using Microsoft email addresses - Again!
Post by: jelv on June 15, 2018, 12:34:05 PM
I hadn't seen those posts.

What I did was compare the source of 2.0.15 with the source of 2.0.1. I could see that the search lines had been totally changed to use a different method of validating the email format, but that the surrounding lines had not been changed. I just replaced the code with the new version in both the search and the add.

Edit: I've looked at 133.

Code: [Select]
elseif (empty($regOptions['email']) || preg_match('~^[0-9A-Za-z=_+\-/][0-9A-Za-z=_\'+\-/\.]*@[\w\-]+(\.[\w\-]+)*(\.[\w]{2,6})$~', $regOptions['email']) === 0 || strlen($regOptions['email']) > 255)
is the old version from 2.0.1

Code: [Select]
$reg_errors[] = array('lang', 'profile_error_bad_email');
Looks like a line was omitted by a copy/paste error in the quoted post. That line is in my modified version.