transport-email unable to send to GMail recipient
-
@CJ Is it possible you can set up your plug-in so it will e-mail two external addresses simultaneously, one to your problematic Gmail account and one to one of the other accounts that work?
With it set up that way, if you receive the message on your non-Gmail account but do not on your Gmail account, I would suspect there's something going on with Gmail receiving the messages and not with XO sending out the messages. Since the sending server is the same for both, if it successfully relays over to one recipient but not the second, I doubt that your sending server is the issue as it would block both messages if there was some problem with the handoff from XO to the message queue.
Have you checked to make sure you haven't been inadvertently blacklisted on Google's mail servers?
It's a shame you don't have access to the SMTP server you are using as the relay, as being able to perform a message trace would be a huge help. However, given that the issue lies only with Gmail and nothing else, I'm going to point a finger at their service as the cause of the issue, followed by your SMTP relay, as XO is simply passing the message over to the relay and, if it works for everything but Gmail, I would think that is the culprit.
-
@JamfoFL I'm not sure what sending to two addresses at once would accomplish that sending individually wouldn't. I'm not changing the SMTP settings in between tests.
Considering that all of my other machines, which are using the same account and SMTP settings, still work, I'm pretty sure I'm not blacklisted by gmail. And while I would love to lay the blame on Google, the fact that it's only XCP-ng that's having this problem despite using identical settings is what keeps bringing me back to it.
That said, I did discover something. XCP-ng emails only send to other accounts on the domain. Test emails to anything other than that just disappear. So it's not limited to gmail. Additionally, I can send to gmail and all of the other accounts just fine using the webmail for the account I'm sending with.
-
Another thing I just realized while comparing sources is that transport-email includes an attachment. None of my TrueNAS emails to but my ssmtp emails do. The only other differences I can find are minor differences in the MIME section.
-
@CJ OK, the added info that it's all external accounts and not just Gmail does eliminate Gmail as a possible source.
The next thing to look at will be your SMTP server. From what I can see (and others with much more knowledge of XO's internal workings can correct me if I'm wrong), XO does not have its own, internal SMTP server... it simply creates a message and then forwards it to the SMTP server you've configured and that server is what performs the actual relay of the e-mail message. Since your SMTP server is happily forwarding those messages to internal recipients on the same domain but is not forwarding those messages outside of your domain, we may need to look at the configuration settings on the mail server itself.
It is not uncommon for mail SysAdmins to configure their mail servers to only allow external relay from specifically defined IP Addresses. Maybe the IP Address for your XO server needs to be added?
One thing I would try that will completely take the XO application itself out of the equation is utilizing Telnet to test your SMTP connection. You can check out this site for instructions:
How to check SMTP connection from Command Line (Linux)SSH into the command line interface for the Linux server that is hosting XO and scroll down to Step 1: Checking SMTP Connection Using Telnet. Use your SMTP server's information and try to send yourself a test message through the telnet command line. This takes everything related to the XO application and the transport plug in out of the equation. If you can successfully send an e-mail that is received by Gmail, then we can definitely say that the is a problem in XO and the plug in. If you can't, hopefully you'll be able to get some useful information from the rejection message you'll receive in telnet when attempting to send the message. That should give us a lot more information of where the potential problem may lie, and will give you some actual feedback of what is happening when you try to send out from that server.
-
@JamfoFL This whole saga continues to get even more weird.
I borrowed a domain and created an email account there and then configured the plugin to use it. The initial test failed due to SPF/DKIM error. That address also received a bounce from gmail with the same information. Fixing the SPF/DKIM allowed the email to show up in gmail.
The original domain had one of SPF or DKIM configured but I don't recall which but was already working for gmail with the other servers. I had originally thought that might be the problem so I had already fixed it. Additionally, if SPF/DKIM were the issue I would have expected to receive a bounce response like today.
And my TrueNAS continues to be able to send email to any domain using the same account and settings that continue not to work with XCP-ng.
-
@CJ Just to be thorough, I sent myself a couple test messages from my XO e-mail transport plug-in, both using the telnet command line and the testing feature in the plug-in itself. I have a Gmail account I use personally, and was able to receive both test messages with no issues.
My SMTP server is a hosted Office 365 Exchange server following Microsoft's best practices guide for setting up a relay.
I am also on the latest and greatest versions of XO from Sources.
So, I don't see any issue with the plug-in itself, so we need to look into the relationship between the plug-in in your environment and the SMTP relay server. Hopefully running the telnet test will generate some helpful messages we can use to help get to the bottom of the issue!
-
@JamfoFL I had to remove the
-starttls smtpsection because I'm using TLS but I got a successful connection.I would be surprised if that didn't work considering it's accepting my emails and whenever I had the connection info incorrect I would get an error in the XOA logs.
-
It just keeps getting better and better. I tried manually sending an email from the command line and immediately got back a bounce message showing that I was now on the Spamhaus CSS Blocklist.
-
@CJ Ugh... that definitely sucks! I understand why those things exist, but man are they a PITA when they bite you like that!
-
@JamfoFL I've successfully gotten removed from the list but that also means I'm not manually testing sending emails anymore.
Any other suggestions as to why XCP-ng emails only don't work?
-
@CJ At this point, without being able to pull logs from the SMTP server, it's going to be hard to tell what is going on. Why the SMTP server is taking the message and forwarding it to internal addresses but is not to external addresses is a mystery and having the logs from the SMTP server would help diagnose the root cause.
There's definitely something amiss on the SMTP server and how it communicates with your XO server. The fact that simply sending a raw e-mail message using Telnet got your site blacklisted is a clue (remember, when you sent that message using Telnet on the Linux host you totally removed any and all XO software and plug-ins from the equation... if you set up a raw Linux host with no other software installed and ran the same Telnet test you should be able to send an e-mail).
I'm curious if you're running the supported version of Orchestra (XOA), or the self-build version from sources (XO)?
The last thing I can think of with the info we have is something I had mentioned earlier, and that is the settings on your mail server. Many sites will lock down the ability of a mail server to act as a relay, that is, to accept messages from some device and then forward that as an SMTP message. For example, I mentioned that I use Microsoft 365 hosted Exchange servers as my SMTP relay for XO and a handful of other internal devices. In my Exchange settings, I have it locked down so ONLY devices with a specific IP Address (like the IP of my XO server) can relay... anything else on my network can NOT relay. This helps prevent something like a rogue agent on a user's computer from acting as a spam sender. If your e-mail server has something similar, it may be set up to allow local traffic, but will not allow relay to external addresses. It's also possible that stuff coming from XO is not formatted in a way that your SMTP server can handle and outgoing messages are carrying improper headers that make it look like spam and, therefore, you're getting blacklisted. However, since you were blacklisted while running the Telnet SMTP test (which sends out the most basic, low-level type of e-mail you can have), I think something else is at play.
I know you said you can't see any logs because your mail server is part of a shared hosting plan... but is there anyone else you can reach out to that can pull those logs for you? At this point, we really need to see what is happening on that SMTP server the moment it receives a message from the XO server and what happens as it tries to relay it out to the Internet.
-
@JamfoFL To clarify, my site's IP did not get added to the list. It was my IP and because if my attempts to manually send an email. Following the instructions in the previously linked article caused my SMTP server to flag my IP as misconfigured, malware, or something similar. I did manage to actually send a test email and that's when I got the notification that it was blocked and I was on the spam list.
Also, I'm still unclear what manually sending an email via telnet accomplishes. We know the issue has something to do with the interplay between the plugin and my SMTP server as using a different SMTP server worked and my TrueNAS servers are able to use my SMTP server with the same account and SMTP settings.
I'm running a source built version of XOA on commit https://github.com/vatesfr/xen-orchestra/commit/2978ad148618b877db4695a66acc2b7c5d01a2fc
The mail server isn't locked down to my IP and all of my email would be coming from the same IP. XCP-ng, TrueNAS, ssmtp and my email client are all on my local network. The SMTP server is on my hosting providers network.
I definitely think it's something about the way the plugin message is formatted that my SMTP server doesn't like. But as I mentioned earlier, the only differences I could find were the attachment and minor differences in the MIME section. Since my ssmtp setup sends attachments using the same account and SMTP server I don't think that's the issue. I've considered attempting to tweak the plugin email to use MIME formatting like my TrueNAS emails, but I'm a bit hesitant to attempt a manual send with it in order to avoid getting blocked again.
-
@CJ Just trying to eliminate all possibilities in a systemic fashion. What really is the most confusing aspect to this whole thing (and why I was suggesting trying other testing methods) is that you say you CAN send messages from XO to your own domain, no problem, but anything to any other domain just disappears.
When you set up your information in XO, it has no idea if a message is on your domain or not... it's going to send the message out the same way no matter what the recipient address might be. There's nothing going on in the plug-in that says, "format a message one way for this domain, format it another way for all other domains." So, trying to figure out just what is the difference, what is coming from XO, how is the SMTP server interpreting that information, and what the SMTP server is doing differently when it sends something to your internal domain vs. sending something to an external domain, is the crux of the problem.
Again, we have to ask the question... if there is something about the way the plug-in message is formatted that your SMTP server doesn't like, why does the SMTP server successfully send the messages to your own domain? I would think if your SMTP server didn't like the encoding or some aspect of the message from the plug-in it would fail for all the test messages you try to send, not just those going out to other domains. That's the only reason I've tried other testing... if you said that it simply didn't send at all, to any recipient on any domain, then I would definitely lean towards some kind of encoding issue. It's the fact that it does work for your local domain is what confuses the situation.
Unfortunately, this is where being able to run a message trace from the SMTP server would be nice... we would be able to read the success/failure/rejection messages and that would give us a definitive answer as to why the server is rejecting that message. It might even help point to MIME issues if that was the case. Otherwise, at this point, we're just stumbling in the dark.
If it doesn't give anything away and is something you are comfortable revealing, who is the developer/host of your SMTP server?
-
@JamfoFL I definitely agree that it's a frustrating and confusing experience. It doesn't help that there's no bounce messages and the host doesn't provide any deliverability trace options.
I'll see if I can poke around so more and possibly reach out to them to figure out what the issue is. I haven't reached out to them yet as I've generally found them lacking but I haven't been able to find any good hosting recommendations. It seems everyone dislikes their host, or if they did happen to like them, the host ends up gobbled up by that group who's acquiring them all.
-
@CJ Cool. If you get anything useful from them, I'll be happy to look... even though I imagine with all your private info in it, it's probably not something you'll want to share. Fingers crossed they send you something useful and, when you look, you'll have that "A-ha!" moment that will finally make this all make some sense!
-
@JamfoFL After a bunch of back and forth with tech support they said that the emails are getting sent to /dev/null and they have no idea why. Now I have to wait for more senior techs to investigate and attempt to figure out the issue.
-
@CJ Yeesh... I have my fingers and toes crossed that they can figure out what is going on for you, or at least give us more info for a place to look!
-
@JamfoFL It turns out that the issue is because XCP-ng sends HELO 127.0.0.1 and the SMTP server considers that suspicious and spammy. Therefore the email gets dropped.
TrueNAS appears to use a HELO of the server name, which is apparently acceptable to the SMTP server. Which is odd as support wanted me to use a HELO of the actual SMTP server.
-
Is there anything we can do on our side?
-
@CJ @olivierlambert Looking at the RFC, the SMTP HELO command needs to be followed by a domain name, not an IP address.
Testing my XO (source) install, it does send the host/domain name of the XO machine in the HELO message and not 127.0.0.1. So the the question is, why is your install sending 127.0.0.1 or why is the SMTP server seeing that in the HELO message.
I also test XCP-ng (which uses
ssmtp) and it also sent the name, not IP.
