transport-email unable to send to GMail recipient
-
@CJ At this point, without being able to pull logs from the SMTP server, it's going to be hard to tell what is going on. Why the SMTP server is taking the message and forwarding it to internal addresses but is not to external addresses is a mystery and having the logs from the SMTP server would help diagnose the root cause.
There's definitely something amiss on the SMTP server and how it communicates with your XO server. The fact that simply sending a raw e-mail message using Telnet got your site blacklisted is a clue (remember, when you sent that message using Telnet on the Linux host you totally removed any and all XO software and plug-ins from the equation... if you set up a raw Linux host with no other software installed and ran the same Telnet test you should be able to send an e-mail).
I'm curious if you're running the supported version of Orchestra (XOA), or the self-build version from sources (XO)?
The last thing I can think of with the info we have is something I had mentioned earlier, and that is the settings on your mail server. Many sites will lock down the ability of a mail server to act as a relay, that is, to accept messages from some device and then forward that as an SMTP message. For example, I mentioned that I use Microsoft 365 hosted Exchange servers as my SMTP relay for XO and a handful of other internal devices. In my Exchange settings, I have it locked down so ONLY devices with a specific IP Address (like the IP of my XO server) can relay... anything else on my network can NOT relay. This helps prevent something like a rogue agent on a user's computer from acting as a spam sender. If your e-mail server has something similar, it may be set up to allow local traffic, but will not allow relay to external addresses. It's also possible that stuff coming from XO is not formatted in a way that your SMTP server can handle and outgoing messages are carrying improper headers that make it look like spam and, therefore, you're getting blacklisted. However, since you were blacklisted while running the Telnet SMTP test (which sends out the most basic, low-level type of e-mail you can have), I think something else is at play.
I know you said you can't see any logs because your mail server is part of a shared hosting plan... but is there anyone else you can reach out to that can pull those logs for you? At this point, we really need to see what is happening on that SMTP server the moment it receives a message from the XO server and what happens as it tries to relay it out to the Internet.
-
@JamfoFL To clarify, my site's IP did not get added to the list. It was my IP and because if my attempts to manually send an email. Following the instructions in the previously linked article caused my SMTP server to flag my IP as misconfigured, malware, or something similar. I did manage to actually send a test email and that's when I got the notification that it was blocked and I was on the spam list.
Also, I'm still unclear what manually sending an email via telnet accomplishes. We know the issue has something to do with the interplay between the plugin and my SMTP server as using a different SMTP server worked and my TrueNAS servers are able to use my SMTP server with the same account and SMTP settings.
I'm running a source built version of XOA on commit https://github.com/vatesfr/xen-orchestra/commit/2978ad148618b877db4695a66acc2b7c5d01a2fc
The mail server isn't locked down to my IP and all of my email would be coming from the same IP. XCP-ng, TrueNAS, ssmtp and my email client are all on my local network. The SMTP server is on my hosting providers network.
I definitely think it's something about the way the plugin message is formatted that my SMTP server doesn't like. But as I mentioned earlier, the only differences I could find were the attachment and minor differences in the MIME section. Since my ssmtp setup sends attachments using the same account and SMTP server I don't think that's the issue. I've considered attempting to tweak the plugin email to use MIME formatting like my TrueNAS emails, but I'm a bit hesitant to attempt a manual send with it in order to avoid getting blocked again.
-
@CJ Just trying to eliminate all possibilities in a systemic fashion. What really is the most confusing aspect to this whole thing (and why I was suggesting trying other testing methods) is that you say you CAN send messages from XO to your own domain, no problem, but anything to any other domain just disappears.
When you set up your information in XO, it has no idea if a message is on your domain or not... it's going to send the message out the same way no matter what the recipient address might be. There's nothing going on in the plug-in that says, "format a message one way for this domain, format it another way for all other domains." So, trying to figure out just what is the difference, what is coming from XO, how is the SMTP server interpreting that information, and what the SMTP server is doing differently when it sends something to your internal domain vs. sending something to an external domain, is the crux of the problem.
Again, we have to ask the question... if there is something about the way the plug-in message is formatted that your SMTP server doesn't like, why does the SMTP server successfully send the messages to your own domain? I would think if your SMTP server didn't like the encoding or some aspect of the message from the plug-in it would fail for all the test messages you try to send, not just those going out to other domains. That's the only reason I've tried other testing... if you said that it simply didn't send at all, to any recipient on any domain, then I would definitely lean towards some kind of encoding issue. It's the fact that it does work for your local domain is what confuses the situation.
Unfortunately, this is where being able to run a message trace from the SMTP server would be nice... we would be able to read the success/failure/rejection messages and that would give us a definitive answer as to why the server is rejecting that message. It might even help point to MIME issues if that was the case. Otherwise, at this point, we're just stumbling in the dark.
If it doesn't give anything away and is something you are comfortable revealing, who is the developer/host of your SMTP server?
-
@JamfoFL I definitely agree that it's a frustrating and confusing experience. It doesn't help that there's no bounce messages and the host doesn't provide any deliverability trace options.
I'll see if I can poke around so more and possibly reach out to them to figure out what the issue is. I haven't reached out to them yet as I've generally found them lacking but I haven't been able to find any good hosting recommendations. It seems everyone dislikes their host, or if they did happen to like them, the host ends up gobbled up by that group who's acquiring them all.
-
@CJ Cool. If you get anything useful from them, I'll be happy to look... even though I imagine with all your private info in it, it's probably not something you'll want to share. Fingers crossed they send you something useful and, when you look, you'll have that "A-ha!" moment that will finally make this all make some sense!
-
@JamfoFL After a bunch of back and forth with tech support they said that the emails are getting sent to /dev/null and they have no idea why. Now I have to wait for more senior techs to investigate and attempt to figure out the issue.
-
@CJ Yeesh... I have my fingers and toes crossed that they can figure out what is going on for you, or at least give us more info for a place to look!
-
@JamfoFL It turns out that the issue is because XCP-ng sends HELO 127.0.0.1 and the SMTP server considers that suspicious and spammy. Therefore the email gets dropped.
TrueNAS appears to use a HELO of the server name, which is apparently acceptable to the SMTP server. Which is odd as support wanted me to use a HELO of the actual SMTP server.
-
Is there anything we can do on our side?
-
@CJ @olivierlambert Looking at the RFC, the SMTP HELO command needs to be followed by a domain name, not an IP address.
Testing my XO (source) install, it does send the host/domain name of the XO machine in the HELO message and not 127.0.0.1. So the the question is, why is your install sending 127.0.0.1 or why is the SMTP server seeing that in the HELO message.
I also test XCP-ng (which uses
ssmtp) and it also sent the name, not IP. -
@Andrew @olivierlambert If anyone can provide me with the proper place to look for logs, etc, I can attempt to determine what's going on. My XOA is a source install as well.
Also, now that I know what's causing the problem, I can use my own domain to review the test messages as they still work.
-
So if XO from sources it's NOT XOA
(XOA is only meant for the virtual appliance we deliver). I'll be curious to see if you have the same issue with XOA, that would help to rule out a XO-related issue 
-
@olivierlambert Sorry, didn't realize.
How can I test it in XOA? I don't have the transport-email plugin.
-
This post is deleted! -
@Andrew I think I figured out why it's happening. It looks like NodeMailer is getting the hostname and because mine isn't a FQDN then it sets the default hostname to 127.0.0.1
https://github.com/nodemailer/nodemailer/blob/master/lib/smtp-connection/index.js#L1788
Is your hostname a FQDN?
Apparently Debian thinks hostname should not return a FQDN. https://www.debian.org/doc/manuals/debian-reference/ch03.en.html#_the_hostname
-
@olivierlambert @Andrew That did it. If the hostname of the machine doesn't contain a . then NodeMailer sets it to 127.0.0.1 and therefore sends helo 127.0.0.1 when it connects to the SMTP server.
It looks like if a client name is passed to NodeMailer it skips this check, so would it be possible for XO to expose the field as an optional parameter?
-
Thanks for your feedback, let me add @julien-f in the convo
-
@CJ Which field would you want to be optional?
-
@julien-f The client hostname, although I don't know that that's really the best solution. This seems like a weird edge case that doesn't seem to affect most people.
Right now I've added the domain to my /etc/hostname file even though it's against Debian convention.
-
@CJ Please test the branch
email-local-hostnamewhich make the local hostname configurable and keep me posted.
