Good news everyone!
For anyone that comes across this after the fact. From what I can gather, the issue is in fact it is looking for a valid certificate when going to an IP to get the stats. So even a valid certificate will fail.
The solution that I have found, is the "Enable Unauthorized Certificates" option in Settings > Server.
And to be clear, it is obvious in hindsight...
What tripped me up is the fact that I'm now only connected to the pool master (After joining the second host), which has valid certificates.
So why would I allow unauthorized certificates?
Well this is why, so when the calls go out from the master with an IP address, it doesn't get tripped up.
Cheers!