Figured it out after some digging, appears it's related to Netbox having ipv4 translated into ipv6 for that field, so to use IPv4 you have to have :ffff:x.x.x.x/128 as your IP address (the x's being your octet).
Once you do this it works without issue.