Deploying firewall to XCP-NG with rescue

ditzy-olive

Hi, just wanted to share this script I am using to deploy firewall to xcp-ng.
I'm rather a UFW person or MikroTik with safe mode, but I don't have that on xcp-ng, so meddling with the firewall was always increasing my anxiety levels. So I was using the Hetzner built in vSwitch firewall which is a pain and allows only 10 rules.

So I made (actually I just told Claude to create this) script that will take the local file called iptables and push that to one or all of my servers I have with hetzner. The script is intended to be called from you local machine, not on the server.

To be safe that I didn't break anything, it schedules an automatic rollback in 2 minutes, applies the new firewall, checks that ssh is still accessible, then removes the planed rollback.

#!/bin/bash
# XCP-ng Firewall Deployment Script
# Deploys the unified firewall configuration to all hosts
# Usage: ./deploy-firewall.sh [test|x1|x2|x3|all]

set -e

# Host definitions
X1_IP="167.235.xx.xx"
X2_IP="167.235.xx.xx"
X3_IP="167.235.xx.xx"

FIREWALL_FILE="iptables"
BACKUP_SUFFIX="backup.$(date +%Y%m%d_%H%M%S)"

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

print_header() {
    echo -e "${GREEN}================================${NC}"
    echo -e "${GREEN}$1${NC}"
    echo -e "${GREEN}================================${NC}"
}

print_warning() {
    echo -e "${YELLOW}WARNING: $1${NC}"
}

print_error() {
    echo -e "${RED}ERROR: $1${NC}"
}

print_success() {
    echo -e "${GREEN}SUCCESS: $1${NC}"
}

deploy_to_host() {
    local HOST_IP=$1
    local HOST_NAME=$2

    print_header "Deploying to $HOST_NAME ($HOST_IP)"

    echo "1. Checking if 'at' command is available..."
    ssh root@$HOST_IP "command -v at > /dev/null" || {
        print_error "'at' command not found on $HOST_NAME"
        print_warning "Installing 'at' package..."
        ssh root@$HOST_IP "yum install -y at && systemctl enable --now atd" || {
            print_error "Failed to install 'at' on $HOST_NAME"
            return 1
        }
    }

    echo "2. Ensuring atd service is running..."
    ssh root@$HOST_IP "systemctl is-active atd > /dev/null || systemctl start atd" || {
        print_error "Failed to start atd service on $HOST_NAME"
        return 1
    }

    echo "3. Copying firewall configuration..."
    scp $FIREWALL_FILE root@$HOST_IP:/root/iptables.new || {
        print_error "Failed to copy file to $HOST_NAME"
        return 1
    }

    echo "4. Backing up current configuration..."
    ssh root@$HOST_IP "cp /etc/sysconfig/iptables /etc/sysconfig/iptables.$BACKUP_SUFFIX" || {
        print_error "Failed to backup on $HOST_NAME"
        return 1
    }

    echo "5. Creating rollback script..."
    ssh root@$HOST_IP "cat > /root/firewall-rollback.sh << 'ROLLBACK_EOF'
#!/bin/bash
# Automatic firewall rollback script
echo \"[\$(date)] FIREWALL ROLLBACK: Restoring previous configuration\" >> /var/log/firewall-rollback.log
cp /etc/sysconfig/iptables.$BACKUP_SUFFIX /etc/sysconfig/iptables
systemctl restart iptables
echo \"[\$(date)] FIREWALL ROLLBACK: Completed\" >> /var/log/firewall-rollback.log
ROLLBACK_EOF
chmod +x /root/firewall-rollback.sh" || {
        print_error "Failed to create rollback script on $HOST_NAME"
        return 1
    }

    echo "6. Scheduling automatic rollback in 2 minutes..."
    local AT_JOB_ID=$(ssh root@$HOST_IP "echo '/root/firewall-rollback.sh' | at now + 5 minutes 2>&1 | grep 'job' | awk '{print \$2}'")
    if [ -z "$AT_JOB_ID" ]; then
        print_error "Failed to schedule rollback on $HOST_NAME"
        return 1
    fi
    print_warning "Rollback scheduled with job ID: $AT_JOB_ID"
    print_warning "If SSH connection is lost, firewall will auto-rollback in 2 minutes!"

    echo "7. Installing new configuration..."
    ssh root@$HOST_IP "cp /root/iptables.new /etc/sysconfig/iptables" || {
        print_error "Failed to install new config on $HOST_NAME"
        return 1
    }

    echo "8. Restarting iptables service..."
    ssh root@$HOST_IP "systemctl restart iptables" || {
        print_error "Failed to restart iptables on $HOST_NAME"
        print_warning "Manual rollback will occur in 2 minutes..."
        return 1
    }

    echo "9. Waiting 5 seconds before testing SSH..."
    sleep 5

    echo "10. Verifying SSH access..."
    if ssh root@$HOST_IP "echo 'SSH test successful'" 2>/dev/null; then
        print_success "SSH verification successful!"
    else
        print_error "SSH verification failed on $HOST_NAME"
        print_warning "Automatic rollback will occur in ~2 minutes"
        print_warning "You can also use console access to verify or manually rollback"
        return 1
    fi

    echo "11. Canceling automatic rollback..."
    ssh root@$HOST_IP "atrm $AT_JOB_ID" || {
        print_warning "Failed to cancel rollback job - but SSH works, so manual cancellation recommended"
    }
    print_success "Automatic rollback cancelled - firewall is stable!"

    echo "12. Displaying active rules..."
    ssh root@$HOST_IP "iptables -L RH-Firewall-1-INPUT -n | head -20"

    print_success "Deployment to $HOST_NAME completed!"
    echo ""
    return 0
}

test_connectivity() {
    local HOST_IP=$1
    local HOST_NAME=$2

    echo "Testing connectivity to $HOST_NAME..."
    if ping -c 2 $HOST_IP > /dev/null 2>&1; then
        print_success "$HOST_NAME is reachable"
    else
        print_error "$HOST_NAME is not reachable"
        return 1
    fi

    if ssh -o ConnectTimeout=5 root@$HOST_IP "echo 'SSH OK'" > /dev/null 2>&1; then
        print_success "SSH to $HOST_NAME is working"
    else
        print_error "SSH to $HOST_NAME failed"
        return 1
    fi

    return 0
}

show_usage() {
    echo "Usage: $0 [test|x1|x2|x3|all|rollback]"
    echo ""
    echo "Commands:"
    echo "  test     - Test connectivity to all hosts without deploying"
    echo "  x1       - Deploy to x1 only ($X1_IP)"
    echo "  x2       - Deploy to x2 only ($X2_IP)"
    echo "  x3       - Deploy to x3 only ($X3_IP)"
    echo "  all      - Deploy to all hosts (x1, x2, x3)"
    echo "  rollback - Manually rollback firewall on specified host"
    echo ""
    echo "Examples:"
    echo "  $0 x3                    # Test deployment on x3 first"
    echo "  $0 all                   # Deploy to all hosts"
    echo "  $0 rollback x3           # Manually rollback x3"
    echo ""
    echo "Safety Features:"
    echo "  - Automatic rollback scheduled for 2 minutes after deployment"
    echo "  - Rollback is cancelled only if SSH verification succeeds"
    echo "  - If you lose SSH access, firewall auto-reverts in 2 minutes"
    exit 1
}

manual_rollback() {
    local HOST_IP=$1
    local HOST_NAME=$2

    if [ -z "$HOST_IP" ]; then
        print_error "Please specify host: x1, x2, or x3"
        echo "Example: $0 rollback x3"
        exit 1
    fi

    print_header "Manual Rollback on $HOST_NAME ($HOST_IP)"

    echo "1. Finding latest backup..."
    local BACKUP_FILE=$(ssh root@$HOST_IP "ls -t /etc/sysconfig/iptables.backup.* 2>/dev/null | head -1")

    if [ -z "$BACKUP_FILE" ]; then
        print_error "No backup file found on $HOST_NAME"
        echo "Available files:"
        ssh root@$HOST_IP "ls -la /etc/sysconfig/iptables*"
        return 1
    fi

    echo "Latest backup: $BACKUP_FILE"

    echo "2. Restoring backup..."
    ssh root@$HOST_IP "cp $BACKUP_FILE /etc/sysconfig/iptables" || {
        print_error "Failed to restore backup"
        return 1
    }

    echo "3. Restarting iptables..."
    ssh root@$HOST_IP "systemctl restart iptables" || {
        print_error "Failed to restart iptables"
        return 1
    }

    echo "4. Verifying SSH access..."
    sleep 2
    ssh root@$HOST_IP "echo 'SSH test successful'" || {
        print_error "SSH verification failed - you may need console access"
        return 1
    }

    print_success "Rollback completed successfully on $HOST_NAME!"

    echo "5. Current rules:"
    ssh root@$HOST_IP "iptables -L RH-Firewall-1-INPUT -n | head -10"
}

# Main script
if [ ! -f "$FIREWALL_FILE" ]; then
    print_error "Firewall configuration file '$FIREWALL_FILE' not found!"
    exit 1
fi

case "$1" in
    test)
        print_header "Testing Connectivity"
        test_connectivity $X1_IP "x1" || echo ""
        test_connectivity $X2_IP "x2" || echo ""
        test_connectivity $X3_IP "x3" || echo ""
        ;;
    x1)
        deploy_to_host $X1_IP "x1"
        ;;
    x2)
        deploy_to_host $X2_IP "x2"
        ;;
    x3)
        deploy_to_host $X3_IP "x3"
        ;;
    rollback)
        case "$2" in
            x1)
                manual_rollback $X1_IP "x1"
                ;;
            x2)
                manual_rollback $X2_IP "x2"
                ;;
            x3)
                manual_rollback $X3_IP "x3"
                ;;
            *)
                manual_rollback "" ""
                ;;
        esac
        ;;
    all)
        print_warning "This will deploy to ALL hosts!"
        read -p "Are you sure? (yes/no): " confirm
        if [ "$confirm" != "yes" ]; then
            echo "Deployment cancelled."
            exit 0
        fi

        deploy_to_host $X3_IP "x3" || exit 1
        sleep 2
        deploy_to_host $X1_IP "x1" || exit 1
        sleep 2
        deploy_to_host $X2_IP "x2" || exit 1

        print_header "Deployment Summary"
        print_success "All hosts deployed successfully!"
        echo ""
        echo "Next steps:"
        echo "1. Test VM internet connectivity on all hosts"
        echo "2. Test VM migration between hosts"
        echo "3. Disable Hetzner vSwitch firewall"
        ;;
    *)
        show_usage
        ;;
esac

bleader

@ditzy-olive Hello, be careful with that, your script installs it to /etc/sysconfig/iptables which is part of the iptables-services package, so it could be overwritten by a package update.

Granted, I don't think we ever updated it, but an upgrade to a newer version (when v9.0 comes) will for sure replace it.

ditzy-olive

@bleader thanks for the tip. You see my knowledge in redhat based systems is almost non existant. Think I'm almost 99% debian and 1% ubuntu (and that not by choice)
So I will take care of redeploying the script as soon as I upgrade, should be easily noticeable since monitoring will loose access and yell at me.