How can I avoid VRRP packet sending failure when hot loading firewall？

[qqliuxiaoran]

Hello,
My notify_master script will hot reload firewalld(firewall-cmd --reload) when keepalived enter MASTER state.
I have noticed that this behavior can lead to a failure in sending vrrp packets, with an impact time of approximately 2-4 seconds and the loss of several vrrp packets (advert_int 1)

My keepalived configuration is quite simple

global_defs {
  
}

vrrp_instance product {
    state BACKUP
    interface enp65s0f1
    virtual_router_id 114
    priority 100
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass ******
    }
    virtual_ipaddress {
        myVIP/24 dev enp65s0f1
    }

    unicast_src_ip xx.xxx.x.xx1
    unicast_peer {
     xx.xxx.x.xx2
   }

   # reload firewalld when switch to master
   notify_master "/etc/product/bin/reload_firewalld.sh"   
}

How can I avoid VRRP packet sending failure when hot loading firewall？
Looking forward to your reply, thank you!

[pqarmitage]

keepalived runs the notify_master script at the same time as it decides to become master and start sending adverts, so I can only assume that the reload_firewalld.sh script (and this is possibly due to firewall-cmd) takes several seconds to execute.

So far as I can see there are 2 possibilities.

1. Your firewall configuration does not allow sending VRRP packets prior to the reload and so the adverts are rejected until the firewall configuration is complete
2. While firewall-cmd --reload is executing, the current firewall configuration is first deleted, causing VRRP packets to be rejected until the new configuration is loaded.

In order for the VRRP protocol to work properly it is essential that VRRP packets can be sent as soon as keepalived decides that a VRRP instance should become master (otherwise another device may decide to become master if there are more than two VRRP systems involved).

What are your reasons for reloading firewalld? I think you need to look at that first and probably ensure that your firewall configuration is permanently set up so that you don't need to hot load the firewall (I note that you don't have a notify_backup script to update the firewall when transitioning to backup state, so why does the firewall need reloading when transitioning to master state?). Alternatively can you just issue the necessary commands to change the firewall configuration, rather than doing a full reload?

[qqliuxiaoran]

Thank you for your reply！
My application provides users with the ability to modify firewall configurations. When configuration changes, a 'File Change Listener' will use rsync to transfer firewall file to the backup machine. So, when backup machine switch to MASTER state, it needs to hot reload firewall.

By the way, When master firewall configuration changes, my application also execute firewall-cmd --reload.

I try pre execute iptables -I OUTPUT -d xxxx -p vrrp -j ACCEPT but not work when hot loading firewalld. Perhaps the key to solving the problem is to understand the working mechanism of the firewalld.

[pqarmitage]

@qqliuxiaoran Is it possible to make the backup machine reload the firewall configuration immediately after the rsync, rather than waiting for the backup machine switching to MASTER state? The problem seems to be reloading the firewall configuration at the same time as transitioning to master.

We have in the past seen problems with systems not being able to receive VRRP adverts until they become master and send an advert; once they have sent an advert, they can start receiving them and if they are lower priority than the true master, they then revert to backup state. This appears to be due to their initially being no conntrack entry for the VRRP adverts, and only once an entry has been created due to sending an advert can adverts then be received (their is probably a rule something like -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT that allows receipt of adverts once one has been sent). The way around this problem is to have an explicit rule in the firewall to allow VRRP adverts to be received.

Do you know how long the firewall-cmd reload takes to execute? You state that adverts cannot be sent for 2-4 seconds, and so presumably the firewall reload is taking that long. If the reload is taking 2-4 seconds, then reloading the firewall when keepalived transitions to master is just not going to work. I have had a quick look at the firewall-cmd code, and it seems to be doing quite a lot during a reload.

[qqliuxiaoran]

Many thanks for your reply!

Is it possible to make the backup machine reload —— Yes, it help not happen vrrp package loss when backup switch to master, i'm working for some code changes now

Do you know how long the firewall-cmd reload takes to execute? —— Very fast. After typing the command and entering, it was processed and exited quickly, about 2 seconds

We have in the past seen problems with systems not being able to receive VRRP adverts until they become master and send an advert; once they have sent an advert, they can start receiving them and if they are lower priority than the true master, they then revert to backup state. This appears to be due to their initially being no conntrack entry for the VRRP adverts, and only once an entry has been created due to sending an advert can adverts then be received (their is probably a rule something like -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT that allows receipt of adverts once one has been sent). The way around this problem is to have an explicit rule in the firewall to allow VRRP adverts to be received. —— I feel a bit understanding now After reading your case and explanation， Maybe I should take the time to see how the firewall works.