How can we clear stale entries in zookeeper ?

Deploying a proxy sometimes can take more time than usual. When the management server logs are checked - the following errors are observed :

"ConfigurationService.validateEvent() - Unable to find application {} in apprepo for organization {} but entry exists in zk, ignoring it. " + "Please use force undeploy to remove these stale entries"

The option of force undeployment is suggested to remove the stale entries.

Why are stale entries built up in ZK ?

Is there a cleaner way of clearing them other than force undeployment

Solved Solved
1 4 1,800
1 ACCEPTED SOLUTION

Not applicable

Hi @Venkataraghavan , App bundles are stored in C* where as active revision details are stored in ZK , During undeployment the corresponding entries get deleted from both C* and ZK .

AFAIK If there are any network issues or time outs to ZK , this issue might happen .

Force delete will make sure the stale or missed details are deleted .

Force un deployment is the cleaner approach in this context however you can definitely delete the node from ZK which is not recommended but you can rely on that option as your last if nothing works out .

View solution in original post

4 REPLIES 4

Not applicable

Hi @Venkataraghavan , App bundles are stored in C* where as active revision details are stored in ZK , During undeployment the corresponding entries get deleted from both C* and ZK .

AFAIK If there are any network issues or time outs to ZK , this issue might happen .

Force delete will make sure the stale or missed details are deleted .

Force un deployment is the cleaner approach in this context however you can definitely delete the node from ZK which is not recommended but you can rely on that option as your last if nothing works out .

Thanks @Maruti Chand - one last question - Are there any other events besides undeployment that would lead to the entries getting deleted in C* ?

were you able to get any information in this issue ?

You could try to remove stale entries in ZooKeeper using following approach:

1. Find all servers of OPDK deployment and their UUIDs via the admin API:

curl -u sysadmin@email.com "http://management-server-host:8080/v1/servers?pod=gateway&region=region-name"
curl -u sysadmin@email.com "http://management-server-host:8080/v1/servers?pod=central&region=region-name"
curl -u sysadmin@email.com "http://management-server-host:8080/v1/servers?pod=analytics&region=region-name"

2. Check the current ZooKeeper Tree:

ssh {zookeeper-host}
/opt/apigee/apigee-zookeeper/contrib/zk-tree.sh

3. Take backups of all ZooKeeper nodes:

https://docs.apigee.com/private-cloud/v4.19.01/how-perform-backup

4. Identify any stale entries and remove them:

ssh {zookeeper-host}
/opt/apigee/apigee-zookeeper/bin/zkCli.sh 
ls [path] 
rmr [path]