while upgrading 4.52 zookeeper not starting, in logs IOException, no snapshot found

[hemanthteejay]

@anilsagar @dchiesa1 

I tried upgrading Apigee version 4.51 to 4.52. here after upgrading apigee-setup I ran command to upgrade cs and zk, both were upgraded but zookeeper is not starting and while checking the logs it is showing as "IOException, no snapshot found, but there are log entries, something is broken!"

Can someone help resolve this issue

[neeldey]

Hello,

Can you try the following?

1. On the ZK node where this is occurring, navigate to "/opt/apigee/customer/application"

2. Create or edit file "zookeeper.properties" and add line "lib_settings_zk_jvm_opts=-Dzookeeper.snapshot.trust.empty=true"

3. Save the file and ensure it is readable by "apigee" user. If needed, change ownership to apigee user. "chown apigee:apigee /opt/apigee/customer/application/zookeeper.properties"

4. Start apigee-zookeeper. "apigee-service apigee-zookeeper start". It should now come up.

5. Once ZK comes up, you can decide to leave this property as is or remove this property from the file. If the zookeeper.properties file didn't exist earlier and you created it in #2, you can delete this file. If you decide to remove the property or the file, restart apigee-zookeeper for the change to take effect.

6. Repeat this procedure on any ZK node where issue is occurring.

Note: The ZK upgrade script should have already taken care of this. I cannot say why the upgrade let this error go through. If possible, can you share list of files in "/opt/apigee/data/apigee-zookeeper/data/version-2/" and also relevant ZK logs from the date upgrade was tried? You can redact any sensitive details like IP addresses from logs before posting here. The internal Google bug which concerns this is 237080432 for future reference.