Apigee Shared Flows and Products missing after upgrade from 4.19.06 to 4.51.00

Hi All,

We have upgraded Apigee from 4.19.06 to 4.51.00 recently and we could see the data intact under Admin Tab but all other data under APIs and Publish Product or Developers Tab is missing. The error popping up on UI reads that "Organization does not exist". We could see the organization exists in Cassandra but it does not seem to be in sync with UI. Can anyone help on how to resolve this issue?

What would have broken while upgrade due to which we could not see APIs Proxies/shared flows etc on Apigee Dashboard even when everything seems to be in place on cassandra nodes. Also, we did not get any error while performing upgrade.

Do let me know if I need to share any further details to get help on this scenario. Any pointers will be appreciated.

0 2 112
2 REPLIES 2

It seems like there might have been a synchronization issue between the Apigee UI and the underlying Cassandra data store after the upgrade. Di you review the upgrade logs to see if there were any errors during the upgrade process & also run below management api's to validate organization.

https://apidocs.apigee.com/operations#organizations

Also check cassandra logs & perform  https://docs.apigee.com/private-cloud/v4.50.00/apache-cassandra-maintenance-tasks

IMP: Post upgrade you will need to restart all components as per https://docs.apigee.com/private-cloud/v4.52.00/update-apigee-edge-45100-45200

It is hard with out enough information..

Thanks for looking into this. I have reviewed the upgrade logs and there were no errors found. I have already run the suggested management api's and could see the organization details but somehow it is not syncing with edge-ui. I have run cassandra maintenance on all the cassandra nodes. PFA output of repair command. Also, when i am executing zktree I am able to see the proxies. PFB sample below.

Sample proxies in zktree: organizations/ihsmarkit/environments/dev/apiproxies/APICoreRequest/revisions/1/spec/_deploymentBean: {"organization":"ihsmarkit","application":"APICoreRequest","revision":"1","path":"/","env":"dev","trafficPct":1,"hasPath":true,"subType":"SharedFlow","configVersion":"SHA-512:2f788f955995523bf797e236e02e565c65c64c3ff32c63845f355ec31b8c1f8ed399e4db7a919af0a4b196f79d028895b21e4c49f37d1f58af3371f17494c968","hostedTargetRoutingDetails":null,"bundleStorageInfo":null,"servers":[],"abprimaryRevision":null,"abtest":false}
/organizations/ihsmarkit/environments/dev/apiproxies/xref: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses/8d973c35-e6ef-48d0-9a3c-9d5bf533afef: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses/8d973c35-e6ef-48d0-9a3c-9d5bf533afef/status: success
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses/115d18f5-9689-4305-ad96-6ee6c2b66102: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses/115d18f5-9689-4305-ad96-6ee6c2b66102/status: success
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses/57f8d37d-e7c1-47c9-9691-5e38a50b81cb: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses/57f8d37d-e7c1-47c9-9691-5e38a50b81cb/status: success
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses/7636793d-e001-47b8-bc3e-7107cd9179dd: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/statuses/7636793d-e001-47b8-bc3e-7107cd9179dd/status: success
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/state: deployed
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/spec: 
/organizations/ihsmarkit/environments/dev/apiproxies/xref/revisions/1/spec/spec.xml: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Configuration>
<ConfigVersion>SHA-512:b16951a4c07fea50af6f6405e8a9b63af953ee35d661d47e801fe03ee905e7d712e1dc81dd221a038f50d607cfc87111c99a42c2b97ed8d3f4ecf987a4517a43</ConfigVersion>
<BasePath>/</BasePath>
</Configuration>

 

PFB output of cassandra repair on one node:

root@md-xxxx:/home/xxxx # /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra apigee_repair -pr
Storage Used: 10%
Nodetool Repair Started . . .
[2023-04-27 08:27:18,270] Starting repair command #1, repairing 1 ranges for keyspace apimodel (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:18,374] Repair session 52911440-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:18,374] Repair command #1 finished
[2023-04-27 08:27:18,381] Starting repair command #2, repairing 1 ranges for keyspace analytics (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:18,397] Repair session 52a140e0-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:18,397] Repair command #2 finished
[2023-04-27 08:27:18,401] Starting repair command #3, repairing 1 ranges for keyspace counter (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:18,426] Repair session 52a42710-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:18,426] Repair command #3 finished
[2023-04-27 08:27:18,431] Starting repair command #4, repairing 1 ranges for keyspace devconnect (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:18,443] Repair session 52a8baf0-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:18,443] Repair command #4 finished
[2023-04-27 08:27:18,449] Starting repair command #5, repairing 1 ranges for keyspace audit (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,055] Repair session 52ab7a10-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,055] Repair command #5 finished
[2023-04-27 08:27:19,060] Starting repair command #6, repairing 1 ranges for keyspace cache (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,085] Repair session 5308b540-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,085] Repair command #6 finished
[2023-04-27 08:27:19,089] Nothing to repair for keyspace 'system'
[2023-04-27 08:27:19,093] Starting repair command #7, repairing 1 ranges for keyspace user_settings (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,099] Repair session 530dbe50-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,099] Repair command #7 finished
[2023-04-27 08:27:19,103] Starting repair command #8, repairing 1 ranges for keyspace dek (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,108] Repair session 530f44f0-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,108] Repair command #8 finished
[2023-04-27 08:27:19,112] Starting repair command #9, repairing 1 ranges for keyspace apprepo (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,200] Repair session 5310a480-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,200] Repair command #9 finished
[2023-04-27 08:27:19,204] Starting repair command #10, repairing 1 ranges for keyspace keyvaluemap (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,215] Repair session 531eae40-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,215] Repair command #10 finished
[2023-04-27 08:27:19,218] Starting repair command #11, repairing 1 ranges for keyspace kms (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,262] Repair session 5320f830-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,262] Repair command #11 finished
[2023-04-27 08:27:19,266] Starting repair command #12, repairing 1 ranges for keyspace identityzone (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,273] Repair session 53282420-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,273] Repair command #12 finished
[2023-04-27 08:27:19,277] Nothing to repair for keyspace 'system_auth'
[2023-04-27 08:27:19,279] Starting repair command #13, repairing 1 ranges for keyspace system_traces (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,283] Repair session 532a1ff0-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,283] Repair command #13 finished
[2023-04-27 08:27:19,287] Starting repair command #14, repairing 1 ranges for keyspace auth (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,292] Repair session 532b5870-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,292] Repair command #14 finished
[2023-04-27 08:27:19,295] Starting repair command #15, repairing 1 ranges for keyspace ax_custom_report_model (parallelism=SEQUENTIAL, full=true)
[2023-04-27 08:27:19,314] Repair session 532c90f0-e4d5-11ed-b5bf-9dfb03180332 for range (113427455640312821154458202477256070485,0] finished
[2023-04-27 08:27:19,314] Repair command #15 finished