Upgrading Apigee Hybrid in a Multi-Region environment

[rhawley]

I am going to be updating an Apigee Hybrid 1.4.2 to Apigee Hybrid 1.5.3, on Anthos GKE-On-Prem 1.7.0.  Lower environments have upgraded successfully, but I have not seen anything in the upgrade documentation about multi-region environments.  We have two regional clusters in production, with Cassandra replicated between them.  Do I upgrade them one at a time? Do I need to stop one of the clusters while I upgrade the other?  Would appreciate any insights from others with this same setup.

[sunilomanakutta]

We have similar setup in one of our projects. 

We have not upgraded both the clusters but once we had to reinstall one of the clusters.   Upgrade is something we are planning in near future as well.

When we did reinstall of one of the clusters (dc2)- we first drained the  cassandra nodes  (nodetool) in the  dc2 and then we did reinstall.  Once it cluster was up - for apigee we rejoined the dc1 cassandra ring as per the steps given in(Multi-region deployment on GKE and GKE on-prem  |  Apigee X (google.com)).  Please note this was not an upgrade - but a reinstall.

For upgrade you can do one cluster at a time. You should not be needing to do anything extra(like draining nodes etc) as upgrade can be done inplace. It should not cause any problem since Cassandra version is same in Apigee hybrid 1.4.x as well as in Apigeehybrid 1.5.3 ( which is 3.11.6-- Apigee hybrid: supported platforms  |  Apigee X  |  Google Cloud) . Also recommend that since this is production you can confirm the steps with APIGEE support before you do it.

[sunilomanakutta · Accepted solution]

Just adding to the earlier post - we did the upgrade ( GKE onPrem  + APIGEE Hybrid)  in PROD- one cluster at a time - in place- infact we had around 3-4 days between the 2 cluster(/regions) upgrade.  Most of the workloads have RollingUpgrade (especially Cassandra) configured - so you will see minimal disruption.  For  added buffer  we had traffic routed to second cluster while doing first cluster upgrade. 

Please note we also confirmed with support on the steps before upgrade. We also confirmed with support that there were no schema differences  ( for cassandra) between the versions that we upgraded (1.4.1 to 1.5.5). GKE was upgraded to 1.7.5 from 1.5.2 ( also we upgraded  ASM to 1.8.6 from 1.7.6)