Data transfer from one cluster to another

Not applicable

Dear All,

I have a 13 Node cluster (15.07 version) which we'll be upgrading soon. We also aim to increase number of nodes Below is how we are aiming it:

1. We plan to create a 18 Node cluster.

2. Migrate the data (cassandra, ldap & postgres) from 13 Node to 18 Node.

3. Once testing is successful after data migration we will point our ELB's to 18 Node.

4. After monitoring for few days we will sunset our 13 Node servers.

Does anyone know how to migrate data from one cluster to another cluster ?

I've tried backup & restore and it's not working. I suspect backup and restore also restores UUID's hence it is creating issues.

I've also tried pg_dump for postgres but it was not working as well. Or maybe i didn't used it currently ???

Also we require steps for openldap.

Will be grateful if someone has tried it and can share the steps.

1 6 2,584
6 REPLIES 6

Two good ways to tackle this are with backup and restore or by migrating data at the management API level.

If you use backup and restore, you need to mirror the data from your 13 node cluster to 13 nodes in your 18 node cluster as appropriate, then build out the remaining 5 nodes and join them to the existing planet. There are some limited situations where you might be able to move the data to different nodes in the new cluster and build something that doesn't look like the source cluster, but it's safest to restore to a cluster that is identical to your source cluster and then add nodes from there.

You could also use the organization migration tool at https://github.com/apigeecs/apigee-migrate-tool to export and import data at the application level. Using this method, you would simply build your 18-node cluster from scratch and then import your data through the management API. Note that you would not carry over some data such as stored analytics data using this method. You could dump your PostgreSQL database and restore to the new cluster if needed.

Hi Paul,

Thanks a lot for your reply. I have already successfully tested:

https://github.com/apigeecs/apigee-migrate-tool

But this only transfers one org at a time. If we have large number of orgs then it's a issue. Also i aim to migrate LDAP & Postgres Data.

I used the pg_dump utility to transfer Postgres Data but after that my proxies from both source and destination cluster stopped to work. When i try to hit the proxies it give "this site is not secure" error.

A message indicating that the site is not secure is probably coming from your browser in response to a self signed certificate. Are you using SSL to secure your virtual hosts? Did you import your certificates into the new target installation?

Hi Paul,

I've enabled SSL for both the clusters. Also I've done Virtualhost configurations using same certificates.

FYI, both the clusters have same admin email, admin password, ldap password. Below is a quick summary of this problem:

1. I took pg_dump from source.

2. I restored the pg_dump to the destination. Once i restored the pg_dump the source cluster proxies started to give "secure connection error"

3. The proxies in the destinationc cluster was working & i was able to see the analytics data of source cluster in the dashboard.

4. The analytics data was not changing in destination cluster even though proxies were hitting. hence i restarted the management & qpid services.

5. After services restart there was a change in analytics data in the dashboard but after that even the proxies in destination cluster started to give secure connection error.

6. One more thing the analytics data in both the source and destination are not changing eventhough proxies are getting failed. It should atleast show the failed proxies data.

Any thoughts how i can resolve this error ?

Best Regards,

Kris

JFYI, I have a support case also in place :

1063713

I've updated the case with my recent comment.

A support case is the best way to pursue this for now. It sounds like there may be a few factors involved and a troubleshooting session is the best way to discover which pieces are working and which are not.