How to cleanup old router uuid definitions?

Not applicable

Recently our deployments have started failing on our internal OPDK cluster with 6 routers/message processors. The deployment API is returning:

...    
}, {
      "status" : "undeployed",
      "type" : [ "router" ],
      "uUID" : "3322d9ce-66cd-4c84-9e1f-236518062ed4"
    }, {
      "status" : "undeployed",
      "type" : [ "router" ],
      "uUID" : "8ca2c2bf-982c-4ae1-8b8a-6a9003b1f041"
    }, {
      "status" : "undeployed",
      "type" : [ "router" ],
      "uUID" : "2f63ab90-221d-4054-a96a-66c3c79bdb64"
    }, {
      "error" : "com.apigee.zookeeper.ZooKeeperException{ code = zookeeper.ErrorSettingData, message = Error while setting data for path : /organizations/MYORG/environments/prod/apiproxies/MYPROXY/revisions/3/statuses/78ebd00d-2663-4596-a2de-27df6c47e93a/status, associated contexts = []}",
      "status" : "error",
      "type" : [ "router" ],
      "uUID" : "78ebd00d-2663-4596-a2de-27df6c47e93a"
    }, {
      "status" : "undeployed",
      "type" : [ "router" ],
      "uUID" : "a1802e7d-8c23-47e7-8abb-e32cb97c13f1"
    }, {
      "status" : "undeployed",
      "type" : [ "router" ],
      "uUID" : "4a4b2e88-c486-4381-8f40-ae627e197205"
    }, {
      "error" : "com.apigee.kernel.exceptions.spi.UncheckedException{ code = application.bootstrap.FailedToConfigure, message = Configuration failed, associated contexts = []}",
      "status" : "error",
      "type" : [ "router" ],
      "uUID" : "fe55b507-9399-40bf-9b2d-bd99ff088a25"
    } ],
    "state" : "error"
  } ],

There's deployment errors on two routers. But, it's listing 7 routers when we only have 6. The management server API GET /v1/o/MYORG/e/prod/servers returns a list of servers, where only 6 are routers.

We've tried manually deleting the extraneous router guid, restarting zookeeper, restarting routers/message processors, but the deploys still fail with the same error. It's trying to deploy to routers which don't exist.

How can we properly clean up these extraneous routers? Is there some cache in front of zookeeper that we need to clear?

This is blocking our ability to perform deployments.

0 3 829
3 REPLIES 3

remeeshnair
Participant IV

Hi Eric,

Please find attached, are the steps we have used to clean up the old RMP nodes from registration as advised by apigee customer success team. We have 5 router multi DC set up. But if you are attempting this on production environments I would suggest to get it verified with Apigee Customer Success team before attempting.

Regards,

Remeesh.

Thanks. I think we'll try this. How would we register the router again? Will it register itself on start-up?

adas
Participant V

You can make the following call to delete the stale uuid entry:

curl -v http://mgmtIp:mgmtPort/v1/servers/{uuid} -X DELETE

This would DELETE the stale entry from zookeeper and your management api call for deployment would no longer attempt to deploy the apiproxy to that router any more. Hope this helps.