Error fetching Status Datastore not added to the pods of the organization

Not applicable

I have a multi-dc on prem install that displays "Error fetching Status Datastore not added to the pods of the organization" upon login to ui in either dc. Any ideas what needs to happen? Do I need to associate the datastore in a multi-dc install? I've never seen this message on an install in a single dc environment.

0 12 722
12 REPLIES 12

Not applicable

@Michael McCann , I have never seen that error message but if you run the below 2 APIs on one for your mgmt servers and share the output, I can compare with my working 2 dc setup.

curl -v http://localhost:8080/v1/servers -u maruthi

curl -v http://localhost:8080/v1/servers?region=dc-2 -u maruthi

Here is servers

> GET /v1/servers HTTP/1.1 > Authorization: Basic YXBpZ2VlQGRpcmVjdHYuY29tOkBwIWczM2R0dg== > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < Date: Thu, 24 Mar 2016 20:47:26 GMT < Content-Length: 3524 < [ { "internalIP" : "147.22.34.46", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "06c7f58a-47d9-475d-80cc-7d9bbefdf9c4" }, { "externalHostName" : "localhost", "externalIP" : "147.22.34.47", "internalHostName" : "localhost", "internalIP" : "147.22.34.47", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ { "name" : "started.at", "value" : "1458844992378" }, { "name" : "startup.time", "value" : "16 second(s)" }, { "name" : "Profile", "value" : "ManagementServer" }, { "name" : "rpc.port", "value" : "4526" }, { "name" : "http.management.port", "value" : "8080" }, { "name" : "jmx.rmi.port", "value" : "1099" }, { "name" : "up.time", "value" : "2 hour(s) 4 minute(s) 14 second(s)" } ] }, "type" : [ "management-server" ], "uUID" : "dc25a06b-ceb1-438e-9f38-06c56af5114a" }, { "externalHostName" : "localhost", "externalIP" : "147.22.34.47", "internalHostName" : "localhost", "internalIP" : "147.22.34.47", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ { "name" : "started.at", "value" : "1458845965084" }, { "name" : "startup.time", "value" : "15 second(s)" }, { "name" : "Profile", "value" : "ManagementServer" }, { "name" : "rpc.port", "value" : "4526" }, { "name" : "http.management.port", "value" : "8080" }, { "name" : "jmx.rmi.port", "value" : "1099" }, { "name" : "up.time", "value" : "1 hour(s) 48 minute(s) 1 second(s)" } ] }, "type" : [ "management-server" ], "uUID" : "b2697756-a434-470d-bab0-e7fd3f62a496" }, { "externalHostName" : "localhost", "externalIP" : "147.22.34.4", "internalHostName" : "localhost", "internalIP" : "147.22.34.4", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457119836341" }, { "name" : "startup.time", "value" : "7 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4529" }, { "name" : "http.management.port", "value" : "8083" }, { "name" : "jmx.rmi.port", "value" : "1102" }, { "name" : "up.time", "value" : "481 hour(s) 16 minute(s) 50 second(s)" } ] }, "type" : [ "qpid-server" ], "uUID" : "dee80332-99c3-4dd7-bba5-76963c51cbd1" }, { "internalIP" : "147.22.34.44", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "ec9191bd-5174-4da6-b947-25f77dc58694" }, { "internalIP" : "147.22.34.42", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "a76ab194-a60d-48d2-b262-64ca01ab2f25"

Here is servers dc-2

> GET /v1/servers?region=dc-2 HTTP/1.1 > Authorization: Basic YXBpZ2VlQGRpcmVjdHYuY29tOkBwIWczM2R0dg== > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < Date: Thu, 24 Mar 2016 20:49:34 GMT < Content-Length: 5201 < [ { "externalHostName" : "localhost", "externalIP" : "10.55.2.189", "internalHostName" : "localhost", "internalIP" : "10.55.2.189", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1458839942522" }, { "name" : "startup.time", "value" : "14 second(s)" }, { "name" : "Profile", "value" : "ManagementServer" }, { "name" : "rpc.port", "value" : "4526" }, { "name" : "http.management.port", "value" : "8080" }, { "name" : "jmx.rmi.port", "value" : "1099" }, { "name" : "up.time", "value" : "3 hour(s) 30 minute(s) 31 second(s)" } ] }, "type" : [ "management-server" ], "uUID" : "4c6113ff-167b-4c33-ac94-cdd88b9c6b1b" }, { "externalHostName" : "localhost", "externalIP" : "10.55.2.198", "internalHostName" : "localhost", "internalIP" : "10.55.2.198", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457120501316" }, { "name" : "startup.time", "value" : "7 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4529" }, { "name" : "http.management.port", "value" : "8083" }, { "name" : "jmx.rmi.port", "value" : "1102" }, { "name" : "up.time", "value" : "481 hour(s) 7 minute(s) 52 second(s)" } ] }, "type" : [ "qpid-server" ], "uUID" : "aa5085fa-4a64-43d7-a37d-9ba6d1acd24c" }, { "internalIP" : "10.55.2.195", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "29d53d91-9d8b-42e8-bf81-9f6be9db2e76" }, { "externalHostName" : "localhost", "externalIP" : "147.22.32.103", "internalHostName" : "localhost", "internalIP" : "147.22.32.103", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457392748891" }, { "name" : "startup.time", "value" : "15 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4529" }, { "name" : "http.management.port", "value" : "8083" }, { "name" : "jmx.rmi.port", "value" : "1102" }, { "name" : "up.time", "value" : "405 hour(s) 30 minute(s) 25 second(s)" } ] }, "type" : [ "qpid-server" ], "uUID" : "fd354de1-847c-4b8e-8443-64062fa54d96" }, { "internalIP" : "10.55.2.197", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "c0004f1c-764f-4d14-bb98-012f5059f867" }, { "internalIP" : "10.55.2.196", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "959d3c4d-0b70-4d7b-bfde-49699ae3fd5a" }, { "externalHostName" : "localhost", "externalIP" : "10.55.2.18", "internalHostName" : "localhost", "internalIP" : "10.55.2.18", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457388497164" }, { "name" : "Profile", "value" : "ManagementServer" }, { "name" : "startup.time", "value" : "15 second(s)" }, { "name" : "rpc.port", "value" : "4526" }, { "name" : "http.management.port", "value" : "8080" }, { "name" : "jmx.rmi.port", "value" : "1099" }, { "name" : "up.time", "value" : "406 hour(s) 41 minute(s) 17 second(s)" } ] }, "type" : [ "management-server" ], "uUID" : "8fb9a217-150e-414e-8037-16fd7a195901" }, { "externalHostName" : "localhost", "externalIP" : "10.55.2.17", "internalHostName" : "localhost", "internalIP" : "10.55.2.17", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457388816345" }, { "name" : "startup.time", "value" : "7 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4529" }, { "name" : "http.management.port", "value" : "8083" }, { "name" : "jmx.rmi.port", "value" : "1102" }, { "name" : "up.time", "value" : "406 hour(s) 35 minute(s) 57 second(s)" } ] }, "type" : [ "qpid-server" ], "uUID" : "1495d07f-a919-444f-bdf9-e218692cb291"

Here is servers

> GET /v1/servers HTTP/1.1 > Authorization: Basic > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < Date: Thu, 24 Mar 2016 20:56:08 GMT < Content-Length: 3526 < [ { "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "06c7f58a-47d9-475d-80cc-7d9bbefdf9c4" }, { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ { "name" : "started.at", "value" : "1458844992378" }, { "name" : "startup.time", "value" : "16 second(s)" }, { "name" : "Profile", "value" : "ManagementServer" }, { "name" : "rpc.port", "value" : "4526" }, { "name" : "http.management.port", "value" : "8080" }, { "name" : "jmx.rmi.port", "value" : "1099" }, { "name" : "up.time", "value" : "2 hour(s) 12 minute(s) 55 second(s)" } ] }, "type" : [ "management-server" ], "uUID" : "dc25a06b-ceb1-438e-9f38-06c56af5114a" }, { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ { "name" : "started.at", "value" : "1458845965084" }, { "name" : "startup.time", "value" : "15 second(s)" }, { "name" : "Profile", "value" : "ManagementServer" }, { "name" : "rpc.port", "value" : "4526" }, { "name" : "http.management.port", "value" : "8080" }, { "name" : "jmx.rmi.port", "value" : "1099" }, { "name" : "up.time", "value" : "1 hour(s) 56 minute(s) 43 second(s)" } ] }, "type" : [ "management-server" ], "uUID" : "b2697756-a434-470d-bab0-e7fd3f62a496" }, { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457119836341" }, { "name" : "startup.time", "value" : "7 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4529" }, { "name" : "http.management.port", "value" : "8083" }, { "name" : "jmx.rmi.port", "value" : "1102" }, { "name" : "up.time", "value" : "481 hour(s) 25 minute(s) 31 second(s)" } ] }, "type" : [ "qpid-server" ], "uUID" : "dee80332-99c3-4dd7-bba5-76963c51cbd1" }, { "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "ec9191bd-5174-4da6-b947-25f77dc58694" }, { "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "a76ab194-a60d-48d2-b262-64ca01ab2f25"

this is a test reply; please ignore.

this is a test reply; please ignore.

this is a test reply; please ignore.

Here is servers dc-2

> GET /v1/servers?region=dc-2 HTTP/1.1 > Authorization: Basic > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < Date: Thu, 24 Mar 2016 20:59:09 GMT < Content-Length: 5189 < [ { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1458839942522" }, { "name" : "startup.time", "value" : "14 second(s)" }, { "name" : "Profile", "value" : "ManagementServer" }, { "name" : "rpc.port", "value" : "4526" }, { "name" : "http.management.port", "value" : "8080" }, { "name" : "jmx.rmi.port", "value" : "1099" }, { "name" : "up.time", "value" : "3 hour(s) 40 minute(s) 6 second(s)" } ] }, "type" : [ "management-server" ], "uUID" : "4c6113ff-167b-4c33-ac94-cdd88b9c6b1b" }, { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457120501316" }, { "name" : "startup.time", "value" : "7 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4529" }, { "name" : "http.management.port", "value" : "8083" }, { "name" : "jmx.rmi.port", "value" : "1102" }, { "name" : "up.time", "value" : "481 hour(s) 17 minute(s) 27 second(s)" } ] }, "type" : [ "qpid-server" ], "uUID" : "aa5085fa-4a64-43d7-a37d-9ba6d1acd24c" }, { "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "29d53d91-9d8b-42e8-bf81-9f6be9db2e76" }, { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457392748891" }, { "name" : "startup.time", "value" : "15 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4529" }, { "name" : "http.management.port", "value" : "8083" }, { "name" : "jmx.rmi.port", "value" : "1102" }, { "name" : "up.time", "value" : "405 hour(s) 40 minute(s) " } ] }, "type" : [ "qpid-server" ], "uUID" : "fd354de1-847c-4b8e-8443-64062fa54d96" }, { "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "c0004f1c-764f-4d14-bb98-012f5059f867" }, { "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ ] }, "type" : [ "application-datastore", "auth-datastore", "apimodel-datastore", "user-settings-datastore", "audit-datastore" ], "uUID" : "959d3c4d-0b70-4d7b-bfde-49699ae3fd5a" }, { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457388497164" }, { "name" : "Profile", "value" : "ManagementServer" }, { "name" : "startup.time", "value" : "15 second(s)" }, { "name" : "rpc.port", "value" : "4526" }, { "name" : "http.management.port", "value" : "8080" }, { "name" : "jmx.rmi.port", "value" : "1099" }, { "name" : "up.time", "value" : "406 hour(s) 50 minute(s) 52 second(s)" } ] }, "type" : [ "management-server" ], "uUID" : "8fb9a217-150e-414e-8037-16fd7a195901" }, { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "central", "reachable" : true, "region" : "dc-2", "tags" : { "property" : [ { "name" : "started.at", "value" : "1457388816345" }, { "name" : "startup.time", "value" : "7 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4529" }, { "name" : "http.management.port", "value" : "8083" }, { "name" : "jmx.rmi.port", "value" : "1102" }, { "name" : "up.time", "value" : "406 hour(s) 45 minute(s) 32 second(s)" } ] }, "type" : [ "qpid-server" ], "uUID" : "1495d07f-a919-444f-bdf9-e218692cb291"

I see you have 1 qpid in dc-1 and 3 qpid in dc-2, I suspect something to do with analytics setup

but we can confirm if you can send the below output too

curl -v http://localhost:8080/v1/servers?pod=analytics -u maruthi

curl -v http://localhost:8080/v1/analytics/groups/ax -u maruthi

@Michael McCann

I only see 1 qpid in each dc when I look at servers. Maybe I'm missing something. I have one postgres server in each dc to go with each qpid. dc-1 is setup as the master.

Here is servers?pod=analytics

> GET /v1/servers?pod=analytics HTTP/1.1 > Authorization: Basic > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < Date: Fri, 25 Mar 2016 16:24:51 GMT < Content-Length: 1666 < [ { "externalHostName" : "localhost", "externalIP" : "", "internalHostName" : "localhost", "internalIP" : "", "isUp" : true, "pod" : "analytics", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ { "name" : "started.at", "value" : "1458057005551" }, { "name" : "startup.time", "value" : "5 second(s)" }, { "name" : "Profile", "value" : "AXPlatformAgentService" }, { "name" : "rpc.port", "value" : "4530" }, { "name" : "http.management.port", "value" : "8084" }, { "name" : "jmx.rmi.port", "value" : "1103" }, { "name" : "up.time", "value" : "240 hour(s) 34 minute(s) 45 second(s)" } ] }, "type" : [ "postgres-server" ], "uUID" : "19d7f4ff-c1e0-43be-9f21-5effc01b6f0c" }, { "internalIP" : "", "isUp" : true, "pod" : "analytics", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "reportcrud-datastore", "analytics-datastore" ], "uUID" : "567fd352-0bc6-4332-9047-dc91384f8266" }, { "internalIP" : "", "isUp" : true, "pod" : "analytics", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "reportcrud-datastore", "analytics-datastore" ], "uUID" : "5d5dce4e-e25d-48ef-aa43-ea44a6119d87" }, { "internalIP" : "", "isUp" : true, "pod" : "analytics", "reachable" : true, "region" : "dc-1", "tags" : { "property" : [ ] }, "type" : [ "reportcrud-datastore", "analytics-datastore" ], "uUID" : "753ff39d-cff9-40ef-8a26-acfffcb944bf" * Connection #0 to host localhost left intact * Closing connection #0 } ]

Here is analytics/groups/ax

[ { "name" : "axgroup001", "properties" : { "consumer-type" : "ax" }, "scopes" : [ "eDirectv~R1" ], "uuids" : { "aries-datastore" : [ ], "postgres-server" : [ "19d7f4ff-c1e0-43be-9f21-5effc01b6f0c:59eb2361-ec29-401d-a235-a0422f0409b0" ], "dw-server" : [ ], "qpid-server" : [ "aa5085fa-4a64-43d7-a37d-9ba6d1acd24c", "fd354de1-847c-4b8e-8443-64062fa54d96", "dee80332-99c3-4dd7-bba5-76963c51cbd1", "1495d07f-a919-444f-bdf9-e218692cb291" ] }, "consumer-groups" : [ { "name" : "consumer-group-001", "consumers" : [ "aa5085fa-4a64-43d7-a37d-9ba6d1acd24c", "fd354de1-847c-4b8e-8443-64062fa54d96", "dee80332-99c3-4dd7-bba5-76963c51cbd1", "1495d07f-a919-444f-bdf9-e218692cb291" ], "datastores" : [ "19d7f4ff-c1e0-43be-9f21-5effc01b6f0c:59eb2361-ec29-401d-a235-a0422f0409b0" ], "properties" : { } } ], "data-processors" : { }

I am also getting the same error.

We also have 2 datacenters -> Once contains Router and Message Processor and other contains the rest components.

To solve this error we also tried:

  1. Restarting the components
  2. Running Cassandra repair tool
  3. Creating the Env. again
  4. Manually adding the cassandra

But still facing the issue

Not applicable

did you use this process to create organization?

/opt/apigee/apigee-service/bin/apigee-service apigee-provision setup-org -f configFile