Postgres DB and Server working but API check is returning isUp='false'

[ajha]

Hi,

We have 13 node HA cluster, and recently we changed the password for Postgres DB(both postgres/apigee) user, since then even though Postgresql/postgres-server are both running fine and am able to see the analytics in UI, if I try to check the service health through curl API commands...its showing

curl http://postgres_IP:port/v1/servers/self/up

"message" : "Service not up yet.", "contexts" : [ ]

If I see through */v1/server/self, I can see:

"isUp" : false, "pod" : "analytics", "reachable" : true, "region" : "dc-1",

If I see the postgres-server log, this is what I get:

Oct 25, 2017 2:28:01 PM org.apache.cxf.endpoint.ServerImpl initDestination INFO: Setting the server's publish address to be /alpha Oct 25, 2017 2:29:14 PM org.apache.cxf.jaxrs.utils.JAXRSUtils findTargetMethod WARNING: No operation matching request path "/v1/servers/self/health/" is found on subresource, Relative Path: /servers/self/health/, HTTP Method: GET, ContentType: application/octet-stream, Accept: */*,. Please enable FINE/TRACE log level for more details

Not sure, from where these service APIs get data, so not able to proceed any further, would greatly appreciate any support on this.

Thanks,

Amit.

[cmbrown]

@Amit Jha is there a chance you could turn on debig level for the PG logs? also take a look at the qpid and mp log files to look for errors

[ajha]

Hi Christin,

Thanks for reply.

We enabled the 'Debug' level log for Postgres-server(earlier I was doing with 'trace' and not able to see much). Now I can see below:

2017-10-30 11:24:46,504 main INFO c.a.a.d.c.ServerHandle - ServerHandle.createConnectionPool() : Initialized the connection pool for lookup pos 0 with url jdbc:postgresql://post-serverIP:port/apigee

---------------------------------------------------------

Exception: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (FATAL: password authentication failed for user "apigee")

Which means, its not able to make an jdbc connection for 'apigee' user.

Seems in changing password, change at some other place is also required..

[cmbrown]

how did you change the pw ? if you could let me know the steps can see where to look

[ajha]

Hi Christin,

We changed according to Apigee documentation

https://docs.apigee.com/private-cloud/v4.17.05/resetting-edge-passwords

Here "Resetting PostgreSql Password" Header content

We have Postgres setup as Master/Standby. We changed only at master for both the users postgres/apigee and got the same changed to mgm-server/qpid-server/postgres-server properties file and started the services in the same sequence as suggested in docs.

Initially we faced the issue, when using password with special chars(*$@ etc...), so we changed it to contain only Alphabet(small/caps) and digits.

So now Analytics UI is up and showing the data and all services too are up, only check through API is failing.

[ajha · Accepted solution]

Hi Christin,

Post further digging was able to find the root cause and fix this issue.

As per component:

• conf_pg-agent_password=newEncryptedPasswordForPostgresUser
• conf_pg-ingest_password=newEncryptedPasswordForPostgresUser
• conf_query-service_pgDefaultPwd=newEncryptedPasswordForPostgresUser
• conf_query-service_dwDefaultPwd=newEncryptedPasswordForPostgresUser
• conf_analytics_aries.pg.password=newEncryptedPasswordForPostgresUser

Postgres/apigee user use above mentioned corresponding user.

While in our installation, it was changed to use opposite users as below:

• /opt/apigee/token/application/postgres-server.properties:conf_pg-agent_userName=apigee
• /opt/apigee/token/application/postgres-server.properties:conf_pg-ingest_userName=apigee
• /opt/apigee/edge-postgres-server/token/default.properties:conf_query-service_pgDefaultUser=postgres
• /opt/apigee/edge-postgres-server/token/default.properties:conf_query-service_dwDefaultUser=postgres
• /opt/apigee/token/application/postgres-server.properties:conf_analytics_aries.pg.username=apigee

Also below note in docs is WRONG:

https://docs.apigee.com/private-cloud/v4.17.05/resetting-edge-passwords

7. c. Add the following properties to the files:
Note: All of these properties take the encrypted 'postgres' user password.

It should be corrected to say, use the passwords for corresponding users as per your installation.

Thanks,

Amit.

[cmbrown]

thanks for posting @amit I submitted a docs edit for your post as well.