4.15.04.03 patch no analytics data

ccovney · 09-30-2015 09:26 AM

Hi everyone,

We recently upgraded to the On prem Private Cloud 4.15.04.03 Edge service pack, and we notice that there is no analytics data coming through to the management server.

We are running java 1.7 and i believe analytics was coming in just fine before the patch upgrade. Any idea why this might be the case? Thanks!

mschreuder

Hi @Chris Covney, I don't have a direct answer for you but I remembered a related question had been asked - check out Analytics not showing up in newly created orgs, especially the point about the 15.04 specific script: apigee-postgres-register.sh that you need to run if you have not.

Let us know if you don't get any success from the answers to that other question.

ccovney

@mschreuder

It looks like the ax status is all good:

{
  "environments" : [ {
    "components" : [ {
      "hosts" : "[localhost, localhost]",
      "message" : "success",
      "name" : "qs",
      "status" : "SUCCESS",
      "uuid" : "[9be9a4e6-3d18-4d2b-8090-6f17de5daece, 7c251b96-243a-4de8-a76e-02e19b0a8add]"
    }, {
      "hosts" : "[localhost:localhost]",
      "message" : "success at Mon Sep 28 15:19:23 EDT 2015",
      "name" : "pg",
      "status" : "SUCCESS",
      "uuid" : "[ab448e45-5af2-48c8-badc-7b8dc166c3b4:494365b0-3efc-4620-a7bd-077ad99ab2ac]"
    } ],
    "message" : "",
    "name" : "prod"
  } ],
  "organization" : "Intralinks",
  "status" : "SUCCESS"
}

In the Qpid servers system.log we see this error:

2015-09-30 15:45:45,021  Dispatcher-0-Conn-27 WARN  c.a.a.m.MessageConsumer - MessageConsumer.process() : Sending message batch to the DLQ.
com.apigee.analytics.persistence.api.PersistenceException: No Sink found  for qualifier MsgQualifier{queueName='ax-q-axgroup001-consumer-group-001', groupName='axgroup001', cg=consumer-group-001}. Rejecting messages

I know that java needed to be upgraded on the Management Servers for the .03 patch. We upgraded java on ALL the Apigee nodes for consistency, do you think this is breaking analytics?

ccovney

@mschreuder

It says access denied when i try to access this community article, and judging by the title, it appears like it might be relevant to this investigation. Can you ascertain the contents of this article?

https://community.apigee.com/articles/8512/qpid-re...

https://community.apigee.com/articles/8512/qpid-re...

mschreuder

I do have access to that community article, you are correct it's talking about the same qpid error "no sink found for..." here's what it says as the solution:

-Run the following sript:

${APIGEE_ROOT}/apigee4/share/installer/apigee-postgres-register.sh

-Restart qpid services

It's also worth noting the the OPDK Documentation mentions the following points about the procedure:

The restart is required to create new consumer group queues in existing consumer group QPIDs. On occasion, queue creation can take up to 15-20 minutes. Checks mentioned in #11 should be repeated a few times till the confirmation is obtained. (This is referring to running the qpid-stat -q command until you see the Consumer group in the QPid.)

Not sure though whether this extra insight into what happens after running the script and the below are related, or independent.

birute

@Chris Covney can you give it another try to access the article?

ccovney

@mschreuder

also, this might be an issue. not sure if i've run the commands correctly though:

-bash-4.1$ sh /apps/apigee4/bin/psql.sh 

psql (9.3.9)

Type "help" for help.




apigee=# \d analytics."accountmanagement.Common.agg_app"

Did not find any relation named "analytics."accountmanagement.Common.agg_app"".

apigee=# \d analytics."accountmanagement.Common.fact"

Did not find any relation named "analytics."accountmanagement.Common.fact"".

apigee=# \d analytics."accountmanagement.Common.agg_app" \d analytics."accountmanagement.Common.fact";

Did not find any relation named "analytics."accountmanagement.Common.agg_app"".

ccovney

@mschreuder

lol sorry for inundating you with messages, but i also am seeing errors in the PG sys logs when i bounce PG server:

2015-09-30 16:37:12,055  main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path

2015-09-30 16:37:12,056  main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [onboarding-module]

2015-09-30 16:37:12,057  main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: []

2015-09-30 16:37:12,057  main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway.



2015-09-30 16:37:14,041  main WARN  REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts

2015-09-30 16:37:14,438  qtp39112485-30 - /v1/servers/self/uuid ERROR REST - ExceptionMapper.toResponse() : Error occurred : null


	org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336)

	org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211)

	org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92)

	org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

	

2015-09-30 16:37:14,455  qtp39112485-32 WARN  REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/

2015-09-30 16:37:14,456  qtp39112485-32 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null




	org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336)

	org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211)

	org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92)

	org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)




2015-09-30 16:37:15,254  main ERROR KERNEL - MicroKernel.deployAll() : Some services were not started due to unresolved dependencies:

2015-09-30 16:37:15,254  main ERROR KERNEL - MicroKernel.deployAll() : AccessControl was not started because of missing dependencies: [CacheService]

2015-09-30 16:37:15,254  main ERROR KERNEL - MicroKernel.deployAll() : SecurityService was not started because of missing dependencies: [UserService, ResourceService]

mschreuder

@Alex Toombs saw you had been helping with similar issues, are you able to help @Chris Covney with this one?

atoombs

@mschreuder and @Chris Covney running the ${APIGEE_ROOT}/apigee4/share/installer/apigee-postgres-register.sh script should resolve this.

The only other suggestion I have on this is to restart all components (RRTs/MPs, not just AX components) so that everything picks up these changes. Even after doing this, are you still seeing the "No Sink found" errors?

ccovney

@Alex Toombs @mschreuder

Thanks guys! I will run this script and let you know if it fixes the issue. Just to be clear, after running this script, should do a rolling bounce of all components?

Thanks!

atoombs

@Chris Covney yes, all AX components (PG / QPID), and the Routers and MPs. In the past, that's been required for all the changes to be picked up by all needed components.

Please advise if you still get "No sink found" errors after that, and we can proceed from that point if so.

Thanks.