PG <> found in more than one group

samuelucich
Participant IV
This is a brand new AIO install. Created an org called 'acclaim'. I'm able to access the org and create and test a proxy, but analytics seems to be broken.

When I log into the UI I see error messages like "Error fetching analytics data".

When I look back at the console output from the setup-org command, I see this:


!!!! Error !!!!
HTTP STATUS CODE: 400
{
  "code" : "analytics.dataapi.service.PGFoundInMultipleGroups",
  "message" : "PG aa432f90-fe53-47d2-a623-e66ff0a06427 found in more than one group",
  "contexts" : [ ]
}
{
  "name" : "ax-grp-001",
  "properties" : {
  },
  "scopes" : [ "acclaim~prod" ],
  "uuids" : {
    "qpid-server" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "postgres-server" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "datastores" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }
}

This validation shows a FAILURE at the end:

 curl -u samu@allinone.local:xxxx http://localhost:8080/1/organizations/acclaim/environments/prod/provisioning/axstatus
{
  "environments" : [ {
    "components" : [ {
      "hosts" : "[null]",
      "message" : "success",
      "name" : "qs",
      "status" : "SUCCESS",
      "uuid" : "[eba30196-75b5-4252-ad22-50ec443e7e6c]"
    }, {
      "hosts" : "[null]",
      "message" : "",
      "name" : "pg",
      "status" : "UNKNOWN",
      "uuid" : "[aa432f90-fe53-47d2-a623-e66ff0a06427]"
    } ],
    "message" : "",
    "name" : "prod"
  } ],
  "organization" : "acclaim",
  "status" : "FAILURE"
}


Thanks, Sam
0 9 548
9 REPLIES 9

Not applicable

Hi @samuel ucich,

Can you restart the Management Server and then share the output of

curl -u adminEmail:adminPwd http://<ms_ip>:8080/v1/analytics/groups/ax

samuelucich
Participant IV

Hi Mauro,

Thanks for your assistance!

The management server was restarted, and below is the output from the recommended command:

[root@server1 /]# curl -u samu@allinone.local:xxxx http://localhost:8080/v1/analytics/groups/ax
[ {
  "name" : "axgroup-001",
  "properties" : {
  },
  "scopes" : [ "VALIDATE~test" ],
  "uuids" : {
    "qpid-server" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "postgres-server" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "datastores" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }
}, {
  "name" : "ax-grp-001",
  "properties" : {
  },
  "scopes" : [ "acclaim~prod" ],
  "uuids" : {
    "qpid-server" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "aries-datastore" : [ ],
    "postgres-server" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ],
    "dw-server" : [ ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "datastores" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }
} ][root@server1 /]#

The configuration looks good, as it is an AIO you can restart all components and see if the problem persists.

If it continues you can take a look to the Qpid Server and Management Server system logs to see if there's something else. Also you can run qpid-stat -q to verify that the queues for the two groups were created.

samuelucich
Participant IV

Hi,

This exception was in the system log under edge-management-server/logs

2017-12-01 08:39:58,534 org:acclaim env:prod  pool-3-thread-28 ERROR REST - CustomJAXRSInvoker.performInvocation() : CustomJAXRSInvoker.performInvocation : Method com.apigee.analytics.dataapi.impl.DataAPIResourceHandlerImpl.getStats threw an exception.
com.apigee.rest.framework.DataException: Data Access Error - Data Access Error
        at com.apigee.analytics.dataapi.impl.DataAPIResourceHandlerImpl.getStats(DataAPIResourceHandlerImpl.java:344) ~[analytics-dataapi-impl-1.0.0.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173) [cxf-rt-core-2.4.7.jar:2.4.7]
        at com.apigee.rest.framework.CustomJAXRSInvoker.performInvocation(CustomJAXRSInvoker.java:126) ~[rest-1.0.0.jar:na]
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) [cxf-rt-core-2.4.7.jar:2.4.7]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:241) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) [cxf-rt-core-2.4.7.jar:2.4.7]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
        at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) [cxf-rt-core-2.4.7.jar:2.4.7]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106) [cxf-rt-core-2.4.7.jar:2.4.7]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263) [cxf-api-2.4.7.jar:2.4.7]
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-rt-core-2.4.7.jar:2.4.7]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:232) [cxf-rt-transports-http-2.4.10.jar:2.4.7]
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:209) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:152) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:114) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:113) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [javax.servlet-api-3.1.0.jar:3.1.0]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:830) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:543) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:530) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1584) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandle

Maybe some problems when the AX were enabled for that particular environment and the ax-group. You could try removing the scope from the group and add new ax group and provision that scope (org~env) to avoid going deeper and more time because it is an AIO.

Also you can remove the scope and add it to the original one. In an AIO you have the same servers so I don't see any need to break the AX into different groups.

curl -u sysAdminEmail:passWord -X DELETE "https://<MS_IP>:8080/v1/analytics/groups/ax/ax-grp-001/scopes?org=org_name&env=env_name" -H "Content-Type: application/json"

Hi,

Tried you suggestions, but no luck.

I removed and added scope, restarted, management server, postgres, qpid...but still same errors

I then tried to add a new ax group (axgroup-002) see below...but still same errors.

curl -u samu@allinone.local:Brenda99 http://localhost:8080/v1/analytics/groups/ax
[ {
  "name" : "axgroup-002",
  "properties" : {
  },
  "scopes" : [ "acclaim~prod" ],
  "uuids" : {
    "qpid-server" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "aries-datastore" : [ ],
    "postgress-server" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ],
    "postgres-server" : [ ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-002",
    "consumers" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "datastores" : [ ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }
}, {
  "name" : "axgroup-001",
  "properties" : {
  },
  "scopes" : [ "VALIDATE~test" ],
  "uuids" : {
    "qpid-server" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "postgres-server" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "datastores" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }
}, {
  "name" : "ax-grp-001",
  "properties" : {
  },
  "scopes" : [ ],
  "uuids" : {
    "qpid-server" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "postgres-server" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "eba30196-75b5-4252-ad22-50ec443e7e6c" ],
    "datastores" : [ "aa432f90-fe53-47d2-a623-e66ff0a06427" ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }
} 

samuelucich
Participant IV

Epilogue: In the end I removed everything and re-installed. This time I did not do the Validate stuff, and just provisioned my own org, using the default Ax group (axgroup-001), and Analytics do work now!

Not sure if have a Validate org and an org provisioned by 'apigee-provision' cmd, but at this point i'm not going to complain.

The key thing here is that you ran apigee-validate and I'm guessing didn't then clean up afterwards?

During validation it creates axgroup-001 and assigns all postgres/qpid servers to it. Those servers then cannot be assigned to another analytics group - hence the error you're seeing.

@samuel ucich, I had to do the same.