Analytics Groups missed

[engmohamedeltok]

We have issue with Apigee MPs. The issue is that the analytics details are not retrieved successfully for all analytics groups for all scopes and this causes no AxDataPusher and also no watchers are created for those scopes and accordingly analytics data will not be collected for those scopes as shown in below sample log.

 2021-12-12 12:55:46,803 org:mof  Apigee-Timer-0 ERROR SERVICES.COLLECTION - Collector.buildGroupDataPushers() : Failed to build all groups for mof

2021-12-12 12:55:46,803 org:mof  Apigee-Timer-0 ERROR SERVICES.COLLECTION - Collector.buildGroupDataPushers() : Scope mof~ not found in any AX group. Analytics is enabled with send to queue and analytics data cannot be sent to QPID

2021-12-12 12:55:46,803 org:mof  Apigee-Timer-0 ERROR SERVICES.COLLECTION - Collector.startCollection() : No AxDataPusher created for scope ServerScope{organization='mof', environment=''}. Analytics data will not be collected.

[DigitalMonk]

I am running into the same issue after upgrading to 4.51.0.0.  When I check the AX group through the admin api it does show the environment in scope.  Wondering if you were able to resolve this?  Thanks!

[yuriyl]

I and @ncardace were recently involved in the investigation of this issue.

Long story short, 4.51 upgrade introduces a regression and race condition in ZK reads during MP startup. The tell-tale signal is the message above. 

Apigee OPDK/AX team created the fix. It is going to be part of the next fix release. That's provisionally next week. 

[DigitalMonk]

OK, glad it's not just me. 🙂

Do you know if we will be able to retrieve the "lost" analytic data after applying the fix?  Unfortunately, it took us a while to realize that analytics weren't being captured, so we're about 2 weeks behind now.  Thanks!

[yuriyl]

 > , glad it's not just me. 🙂

Well, yes, I know, it's of a little consolation but yes. 

> Do you know if we will be able to retrieve the "lost" analytic data after applying the fix?  U

So for the authoritative answer you would need to contact Support, but as per my understanding, as AxDataPusher were not configured, the AX request batches were never sent to the QD. Thus no way to retrieve it.

>  Unfortunately, it took us a while to realize that analytics weren't being captured,

For the future of course, as a lesson learned, a logged ERROR messages better trigger alert and get analysed. I know it's a banal wisdom... 

[yuriyl]

FYI, on Feb 16, a new release of Apigee Private Cloud, 4.51.00.04 was released. 

https://docs.apigee.com/release/notes/4510004-private-cloud-relnotes#fixes

The Analytics fix is the part of it:

[DigitalMonk]

I know its been a while, but I just wanted to follow up on this to say that we upgraded and it did correct the issue.  It also appears that we did get back some amount of the "lost" analytics (at least 30 days worth).  Thanks again for your help!