Analytics big problem, not updating

[apiphu]

Hello!

We have run into the problem that the traffic on analytics for healtcheck has dropped significantly 1 month ago.
Traffic has dropped by about half. Analytics do not show the reality that they should.
We restarted qpid server, since then analytics doesn't really want to work.
Any guesses what could have caused this problem? Where should we look?

Thank you very much!

[dchiesa1]

analytics doesn't really want to work.

Hmm, well I'd suggest you look in the logs for Qpid, and Postgres, check the disk space available on all of your nodes (MPs, Qpid, Datastore machines). and then contact Apigee support for further assistance.

[API-Evangelist]

Try below reference which may help you.

https://docs.apigee.com/api-platform/troubleshoot/analytics/data-not-showing-analytics-dashboards

https://docs.apigee.com/api-platform/troubleshoot/analytics/analytics-data_stuck_qpidd

[apiphu]

Hello!

Unfortunately we have not been able to solve the problem yet.
It turns out that one of the linux system engineers deleted the qpid-proton-c package during a weekend maintenance update. We only restored it on the Qpid servers. Is it possible that qpid-proton-c is required on other components (MS, MP, router, etc)?
On proxy performance it shows only 1 out of 6 on the stats.

https://docs.apigee.com/api-platform/troubleshoot/analytics/analytics-data_stuck_qpidd

We looked at this, there were indeed stuck messages, but it didn't solve the problem.

https://docs.apigee.com/api-platform/troubleshoot/analytics/data-not-showing-analytics-dashboards

When the "apiproxy?_optimized" curl was called, there were response_status_code included, there were  401, 429, 503 codes. Does this help?

I have a suspicion that there is a problem with postgresql, because the standby server is not working properly, there were regular sql errors. When we shut down the standby server, all of a sudden 10 thousand traffic came on analytics, but then the data died again.

We have not found the cause of the problem based on logs.

apigee version: 4.19.06

[API-Evangelist]

General comment..Migrate to latest version -https://docs.apigee.com/private-cloud/v4.51.00/update-apigee-edge-41906-45100 (It is immediate as there is no support)

AFAIK qpid-proton-c primarily for analytics components only.

Have seen such issues and you would need to provide some guidance to infra team during patching time.

Apigee application has a qpid-proton-c package installed.

Example:

Red hat Satellite sees the version your application installed, and considers it a needed update for a piece of software that is shipped as part of satellite.Machine does not have the satellite-provided software installed, nor can it be installed, because it conflicts with the qpid-proton-c rpm installed by your application.This causes satellite to always say that your machine is missing a patch, and it is impossible to resolve the inconsistency.

Infra team should able to address via  /etc/yum.conf to exclude  to patch qpid-proton-c.

Would encourage to connect with support as it will be easy to look at the infra and triage as this needs additional data to assist here.

You may move old pgdata and re-run db_upgrade or few additional steps which support can guide.

[apiphu]

Hi!

Thanks, but we can't suddenly upgrade to the latest version.
However, I forgot to tell you that sandbox analytics works perfectly.