How do I resize the Qpid queues in Apigee Edge OPDK 4.15.01

[#1]

There are occasions when the Qpid queues will fill up and no more data can flow between the Message-Processor (MP) and Postgres (PG). This will be reflected when viewing the Dashboard within Apigee Edge. No data will display, or it will appear as if traffic has suddenly dropped.

A good indication that the queue may be full is if a message similar to the following is seen in the MP system.log file.

Enqueue capacity threshold exceeded on queue "<queue_name>", where <queue_name> is the name of the actual Qpid queue that is full or not draining.

You may also get statistics on the Qpid queues by running the following OPDK script.

/<install_root>/apigee4/bin/qpid-stat.sh -q

Run that script a few times and if the count in the "msg" column does not seem to be going down, then this may be a good indicator that the queue is not draining, a sign that the queue may be full or stuck.

Resizing that specific queue will fix the issue and cause data to once again flow to PG. In order to resize the queue perform the following steps.

1. Stop Qpid $ sudo /<install_root>/apigee4/etc/init.d/apigee-qpidd stop
2. The next few steps require to be run as root. $ sudo su
3. Currently, our scripts do not set the PYTHONPATH, so this must be done manually. $ export PYTHONPATH="/<install_root>/apigee4/share/apache-qpid/lib64/python2.6/site-packages:$PYTHONPATH"
4. Resize the Qpid queue* $ /<install_root>/apigee4/share/apache-qpid/libexec/qpid/resize --num-jfiles=6 --jfile-size-pgs=6000 /<install_root>/apigee4/data/qpid/data/rhm/jrnl/001a/<queue_name>/

   Where <queue_name> is the same as shown in the system.log file. Hint: You need to sift through the directories to find the qpid journals that are most recently modified and apply the resize to each directory. For example, the directory would be under /<install_root>/apigee4/data/qpid/data/rhm/jrnl/<aphanumeric_dir>. In the above example it is 001a, your install may have a different folders with <alphanumeric_dir>. For each <queue_name>, run the resize command. Here is example syntax for running the resize command for each of the <queue_name> containing consumer-group-001 and consumer-group-002: /opt/apigee4/share/apache-qpid/libexec/qpid/resize --num-jfiles=6 --jfile-size-pgs=6000 /data/apigee4/data/qpid/data/rhm/jrnl/0013/ax-q-axgroup001-consumer-group-001

   /opt/apigee4/share/apache-qpid/libexec/qpid/resize --num-jfiles=6 --jfile-size-pgs=6000 /data/apigee4/data/qpid/data/rhm/jrnl/0014/ax-q-axgroup001-consumer-group-002

5. Go to the queue directory $ cd /<install_root>/apigee4/data/qpid/data/rhm/jrnl/001a/<queue_name>/
6. After resize, the journal files will now be owned by root. Change them to apigee user. $ chown apigee:apigee JournalData*
7. Exit from root user $ exit
8. Start Qpid $ sudo /<install_root>/apigee4/etc/init.d/apigee-qpidd start

*Please note that the value for --jfile-size-pgs is by default 4000. In the cloud, we typically set this to 6000 when we resize. However, Apache Qpid is open source, and more information on various configuration parameters can be found in the Messaging User Guide.

If for any reason you need assistance with any of these steps, please open a support case with Apigee Support and we will help you out.

[akashtp]

For versions 16.0X, we would need to install the following executable that is a part of the package :

qpid-cpp-server-store-0.14-16.el6.x86_64

Once you install that, we can follow the same procedure that is a part of the above article to resize the qpidd queue.

[jghunt]

NOTE: For Apigee Edge version 17.01 and newer, we do not recommend the resize of Qpid queues, the version of Qpid in this version automatically sets the size of the queue. If qpid capacity becomes a problem, we recommend adding new Qpid nodes.

[rajeevyes]

@Akash Prabhashankar @jhunt@apigee.com @Dave Newman where can i find the following script on opdk 4.16.09?

/<install_root>/apigee4/share/apache-qpid/libexec/qpid/resize

[#5]

After upgrade from 4.17.09 to 4.18.05, our analytics data is showing 40 mins delayed and the load in UI is taking longer. I don't see the full error in logs as we can see data. Is this a reason of qpidd is almost full? Could you please suggest how to resolve the issue?