* This is a TADA article. TADA stands for "Troubleshooting And Debugging for Apigee".
For troubleshooting connection between Target endpoint and the backend itself, most problems with Apigee Proxies are solved by using the TRACE tool of the Edge UI.
More complex problems require usage of a Debug logging
The most complex problems call for usage of tcpdump. For Apigee hybrid, the scenario is complicated by the fact that apigee-runtime container runs as an apigee:apigee user. Good for your security, not so for tcpdump, which require root's permissions.
We are going to use tcpdump utility to capture and analyse southbound request-response for a target IP address traffic.
We will capture the message that uses ip address URL, then we will add a Host header to the response using Assign Message policy and then will capture the messages again.
This article describes the key steps for the troubleshooting technique. you can refer to the full walkthrough version for step-by-step instructions and detailed explanations at https://github.com/apigee/ahr/wiki/tada-apigee-hybrid-tcpdump
For details: How to get tcpdump for containers inside Kubernetes pods https://community.pivotal.io/s/article/How-to-get-tcpdump-for-containers-inside-Kubernetes-pods?lang...
1. Find the container name and node where your pod is deployed
kubectl describe pod $POD -n apigee|grep -E "Node:|Containers:|Container ID:|apigee-runtime:" Node: gke-dc1-cluster-apigee-runtime-f80b30bd-btxy/10.142.0.19 Init Containers: Container ID: docker://f551fe5cf7c23b138966d656fe7891bd620a62c7181d8b0e7f56817f578f31fd
2. Ssh into the node
gcloud compute ssh gke-dc1-cluster-apigee-runtime-f80b30bd-btxy --zone us-east1-b yuriyl@gke-dc1-cluster-apigee-runtime-f80b30bd-btxy ~ $
3. Using container Id, find the pod's unique network interface index inside it's container.
export CID=95f76d7ffb881d19ff77117f3f9a657a91843c25ab49009cf5e9b71a3e1af97b docker exec $CID /bin/bash -c 'cat /sys/class/net/eth0/iflink'
Output:
60
4. Take the result from that and locate that interface on the worker
for i in /sys/class/net/veth*/ifindex; do grep -l 60 $i; done
Output:
/sys/class/net/veth052c835c/ifindex
We need two terminal sessions:
Using interface id from the previous section, capture incoming and outgoing traffic for httpbin.org IP address we found earlier
1. Start tcpdump to capture output into a request.pcap file.
tcpdump -i veth052c835c host 3.211.1.78 -w request.pcap
2. Execute request using curl in a separate session
3. Press Ctrl+C to interrupt the tcp dump capture.
Output:
^C9 packets captured 9 packets received by filter 0 packets dropped by kernel
4. Copy capture file out of toolbox to the node See also: https://cloud.google.com/container-optimized-os/docs/how-to/toolbox#getting_files_into_and_out_of_to...
5. Exit from toolbox session. Notice the container name.
root@gke-dc1-cluster-apigee-runtime-f80b30bd-btxy:~# exit logout Container yuriyl-gcr.io_google-containers_toolbox-20180918-00 exited successfully.
6. At the node, execute sudo cp
to copy request.pcap file from container to the node file system
sudo cp /var/lib/toolbox/yuriyl-gcr.io_google-containers_toolbox-20180918-00/root/request.pcap .
7. Exit the node and copy request.pcap file to your work machine
exit
8. Copy capture file to your work machine with installed wireshark.
gcloud compute scp gke-dc1-cluster-apigee-runtime-f80b30bd-btxy:~/request.pcap . --zone us-east1-b
9. Open the file in wireshark, select the GET line and right-click follow the HTTP stream.
1. Using Edge UI, add the following Assign Message Policy to the Target Endpoint PreFlow event.
<AssignMessage async="false" continueOnError="false" enabled="true" name="AssignMessage-SetHost"> <AssignVariable> <Name>target.header.host</Name> <Value>xxx.yyy.com</Value> </AssignVariable> <AssignTo createNew="false" transport="http" type="request"/> </AssignMessage>
2. Save and Deploy a new version of the httpbin API proxy
3. Repeat tcpdump capture and analysis for a new curl request
Observe that the request now uses an overridden SNI-compliant host value.