Apigee Hybrid Container Traffic Analysis with tcpdump for Target Requests

1 0 556

* This is a TADA article. TADA stands for "Troubleshooting And Debugging for Apigee".

For troubleshooting connection between Target endpoint and the backend itself, most problems with Apigee Proxies are solved by using the TRACE tool of the Edge UI.

More complex problems require usage of a Debug logging

The most complex problems call for usage of tcpdump. For Apigee hybrid, the scenario is complicated by the fact that apigee-runtime container runs as an apigee:apigee user. Good for your security, not so for tcpdump, which require root's permissions.

We are going to use tcpdump utility to capture and analyse southbound request-response for a target IP address traffic.

We will capture the message that uses ip address URL, then we will add a Host header to the response using Assign Message policy and then will capture the messages again.

This article describes the key steps for the troubleshooting technique. you can refer to the full walkthrough version for step-by-step instructions and detailed explanations at https://github.com/apigee/ahr/wiki/tada-apigee-hybrid-tcpdump

Identify container interface

For details: How to get tcpdump for containers inside Kubernetes pods https://community.pivotal.io/s/article/How-to-get-tcpdump-for-containers-inside-Kubernetes-pods?lang...

1. Find the container name and node where your pod is deployed

kubectl describe pod $POD -n apigee|grep -E "Node:|Containers:|Container ID:|apigee-runtime:"

Node:               gke-dc1-cluster-apigee-runtime-f80b30bd-btxy/10.142.0.19
Init Containers:
    Container ID:  docker://f551fe5cf7c23b138966d656fe7891bd620a62c7181d8b0e7f56817f578f31fd

2. Ssh into the node

gcloud compute ssh gke-dc1-cluster-apigee-runtime-f80b30bd-btxy --zone us-east1-b

yuriyl@gke-dc1-cluster-apigee-runtime-f80b30bd-btxy ~ $ 

3. Using container Id, find the pod's unique network interface index inside it's container.

export CID=95f76d7ffb881d19ff77117f3f9a657a91843c25ab49009cf5e9b71a3e1af97b
docker exec $CID /bin/bash -c 'cat /sys/class/net/eth0/iflink'

Output:

60

4. Take the result from that and locate that interface on the worker

for i in /sys/class/net/veth*/ifindex; do grep -l 60 $i; done

Output:

/sys/class/net/veth052c835c/ifindex

tcpdump: Traffic Capture

We need two terminal sessions:

  • one to run tcpdump
  • one to execute curl requests.

Using interface id from the previous section, capture incoming and outgoing traffic for httpbin.org IP address we found earlier

1. Start tcpdump to capture output into a request.pcap file.

tcpdump -i veth052c835c host 3.211.1.78 -w request.pcap

2. Execute request using curl in a separate session

3. Press Ctrl+C to interrupt the tcp dump capture.

Output:

^C9 packets captured
9 packets received by filter
0 packets dropped by kernel

4. Copy capture file out of toolbox to the node See also: https://cloud.google.com/container-optimized-os/docs/how-to/toolbox#getting_files_into_and_out_of_to...

5. Exit from toolbox session. Notice the container name.

root@gke-dc1-cluster-apigee-runtime-f80b30bd-btxy:~# exit

logout
Container yuriyl-gcr.io_google-containers_toolbox-20180918-00 exited successfully.

6. At the node, execute sudo cp to copy request.pcap file from container to the node file system

sudo cp /var/lib/toolbox/yuriyl-gcr.io_google-containers_toolbox-20180918-00/root/request.pcap .

7. Exit the node and copy request.pcap file to your work machine

exit

8. Copy capture file to your work machine with installed wireshark.

gcloud compute scp gke-dc1-cluster-apigee-runtime-f80b30bd-btxy:~/request.pcap . --zone us-east1-b

9. Open the file in wireshark, select the GET line and right-click follow the HTTP stream.

tcpdump request with ip address

Add AssignMessage Policy to set the Host header

1. Using Edge UI, add the following Assign Message Policy to the Target Endpoint PreFlow event.

<AssignMessage async="false" continueOnError="false" enabled="true" name="AssignMessage-SetHost">
    <AssignVariable>
        <Name>target.header.host</Name>
        <Value>xxx.yyy.com</Value>
    </AssignVariable>
    <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

Edge UI Assign Message to Set Host

2. Save and Deploy a new version of the httpbin API proxy

3. Repeat tcpdump capture and analysis for a new curl request

tcpdump Host Header

Observe that the request now uses an overridden SNI-compliant host value.

Version history
Last update:
‎12-30-2020 07:24 AM
Updated by: