Sending OpenTelemetry spans from Apigee hybrid to ...

ncardace · ‎09-30-2022

Introduction

This article is the second of a series of articles that cover the implementation within Apigee hybrid of Distributed Tracing requirements by leveraging commercially widely available third party providers. If you have not yet read the first article (where the integration with Dynatrace is presented), I encourage you to look at that first. It will help you to gain a better understanding of the common requirements, use cases, the best practices in Google's SRE principles. It also includes the additional resources to learn about the OpenTelemetry protocol specifications and the native tracing capabilities offered by Google Cloud Trace. I have not duplicated them in this article.

The first and extensive article is available here.

A detailed description of Cloud Trace is presented here.

You can find a detailed description on the tracing features of Apigee hybrid, how to enable it from the control plane and consume it from GCP Cloud Trace here. All the default trace variables available in the tracing report are listed here. Detailed steps to customize its configuration are included here.

In the present article I describe how to send the API call traces to New Relic independently on the choice of the Kubernetes flavor used to deploy the hybrid runtime plane. In short, this is achieved in a pretty straightforward way by leveraging the open standards and without enabling the native distributed tracing option in Apigee hybrid.

The New Relic Trace API

Note that there is only one available interface (despite two possible accepted formats for the trace data, i.e. a New Relic-based and a Zipkin-based format) to send data to New Relic and this is by leveraging the New Relic Trace API. An introduction to the New Relic Trace API is provided in their documentation here.

A license key for New Relic is needed.

For the Apigee runtime plane, in order to reconstruct span data from the individual API proxies, I will show you in the last part of the article how to achieve this by adding Apigee policies to the API proxies to produce the trace data in the format ready for New Relic.

Monitoring Kubernetes with New Relic

If you are planning to operate an Apigee hybrid runtime cluster on any supported Kubernetes flavor, then you might also be interested in the instrumentation of the full cluster with New Relic. The product has a dedicated set of documents to describe what can be achieved and how. The documentation is provided here.

Instrument the Apigee hybrid runtime plane for OpenTelemetry

The Apigee runtime cluster has two different points where traces are generated. The first is where the istio-ingress pods are - in a corresponding Kubernetes namespace, the second is instead in the apigee namespace, and it consists of the spans corresponding to the message processor pods, which are the policy enforcement points within the API gateway. If you are not familiar with how the Apigee hybrid runtime plane processes the API calls, proceed to study the detailed description of the API call flow with each runtime kubernetes cluster: refer to this article.

Both components istio-ingress and Apigee hybrid API message processors will need to be instrumented.

a) Instrumentation of the istio-ingress pods:

Instrumenting the istio-ingress pods within the runtime plane is pretty straightforward: in fact, the instrumentation of a native istio-ingress with the OpenTelemetry-standard collector is supported natively and documented here in the official Istio documentation; observability is already built into the Istio design. This is not more complicated than following the steps to enable envoy access logging. Instrumentation of the native istio-ingress is thus achieved with the opentelemetry collector available for envoy; this exports to Dynatrace the metrics needed, collected from the cluster ingress.

b) Instrumentation of the Apigee hybrid API proxies.

Each Apigee hybrid API proxy, for each API call will be able to prepare the payload to feed the New Relic Trace API with an individual HTTP POST call as a PostClient (response) flow operation, without the introduction of any additional latency in the client response. If you are not familiar with the PostClient (response) flow in the Apigee proxies, you can review this document.

The reference to all the definitions of all Apigee hybrid "flow variables" is available in this document.

As the Message Processor pods will be emitting information to New Relic using the OpenTelemetry Protocol (OTLP).

Decorate spans with attributes

In New Relic it is possible to add attributes to trace data sent to the Trace API so that spans display specific properties in the UI. A detailed description is provided here. JSON examples are provided here in the documentation both for the New Relic format and the Zipkin format.

In conclusion

You can prepare the payload variable for the HTTP POST operation using the Apigee Assign Message Policy as described here. You can then progress to assemble the Apigee Service Callout policy following the policy documentation here. In the example below, the Assign Message Policy will populate the flow variable called _requestBody that contains the inner payload of the requestBody. This is subsequently used in the second policy to perform the HTTP POST.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ServiceCallout async="false" continueOnError="false" enabled="true" name="NewRelicCallout">
    <DisplayName>NewRelicCallout</DisplayName>
    <Properties/>
    <Request clearPayload="true" variable="myRequest1">
        <Set>
            <Headers>
                <Header name="Content-Type">application/json</Header>
                <Header name="Api-Key">YOUR_LICENSE_KEY</Header>
                <Header name="Data-Format">newrelic</Header>
                <Header name="Data-Format-Version">1</Header>
            </Headers>
            <Verb>POST</Verb>
            <Payload contentType="application/json" variablePrefix="@" variableSuffix="#">
                {
                  "@_requestBody#"
                }
            </Payload>
        </Set>
        <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    </Request>
    <Response>newRelicResponse</Response>
    <HTTPTargetConnection>
        <Properties/>
        <URL>https://trace-api.newrelic.com/trace/v1</URL>
    </HTTPTargetConnection>
</ServiceCallout>

Full ServiceCallout Policy Example

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ServiceCallout continueOnError="true" enabled="true" name="SC_NEWRELIC">
<DisplayName>SC_NEWRELIC</DisplayName>
<Properties/>
<Request clearPayload="false" variable="newrelic-req">
     <Add>
         <Headers>
             <Header name="Content-Type">application/json</Header>
                <Header name="Api-Key">YOUR_LICENSE_KEY</Header>
                <Header name="Data-Format">newrelic</Header>
                <Header name="Data-Format-Version">1</Header>
         </Headers>
     </Add>
     <Set>
         <Verb>POST</Verb>
         <Payload>
{
  "resourceSpans": [
{
   "resource": {
     "attributes": [
       {
         "key": "service.name",
         "value": {
           "stringValue": "apigee-runtime"
         }
       }
     ]
   },
   "instrumentationLibrarySpans": [
     {
       "spans": [
         {
           "trace_id": "{traceid}",
           "span_id": "{span_id}",
           "parent_span_id": "{parent_span_id}",
           "name": "{request.uri}",
           "kind": 2,
           "start_time_unix_nano": {client.received.start.timestamp}000000,
           "end_time_unix_nano": {client.sent.end.timestamp}000000,
           "droppedAttributesCount": 0,
           "droppedEventsCount": 0,
           "attributes": [
              {
                 "key": "http.host",
                 "value":{
                    "string_value": "localhost"
                 }
                 },
                 {
                 "key": "http.status_code",
                 "value":{
                    "string_value": "{response.status.code}"
                 }
              },
              {
                 "key": "http.method",
                 "value":{
                    "string_value": "{request.verb}"
                 }
              },
              {
                 "key": "http.request_content_length",
                 "value":{
                    "string_value": "{request.header.content-length}"
                 }
              },
              {
                 "key": "http.response_content_length",
                 "value":{
                    "string_value": "{response.header.content-length}"
                 }
              },
              {
                 "key": "http.server_name",
                 "value":{
                    "string_value": "{target.host}"
                 }
              }
           ],
           "events": [
              {
                 "time_unix_nano": {system.timestamp}000000,
                                 "name": "event1",
                                 "attributes": [
                                     {
                                         "key": "exception.message",
                                         "value":{
                                            "string_value": "{escapeJSON(EventMessage)}"
                                            }
                                     }
                                 ]
              }
           ],             
              "status": {
             "code": {StatusCode},
             "message": "{StatusMessage}"
           }
         }
       ],
       "instrumentationLibrary": {
         "name": "local-curl-example"
       }
     }
   ]
}
 ]
}
         </Payload>
     </Set>
     <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
</Request>
<Response>dyna-res</Response>
<HTTPTargetConnection>
     <Properties/>
        <URL>https://trace-api.newrelic.com/trace/v1</URL>
    </HTTPTargetConnection>
</ServiceCallout>

Further resources: