Apigee - Eventarc - Workflow

[ganadurai]

Apigee Event Processing via GCP Eventarc and Workflows

Tracking and Processing Apigee Management Events

Actions on the Apigee management plane get recorded via audit logs to track events/changes that occur. Few to mention:

• Actions on API Proxies (Create, Update, Deploy, etc) 
• Actions on API Products (Create, Update, Deploy, etc) 
• Actions on Developer and Developer Apps 
• and more.

For information about Apigee audit logs, see the details here.

To view a sample of these events in Cloud Logging for your GCP-Project / Apigee-Org execute the below (adjust the query based on your needs):

protoPayload.methodName=~"google.cloud.apigee.v1.Deployment*"
OR protoPayload.methodName=~"google.cloud.apigee.v1.Api*"
OR protoPayload.methodName=~"google.cloud.apigee.v1.Target*"
OR protoPayload.methodName=~"google.cloud.apigee.v1.Developer*"
OR protoPayload.methodName=~"google.cloud.apigee.v1.Environment*"
OR protoPayload.methodName=~"google.cloud.apigee.v1.Project*"
OR protoPayload.methodName=~"google.cloud.apigee.v1.Organization*"
NOT protoPayload.methodName="google.cloud.apigee.v1.RuntimeService.ReportInstanceStatus"
NOT protoPayload.methodName="google.cloud.apigee.v1.EnvironmentService.Subscribe"
NOT protoPayload.methodName="google.cloud.apigee.v1.EnvironmentService.Unsubscribe"
NOT protoPayload.methodName="google.cloud.apigee.v1.DeploymentService.GenerateDeployChangeReport"

There is always a need to better capture the above events and process the events by posting it to a Cloud Function, Pub/Sub, Cloud Run or to an external http endpoint.

One of the ways to achieve the above is by utilizing GCP EventArc. An Eventarc trigger enables capturing specific events from Cloud Logging audit logs and acting on it.

Sample Implementation

Follow the below steps to capture an Apigee Developer create event via EventArc and post it to GCP Workflow. In this example the Workflow posts the audit log payload to an HTTP endpoint. Follow the steps within your GCP Cloudshell.

1. Initialize variables
   
   PROJECT_ID=<GCP Project Id>
   USER_ID=<GCP User email>
   TRIGGER_SA=<Service Account Name, created and used in this setup>
   
   gcloud config set project $PROJECT_ID
   export PROJECT_NUMBER="$(gcloud projects describe ${PROJECT_ID} --format='get(projectNumber)')"
   
   
2. Enable the APIs
   

   gcloud services enable \
   logging.googleapis.com \
   eventarc.googleapis.com \
   workflows.googleapis.com \
   workflowexecutions.googleapis.com \
   pubsub.googleapis.com

   
   
3. Grant user to admin EventArc
   

   gcloud projects add-iam-policy-binding ${PROJECT_NUMBER} \
   --member=user:$USER_ID --role=roles/eventarc.admin
   
   

4. Create Service Account and Assign the needed roles
   

   gcloud iam service-accounts create ${TRIGGER_SA}

   gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member=serviceAccount:${TRIGGER_SA}@$PROJECT_ID.iam.gserviceaccount.com \
   --role=roles/workflows.invoker

   gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member "serviceAccount:${TRIGGER_SA}@$PROJECT_ID.iam.gserviceaccount.com" \
   --role="roles/eventarc.eventReceiver"

   gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member "serviceAccount:${TRIGGER_SA}@$PROJECT_ID.iam.gserviceaccount.com" \
   --role "roles/logging.logWriter"
   
   

5. Create Workflow yaml
   In the below code snippet, replace "<endpoint-to-post-data>" with a valid url that can take in the event payload

   cat <<EOF > workflow.yaml
   main:
       params: [input]
       steps:
       - registerPayload:
           call: http.post
           args:
               body:
                   payload: \${input}
               url: <endpoint-to-post-data>
           result: httpOutput
       - returnOutput:
               return: \${httpOutput.body}
   EOF
   
   

6. Deploy Workflow
   
   gcloud workflows deploy developer-create-trigger-workflow \
   --source=workflow.yaml \
   --location=us-central1 \
   --service-account=${TRIGGER_SA}@$PROJECT_ID.iam.gserviceaccount.com
   
   
7. Create Eventarc trigger that posts data to Workflow
   Wait for few minutes for the enabling of APIs to take effect and then run the below command. If error occurs due to permissions, wait for more time and run the below command.
   
   gcloud eventarc triggers create apigee-developer-create-workflows-trigger \
   --location=us-central1 \
   --destination-workflow=developer-create-trigger-workflow \
   --destination-workflow-location=us-central1 \
   --event-filters="type=google.cloud.audit.log.v1.written" \
   --event-filters="serviceName=apigee.googleapis.com" \
   --event-filters="methodName=google.cloud.apigee.v1.Developers.CreateDeveloper" \
   --service-account="${TRIGGER_SA}@${PROJECT_ID}.iam.gserviceaccount.com"
   
   
8. Validating the setup
   
   1. Create Developer via the Mgmt api or the console
   2. Check Cloud logging for the presence of the audit log in your corresponding GCP project.
   3. Check Eventarc for the trigger invocations
   4. Check Workflow for the executions triggered via Eventarc.