CI/CD for Application Integration

[ssvaidyanathan]

Introduction

Application Integration is an Integration-Platform-as-a-Service (iPaaS) solution in Google Cloud that offers a comprehensive set of core integration tools to connect and manage the multitude of applications and data required to support various business operations.

Application Integration is a low-code/no-code platform that allows developers to create integration flows. Integration flows are sequences of tasks or activities that connect and coordinate the exchange of data between different systems. It is a common practice to develop these assets and promote them to various environments just like source code.

The best practice for CI/CD with Application Integration is to use a GCP project for each phase of SDLC. In this example, we will demonstrate how to automate the promotion of Application Integration and Integration Connectors artifacts between two SDLC phases - development (dev) and qa environments.

Prerequisites

• Application Integration and Integration Connectors are enabled and provisioned. If you have not done so already, please follow this guide
• Basic working knowledge of Application Integration and Integration Connectors.
• GitHub is used as the example source code repository. A GitHub account will be necessary to complete the instructions
• Download gcloud and configure the default project

gcloud config set project $project

Principles

There are a few (opinionated) principles followed in this example:

1. An integration flow developed in the development environment must be deployed unchanged in production. Treat the integration flow like one would source code.
2. The portions of an integration flow that do change from one environment to another are externalized. Only those properties change between environments. For example, the HTTP URL configured for a REST Task.
3. There must be traceability between what is stored in a source code repository and an integration flow version deployed in an environment.
4. Maintain a single repository for each deployable service
5. Automate the deployment between environments
6. Test in a clone of the production environment

Steps to Automate Application Integration Deployments

In this example you will see how to store integration and connector assets to a source code repository, promote those assets from one SDLC environment to another and finally, how such deployments can be automated.

Sample Integration

We will build a sample integration with minimal complexity to demonstrate this use case. This  Integration flow calls an API and publishes the response from the API to a Pub/Sub topic. This sample is meant to illustrate the use of REST and Connector tasks.

Create a topic

gcloud pubsub topics create mytopic

Create a connection for Pub/Sub

In the GCP console, navigate to “Integration Connectors” from the left menu and click the “+ CREATE NEW” button to create a new connector. 

 

Please change the Service Account and Project ID appropriately. Ensure the service account has privileges to publish to the PubSub topic. This may take a few mins

Create an Integration Flow as follows

1. Create a new Integration “sample” with description “Sample Integration for CI/CD” and select your appropriate region. Click “CREATE”

2. In the Integration designer, click the “+ CREATE” in the Variables section to create some config variables. Make sure you select “Config Variable for Integration” in the Variable Type dropdown

3. Similar to the step above, create another config variable and called it “URL”

4. One more config variable called “CONN_NAME”

Once all the variables are created, you should see them in the Variables section

5. Add an API Trigger and wire to the REST task. In the REST Task config, select the URL config variable

Add the config variable “ENV” to a custom http header “X-INTEGRATION-ENV” in the REST task as shown below. NOTE: the variable name will be `CONFIG_ENV`

6. Add a Connectors Task, do not wire it yet. Select the PubSub connection created previously

Select the connection

Click “NEXT”. Click “Actions” as the TYPE. Click “NEXT” again

Select “Publish Message” for “Action” and click “NEXT”

Click DONE. This should configure the Connector task.

In the connector task, change the “Connection Name” to use the “CONN_NAME” variable

7. Add a data mapper task and wire the tasks as shown below

8. Click the Data Mapping task and open the Data Mapper editor. In the Data Mapper Editor, map the output variable of the REST task to the input variable of the Connector task

9. Test the integration to ensure the flow works successfully. To Test, click the “TEST” button and in the input section, provide the following values

`CONFIG_ENV`	dev
`CONFIG_URL`	https://httpbin.org/get
`CONFIG_CONN_NAME`	projects/$PROJECT/locations/$REGION/connections/pubsub

NOTE: Replace PROJECT and REGION with your appropriate values

Now hit “Test integration”. You should get a successful response.

10. Now click “Edit Integration”. In the Variable panel on the left, click “connectorOutputPayload” and select “View Details”

In the Variable panel, under Variable Type, change it from “None” to “Output from Integration” and click “SAVE”

Similarly set the “responseBody” variable type to “Output from Integration” and click “SAVE”

Now Publish the Integration again with the same set of config variable value

`CONFIG_ENV`	dev
`CONFIG_URL`	https://httpbin.org/get
`CONFIG_CONN_NAME`	projects/$PROJECT/locations/$REGION/connections/pubsub

NOTE: Replace PROJECT and REGION with your appropriate values

Now hit “Publish integration”. Click “Test”. You should get a successful response.

In the new response, you will see the output variable printed for both the HTTP call that was made and the response message ID from Pub/Sub. You should see “X-Integration-Env” in the output

In this example, the following details may change between environments:

1. The URL used in the REST task
2. The project where the PubSub connection exists (for ex CONN_NAME variable)
3. The topic name
4. The variable content (for ex X-INTEGRATION-ENV variable)

Prepare Github

Create a new repository. It is recommended that every integration flow uses a different repository.

mkdir app-integration-demo && cd app-integration-demo

git init
git checkout -b dev

In this step, the dev branch is created. The recommended folder structure for Integration and Connectors are as follows:

├── cloudbuild.yaml #the cloud build deployment file
└── <env>
│   ├── connectors
│   │   └── <connector-name>.json #there is one file per connector. the connector name is the file name.
│   ├── config-variables
│   │   └── <integration-name>-config.json #there is one file per integration. 
│   ├── authconfigs
│   │   └── <authconfig-name>.json #there is one file per authconfig. the authconfig name is the file name.
│   ├── endpoints
│   │   └── <endpoint-name>.json #there is one file per endpoint attachment. the endpoint attachment name is the file name.
│   ├── zones
│   │   └── <zone-name>.json #there is one file per managed zone. the managed zone name is the file name.
│   ├── sfdcinstances
│   │   └── <instance-name>.json #there is one file per sfdc instance. the sfdc instance name is the file name.
│   ├── sfdcchannels
│   │   └── <instance-name_channel-name>.json #there is one file per sfdc channel. A combination of sfdc instance name and channel name is the file name.
│   ├── overrides
│   │   └── overrides.json #always name this overrides.json. there is only one file in this folder
│    └── src
│       └── <integration-name>.json #there only one file in the folder. the integration name is the file name.

Introduction to integrationcli 

integrationcli is a tool that lets you interact or manage (create, delete, get, list integrations and connections) with Application Integration, Integration Connectors or Apigee Integration/Connector APIs. This example uses this tool to automate deployments. You can see other examples and options here. 

Install the CLI with the following command:

curl -L https://raw.githubusercontent.com/GoogleCloudPlatform/application-integration-management-toolkit/main/downloadLatest.sh | sh -

Set integrationcli preferences:

token=$(gcloud auth print-access-token)
project=<set DEV project here>
region=<set DEV region here>

integrationcli prefs set -p $project -r $region -t $token

Create a scaffold for the Integration

integrationcli integrations scaffold -n sample -s 1 -f app-integration-demo -e dev

Where “-n sample” is the name of the integration, "-s 1" is the snapshot number, "-e dev" is the environment name which creates a directory under which the files are stored and "-f app-integration-demo" is the folder to generate the artifacts . This command downloads the integration, connections used by the integration, authconfigs, sfdc instances & channels in the folder structure mentioned previously.

Overrides file

Integration flows contain values that can change between environments. For example,

• The Connector task needs to be changed as you migrate from one project to another
• The REST task & Cloud Function tasks may have different values between environments

integrationcli makes it easy to generate an overrides file for values that typically change between environments. 

The file should look like this (./dev/overrides/overrides.json)

{
   "task_overrides":[
      ],
   "connection_overrides":[
      {
         "taskId":"2",
         "task":"GenericConnectorTask",
         "parameters":{
            "connectionName":"pubsub"
         }
      }
   ],
   "param_overrides":[],
   "integration_overrides":{
      "databasePersistencePolicy":"DATABASE_PERSISTENCE_POLICY_UNSPECIFIED",
      "enableVariableMasking":false,
      "cloudLoggingDetails":{
         "cloudLoggingSeverity":"CLOUD_LOGGING_SEVERITY_UNSPECIFIED",
         "enableCloudLogging":false
      }
   }
}

The values of "databasePersistencePolicy", "enableVariableMasking", "cloudLoggingSeverity", "enableCloudLogging" can be overridden when the integration is promoted to other environments. Other overrides not automatically captured from the CLI may be added to the file at this time. Examples like retries, default values etc. may be added.

The scaffold also creates a cloudbuild.yaml that can help you with your pipeline.

Check in all the content to the dev branch.

git add --all
git commit -m 'first draft'
git push

The folder structure will look like this:

├── dev
│ ├── config-variables
│ │ └── sample-config.json
│ ├── connectors
│ │ └── pubsub.json
│ └── overrides
│   └── overrides.json
└── cloudbuild.yaml
└── src
    └── sample.json

Now copy the “dev” directory and paste it as “qa”. The new structure of the repo is

├── dev
│ ├── config-variables
│ │ └── sample-config.json
│ ├── connectors
│ │ └── pubsub.json
│ └── overrides
│   └── overrides.json
├── qa
│ ├── config-variables
│ │ └── sample-config.json
│ ├── connectors
│ │ └── pubsub.json
│ └── overrides
│   └── overrides.json
└── cloudbuild.yaml
└── src
    └── sample.json

In the newly copied “qa” directory, make changes to the sample-config.json, overrides.json and pubsub.json files that are specific to the qa environment. In this example, we will change the variable to qa.

overrides.json fragment:

"integration_overrides": {
	"databasePersistencePolicy": "DATABASE_PERSISTENCE_POLICY_UNSPECIFIED",
	"enableVariableMasking": true,
	"cloudLoggingDetails": {
		"cloudLoggingSeverity": "INFO",
		"enableCloudLogging": true
	}
}

Sample-config.json:

{
	"`CONFIG_CONN_NAME`": "projects/$PROJECT/locations/$REGION/connections/pubsub",
	"`CONFIG_ENV`": "qa",
	"`CONFIG_URL`": "https://mocktarget.apigee.net/echo"
}

Replace the PROJECT and REGION values with your QA Project and region. Notice that the value if CONFIG_ENV and CONFIG_URL have changed.

Commit all the content to the dev branch.

git commit -am 'adding qa configurations'
git push

Create the QA branch

git checkout -b qa
git push

Manual deployments to QA

The artifacts in the qa branch can now be deployed to the QA environment (GCP project). Set integrationcli preferences for the QA environment:

token=$(gcloud auth print-access-token)
project=<set QA project here>
region=<set QA region here>

integrationcli prefs set -p $project -r $region -t $token

integrationcli can be used to apply changes generated by scaffold. This will automatically provision all the necessary entities in the right order

integrationcli integrations apply -f . -e qa --wait=true

Once you test the integration, you should see that the integration should point to the new URL and also have “qa” as the custom header that was set in the sample-config.json file

Integration with Cloud Build

To promote changes into upper environments like QA, Stage, you would need to set up DevOps pipelines for example, Jenkins, Gitlab, or CloudBuild, etc. For this demo purpose, we are going to use Cloud Build, you can learn more about cloud build here.

This example uses a custom cloud builder called integrationcli-builder:

us-docker.pkg.dev/appintegration-toolkit/images/integrationcli-builder:latest

Please follow the instructions here to set up Cloud Build to deploy Application Integration and Integration Connectors.

There are few options in the cloud build YAML to consider:

substitutions:
  _CREATE_SECRET: "false" # do not create Secret Manager secrets
  _GRANT_PERMISSIONS: "true" # grant service account permissions
  _ENCRYPTED: "false"    # use encryption
  _DEFAULT_SA: "false"   # use default service account
  _SERVICE_ACCOUNT_NAME: # the service account for connectors
  _KMS_RING_NAME:        # name of the KMS key ring
  _KMS_KEY_NAME:         # the name of the cloud kms key
  _WAIT: "true"

Please change them to appropriate values esp. if using Cloud KMS for encryption.

Manual triggering of builds

gcloud builds submit --config=cloudbuild.yaml --project=$project --region=$region

NOTE: The integration revision is labeled (userLabel field) with the SHORT_SHA of the commit code in github. This is to provide traceability between what is stored in the source code repository and the published revision.

Automate deployments with triggers

Setup a Cloud Build Trigger as shown in the screenshot below

This will trigger deployments to the qa environment automatically as soon as changes are merged with the qa branch.

Similar to QA, you can create a directory called “prod” and then move it to a “prod” branch using a pipeline to your “prod” environment. 

Advanced Options

When developing integrations, there may be a need to store sensitive information. For example, 

1. An authconfig may contains api keys or passwords
2. Connections to databases, FTP servers etc. require username and password

Read more about advanced options here to use Cloud KMS to encrypt sensitive information.

Special Thanks to Nandan Sridhar for his collaboration on this article and also for all his contribution to integrationcli

[meghangupte]

Hi @ssvaidyanathan ! This is a very informative article. What is the recommended way of handling functional tests with CI/CD? Ideally we would want to run the functional tests on production when we are promoting the integration from dev to prod

[ssvaidyanathan]

@meghangupte - Thank you for the feedback. We are adding some features to the product around this. Stay tuned!

[meghangupte]

Hi @ssvaidyanathan Looks like integrationscli doesn't support config variables. Support for config variables was added in https://github.com/GoogleCloudPlatform/application-integration-management-toolkit/releases/tag/v0.75... but its still in pre release phase. When is the planned release?

[ssvaidyanathan]

Yes - its in Pre-release along with few other features we are adding. We plan to push the main release in a few days.. Until then you can install and use that version

export INTEGRATIONCLI_VERSION=v0.75.0-beta.3
curl -L https://raw.githubusercontent.com/GoogleCloudPlatform/application-integration-management-toolkit/main/downloadLatest.sh | sh -