Continuous Integration for API Proxies

Introduction

Continuous integration is a software development practice where members of a team integrate their work frequently and each integration is verified by an automated build. This results in multiple integrations a day allowing the team to develop cohesive software more rapidly. Therefore it is an essential practice for any agile team.

Continuous integration reduces integration headaches (conflicts, merge issues) and increases the speed of finding and fixing code errors. It’s automated build verification step ensures the system is still working as expected after many integrations and helps building confidence in the code.

The objective of this article is to explain continuous integration practice with respect to API proxy development and details some of the techniques and tools used for achieving it within the Apigee platform.

This article mentions some of the tools that are used by Apigee Customer Success team however it doesn’t necessarily recommend those tools over others. The emphasis should be on the capabilities of the tools rather than the tools themselves.

Virtuous Circle

Continuous integration cycle consists of the following steps:

  1. Source control
  2. Branch push and pull/merge requests triggering automated builds
  3. Static code quality analysis
  4. Unit testing
  5. API proxy deployment to an Apigee environment
  6. Integration testing
  7. API documentation updates

Source Control

Source control is an integral part of any software development project. It is where all source files, configuration, commits and history live - therefore it is where most of the integration happens between multiple developers in a team. All commits for a particular project must be made against a single mainline branch within a repository - we call this the master branch.

Source control is the single source of truth in terms of all code and configuration - even if an artifact control (like artifactory type system) is also being used to store packaged project artifacts. Once the artifact package is constructed from SCM, it can then be reused in subsequent build iterations as a ready-made package bypassing initial package creation from SCM.

More information on source control, especially for API proxy development can be found here: https://community.apigee.com/articles/34868/source-control-for-api-proxy-development.html

So our virtuous cycle starts with developer(s) within a team pushes their feature branch containing a number of commits to the repository (origin).

Static Code Quality Analysis

Apigee has the functionality to execute custom code written in Java, JS and Python placed in any flow (request or response). It also has the functionality to execute node.js code placed as a target to the API proxy. It is recommended that you run static code analysis for these custom code to promote consistency and language specific best practices within the team.

There are various tools available to do this for all of those languages. Most of them allow editor integration and source control hooks that can be adopted in the development environment. Most of the popular task runners offer watchers that can be used to trigger such analysis on changed files only. So there are no excuses really not to run quality analysis on code for local development.

When it comes to integration though we shouldn’t rely solely on local analysis but need to integrate such tools into our continuous integration process so that the process is automated to catch any quality issues if they are missed in the local development environment. Our recommendation is to choose a CI tool that can run your code quality analysis and fail the build fast if it finds any problems.

Code analysis tools come with predefined rules that checks various aspects of the code. Your team may not agree with some of the rules or conventions that tool is enforcing but nowadays most tools have the option of disabling some of the rules so you can configure the tool according to your team’s conventions.

Running such analysis will be especially useful if the whole team or some members are new to that language as you will be warned of common pitfalls, e.g. the use of === in JS.

Unit Testing

There are two main areas where it is recommended to write unit tests for Apigee API proxies:

  1. Scenarios where it is not possible to test the entire API proxy only with integration testing. These are operations that are hidden from the API client, e.g. JS callout to another API.
  2. Testing isolated, complex but crucial code, e.g. JS code that does complex IP blacklisting using various rules, encryption, signature generation

Test coverage is especially important for the second area mentioned above. Integration testing on its own also will not give you any coverage indicators. If you rely on code coverage for such complex code, unit testing is the way to go.

Other advantages of unit testing compared to integration testing for Apigee API proxies are:

  • Code can be tested locally during development without deployment to an Apigee environment first. Editors and source control systems can also be configured to run unit tests automatically during development.
  • Unit testing will execute much faster than integration testing as tests will not incur any network traffic and will be tested in isolation from other systems.

It is important to write unit tests for the custom code that your team is developing rather than for the out-of-the-box policies that are built by Apigee which are already passed through extensive testing before they are made available in the product. So the recommendation is to set the testing boundary to Java Callout, JavaScript Callout, Python Script policies and Node.JS targets only.

More information and implementation tricks on API proxy unit testing can be found:

API Proxy Deployment

The basic means of deploying an API proxy is via Apigee Management API. All other tools, including Apigee Management UI are using APIs to deploy a proxy.

This page from Apigee product documentation explains how API proxies are deployed to Apigee: http://docs.apigee.com/api-services/content/understanding-deployment

For deployment via the Apigee Management API, see http://docs.apigee.com/api-services/content/deploy-api-proxies-using-management-api and http://docs.apigee.com/management/apis/post/organizations/%7Borg_name%7D/environments/%7Benv_name%7D...

For deployment via the command line, see http://docs.apigee.com/api-services/content/deploying-proxies-command-line.

There are also various tools that are built within the community as open source projects to help with deployment:

The recommendation is to integrate one of the above deployment tools with your CI to perform API proxy deployment.

Integration Testing

Integration testing is one of the most obvious and important types of testing for API proxies. The general idea is to have your integration testing tool of choice simulate user requests hitting the API proxy which will then be hitting your target APIs.

We should be designing the integration testing such that it is executing requests on each request and asserting behaviour and data. Examples of behaviour in API proxies are traffic management, OAuth handshake or any particular behaviour exposed by the target or 3rd party APIs. Examples of data are error codes, response payload values and structure.

Please note that an API proxy must be deployed to an Apigee environment before it can serve HTTP requests from integration testing tool. Therefore integration testing step must be executed after deployment step in CI configuration.

Please see the following Apigee community articles for testing strategies and implementation guidance:

API Documentation

As API proxy interfaces are evolving over time, their documentation must also be kept in sync. Apigee Developer Services Portal has a feature called SmartDocs which lets you document your APIs in a way that makes the API documentation fully interactive. Interactive documentation means portal users can:

  • Read about your APIs
  • Send a live request to your API
  • View a live response returned from the API

You can represent your APIs by creating a model using WADL or OpenAPI (formerly Swagger) specification which can be modified during development and pushed to Apigee Developer Services Portal from CI using APIs so that your documentation is kept up to date with the API proxy interface.

Please refer to the following Apigee community articles for documentation strategies and automating API documentation:

API Configuration

A typical Apigee deployment includes modifications to environment configuration together with policies and custom code. These include changes to KVM, cache resources, target servers, products, applications, keystores, truststores, etc. Apigee exposes management API resources that can be used to manage environment configurations.

The recommendation is to automate modifications to Apigee environment configuration during CI builds using Management API resources or deployment tool of your choice.

The benefits include:

  • Configuration stored and maintained in SCM (single source of truth). Configuration changes subject to usual change control procedures (pull/merge requests, approval, etc). Changes implemented in a feature scope also include configuration modifications.
  • Changes applied in an automated fashion eliminating manual steps, mistakes and forgotten entries.
  • Audit of changes kept and managed through source control. Existing Apigee audit reporting do not include all configuration areas.
  • Apigee does not keep backups of your environment configuration. If you lose your configuration (mistakes, org/env removal, etc), you will need to reconfigure the environment from scratch without a master copy. Keeping configuration in source control and applying them automatically will save you time and keep you sane in these scenarios.

Refer to the following resources for implementation:

Continuous Integration Process

The overall process for continuous integration for Apigee proxies looks like the following:

  1. A feature branch is developed, tested and committed to SCM. It is pushed to origin.
  2. CI automatically picks up the change in the feature branch and starts the activities explained above to validate and deploy the implementation in development environment.
  3. Once changes are validated, the developer creates a pull/merge request to merge them into master. Changes are accepted and merged after review process.
  4. CI automatically picks up the change in the master branch and starts the activities explained above to validate and deploy the integration in development environment.
  5. Once integration is validated, CI automatically deploys the new master branch into integration or staging environment.
  6. From this point onwards, changes move through the environments up to the production environment. Each environment progression starts with that change merged with the target environment branch manually or automatically. Please see Source Control for API Proxy article for more information on environment progression. The recommendation is to rely on automated testing/verification instead of pausing the progression for manual testing or interventions in order to make the whole process as agile and fast as possible. This will move you towards continuous delivery.

Best Practices

The following general best practices for Continuous integration also applies to Apigee proxy development:

  • There should be a single source control repository, available for everyone with a mainline that integrates all feature changes.
  • Automation is favoured over manual verifications and deployments.
  • Each change should accompany tests that can be executed automatically by continuous integration tools in order to verify the build.
  • High frequency integrations to mainline is favoured over long living feature branches in order to minimise painful big merges later on.
  • Fixing failed builds on mainline should be the high priority over feature development for the whole team.

Summary

On the whole the main objective of CI is to reduce the risk of breaking changes reaching and deployed to production environment. By relying on an automated process that validates and deploys changes as they are happening increases the trust and improves the speed of new features and improvements delivered to your target audience.

Comments
adas
Participant V

Awesome post @Ozan Seymen. Very insightful and great ideas for proxy/api development.

Not applicable

With Apigee API proxies getting developed (create, configure, and manage) in the cloud directly in the Edge management UI,

1. Can I assume these proxies are not included in the CI process? If not, do you expect the developer to export these proxies (manually or through management APIs) and check-in into git repo?

2. Are only custom plugin developments like custom policies, node.js / java code built outside (of the Egde UI) etc. considered for this CI process?

Thanks.

sidd-harth
Participant V

Yes, we can use Management API calls to manually push the proxy source to SCM. Within CICD pipeline we can Apigee deploy using wide range of plugins like Apigee Maven Plugin.

Please check Apigee Maven Plugin , it can do many things.

https://community.apigee.com/articles/8729/apigee-tools-plugins-apigee-development-made-easy.html

ozanseymen
Staff

Hi @Vet D

1. Edge Management UI doesn't have any integration with source control at the moment. Therefore the only way is to download the revision and push to source control manually. That is why I recommend editing proxy configuration and custom code locally in an editor of your choice instead of using management UI IF it is your intention to do CI/CD.

2. Yes, a deployment "bundle" for Apigee contains all custom code under /resources folder. The root folder ("apiproxy") is zipped which contains all code.

santiago
New Member

Excellent post!

I see there is a case not covered, related to the sync between the development pipeline of the backend system, and the api itself.

What is the best way to ensure changes in both components be aligned?

Thanks

martinsvensson-
New Member

Indeed a great post, thanks for sharing.

dchiesa1
Staff

Timeless guidance, Still valuable and relevant. Thank you!

Version history
Last update:
‎11-25-2016 02:15 AM
Updated by: