Source Control for API Proxy Development

Introduction

The objective of this document is to describe SCM strategies that are employed specifically for API proxy development and deployment by Apigee Customer Success Team. It assumes a distributed version control is used and contains heavy references on git concepts.

Repository Strategy

An SCM repository groups all development and release activities that are independently performed by a specific group of people with varying levels of access. For API proxy development an API proxy with a specific version fits this definition nicely. We want to keep each version of an API proxy separate from other versions of the same API proxy and from other API proxies. Therefore a single SCM repository with its own branching structure per API proxy makes sense.

For example:

customer-v1 (repo)

|- feature-123

|- master

|- intg

|- uat

|- prod

customer-v2 (repo)

|- feature-546

|- master

|- intg

|- uat

|- prod

identity-v1 (repo)

|- feature-10

|- master

|- intg

|- uat

|- prod

Branching Strategy

The recommended branching strategy is to have feature branches for individual user story development, a master branch as the integration point and mainline for stable code, and environment branches to manage code promotion through different environments all the way to production.

The following sections explain each of these branching concepts and discuss how hotfix operations can be performed to those branches.

Master Branch

The master branch is the mainline where all feature branches are created from and merged back into. It is the main integration point for all feature development.

This is the default branch from development point of view: the place you go to when you want the latest stable code, and also the place you go to when you want to start/finish development of a new feature. In git based source control systems it is a convention to call the default branch “master” therefore it is most logical to call the mainline master. However no direct commits should be done to this branch; it should only accept merges from other branches.

Master should always be a healthy branch and should contain stable code, so in theory (and increasingly in practice), we should be able to release after any commit on master branch - of course through a review and automated deployment processes running quality checks and tests.

Code in master branch is usually gets deployed to development environment first and then to “staging” or “integration” environments by automated deployment tools as the code is merged into other SCM branches.

Feature Branches

3931-feature-branch.png

The basic idea of a feature branch is that when you start work on a feature or a user story, you take a branch of the master to work on that feature. This feature branch does not have to be in the main repository where master branch lives; it can be in a completely different repository (as is the case for pull request process in open source projects) but the general approach is to keep all branches including feature branches in the same repository so team members can easily contribute to it. Once the feature is implemented and tested, it can then be merged back to master, therefore integrating the changes with the mainline.

One advantage is that team can choose to exclude or delay a feature completely from a particular release if it is not ready yet or dependent on other components not ready. This is generally done by delaying the merge activity of the feature branch into master branch until such time that it is ready to be released.

Another advantage is that developers can work on their own features in isolation from changes happening elsewhere. They can pull in changes from the master branch at certain intervals if they need to incorporate those changes in their work or to simply make the future integration less painful.

However this isolation is also a disadvantage if not managed correctly. The work in the feature branch will need to be merged with the master branch at some point and this will be a scary task if those two branches diverge further in time. Conflict resolution and merge tools help solve some of the problems but frequent integration of code from master to feature branch helps by enabling very early conflict discovery and eliminating big bang merge operation in the future. This also enforces a best practice in continuous integration which is for everyone to integrate their work with master at least once a day.

Once the feature is ready and tested, it can be integrated with the master branch. This integration point gives the team an opportunity to review and comment on the proposed changes. This process is generally referred to as “pull request” or “merge request” which is an advantage of using feature branches rather than direct commits on master branch.

Code in feature branches are usually deployed to development environment by automated deployment tools.

Environment Branches and Code Promotion

API proxy code needs to be deployed to an Apigee environment for execution - it cannot be executed outside Apigee. This is a dependency for integration and performance testing and therefore a big consideration for the entire continuous integration and delivery process. During the initial design phase of an API program we get together with the customer’s development and ops teams to come up with the environment list and code promotion strategy based on their existing software development lifecycle and requirements.

The idea is then to create a branch per environment and control code promotion by merging code from one branch to another. Such branches are commonly referred to as “environment branches” and are created downstream to master. For example we can have a branch called “staging” and whenever a commit is merged to this branch we can have the CI process trigger a build and the deploy API proxy to the “staging” environment in Apigee.

environment_branches.png

A common branch configuration and environment mapping is as follows:

BranchEnvironmentAPI TargetTesting
feature-123devMock targetFull integration
masterdevMock targetFull integration
intgintgActual DEV backendPartial integration
uatuatActual UAT backendUAT
prodprodActual PROD backendSmoke

See also https://community.apigee.com/articles/23210/coordinating-api-and-app-development-cycles.html and https://community.apigee.com/questions/19380/one-api-team-or-many.html for more views on environment design.

In this configuration a new UAT release can be done by merging intg branch to uat branch, and a production release can be done by merging uat branch to prod branch.

When environment progression is controlled by branch merges, it can be reviewed/approved using pull/merge request process which is a great benefit from SDLC point of view. It also enables scenarios where process is initiated by a specific person/team but approved and finalised by a different person/team.

It is important to make sure that the natural flow of commits is downstream (dev -> intg -> uat -> prod) without bypassing any branches to ensure that everything has been tested on all environments.

Using downstream merges, we can do a diff of different branches to understand the differences between individual releases. For example if we diff uat and prod branches we can see the commits that are scheduled to go live in the next production release. We can also look at history log of an environment branch to see the release history for that environment.

Hotfixing

This section explains the hotfixing process where a bug is found in production environment that needs to be fixed urgently without releasing all other commits in master. We employ different strategies based on the similarity of the code segment between master and production branches.

Production branch is quite close to master

This scenario applies when there are no changes in the buggy code segment in master branch compared to production branch. This is usually seen in teams who are practicing continuous delivery where every push to master branch gets deployed to production in a matter of minutes or in teams where code gets deployed to production passing through all other environments without any human intervention. Human intervention in the delivery lifecycle, e.g. manual testing or UAT process, increases the time it takes for a commit on master to go live therefore increases the possibility of another developer modifying or refactoring code in master branch.

The hotfixing process for this scenario can be summarised as:

  1. Make sure bug is tracked under issue tracking software.
  2. Create a new hotfix branch from master branch.
  3. Deploy and test hotfix in dev environment.
  4. Merge hotfix branch to master branch.
  5. Wait for CI to validate the fix.
  6. Cherry-pick the commit to other environments and let CI deploy the fix through the environments up to production.
  7. Resolve bug tracked under issue tracking software.

Production code is different than master

This scenario applies when the code to be fixed in master branch is quite different than the code in production branch - possibly due to refactoring or new feature implementations.

The hotfixing process for this scenario can be summarised as:

  1. Make sure bug is tracked under issue tracking software.
  2. Create a hotfix branch from production branch.
  3. Deploy and test hotfix in dev environment.
  4. Merge hotfix branch to production branch and let CI deploy to production environment.
  5. Create a new hotfix branch from master
  6. Deploy and test hotfix in dev environment
  7. Merge hotfix branch to master branch.
  8. Wait for CI to validate the fix.
  9. Cherry-pick the commit to other environments below production and let CI deploy the fix to environments. This is required to fix the bug in all other environments apart from production as the bug is already fixed in production on item 4 above.
  10. Resolve bug tracked under issue tracking software.

Pull Requests or Merge Requests

Pull/merge requests are the recommended way of merging one branch to the next downstream branch. Person responsible for the proposed changes in the source branch initiates the process by creating a new pull/merge request and asking an assigned person to merge to the next downstream branch. This forms a natural common place for the whole team to get together to review and discuss the proposed changes.

This process can also be used to share an in-progress work and ask for an early review. In this scenario changes are not ready to be merged downstream but feedback is welcome. Other team members can then discuss the changes, provide review feedback or push new code to the branch under review. This is quite useful and recommended when the work is extremely important for the project/team or if it is highly complex. It can also be used as a training opportunity for junior developers/contributors within the team.

For SCM that support it, voting on pull/merge requests is an excellent way of collecting general feedback for the proposed changes instead of relying on a single assignee’s views. This idea can be taken forward by establishing a minimum “Ready for Merge” criteria that requires 2 or more +1s from the team members.

Most importantly pull/merge requests must not to be used as an opportunity for punishment or trolling. This will create a negative atmosphere around the whole project and discourage participation. Instead what is needed is for everybody to express their opinions with no reservations on the changes proposed to promote healthy discussions within the team.

Multiple deployments on one environment

Apigee identifies an API proxy by it’s virtualhost and basepath combination. Virtualhost defines the domain name, port and TLS properties of the API proxy listening for incoming requests. Basepath is a segment of URI that comes after the domain name (and port).

Therefore it is possible to deploy different code for the same API proxy in the same environment by choosing a different virtual host and/or basepath combination. The approach that is generally followed is to keep the same virtual host but append an identifier to the basepath configuration during deployment, e.g. /customer/v1-oseymen. This identifier can be an issue id, feature branch name, username of the machine or git user, etc.

Using this approach, you can develop and deploy code from multiple feature branches at the same time without clashing or undeploying API proxies belonging to someone else. In this scenario only CI will deploy the API proxies with their correct virtual host and basepath combinations in a specific environment.

See also https://community.apigee.com/articles/26716/api-proxy-team-development-with-maven.html for implementation tips on team development principals using maven.

Best Practices on Commits

Commit Messages

Best practice when writing commit messages is to reflect the intention of the implemented changes rather than the contents of the actual commit which are already visible from file diff. Therefore it is recommended to concentrate on the reason (or the “why”) of the commit rather than the contents (or the “what”).

It is also recommended to include the identifier of the issue from issue tracking software in the commit message.

Clean History

Commit history is extremely important for the whole team especially when it is used to understand the changes between two environments. Therefore it is important to keep it concise and clean as much as we can.

One of the techniques we can use to achieve that is to squash multiple small changes during development and replace them with a single commit. The other is to change the order of changes to make it more logical. The only caveat is to refrain history modifications if the code is already pushed to origin.

Summary

Defining an efficient repository and branching strategy will improve your software development and delivery lifecycle. It is very important to reduce complexity as much as we can so processes are lean and fast to execute while at the same time stay compliant to rules and regulations of the environment we are working in.

Comments
Not applicable

Hi,

I am working on a product which we envisage will have a few set of API proxies; such as,

GET / POST / PUT / DELETE /registrations

GET / POST / PUT / DELETE /flights

GET / POST / PUT / DELETE /airports

Together the above list of API proxies will make an API Product. I was of the opinion that I should have one repository mapped to this API product containing multiple folders with each representing one API proxy. Of course I will adhere to Gitflow workflow branching model but at the level of API product (mapped to Git repository).

However, when I read your article above, you suggesting completely different approach than what I am thinking. You are suggesting;

1 repo = 1 API proxy

but I am thinking 1 repo = 1 API product (= multiple API proxies)

What are your thoughts to what I am thinking?

Thanks in advance,

Sapan

ozanseymen
Staff

Couple of questions:

  1. What do you think will be the name of the repo that contains those 3 proxies in your scenario?
  2. What is your plan for supporting v2 of /flights?
  3. Why do you think each of those resources should be exposed as separate API proxies?
  4. What is the trigger for a CI build when a new revision of /flights is committed?

I'd love to hear your answers but you can see where I am going with these... 🙂

Former Community Member
Not applicable

Hi @ozanseymen, I am planning to use this repo setup for my sharedflows and proxies. I have the following branches: master, non-prod, prod.

I am using apigee deploy maven plugin to deploy them to APIGEE. I have 1 repo for each sharedflow.

I have the sharedflow common code in master. The only two files that are specific to an org and its environments are "parent-sharedflow-pom.xml" and "sharedflow/config.json". I have created and added these two files to each branch (non-prod and prod) with org specific content.

When I make a feature branch and then merge to master, I am able to merge the changes to non-prod without overwriting the org specific files. If i have to merge the same changes from non-prod to prod, it will overwrite the two files also.

How do the files need to organized so that the code promotion from master -> non-prod -> prod works?

optimism
Participant V

Hi, Community!
I'm in the process of choosing a branching, release strategy.
I'm using GitLab CI/CD & Maven plugin for Apigee Hybrid deploys.
There are dev & prod environments of the Apigee Hybrid.

Anything changed & improved in this area since the topic starter asked this question back in 2016?

thanks!

BR, Optimism

malligowd
Community Visitor

Dear community members,

We are also in the process of solutionizing branching strategy for Apigee Hybrid for bitbucket repo. Please do suggest if any improvements made in this area. The current provided solution is way too expensive to maintain when we have 500+ API's to be maintained. Please suggest.

 

Best Regards,

Mallikarjun

jose_chavez1
Explorer

Hi @ozanseymen!

I totally agree with using a trunk-based branching strategy here! Also keeping a repo per Apigee proxy is brilliant, especially if you have multiple teams managing multiple microservices behind the scenes!

One alternative to maintaining environment branches is to just maintain a release branch per version. We can use a Semver versioning scheme to create the branch names. 

For example, if the trunk branch is `master`, then you can create a release branch named `release-1.0.0` for the first API version. All production bug fixes and updates can be committed to the release branch and then also committed back to the trunk (master).

You can then promote the artifacts of the release branch to a test environment to undergo testing. Once testing is complete and passes, you can take the same artifacts and deploy to production.

Once the next release branch is ready, the previous release branch is retired to avoid more changes being applied there. This would reduce the need to maintain separate environment branches.

Version history
Last update:
‎11-16-2016 07:37 AM
Updated by: