Deployment of APIGEE using Codeship

Not applicable

Has any one used Codeship as CI/CD and and retrieve secrets from a google cloud store? Has any one has a bash script samples for deployment

0 1 233
1 REPLY 1

Not applicable

Well , we have a setup where we are developing most of our micro-services on the GCP platform and exposing them using the APIGEE gateway . As a CI/CD setup we are using the Codeship tool. One of the advantages of using Codeship, is it is very straight forward to configure a Codeship corresponding to a commit in a specific branch in GIT or another version control tools. You can configure build triggers either for a specific Branch or Branch name starting with a condition. (You can also use git tags to achieve a similar functionality). Even though Codeship allows us to create environment variables to retrieve username and password during runtime , we would rather use the Google cloud storage to make them more secure ( as we already running on GCP Platform). We built a bash script , which gets triggered on a specific deployment pipeline and does the needful. The script is more of draft version and can be enhanced and optimized for more use cases. On a high level , the script on triggered does this

1. Identifies the commit id and corresponding commit for which the job is triggered

2. Retrieve the Org and environment for which this need to deployed based on the commit parameters

3. Retrieves the username/pwd to deploy from GCS

4. Retrieves the entire list of proxies in that repository

5. Iterates the list of proxies

5.1 Identifies the current version deployed specific to the environment

5.2 registers a rollback command for that version ( if deployment fails , we need to rollback to previous version)

5.3 Deploys the proxy to the org and env , if build fails rollback

5.4 Executes the newman test cases for that environment

5.5 If testcases successful , go to the next proxy , if not rollback

Again this script can be used with multiple CI/CD tools accordingly with minor modifications.

#!/bin/bash --login


set -e
###########################
## Shell script to build/deploy all the project to a specific environment based on tag in a specific repository
##
###########################


mgmtserver=https://api.enterprise.apigee.com
verbosity=2


function get_org {
  case "$CI_BRANCH" in
   PRE-DEV*) org=nonprod ;;
   develop*) org=nonprod ;;
   TEST*) org=nonprod ;;
   REL*) org=prod ;;
   RC*) org=nonprod ;;
   *) org=NODEPLOY ;;
 esac
 printf "$(tput setaf 2)-----DEPLOYING TO ORG - $org-----\n"
}


function get_current_environment {
  case "$CI_BRANCH" in
   PRE-DEV*) env=pre-dev ;;
   develop*) env=dev ;;
   TEST*) env=test ;;
   REL*) env=prod ;;
   RC*) env=uat ;;
   *) env=NODEPLOY ;;
 esac
 printf "$(tput setaf 2)-----DEPLOYING TO ENV - $env-----\n"
}


function set_service_account {


   if [[ ( $env == "prod") ]]
   then
		SECRETS_SERVICE_ACCOUNT=secrets-prod@XXXXX.iam.gserviceaccount.com
		SECRETS_SERVICE_ACCOUNT_KEY_FILE=$HOME/.gcp/${SECRETS_SERVICE_ACCOUNT}.json
		CLOUDSQL_DIRECTORY=$HOME/cloudsql
		environment=prod
		ENVIRONMENT_UPPER=PROD
		REPOSITORY_NAME=apigee_prod
		KEY_NAME=apigee-prod
		echo "[ Info ] Repository Name is '$REPOSITORY_NAME' , environment is '$environment', environment in upper case '$ENVIRONMENT_UPPER' "
   else
		SECRETS_SERVICE_ACCOUNT=secrets-dev@XXXXX.iam.gserviceaccount.com
		SECRETS_SERVICE_ACCOUNT_KEY_FILE=$HOME/.gcp/${SECRETS_SERVICE_ACCOUNT}.json
		CLOUDSQL_DIRECTORY=$HOME/cloudsql
		environment=dev
		ENVIRONMENT_UPPER=DEV
		REPOSITORY_NAME=apigee_non_prod
		KEY_NAME=apigee-non-prod
		echo "[ Info ] Repository Name is $REPOSITORY_NAME , environment is $environment, environment in upper case $ENVIRONMENT_UPPER "
	fi
}


function init {


  # Install Google Cloud SDK
  export CLOUDSDK_PYTHON_SITEPACKAGES=1; export CLOUDSDK_CORE_DISABLE_PROMPTS=1;


  if [ ! -d "/home/rof/google-cloud-sdk" ]; then
    (cd ~/; curl -s https://sdk.cloud.google.com | bash)
  fi


  source ~/google-cloud-sdk/path.bash.inc
  #Set service account key files
  mkdir -p ~/.gcp
  output=SECRETS_SERVICE_ACCOUNT_KEY_${ENVIRONMENT_UPPER}
  eval echo \$$output > ${SECRETS_SERVICE_ACCOUNT_KEY_FILE}
}


function run {


  echo "[ Info ] Update gcloud components"
  gcloud components update -q gcloud


  echo "[ Info ] Activate service account secrets"
  gcloud auth activate-service-account $SECRETS_SERVICE_ACCOUNT --key-file $SECRETS_SERVICE_ACCOUNT_KEY_FILE


  echo "[ Info ] Get secrets"
  curl -v "https://cloudkms.googleapis.com/v1/projects/${projectid}/locations/global/keyRings/$ENVIRONMENT_UPPER/cryptoKeys/${KEY_NAME}-key:decrypt" \
    -d "{\"ciphertext\":\"$(gsutil cat gs://$project-$environment/${REPOSITORY_NAME}.properties.encrypted)\"}" \
    -H "Authorization:Bearer $(gcloud auth print-access-token)"\
    -H "Content-Type:application/json" \
  | jq .plaintext -r | base64 -d > /tmp/${REPOSITORY_NAME}.properties
  
	#cat /tmp/${REPOSITORY_NAME}.properties
	
	source /tmp/${REPOSITORY_NAME}.properties
	
	if [[ $env == "prod" ]]
	then
		 echo "$APIGEE_USER"
		 echo "$APIGEE_PASS"
	else
		echo "$APIGEE_NONPROD_USER"
		echo "$APIGEE_NONPROD_PASS"
	fi
}






MYCURL() {
  local allargs
  local ix=0
  # grab the curl args
  while [[ "$1" ]]; do
    allargs[$ix]=$1
    let "ix+=1"
    shift
  done
  [[ -z "${CURL_OUT}" ]] && CURL_OUT=`mktemp /tmp/apigee-pushapi.curl.out.XXXXXX`
  [[ -f "${CURL_OUT}" ]] && rm ${CURL_OUT}
  [[ $verbosity -gt 1 ]] && echo && echo "curl ${allargs[@]}"
  
  # run the curl command
  
  if [[ $env == "prod" ]]
  then
	CURL_RC=`curl -u ${APIGEE_USER}:${APIGEE_PASS} -s -w "%{http_code}" -o "${CURL_OUT}" "${allargs[@]}"`
  else
	CURL_RC=`curl -u ${APIGEE_NONPROD_USER}:${APIGEE_NONPROD_PASS} -s -w "%{http_code}" -o "${CURL_OUT}" "${allargs[@]}"`
	[[ $verbosity -gt 1 ]] && echo "==> ${CURL_RC}"
  fi
}


function rollback_proxies {


	if [ $env == "prod" ]
	then
			if source /tmp/rollback_prod.sh
				then
					printf "===========================Successfully Rolled back proxies to previous versions==============================\n"
					rm /tmp/rollback_prod.sh
					false
				else
					printf "Script failed while rolling back to previous versions...check the logs\n"
					cat /tmp/rollback_prod.sh
					rm /tmp/rollback_prod.sh
					false
										
			fi
	else
			if source /tmp/rollback_nonprod.sh
				then
					printf "===========================Successfully Rolled back proxies to previous versions==============================\n"
					rm /tmp/rollback_nonprod.sh
					false
				else
					printf "Script failed while rolling back to previous versions...check the logs\n"
					cat /tmp/rollback_nonprod.sh
					rm /tmp/rollback_nonprod.sh
					false
			fi
	fi


}






function get_current_revision {
  
  MYCURL -X GET "${mgmtserver}/v1/o/$org/e/$env/apis/$proxydir/deployments"
    if [[ ${CURL_RC} -eq 200 ]]
    then
	  current_rev=(`cat ${CURL_OUT} | grep revision -A 5 | grep name | tr -d '\n'  | sed -E 's/"name"|[":,]//g'`)
      m=${#current_rev[@]}
		  if [[ $m -gt 0 ]]; then 
			printf "$(tput setaf 6)There are existing revisions of that API Proxy\n"
			printf "${current_rev[@]}\n"
		  fi
    elif [[ ${CURL_RC} -eq 401 ]]; then
		printf "$(tput setaf 1)Invalid username/pwd \n"
		false
    elif [[ ${CURL_RC} -eq 400 ]]; then
		current_rev=1
		printf "$(tput setaf 1)There are no current active revisions deployed of that API Proxy\n"
		printf "Setting the revision to ${current_rev[@]}\n"
	elif [[ ${CURL_RC} -eq 404 ]]; then
		current_rev=1
		printf "$(tput setaf 1)Deploying $proxydir for the first time\n"
		printf "Setting the revision to ${current_rev[@]}\n"
	fi 
  
}
function deploy 
{
  printf "$(tput setaf 6)[Info] $(tput setaf 7) Deploying $CI_COMMIT_ID\n"
  printf "$(tput setaf 6)[Info]  $(tput setaf 7) Deploying all the projects from $CI_REPO_NAME , $CI_BRANCH branch\n"
  cd /home/rof/src/github.com/$CI_REPO_NAME
  for i in $(find . -maxdepth 1 -type d -not -name ".git" -not -name "tmp" -not -name "log" -not -name .)
	do
		#printf "$i"
		proxydir=${i##*/}
		printf "$(tput setaf 6) $proxydir \n"
	    ( 		cd $proxydir; 
				printf " $(tput setaf 6) In $PWD \n";
				    if [ $env == 'NODEPLOY' ]
				    then
						 printf "$(tput setaf 1)Invalid Branch or Tag. Please check \n"
						 false
				    elif [ $env == 'prod' ] 
				    then
						printf "$(tput setaf 6)[Info] $(tput setaf 7)Inside prod deployment \n"
						
						get_current_revision
						printf "$(tput setaf 7)===================Adding rollback command for the proxy : $proxydir=======================\n"
						echo -e "apigeetool deployExistingRevision  -u \"${APIGEE_USER}\" -p \"${APIGEE_PASS}\" -o $org  -e $env -n $proxydir -r ${current_rev[@]}\n" >> /tmp/rollback_prod.sh
						
						if mvn clean install -P${env} -Dusername=${APIGEE_USER} -Dpassword=${APIGEE_PASS} -Dapigee.config.dir=resources/edge-prod -Dapigee.config.options=create
						then
								printf "$(tput setaf 7)===================Depolyment successful for the proxy : $proxydir=======================\n" 
								
						
								cd tests/$env;
								printf "$(tput setaf 6)[Info]$(tput setaf 7)RUNNING TESTS FOR $proxydir\t \n"
									
								if 	newman run $proxydir.json -e $env.json --ignore-redirects
								then 
									printf "$(tput setaf 7)===================SUCCESSFULLY EXECUTED TEST CASES=======================\n"
								else
									printf "$(tput setaf 1)=========================== TESTS FAILED====================================================\n"
									
									printf "===========================$(tput setaf 7)Rolling back proxies to previous versions as tests failed==============================\n"
									rollback_proxies
							    fi
						else
							printf "===========================$(tput setaf 1) Build failed==============================\n"
							printf "===========================$(tput setaf 7)Rolling back proxies to previous versions as build failed==============================\n"
							rollback_proxies
						fi


				    else
						printf "$(tput setaf 6)[Info] $(tput setaf 7)Inside $env (dev,test,uat) deployment \n"
						
						get_current_revision
						
						printf "$(tput setaf 7)===================Adding rollback command for the proxy : $proxydir=======================\n"
						echo -e "apigeetool deployExistingRevision  -u \"${APIGEE_NONPROD_USER}\" -p \"${APIGEE_NONPROD_PASS}\" -o $org  -e $env -n $proxydir -r ${current_rev[@]}\n" >> /tmp/rollback_nonprod.sh
						
						if mvn clean install -P${env} -Dusername=${APIGEE_NONPROD_USER} -Dpassword=${APIGEE_NONPROD_PASS} -Dapigee.config.dir=resources/edge -Dapigee.config.options=create
						then
						
								printf "$(tput setaf 7)===================Depolyment successful for the proxy : $proxydir=======================\n" 
								cd tests/$env;
								printf "$(tput setaf 6)[Info]$(tput setaf 7)RUNNING TESTS FOR $proxydir \t \n"
								if newman run $proxydir.json -e $env.json --ignore-redirects
								then 
									printf "$(tput setaf 7)===================SUCCESSFULLY EXECUTED TEST CASES=======================\n"
								else
									printf "$(tput setaf 1)=========================== TESTS FAILED======================================================== \n"
									
									printf "===========================$(tput setaf 7)Rolling back proxies to previous versions==============================\n"
									rollback_proxies
									
								fi
						else
							printf "===========================$(tput setaf 1) Build failed==============================\n"
							printf "===========================$(tput setaf 7)Rolling back proxies to previous versions as build failed==============================\n"
							rollback_proxies
						fi
				    fi
	    )
	done
 printf "$(tput setaf 2) =========================================SCRIPT COMPLETED SUCCESSFULLY=====================================================\n"
}
##############################################
#    Main
##############################################
printf "$(tput setaf 2)==========================================START OF DEPLOYMENT SCRIPT=========================================================\n"
get_org
get_current_environment
set_service_account
init
run
deploy


I have referred to @Dino bash scripts for reference. So thanks for that @Dino

If you are replicating the same on GCP , all you need to do is to set up service accounts on a specific project , to encrypt and decrypt the secrets.

We encrypted APIGEE related secrets in the format

Key="Value" , one for all environments.

Hope this helps..