Azure Devops CI/CD on Apigee Edge

Am trying to build a CI/CD pipeline for Apigee Edge on Azure Devops. Below is the requirement and what I have done so far.

Requirement:

1. Once Swagger specs are saved in swaggerhub, they need to be validated against custom rules.

2. After validation they need to be automatically imported into Apigee Edge as a basic proxy.

3. If possible this basic proxy needs to have some shared shows incorporated as well.

4. Then the dev team works on the proxy and completes development.

5. The new revision needs to be pushed into Github as a repo.

6. From the repo, the CI/CD pipeline should kick off with code analysis (Apigee bundle linter), Unit tests (postman?), Deploy (apigee-maven-plugin?), Tests (Apache Jmeter / apickli).

Note: The pipeline should support both OAS2.0 and OAS3.0.

Done so far:

Swagger specs validation is possible using a swagger-CLI tool, but API standardization within swaggerhub (available for Enterprise plans) provides inbuilt validation. Question is how to get custom rules built in.

OpenAPI2Apigee tool to import the spec into Apigee Edge has worked for me so far, as it keeps failing at the base URL.

I am trying to use curl commands to export the API proxy revision and then push it to Git to build the pipeline.

Please let me know if any of you have worked on anything similar before. If I could crack the OpenAPI2apigee tool i would have something to go on.

but the tool seems to be an interactive one, and in Azure devOps it keeps canceling the build after erroring out on the base URL.

0 14 3,795
14 REPLIES 14

@Nagashree B Please let me know if you have any leads. Thanks in advance!

@Aishwarya Kozhisherry

1. Firstly I'd suggest you validate your swagger specs against the specs which are provided in the samples - like petstore or mockapi. Did you try openAPI2Apigee using one of these specs. If these specs work, then there could be a problem with your api specs. Try and see what might be missing, like the basepath, schemes, host etc. If this doesnt work post the error here or try option 2 below.

2. Do you have access permissions to install Java on your host VM in Azure devops. If yes, you can use this utility, I have used this in my project to generate the proxy code. I am not sure if this supports openAPI 3.0 as this utility was built a few years ago and there have not been much updates to it, it definitely works for openAPI 2.0 - https://github.com/bharathkumarkn/Swagger-Utility

3. To include sharedflows - Once you have a proxy bundle that's generated, you can use powershell scripts to modify the XML files or add policy XMLs to include the shared flows

4. You can push the generated api proxy code to a git repo and have a .vsts-ci.yml also pushed to the same repo (with desired steps) to automatically create a CI pipeline in Azure devops.

5. I presume, you would be executing the unit tests in your CD pipeline, post deployment of API proxies, with postman collections checked in by the developers into the repo. You can use newman to execute the postman collection and then undeploy the current revision if the tests fail.

Since the API proxy is a bundle of XML files in a specific folder structure, you can also make an attempt to generate the code if you can find a openAPI 3.0 spec parser script in powershell or any other program that you can write likewise

Hope this helps.

@Nagashree B

Thanks a lot for the quick response.

1. I am using the petstore spec for all my trials so far. And no, I could not get openapi2apigee to work for that as well. Below is the error I get and the command used.

Command:

sudo openapi2apigee generateApi petStore -s http://petstore.swagger.io/v2/swagger.json -D -d /temp -b https://api.enterprise.apigee.com/ -o XXX-eval -u username -p password -e prod

8267-build-error.jpg

It fails with this

8268-build-error-2.jpg

I will try the swagger-utility as well today. Thanks!

Looks like the :// characters in the url are getting encoded. Did you try setting the url in the pipeline variables and using it, or try enclosing value in quotes in your command line.

@Nagashree B I tried both - enclosing the url in quotes and using it as a pipeline variable, and in both cases I still get the same error.

And while using the swagger-utility tool, in azure devops, it seems to be an interactive tool, and expects response.

Could you please let me know how you were able to use it in your pipeline.

Thanks again for all the support!

@Aishwarya Kozhisherry

I took the source of the swagger-utility and modified it to take input from a configuration file instead of STDIN, to convert it to be non-interactive.

This is how I am invoking the swagger-utility from a wrapper Java class

SwaggerUtilProxyGenerator.generateAPIProxyFromSwaggerYML(<br> ymlLocation, config.getProxyName(),<br> config.getOutputLocation(), config.getProxyEndpointName(), config.getTargetEndpointName());

and I run my util as

java -jar apigee-util.jar -f c:\apigee-config\config.properties

I did try using openAPI2Apigee, the interactive mode is for the deployment only (-D option). The code generation is non-interactive.

You can try to just generate the source code using the openapi2apigee and then do the deployment with apigeetool or maven plugin. I will try to simulate this in the azure devops CI pipeline and let you know.

@Aishwarya Kozhisherry Attached is the Azure devops CI pipeline YAML (for the agent job) for your reference, the api proxy code is generated and deployed to Apigee. I used openapi2apigee to generate the code and apigeetool to deploy to the Apigee environment.

ci-pipeline-yaml.txt

2019-03-05-14-19-32-proxies-apigee.png

@Nagashree B Thanks a lot! I will try both options!

@Nagashree B Thanks a lot for introducing me to apigeetool. I was able to deploy it to Apigee Edge using your advice 🙂

@Nagashree B Could you please let me know if the apigeetool allows to deploy or import a particular revision onto Apigee Edge?

The Readme tells that we can do it, but on using it, I get a invalid argument command.

@Nagashree B

From the time I posted the question, thanks to your tips, I was able to do most of the points mentioned in this post except for adding shared flows and using the apigee -deploy-maven-plugin for deployment. (Am hoping we crack the shared flows soon, but apigee-deploy-maven-plugin isn't really working out. If apigeetool could deploy the existing revision, it would have made life easier)

But my main question is how to scale this for multiple APIs. Is there a way to automatically create the Azure devops pipelines when I get a new swagger spec from Swaggerhub? It could take some parameters, but it would have to create 7 build pipelines and 1 release pipeline. The reason for having multiple pipelines is due to the different sources / repos for these pipelines.

@Aishwarya Kozhisherry

As far as I know only the CI creation can be automatically done in Azure devops by using the

.vsts-ci.yml pushed to the repo.

The CD creation would still be manual, there is no automated way of doing it.

What's the issue with sharedflows? I have a CD pipeline that deploys the shared flows using apigeetool itself.[ you can ping me offline at nagashree.b@gmail.com, I can share my CD Template]

For the maven plugin, I had no issues trying to deploy it. The only issue I had was - I was not able to separate the CI and CD. maven package works fine for the CI. In the CD maven install was doing the packaging again and I wasn't able to link it to the artifact that was generated from the CI build. So I switched to apigee tool

Regarding the apigeetool - I will reply on that in your new post.

@Nagashree B Thank you! I will explore the .vsts-ci.yml option.

Issue with sharedflows is on figuring out how to add the sharedflows automatically to a proxy that has been converted from a swagger spec. Am exploring the apigee-proxy-dependency-plugin, but meanwhile, we feel this will work only when we are able to define where exactly the sharedflow needs to be added and in what sequence it needs to be done somewhere in the CI flow. But that would mean manual intervention. Not sure if anyone has done this before with this level of automation.

Using the maven package I am trying to deploy the exported bundle only, which is essentially CD part. I will try out the apigeetool based on your response in the other post. Thanks again for the quick turnaround.

I'm just getting started with the CI/CD but I am trying to implement the same flow (OpenAPI to apigee, inject the sharedflow references). Not sure if this helps you, but `xq` is a command that lets you write elements to xml according to "path" (i.e. xq -x .path.to.sharedflows)