Best Practices for Artefact Reuse in Proxy Development

[hanselm]

Introduction

This article discusses the various methods and artefacts that can be used for the purposes of sharing or re-use in the API proxy development process. It also provides links to a set of tools that complement or support the reuse of these artefacts that can be incorporated into the proxy development lifecycle.

Objectives

• What are we sharing?
  • files - Files containing common fragments that contain proxy flows which can be shared or re-used in multiple API proxies
  • proxies - An entire proxy can be re-used or shared by other API proxies, using the Proxy Chaining feature
  • code - Code written in Javascript, Java or other languages can be deployed as resources and shared across proxies within their callout policies
  • configuration data - Data such as target urls, credentials, etc that are needed by multiple proxies
• How will this be implemented and versioned?
  • Implementation details are based on the type of artefact re-used and documented below
  • In general, for all re-used artefacts, version control will be handled by the source-code control system (SCM) that is used. eg. Git
    This includes callout code, proxy implementation, as well as configuration data.
  • In addition, when deploying a proxy to Edge, a revision number is assigned to the proxy for each version of the proxy that is deployed.
• Compare vs Global Resources
  • Using global resources is another way to share code or configuration data artefacts across multiple proxies. They would need to be also version controlled.
    The main issue with sharing global resources, is that any change to the code or config data could potentially impact other proxies (when those proxies get re-deployed) and those proxies would need to be fully regression tested in addition to the proxy that required the change to the global resource.

Implementation

Proxy flow/logic re-use with common Flow Fragments

Using files that contain partial proxy flows called flow fragments, logic that is common to multiple proxies can be shared. This allows re-use, and avoids replication of the same policy steps (flows) in each proxy that needs to implement that flow.

Steps to implement are listed below:

• Write a common proxy that has the same file structure on disk as any other regular proxy
• Inside of its' proxies subdirectory code one or more reusable flow fragments. Each flow fragment must be in a file with a naming convention that ends in .flowfrag. eg. validate_token.flowfrag
• The flowfrag file contains a list of <Step> elements that refer to Edge policies with or without <Condition>'s.

• In order to use these common flow fragments within a proxy, simply refer to the flowfrag filename (without the extension) within a Request/Response section in the body of the proxy/target endpoint xml file using # delimiter characters.

<Request>
	<Step>
		<Name>validateInput</Name>
	</Step>
	#validate_token#
	<Step>
		<Name>XmlToJson</Name>
	</Step>
</Request>

• Edit the pom.xml file and configure the plugin to refer to the folder that contains the common proxy

...
<plugins>
	<plugin>
		...
		<executions>
			<execution>
				...
				<configuration>
					<proxyRefs>
					     <proxyRef>../CommonProxy</proxyRef>
					</proxyRefs>
				</configuration>
			</execution>
		</executions>
	</plugin>
</plugins>

• Finally, run Maven to build and deploy the proxy using the Apigee maven dependency and deploy plugins. See the Tools section below for more details on these plugins.
• The plugin will copy over the referenced flow fragments into the proxy being built, before deploying it to Apigee Edge.
• In addition to Maven, there is also the Grunt search-and-replace-files.js tool that will do the same common fragment replacement like the maven deploy plugin. This is used in conjunction with the node.js Apigee Deploy Grunt plugin tool to deploy API proxies to edge. See the Tools section below for more information.

Proxy re-use using proxy chaining

This method of proxy re-use involves developing a standalone API proxy on Edge, and using this proxy as either a target endpoint or in a service callout policy of another proxy. The second proxy chains to the first proxy where the request sent to the second proxy flows thru to the chained proxy using a local connection. The first or chained proxy must reside in the same Organization and deployed to the same Environment as the second proxy. Chaining to a local proxy in this manner is highly efficient since the connection bypasses any network components such as load balancers, routers, and message processors.

More details on proxy chaining are documented here.

Steps to implement are listed below:

• Configure the Target Endpoint of the second proxy to refer to the first (chained) proxy by proxy name as below:

<TargetEndpoint name="datamanager">
    ...
    <LocalTargetConnection>
        <APIProxy>data-manager</APIProxy>
        <ProxyEndpoint>default</ProxyEndpoint>
    </LocalTargetConnection>
</TargetEndpoint>

• Alternatively, you can also configure the Target Endpoint of the second proxy to refer to the first (chained) proxy by proxy basepath as below:

<TargetEndpoint name="datamanager">
    ...
    <LocalTargetConnection>
        <Path>/v1/streetcarts/foodcarts/data-manager</Path> 
    </LocalTargetConnection>
</TargetEndpoint>

Code re-use using resources

API proxies can share common code written in Javascript, Java, Python, Node.js that are deployed to Apigee Edge as resources. These resources can be deployed at the Organization, Environment, or Proxy level. Based on this scope, the resource is available to either the specific proxy, any proxy deployed to that environment, or all proxies in a given Organization. By deploying a resource to the Org or Env level, they can be shared by multiple proxies. In addition to code resources, .wsdl, .xsd. .xsl type resources are also supported. More details can be found here.

Steps to deploy resources :

• Write the code in the supported language of choice and save it in a file on local disk
• Compiled Java code can be deployed as .jar files
• Using the Apigee Resource management apis upload the resource file to Edge while specifying the name and type of resource
• From within a proxy use the Javascript, Java callout policies to refer to the uploaded resource
• In order to reference a Node.js script in an API proxy, use the <ResourceURL> element in the API proxy's TargetEndpoint configuration
• As mentioned in the comment below, global resources (those that are available to all proxies in an Org), must be carefully version or change managed. Any changes to these resources will be picked up by referencing proxies when those proxies are redeployed, so any backward incompatible changes in the resource should be versioned. Since resources of the same type are kept and accessed from the same location in Apigee Edge, a good practice is to version the resource filename.

Code re-use using Git submodules

Using the submodule feature of Git, you can create a separate common repository of shared code that is used by other API proxies. The common repository is kept in a subdirectory of the main API proxies directory and referenced as a dependent project. Change history and version control of both repositories are independent of each other. Then using Maven or a similar build tool, the correct version of the common repository can be checked out and built, and the main API proxy can be deployed along with the necessary dependencies.

Please refer to additional documentation on this feature in the Tools section below.

Code re-use using NPM Private Repositories

If you are developing API proxies using Node.js, and want to leverage common node modules as dependencies, then consider creating and using NPM private repositories. NPM allows dependencies of a node.js application to be sourced from various locations like local file system, or remotely over the network from git, or other private repositories.

More documentation on NPM private repositories can be found in this tutorial.

Configuration Data re-use

Apigee provides various objects to enable sharing of configuration data across API proxies. These include:

• KVM - A key-value map backed by storage in Cassandra, that allows you to store arbitrary data, which can be later retrieved during the runtime flow of a proxy using the KeyValueMapOperations policy. The KVM is populated using a management API. One example of data that can be shared using a KVM are target endpoint urls, which can be retrieved at runtime from within the Target Endpoint request flow, and can be re-written using flow variables. The KVM can be scoped to the Organization, Environment, Proxy and can be re-used by other proxies in the same Org or Env.
• Vault - Similar to the KVM, but stores data encrypted, so it can be used for sharing sensitive information like credentials etc. Useful for proxies that send requests to the same target endpoints that need credentials for access. As of now, the data in a Vault can only be accessed from within a Node.js script running on Edge.
• Target Servers - You can configure named target servers in an Environment and share them with multiple proxies. A target server wraps HTTP configuration and can be used in a load balancing configuration within a proxy's target endpoint.
• Cache Resource - A cache resource can be created within an Environment to cache dynamic data, populated at runtime using caching policies, and read by multiple API proxies running within that environment. The cache is backed using Cassandra data store and replicated across all the Organization's message processors in the cluster.

Tools

• Maven dependency plugin
• Maven deploy plugin
• Apigee Deploy Grunt Plugin
• Git submodules
• Maven config plugin

[hanselm]

Hi @Sai Saran Vaidyanathan - Please review this article on artefact reuse for Accelerator Methodology. thanks

[ozanseymen]

@Hansel Miranda - one thing to note - especially in section "Compare vs Global Resources" - is that a change in a global resource will not be immediately/automatically picked up by all proxies referencing to it. Such changes will only be picked up when a referencing proxy is redeployed.

I'd also suggest adding our recommendations on why and how to version global resources. We should note that all global resources are stored in one place per type, so I am using file naming conventions, e.g. logToLoggly-v1.js. Version number only changes when there is a backwards incompatible change. CI job needs to be configured to trigger jobs for all dependent proxies for certain environments. This will also bring further advise on how to version control global resources.

Some advise on when to choose global resources managed thru Apigee vs shared resources injected via git submodules would be useful. For example, I would go with shared resources injected via submodules for small artefacts that wouldn't adversely affect deployment times. If I have a big JAR file with certificates, external libraries, etc, I would go with global resources. The main reason is that looking at paragraph above, global resources need a high degree of maintenance which is not natural to many development processes, i.e. parts of code separately installed/maintained and managed.

[hanselm]

Thanks for the feedback @oseymen@apigee.com. I've updated the article accordingly.

[akoo]

Nice piece, @Hansel Miranda. I wanted to add an important note: encrypted KVMs are here. Details are in our documentation: http://docs.apigee.com/api-services/reference/key-value-map-operations-policy . You now have an option for encrypted data without having to use Node.js.