API Governance tooling approaches - How do I automate my API program governance?

[#1]

Need for Governance in API programs

The need for governance in API programs surfaces right from the planning stages of the program. Experience has shown that immediate and short term needs of businesses often take priority over long term planning. Treading along this path could result in silos of APIs that deliver minimal business value to application developers, partners and other consumers of the APIs. Unplanned programs could fail because APIs may not be flexible enough to support business goals, and without early success, business and technical stakeholders won't see the potential value in building APIs and the organizational culture would not evolve as required to support a successful API initiative.

Benefits of API Governance

Organizations should build APIs to a well thought out strategy and priority, for which it needs to be able to identify and prioritize candidate APIs. Governance during the early planning/strategy stages of an API program would allow organizations to identify potential APIs in a managed community that would typically include enterprise architects, business stakeholders, portfolio managers and API solution architects. This would ensure that APIs are built proactively to a plan rather than simply reacting and building APIs that are of less value. This approach would facilitate organizational goals by avoiding a chaotic sprawl of APIs.

Governance Tooling

The detail and rigor required from a governance solution should ideally be subject to variation using configurability of the tooling involved. Governance is a necessity and automated tooling is essential to be able to implement it in a scalable fashion, particularly in large organizations. The configurability of this tooling should function as a knob that you can set to the desired level of governance. This configurability and flexibility of the tooling involved is particularly important because the word "governance" more or less has a derogatory connotation in the modern API economy as past experience has shown it to stall innovation and forward thinking which is critical in today's API economy. At the same time, total absence of governance, which may appear to encourage creativity initially could lead to redundant, low-value, unnecessary or at best silos of APIs in the long term. The best place to set this knob is not something cast in stone, would typically vary from organization to organization and needs to be arrived at based on some amount of experimentation and optimization over a course of time.

Types of Governance

API Governance can be broadly classified into 3 types - planning, design/development and run-time. The output from planning governance needs to be a set of candidate APIs that would feed into the next stage of governance, the design/development governance and the output from design/development governance that translate into run-time artifacts would be applied and enforced at run-time. This is elaborated in the next few sections. It is to be noted that the processes explained in this article are geared towards a heavier side of governance and are more suited to large enterprises embarking on large API programs. Smaller programs may choose to adopt a subset of the processes, for example, design and run-time governance without planning governance or just run-time governance, which means they would miss out on certain benefits such as end-to-end closed loop governance and the smooth automated flow-over from one type of governance to another, however that is a trade-off that smaller programs would need to consider for agility.

Planning Governance

The typical functions of API planning governance and tools that support it are:

• Automation - API initiatives are usually planned in the old-fashioned way - on whiteboards, in hall conversations, email threads or presentations and spreadsheets. These ad-hoc approaches result in untraceable decisions, a bad idea in tightly funded projects. Information gathering and governance approvals need to be automated and reportable. It needs to provide visibility through the alignment of the API initiative, its prioritization, budget and effort allocations. Its data model needs to be extensible to support API planning and role-based governance approval processes.
• Mapping to "As Is" state - There should be the ability to pull information about the current systems of record using discovery, introspection and bulk loading type of tools so that identified candidate APIs can be mapped to existing capabilities.
• Dependency analysis - Map API programs and priorities to potential APIs and further down to existing system capabilities such as systems of record and data assets.
• Identifying candidate APIs - Produce well defined and prioritized candidate APIs for development. Progress candidate APIs through automated governance processes and refine content on the way that stakeholders can use to make traceable and accountable decisions based on meaningful data
• Governance reports - Reporting on API initiatives, the stage at which key APIs are in the planning process with drill down into underlying systems of record and relationships.
• Design/Development hand-off - Automatically provision candidate APIs into the next set of governance processes post initial planning approvals and prioritization. These governance processes need to feedback into planning so that the planning community has a clear view of the implementation progress of API initiatives. This closes the loop and provides end-to-end governance

Design/Development Governance

Post the planning governance, the next set of governance processes would be around the design and development of the candidate APIs that are produced as an output of the previous process. This stage of governance and tooling around it would primarily support the below functions:

• API lifecycle management - Define a set of lifecycle stages or gates in the API development process that the APIs being developed will move through. Tooling should ideally be configurable to extend and modify these stages or gates as per the requirements of an organization. There should be configurable templates for each of these stages to capture metadata about the APIs. Metadata are attributes like name, description, minor and major versions, API type (such as partner, private, public), the API products that they would be made available through and so on along with artifacts like requirements documents, design documents, Open API specifications, API documentation etc. Meta-data modeling rules should dictate what attributes and artifacts should be captured for a particular lifecycle stage and which are mandatory. This could, for example prevent an API from being developed without having a design document, an OpenAPI specification, API documentation and test cases in place.
• Governance approvals - Tooling should provide workflow capability with role-based approvals and sign-offs. Tooling should typically integrate with identity management systems of organizations to pull out users and assign them to governance roles such as Security Architect, API Product Manager, API Solutions Architect, API Developer, API tester and so on. Approvals should be configurable with support for sequential, parallel and multi-level approval hierarchies. Approval decisions need to be captured through web-based forms and made traceable as an API moves through its lifecycle
• Runtime policy negotiation - API proxies are usually developed in an API Gateway like Apigee Edge that has rich support for different types of run-time policies such as security, authentication, access control, threat protection etc. However without proper governance, developers are free to use any policies of their choice, secure them in any way they want or worse not secure them at all, get away with inconsistency in threat protection, spike arrest policies, quotas and the like or at best implement these based on verbal discussions, ad-hoc Email conversations or non-standard ways of capturing such decisions. In the absence of governance and associated tooling, there is no way to control the quality of the APIs being developed from the perspective of the policies they use, the configuration of these policies and whether all the required policies are in place. Tooling should be able to capture these decisions even before the development of an API is started. These decisions would typically be approved by the required stakeholders such as Security Architects, API solution architects etc before an API is provisioned into the API Gateway by the tool for development. Organizations typically need to have a core governance team comprising key stakeholders from multiple departments across the organization who would collectively make these governance decisions around policies to be used - for example, all partner facing APIs need to be secured with OAuth, all public APIs need to be secured with API Keys and protected using spike arrest and quotas with the values or value ranges to be used in these policies. These decisions, approvals and data related to them would need to be captured by the tool and stored in its data repository.
• Automation - Tooling should be able to automatically produce run-time API development artifacts in an API Gateway from design specifications such as Open API specifications and policy meta-data captured from the previous step. For example, the tooling would generate API proxy definitions in an API Gateway like Apigee Edge, automatically attach the required API policies that have been captured and approved and configure them. It should ideally also handle issues like common policies and provisioning them by integrating with a framework such as Apigee Edge's Common Proxy Framework (https://github.com/apigee/proxy-dependency-maven-plugin). It would then automatically create necessary products and resources along with developer apps based on the captured metadata. This level of automation removes the burden off API developers and assures the quality of the APIs developed by ensuring that APIs and all related artifacts are created off metadata approved by stakeholders with traceability and accountability. In addition to integrating with the API Gateway, this tooling should also integrate with the API developer portal and create the required artifacts in the portal to expose the API to public, private or partner communities. This would include publishing the Open API specifications to the developer portal and the API documentation that was captured during the design phase of the API. The tooling would also need to automatically generate test cases using Test Driven Development (TDD) frameworks such an Mocha or Business Driven Development (BDD) frameworks such as Yadda or Cucumber.
• Consumption negotiation - For internal APIs, the tool would also need to provide consumption negotiation where internal teams can apply for access to the APIs and request the quota or allowance that their team needs. The tooling could define consumption categories such as Low Volume Consumption, High Volume consumption etc (defined by quota ranges), for example. The tooling could also define collective patterns prescribing, say security and quotas/allowances. For example, there could be a category such as "Standard Internal Consumption" that is basically a "policy set" comprising multiple policies such as API Key security and Low Volume quota or a "Standard External Consumption" category comprising OAuth security and High Volume quota. Such pre-defined sets of policy categories can be defined by governance tooling letting consumer teams select from one of these categories. There is further scope to integrate this with the monetization capabilities offered by a powerful monetization engine such as Apigee Edge Monetization. After an internal team has registered a consumption request, it would typically go through 2 levels of approval - one by the Project Manager of the consuming team to ensure that the consumption makes sense in the context of the Application that the team is building and the Project Manager of the API development team to notify them of the new consumer so that they can plan for the necessary capacity to embrace this new consumer or reject the request, if for example, they are short on capacity. Again, as previously mentioned, these decisions would need to be recorded and reported by the tooling, thereby providing the required visibility, monitoring and traceability.
• Search / Discovery - As the governance tooling becomes a central repository of API assets and related metadata along with governance decisions, it has the potential to provide a powerful search repository and reporting engine.
• Dependency relationships - As the governance tooling has information about the APIs, their consuming applications, the back-end interfaces they are built on, the stakeholders and approval decisions, it has the ability to depict these dependency relationships which can be useful for future impact analysis.
• Integration with Continuous Integration and Continuous Deployment tools - The governance tooling can integrate with CICD tools such as Jenkins to drive promotion of APIs across environments while enforcing and capturing necessary approvals by stakeholders during the promotion process (for example, promotion to production environment) and can integrate with a deployment framework such as Apigee Edge's Maven deploy plugin (https://github.com/apigee/apigee-deploy-maven-plugin) to drive these promotions.
• Integration with Source Code Management repositories - Governance tooling can also integrate with source code management repositories like Git, SVN etc as it is responsible for producing run-time deployable artifacts.It can version, checkin and checkout these artifacts from source code repositories.

Runtime Governance

Most of runtime governance is automatically enforced if the previous 2 stages of governance is taken care of. For example, run-time policies such as security policies, message protection policies and policies impacting API consumption such as quotas are automatically added by governance tooling based on metadata captured and approved in the previous stage of governance as explained in the last section and enforced at run-time.

Conclusion

A governance solution such as this with accompanied tooling would provide end-to-end governance with improved visibility, much needed automation, reporting and vastly improve the quality of APIs developed by organizations and help evolve an organizational culture required to support a successful API initiative. Not all organizations would need to implement all of the steps mentioned above at once as part of a governance solution. Governance should start on a small scale and grow organically for it to be adopted, successful and meet its goals. This is where the configurability of governance tooling becomes important. It helps set the governance model at the desired level, then fine tune it over a period of time to arrive at the optimum governance model that works for an organization, as governance is not a one-size-fits-all initiative.