Program executives and managers hear every day about Agile, and while many organizations try to adopt Agile methodologies in their digital initiatives, API teams are often faced with challenges that can hinder this adoption.
These challenges can largely be split in two groups:
This article seeks to expose how challenges around requirements can be met with what the Agile world calls the Product Backlog.
The Product Backlog and its day-to-day management, refinement, documentation, and visibility to all API teams and stakeholders is at the center of any successful Agile delivery. It is the heart of any Agile approach, the very starting point. Efficient management of the Product Backlog ensures that teams are aligned, knowledge is shared, that requirements are clear, understood, and their Value to Business can be assessed. Moreover, the Product Backlog is the foundation and core aspect of the delivery activities.
This article assumes the use of Scrum but is applicable to other Agile methodologies.
Traditionally a project would start with a requirements matrix in Excel or Word: a very big and very unreadable table, full of “stuff”, hundreds and hundreds of rows, sometimes dozens of columns.
The requirements will likely be split into large chunks of functionality or larger components of this functionality.
Often, you might have a number of requirements documents, which will cover one large item of functionality or component, sometimes with visuals (wireframes, flowcharts, and the like).
Any particular comments or refinements on these are often recorded in a requirements matrix, sometimes without recording at which time any additions were made to the file, or by whom. Often a new version of the file is created and the old one is discarded. There is often is no indication of why any changes were made; a well-kept Change Log on page 2 is a rarity. The priority order for these requirements is often captured using something like MoSCoW.
The capture of requirements in itself results in confusion and information overload. External project members who have not been involved in this capture have a difficult time digging into these documents. Only a handful of people know their way around in what should be widely understood and clear to all. And because Waterfall methodologies work in a phased approach, requirements definition activities are done by business stakeholders, business analysis, and project managers, with little if any involvement from technical or QA teams.
From there, project teams are reliant on business analysts and technical architects to get the right level of detail in terms of technical design: user scenarios, interactions with the system, dependencies, and what takes place behind the scenes. We end up with more specifications documents that are rigid, not easily modified, and it seems we still can’t begin any work as more questions remain open. The work items are still not properly broken down into useful artifacts that a team can work with, and we often see additional discussions back and forth between business and technical teams before work can begin.
Because business requirements and technical architecture and design artifacts are built in isolation, specifications are set regardless of potential dependencies that can block the build activities and involve costly delays.
Finally, there is often no indication whatsoever of the criteria by which a requirement can be defined as met: what is referred to as Acceptance Criteria. Testing is left for the very end of the project, and only once the product is built, or near completion, do we present it to testers and end users for user acceptance. On many occasions no test scenarios are written until the actual testing phase: Requirements are built outside of QA activities.
To sum things up, in traditional waterfall projects:
Agile frameworks seek to fill these gaps, and the capture of all the information necessary to begin work is managed using The Product Backlog.
In its simplest definition, the Product Backlog is an ordered list of all things that needs to be done within the project. It replaces the traditional requirements specification artifacts. The Product Backlog, kept in a single repository, is a living entity that is refined over time and ideally has the following attributes:
Agile teams use collaborative tools that give all members access to every piece of information, and team members can see how information has evolved over time (such as functionality changes). These tools must be easy to access and use for all team members, suppliers, and customers, both technical or non-technical.
In a Product Backlog, requirements are captured as User Stories and Tasks, and are grouped into Epics.
Epics contain User Stories which contain Tasks and sub-tasks. Let’s cover this next:
Epics are large feature categories that contain features. They are used to group work items (example of an Epic: "User Login” or “Authentication”, "SSO").
A user story represents a single piece of functionality (e.g. “user sign-up”, "admin login").
The user story is recorded as a user-centric interaction: As a <persona>, I want to <perform an action> so that I can <derive a benefit>.
Example: As an API Consumer I want to be able to retrieve details about the weather in my users location.
The user story can then be broken into work tasks that are needed to deliver the functionality. I can include all types of tasks that may be the responsibility of any of the project's teams.
Finally the user story is given one or more test scenarios or acceptance criteria to validate that the requirements have been met. For example:
Given that I am on the site and I go to the signup page, and given that I’ve filled in all the required fields, and given that I clicked the submit button, I should receive all the necessary API access keys to use all the features of the Weather API.
The test scenarios should be as precise as possible and include all positive and negative outcomes of the interactions between user and system.
As the requirements around a single piece of functionality are refined, the user story is refined and more detailed descriptions can be added (interaction screens, wireframes, form labels, alert messages, front-end design, and so on.)
Dependencies are recorded and made visible to all as well. User Stories can be set up as successive steps where a user story cannot start before another one is done.
The business value requirement details of a user story is recorded, and the feature sizing is assessed by all members of the teams that are to deliver / build the functionality.
This is what is referred to as the "Definition of Ready".
Every user story is also given a Definition of Done, by which the team can mark its delivery as complete.
A typical Definition of Done can include:
Because Agile programs based around Scrum are built in iterations (or Sprints), the refinement activities are done over time, as the following image depicts. Requirements, Test Definitions, and Build activities are done in parallel:
In the Product Backlog, we typically go from items that don't adhere to the Definition of Ready at the bottom of the backlog to items that have the required amount of detail to be accepted by delivery teams at the top of the backlog. This refinement is done over time through the entire duration of the program.
And as the following image shows, the Product Backlog is constantly re-ordered in light of the information gathered by the team. Items are defined in more detail so they can move up in priority (that is, they're "ready" and can be handed over to the delivery teams) or prioritized by Business stakeholders (that is, requirements that add value). Items can be dropped (a requirement that no longer makes sense to the business or to the technical environment) or items can be added midway through the program as new requirements emerge from the business or technical environment.
In many programs, requirements are recorded by business owners, product managers, business analysts, technical architects, various stakeholders, and other team members.
In Agile frameworks, a Product Owner (PO) is designated to be the single point of contact for all the various sources of information. The PO is responsible for making sure the requirements:
As the requirements evolve, the PO is responsible for recording all changes; for example, who was responsible for a change/refinement, when did the change/refinement happen, and for what reasons.
Often the requirements will be validated by stakeholders and/or end users, and the PO is responsible for the validation of the requirements before they can be accepted by build teams.
These meetings are required to capture the right level of requirements details, to share the information with the team on future requirements, and to report on changes. They are all the responsibility of the PO.
In Scrum, there are typically two types of backlog management meetings:
These meetings are held as required (often once per Sprint but may be less frequent), and the backlog is enriched by the various teams as the Sprint progresses. Business stakeholders assess the value of each requirement at each iteration in order to adapt the end product to the realities of internal and external factors. This allows the requirements to have the flexibility to evolve over the Sprint cycles.
As a result of Product Backlog creation and management, the benefits of managing requirements in an Agile project can be summed up in the following way:
The above summaries are somewhat idealized, and going from a classic Waterfall to a more Agile approach to requirements management comes with challenges. Adopting Agile requires a different approach, a new kind of team organization, as well as a different set of tools.
See also Agile Assurance: Advice for starting the Agile journey
In Scrum, the delivery of every artefact, whether a piece of information in the Product Backlog to the definition of a work task, is the responsibility of the entire team. This requires a different mindset and different team organization.
Team members take a collaborative approach and should be as self-organized as possible. The need for a Project Manager who acts as an orchestrator and leader must be minimized as much as possible and is an anti-pattern for successful Agile programs. This comes with obvious problems for teams that haven't been working in an Agile way. Often, team members will expect the organizational activities to be done by a manager. In Scrum, all members are given equal status, they are all given the voice and place to influence the project in a positive way.
To meet this challenge, the team is meant to be organized in a new way with precise roles and responsibilities. A typical team in charge of API development and delivery could look like the following:
Adopting Agile / Scrum and making it successful also directly depends on the adoption of a set of collaborative tools that enable the team to track and document progress.
Documentation is typically supported by collaborative wiki-like software accessible to all (at Apigee we use Confluence). Teams collaborate on this documentation directly on wiki pages instead of exchanging Word or Excel documents. Changes are tracked automatically, versioning is built-in, and comment threads are available, all of which enable the team to easily collaborate on program artifacts.
The Product Backlog is managed using software such as JIRA to create, maintain, and track requirements, features, and work tasks. Epics, User Stories and Tasks, as well as all dependencies, are all recorded and tracked there.
It is key that such tools must be adopted by all team members, technical and non-technical alike. All recorded communications must be done there as well, and any reluctance in understanding these tools and learning how to use them must be addressed early on for the program's success.
The Product Backlog provides a single source of truth for all members of the wider API team and allows requirements to be defined, prioritized, refined, and reprioritized in one place.
The Product Backlog allows for the inevitable change associated with any digital transformation to be managed while maintaining the velocity required to make programs successful.