Obscure risk found in a Security Audit - anyone got any mitigations?

[johnferguson-1]

During a security audit of a newly built Apigee Edge platform, a risk was discovered. While the Likelihood of the risk is low, the impact could actually be quite high. I am struggling to think of proactive mitigations to this risk.

Background:

We deploy our API Proxies into Dev using an automated build pipeline. The pipeline uses the maven_deploy_plugin. We allow the developer to set their build options as either "validate,update" or "validate,override" depending on whether this is a minor change or more significant.

The build pipeline includes a script called "Environment Leak Protection". This checks the current deployed revision number in Dev and the current revisions in Staging and Prod. If any of the revision numbers match, if the user has chosen "update" as a build option, this is automatically switched to "override".

This prevents an update to a revision in Dev from leaking into Prod if the two environments have the proxy on the same revision.

The Risk:

The audit raised a risk that a rogue developer could simply remove the Leak protection script from their pipeline, align the revisions of the proxies, set their deploy options to 'update', then bypass sign-off and promotion processes by simply pushing an update change into Prod, despite not having the credentials to touch Production assets. Because the revision number would not have changed, this activity would actually be quite difficult to spot, and if the change is subtle, it could actually take some time to become apparent.

How do other people deal with this? If at all?

It may be that we simply have to acknowledge the risk (as obscure and unlikely as it is) and state that any responses would be reactive and punitive. But I would prefer to be able to say that there was a way to proactively stop it, or at the very least, spot that it had occurred in a timely manner.

[dchiesa1]

@Sai Saran Vaidyanathan maybe you have some comment here.

I don't understand the "Environment Leak Protection" thing. I also do not understand the distinction you make between update and override.

the maven deploy plugin can import and deploy, is that right? Are you using it to import and deploy?

Every time you import a proxy, you get a new revision number.

Ideally you use the plugin to deploy the same revision that. you imported.

Eg, import proxy1 into org1, and Apigee tells you "ok got it, that is revision 8".

Then deploy revision 8 to dev. It is not possible that revision 8 is deployed to test or stage or prod, because revision 8 is brand new - it is the result of the import.

If you are NOT importing a new revision, and you are only deploying.... then.... deploying rev 8 to "dev" will not affect test, or stage or prod. They are different environments.

[johnferguson-1]

Hi @dino-at-Google

As I understand it (and as I have seen through playing around through trial and error)

If revision 8 of ProxyA is deployed to both Dev and Prod, and I perform an 'update' to ProxyA via Maven, then the changes I am importing are reflected in revision 8 in both environments. It doesn't create revision 9 unless override is chosen as an Maven deploy option.

This means that even if I don't have the permission to deploy anything into Prod, I can affect the deployed proxy in Prod by updating the same revision in a lower environment.

[dchiesa1]

Ahh, I see now. I'm sorry, I misunderstood initially.

Per this answer from Sai, The "update" option is not recommended in production. I guess you have "prod" in the same org as "dev" and "test", which means, to avoid the problem you described, you should never use update. I will ask @Sai Saran Vaidyanathan to confirm here.

For my part, I never liked the idea of "updating an existing proxy revision". There are problems with this behavior, and in fact the engineering team have removed this as a possibility in Apigee X (latest version of Apigee)! From the documentation on the plugin, it seems like the reason people might want to use "update" is "to avoid creating a new revision". In my opinion, that doesn't seem like a good thing to avoid! Yes, it can lead to a growth in the number of revisions of each API proxy in the organization, but that is an easy problem to solve with other tools. (example)

[ssvaidyanathan]

Thanks for tagging me @Dino-at-Google

Reading this - I also understand that in this particular example, both dev and prod environments are in the same org. And this problem could occur.

Most of the customers I have seen have a separate org for non prod and prod use so the revisions don't clash.

For your case, the best thing to do is to not use update at all. I recommend update only for dev environment (or lower environment) as most changes happen there and we don't want a revision creep. In your case, use override as the only option. Update your pom profile to hardcode the "options" as "override" so that no one can pass it as a parameter to change that through the build.

The other option I can think is create different pom profiles for each environment. Lets say you have dev, qa and prod. Create separate profiles and harcode the apigee.options values for each. You can have it as update for dev and override for qa and prod. With this when you call your build with the dev profile, it will only update the revision of the proxy that was deployed to dev. Lets say Rev 8 was deployed to dev. Any change you push will only update Rev 8 and dev. For higher environments it will always be a new revision and will never conflict with another environment. Hope that makes sense.

Take a look at this example. I have two profiles for my two environments and the apigee.options are hardcoded

Hope this helps !

[johnferguson-1]

@Sai Saran Vaidyanathan & @Dino-at-Google

Thanks for the feedback.

The Apigee standard licence (which my company has bought) only offers one org. So if the best practice is to separate prod and non-prod environments into different orgs, it might be worth discussing this with the Sales guys. It seems incongrious for licensing limitations to force customers to have a platform architecture that's contrary to recommended practice.

To be honest, having non-prod and prod in the same org is fine for the most part; the 'update' option is the only real issue, and the 'Environment Leak' script I wrote is designed to prevent accidental misuse of this. However, the risk raised in the audit revolved around intentional misuse by "rogue developers".

The issue is that the Maven pom files are part of the project source. So while I could prohibit the use of 'update' and hardcode the values in the default pom files, there's no way for me to enforce this. The hypothetical rogue developer can simply overwrite the pom file to use 'update' in their project repo and there's no practical way of me stopping (or even seeing) this.

My hope was that there was some way of controlling this on the Apigee side; i.e. through RBAC, as it seems an oversight that the permissions can be circumvented in this way. Dino's comments regarding Apigee X are reassuring. While I see the rationale for 'update' (i.e. fixing a typo in a description doesn't generate a needless revision change), it does seem more trouble than it's worth.

==

On a personal note, I just wanted to say it is great to be back in the Apigee Community. I was a customer back before Google bought the company, back when @Dino-at-Google was just "Dino", and the community was always a great source of help and insight. I have managed to persuade my new company to buy Apigee, and it's great to see you're still here and still as helpful as ever!