KVM monitoring

we are looking at creating a monitor to capture any changes be it authorized or unauthorized to the KVM. We are thinking we could poll logs, curl command or use one of the management apis. has anybody every done this before?

Solved Solved
2 8 1,150
1 ACCEPTED SOLUTION

Well first, what do you mean by "unauthorized"? The KVM is a protected resource and through RBAC there is no "unauthorized" changes allowed to the KVM. You may have a different definition of "unauthorized" than I do. Maybe you mean "unexpected" ?

In any case, there are a couple of approaches you might consider.

  1. First, you could examine administrative changes via the audit log. Every time an admin user - human user via the UI, or automated script via the API - changes a KVM, that action is audited. Apigee Edge keeps an audit log of all administrative changes, including changes to KVMs. It looks like this:

    5482-kvm-update-audit-entry.png

    The audit log for an organization is readable via an API. And you can filter the audit records to examine only records that deal with KVM. A cron job or similar would be able to give you the ability to examine that audit trail every minute, every 10 minutes or whatever. I've seen people hook up that cron job to a Slack channel to get a notification in the channel when there is any administrative change. So this is one option for you.
    But this audit log includes only administrative changes. as you probably know, it is possible for an API Proxy to include a KeyValueMapOperations policy in it, and that KVM policy can read or write the KVM. Any updates made by the policy will not be recorded in the administrative audit log.

  2. You can read the KVM via the administrative API, and compare the contents with the desired state. Again you would be able to run this as a cron job, every minute, every 10 minutes, or on whatever interval you desire. This doesn't give you notice of WHO changed the KVM, but it does give you notice whether the change was made in a policy or via the administrative API. If you build in the smarts you could make it so that any diversion from the desired state is immediately rectified - you could update the KVM with the desired state when there is any delta.

    One possible surprise here: the KVM reads done within policies are cached. Suppose you use the Administrative API to set the KVM value back to desired state. If you do that, the KVM cache within the message processor is not reset. The newly-written value will become visible in the policy only after the time-to-live on the KVM cache expires. This cache is specified in the policy configuration itself. The caching behavior van be surprising to some people, but this is as designed.

Helpful?

View solution in original post

8 REPLIES 8

Well first, what do you mean by "unauthorized"? The KVM is a protected resource and through RBAC there is no "unauthorized" changes allowed to the KVM. You may have a different definition of "unauthorized" than I do. Maybe you mean "unexpected" ?

In any case, there are a couple of approaches you might consider.

  1. First, you could examine administrative changes via the audit log. Every time an admin user - human user via the UI, or automated script via the API - changes a KVM, that action is audited. Apigee Edge keeps an audit log of all administrative changes, including changes to KVMs. It looks like this:

    5482-kvm-update-audit-entry.png

    The audit log for an organization is readable via an API. And you can filter the audit records to examine only records that deal with KVM. A cron job or similar would be able to give you the ability to examine that audit trail every minute, every 10 minutes or whatever. I've seen people hook up that cron job to a Slack channel to get a notification in the channel when there is any administrative change. So this is one option for you.
    But this audit log includes only administrative changes. as you probably know, it is possible for an API Proxy to include a KeyValueMapOperations policy in it, and that KVM policy can read or write the KVM. Any updates made by the policy will not be recorded in the administrative audit log.

  2. You can read the KVM via the administrative API, and compare the contents with the desired state. Again you would be able to run this as a cron job, every minute, every 10 minutes, or on whatever interval you desire. This doesn't give you notice of WHO changed the KVM, but it does give you notice whether the change was made in a policy or via the administrative API. If you build in the smarts you could make it so that any diversion from the desired state is immediately rectified - you could update the KVM with the desired state when there is any delta.

    One possible surprise here: the KVM reads done within policies are cached. Suppose you use the Administrative API to set the KVM value back to desired state. If you do that, the KVM cache within the message processor is not reset. The newly-written value will become visible in the policy only after the time-to-live on the KVM cache expires. This cache is specified in the policy configuration itself. The caching behavior van be surprising to some people, but this is as designed.

Helpful?

Thanks @Dino this is perfect. If we are ever at an event together I owe you a drink.

Right on! Ben, I knew you were a quality person, the minute I read your question!

I see the proxies being updated but KVM's aren't available in the cloud based apigee. I think it has to do with the hyphens in the KVM name within the url

for instance in the screenshot above the kvm is this way https://enterprise.apigee.com/platform/zzzz/environments/dev/keyvaluemaps but ours is https://enterprise.apigee.com/platform/xxxx/environments/dev/key-value-maps

Hi Ben,

I'm sorry I'm not following.

I see the proxies being updated but KVM's aren't available in the cloud based apigee.

I don't get it. KVMs are available in the cloud-based Apigee Edge SaaS. And you should be able to administer the KVMs in the Apigee Edge administrative UI. I've just searched for a screencast (4MV4D) that shows how to administer and use KVMs, but... wow! I could not find one. That's unfortunate. If you need some basics on how to use them, I'd be glad to provide it. But ask your question in a different way, because I'm not clear what you need.

Sorry my work day is starting earlier than normal 🙂 so i was a little unclear. I meant I can see the proxy updates within the Organization History and with the audit api calls but I can't see within those same logs any changes made to KVMs. When I make a call the audit api and request any information from changes made from the day I see changes to proxies and I would expect to see similar changes to the KVMs but I dont'. I researched a little further and found this user two years ago had a similar problem

https://community.apigee.com/questions/5016/audit-logs-not-showing-kvm-operations.html

First thing - the audit trail is not transactional. via a script, if you make an update and then query the audit trail immediately, you may not see the update in the audit trail. You may have to wait a bit. 2 or 3 seconds.

Ben, can you run a bash script? If not, powershell?

Tell me which is preferable, and let me come up with a quick script that ought to demonstrate KVM updates being recorded in the audit trail.

-Dino

edit: see attached.

kvmaudittest.zip

script works like a champ in my personal apigee cloud but has problems within the company's cloud instance. I worked directly with apigee support and received the following reply after "...discussion with the internal engineering team and I got the confirmation that as of today the audit APIs are unreliable when it comes to entities like KVM."


there is a bug to track the issue

thank you for the help