detailed data on traffic

[#1]

I need to export transaction level data for integration with usage data on other (non-apigee) tools ... so looking to dump a weeks worth of transaction data at a time (~100k-1Mil records) of the general form: (transaction id, timestamp, userid, response size, response processing latency, and about a half dozen other custom fields.

Been fiddling with custom reports in analytics, but seems not to have any way to drill to that level of detail -- feel like I might be barking completely up the wrong tree. Anyone done something similar, or have a better idea where to start?

[mschreuder]

Hi @John.Duncan welcome to Apigee Community!

To my knowledge we don't have a way for you to bulk export individual transaction-level data but my colleague @Sanjoy Bose will be able to confirm best.

Are you able to share a bit more detail about your use case?

I know you mentioned you had been looking at custom reports; what I was wondering is whether there might be a combination of custom reports that together could give you a variety of aggregated data views that could address your use-case.

Another alternative could be to add a service callout in your API Proxy response flow that makes an API call. I have previously seen an implementation where this was done with the service callout making a GET call with a bunch of query parameters representing the various metrics you want to collect. The target in this case was an Apache server and the only purpose it had was to capture access logs of those incoming GET requests. That Apache server's log files were subsequently post-processed to extract the transaction-level data. This is quite an elaborate and complicated implementation though.

Interested to see the experiences of others and what other options could exist to help address your scenario.

And looking ahead, once you see an answer that you feel best addresses your question, please click the Accept link. This will help others in future who may have similar questions.

[#3]

Other suggestions in line to what @mschreuder posted above is to integrate your API Proxy pipeline with Log Management solutions (Loggly, Splunk, AWS, or even Apigee Cached Logs). This can be done without impacting the latency of your APIs by leveraging asynchronous features in Apigee. Check out this tutorial, which provides an overview of how this can be achieved with Node.js and Logging frameworks such as Winston and Bunyan.

https://community.apigee.com/content/kbentry/3352/api-log-management-push-model.html

For a similar implementation of the pushing log externally with Apigee XML Policies, you can leverage MessageLogging, JavaScript Asynchronous API calls or even Java Callouts.

Hope it helps!