BigQuery - Apigee Connector

[nicos97]

I want to integrate my table request of bigquery in APIGEE API proxy.

I create an integration and a collector for my table in google cloud, I connected the APIGEE proxy to connector.

It works APIGEE response me with data of table, but with a problem, the list have a maxinum of 200 items(are more result). I tryed to make a multiple pages logic, but I found a problem.

The connector used for bigquery have an output parameter called "listEntitiesNextPageToken" that should contains the token of next page, but effectively remain the default value after run.

I search everywhere but all documentation of this parameter is this ("https://cloud.google.com/apigee/docs/api-platform/integration/connectors-task")

[Former Community Member]

I think you will need to string together a couple of tasks to meet this use case. Assuming you are going to trigger this from Apigee, 

1. Define necessary Integration input and output variables
2. Begin with an API Trigger
3. Use a Data Mapper if you want to set filter conditions, pageSize etc.
4. Use a While task to start a loop. The loop condition would be to the effect of "until pageToken" is empty
5. Use a Connector task to Big Query within the loop
6. Use a data mapper to map the output variable of the connector task to store the results into an output variable (also within the loop)

Please remember we have a 8MB data size limit when fetching data from sources.

[nicos97]

I tried, but "listEntitiesNextPageToken" mantain the default value(empty) after connector run, the cicle return me the same result every time.

And data are smaller than 8MB.

[Former Community Member]

Is it possible there aren't more than 200 records to fetch? 

[nicos97]

Results are more than 200 rows.

Can APIGEE evaluetion limit the function?

[nicos97 · Accepted solution]

I solved using Call REST endpoint task instead Connector task.

This task not require a cicle to obtain all results, but require 2 REST call:

-first create a job

-second obtain the result of the query into the job