Pass offset param to target URL

[praviningawale]

The target endpoint accepts limit and offset as query param in the request.

The limit specifies the number of records to be returned on each hit and offset Specifies the starting row for the result set. Used by the limit parameter.

In the initial request for the first page of results, omit the offset parameter or set it to zero.

When we hit the target endpoint from apigee first time we set offset to zero and when I change the offset to some value it gives me error as

error: Requests with offsets need to be run after an initial request without an offset

It means that the request is treated a new request each time and expect offset as zero.

I am not sure if there is something related to cache and we do not have cache management enabled. 

Is there a way out to resolve this issue ?

[dchiesa1]

This is a problem with the interface exposed by the target endpoint.  

I suggest that you try invoking it from outside the Apigee context - use curl or something like that - to sort this out. 

The documentation tells you to specify a non-zero offset, but when you do so, it sends back an error. That suggests that what is the documented as the interface for that target is not correct. It is requiring some other parameter. 

In my experience, when this kind of error occurs, the target is expecting a Cookie. this is just a guess on my part.  If I'm right, the interface is designed for a browser client. 

To review, Browsers do cookie management automatically.  The general way it works: 

1. the browser sends the initial rrequest with offset=0 or no offset parameter
2. the server responds with data and with a Set-Cookie header
3. the browser transparently stores that cookie for this particular server
4. when you use the browser to invoke the server again with a non-zero offset, the browser transparently also sends a Cookie header
5. the server inspects both the offset param and the Cookie before responding

Apigee is not intended to be used as a gateway for Web interfaces. It's for APIs.  Apigee does not manage cookies like a browser does.  If the upstream interface is expecting a Cookie, that interface is not designed to be used as an API. (not correctly designed anyway) 

If the upstream system *does* require a Cookie, then you *could* fiddle around with your Apigee proxy to make it work.  Basically you would have to modify the proxy so that it performs Cookie management internally, to simulate it acting like a browser. On the response flow, inspect the response for Set-Cookie headers. If there is one, extract the cookie and stuff it into cache.  On the subsequent request, look into the cache for a cookie. If there is one, apply it to the outbound request. 

This is a pain in the neck which is why Cookie dependency should not exist in a developer-friendly API.  

But as I said, I am just guessing. 

[praviningawale]

Thanks for brief explanation.

To add to discussion below are few points:

The data from target endpoint is fetched using WQL(Workday Query Language)
We pass SELECT COLUMN1 , COLUMN2 , .... FROM TABLE  in request  body and this query returns only 10K rows in response.

I was thinking if we can make 2 calls to WQL

In first call, we can call an endpoint internally in loop by changing offset value within proxy until we reach empty response(limit value will remain fixed like 10K)

This first call will fetch only EMPLID and it will be a json payload as below

{
    "total": 106273,
    "data": [
        {
            "EMPLID": "1000001"
        },
{
            "EMPLID": "1000034"
        },
        {
            "EMPLID": "1000059"
        }
]
}

Then parse this json response and generate a comma separated string(max 10K emplid) of EMPLID based on limit and offset.
Then call WQL again in which we pass this comma seperated string of EMPLID in where clause.

OR

IS there any way we can loop through Target endpoint by changing offset value internally each time until all response is received.

I am not sure if this makes sense.