How to get paginated results from BAAS when SELECTing specific fields?

Question

"Regular" requests (i.e. without as SQL-like parameter) to GET data from BAAS return a "cursor" property on the response object for supplying in subsequent requests to get the next set of entities.

If I issue a request to BAAS that includes an SQL string within the "ql" parameter, crucially, that selects specific fields as opposed to all fields, there is no cursor in the response.

Based on the the docs here:

http://apigee.com/docs/app-services/content/workin...

It suggests using a LIMIT clause within the SQL, which would be fine if there was also a way of doing OFFSET, but that does not seem to be the case (at least I cannot find this within the documentation). In the type of query where selecting specific fields, how do I get the data from the next set of results?

This combined with the fact that the maximum number that can be given to a LIMIT clause is 1000 leave me wondering how to do this?

Thanks,

-Rob

Answer 1

@asagar @Robert Baker

There is a way, it involves defining a function that is smarter about paginating through UG collections. Here's what I use:

/*
* ugCollectionForEach
*
* iterates through all items in a collection within Apigee Edge BaaS.
* Uses the Usergrid client object from the usergrid module.
*
* @param ugClient - the authenticated client object
* @param options - the options for a collection. Pass type and qs.
* @param f - function called with each UG entity. Accepts a single argument.
* @param doneCb - called in case of error or success.
*
*********************************************/
function ugCollectionForEach (ugClient, options, f, doneCb) {
  var results = {count: 0, failCount: 0};
  if ( ! options.qs) {
    doneCb(new Error('missing qs property in the options argument'), null);
  }
  if ( ! options.type) {
    doneCb(new Error('missing type property in the options argument'), null);
  }


  ugClient.createCollection(options, function (e, collection) {
    var e2;
    function doOnePage(collection, cb) {
      while(collection.hasNextEntity()) {
        f(collection.getNextEntity(), results);
        results.count++;
      }
      if (collection.hasNextPage()) {
        collection.getNextPage(function(e){
          if (e) {
            e2 = new Error('could not get next page of entities');
            e2.wrappedError = e;
            cb(e2, results);
          }
          else {
            doOnePage(collection, cb);
          }
        });
      }
      else {
        cb(null, results);
      }
    }


    if (e) {
      e2 = new Error('could not make or get collection');
      e2.wrappedError = e;
      doneCb(e2, null);
    }
    else {
      doOnePage(collection, doneCb);
    }
  });
}


//////////////////////////////////////////////////////////////////
//
// EXAMPLE Usage follows
//


ugCollectionForEach(ugClient,
                    {
                      type:'warehouses',
                      qs:{ql:'order by warehouseId', limit:50}
                    },
                    handleOneEntity,
                    function(e, info) {
                      if (e) {
                        console.log("error:" + JSON.stringify(e, null, 2));
                      }
                      else {
                        console.log("all done." + JSON.stringify(info, null, 2));
                      }
                    });




function handleOneEntity(entity, status) {
 ...
}