Hosted targets and Zip Unzipping issue returning generic 500 error

[pjoseph]

Hi I have a hosted target application that Im trying to use to take the contents of a .xlsx file (a Zipped file used by Microsoft excel) that has been uploaded via express, parse it and do some transformative things to it, and then store the results in an external server somewhere.

The module that im using to read the .xlsx file in memory is called 'exceljs'. It uses an unzip module 'unzipper', that seems to work using Node streams. Everytime i try to parse this fairly compact .xlsx file that was uploaded into memory via 'exceljs's Unzipper, i get a 500 error.

The logs dont show any errors or exceptions, even though I am catching other exceptions just fine, but i was able to trace it into the module that is trying to unzip the Buffer data block that I provided. The 500 error is thrown after 20 seconds of waiting.

Is there anything I can do to debug this more, or is unzipping and zipping a buffer considered streaming?

The 500 error is generic:

Error: Server Error

The server encountered an error and could not complete your request.

Please try again in 30 seconds.

[jcwhitlock · Accepted solution]

This sounds like an application error, as in your application isn't starting up properly or is failing to continue to run without issue. Please look at your runtime logs and see what you can find out. If you have a specific error message that you need help with, please let us know.

[pjoseph]

Hi Jeremy,

I just determined that this is an error that was being thrown by Google App Engine due to memory restrictions. The specific memory restriction, I'm not sure of, but the Excel file in question when uncompressed exceeds 32Megs of memory and thus it would appear the Server error is thrown when it is trying to decompress the file and hold it in local memory.

The uncompressed file size is less than 3Mbs however and I was able to find a way to circumvent this by writing the compressed data file to local storage, then i used a streaming api to read it via 'Exceljs'.

Now the transformation call that I implemented sometimes "times out" from apigee side, but it continues on GAE and sucessfully completes.

Thanks for your assistance here.