upload larger file through apigee proxy

[sfathima]

Hi there, 

Could anyone please help me to solve the payload limit to 10 MB exceeds when we upload a file through apigee.

I understood we can have streaming option but streaming is not working for file size like 25MB 

next we have signed url to upload larger file but using signed url how to upload multiple files at a same time and also signedurl we need two endpoints inorder to upload a single file 

please give some suggestion on the above issue my requirement is I need to upload multiple files which is more than 10MB through apigee

[DChiesa2023]

As you know, there is a limit to the size of the payload that Apigee can handle. Google documents this and the other limits for Apigee.  

Previous discussions on the Apigee community have explained the use of "signed urls" for handling an authenticated, authorized transfer of large data payloads.  Basically the approach is to  use the Apigee gateway as a control channel, while using a different endpoint, outside of Apigee, as the data channel. 

• App sends an API request to Apigee, "I want to upload"
• Apigee validates credentials (token, APIKey, etc)  on that request, then, if credentials are valid, generates a signed URL. This URL can be timestamped, time-limited, single-use. 
• Apigee sends back the signed URL to the original caller
• the original caller posts the large payload to the signed  URL endpoint.

It is up to YOU to obtain or implement an endpoint for the data channel.  One good option is to use Google Cloud Storage. GCS supports signed URLs,  And you can use an Apigee API proxy to generate those signed URLs. 

If you use GCS, or if you use your own implementation of a "data channel", a single endpoint can handle multiple different file uploads. The signature part of the URL will be different, but the endpoint will be the same. 

Here is a recent post describing the use of signed urls with Google CLoud storage, in detail.

Good luck.

[sfathima]

Hi there, 
I tried to generate a signed url to upload  a file with this repo as an example. in this repo example the have generated the signed url for downloading but where I tried to generate signedurl for uploading a file but the url generated and redirected using AM-Redirectupload gives me malformed header error and also I tried to add the file before generating the signed url it is throwing me payload file too large error... I seek some help...

[DChiesa2023]

the url generated and redirected using AM-Redirectupload gives me malformed header error

---

Can you be more specific about the error you are observing, and what you are doing when you observe the error?  What thing generates the error?  And what program are you using when you see it?  Where do you see it?  Specifically and exactly, what is the error you see? a screenshot might help. 

it is throwing me payload file too large error...

---

Same here. "IT is throwing an error".  What is IT ?  And specifically and exactly, what is the error?  be very specific please.

If I were trying to work through this problem - creating a signed URL for upload - I would first use the gsutil utility to create the signed URL , and verify that it works properly with a subsequent PUT request. I'd probably follow along in this document in order to accomplish that. Once I have that working, I'd convert the example at the repo you cited, to create the signed URL in the same way, using Apigee debugsession (aka trace) to verify that the URL that Apigee generates looks generally the same as the URL generated by gsutil.  Then I'd check that it all works. 

break it down and solve it in parts.

[sfathima]

Dear DChiesa....

First of all Thanks for the quick response, and very sorry about not showing the clear picture of the issue I am facing...  

please find the link of the repo 

this is the error I face when I hit the proxy url in postman 

this is the screen capture of the apigee debugging

[DChiesa2023]

I understand that the error is coming from the storage.googleapis.com endpoint.  The error message seems pretty clear to me.  The GCS endpoint is telling you: the signed URL indicated there would be a "content-type" header in the request, but it didn't find one.

Looking at your python code, it appears to include "content-type: application/octet-stream" in the list of headers it signs.  Did you also include that header in the request you sent from Postman? 

Thanks for showing me the error, and the trace.  But it seems like you're omitting or neglecting to consider some significant context here. Just to make sure, I would like to clarify my understanding of how all this works.

• There are 2 requests.  The first request goes to Apigee to generate a Signed URL.  The second request goes to the signed URL, a URL. which uses the hostname of storage.googleapis.com
• The first request seems to succeed. You receive a valid, correctly-structured URL in response. 
• When you invoke the second request, you receive an error

You have not provided any information as to how you invoke the second request. Are you using PUT as the verb?  What headers do you pass?  Can you show a trace from postman, or can you invoke it via curl and show me what you've done? 

Did you try any of the suggestions I offered in my previous reply?  did you try using gsutil? 

and you didn't. 

[sfathima]

Dear DChiesa,

For generating signed url I am able to follow your youtube link and generate and also tried uploading the object. I need some help for my current requirement... actually my organization requirement is using postman with single request we need to upload a big file through apigee. Based on my understanding if I upload before generating signedurl apigee will throw payload limit exceeds error so first we need to generate signed url and then using that url we need to upload the object, which could not be done using single request.

please clarify if there is a way to upload a big file through apigee in a single request 

thanks a lot for your patience and responding me 

Much appreciate to help me understand better. 

[dchiesa1 · Accepted solution]

For generating signed url I am able to follow your youtube link and generate and also tried uploading the object.

OK, great!

I need some help for my current requirement... actually my organization requirement is using postman with single request we need to upload a big file through apigee.

There are limits to the ize of the payload you may pass through Apigee for a single API call. you cannot satisfy that requirement as stated ("we need to upload a big file through Apigee"), if "Big" implies "greater than 10mb payload". Check the documented product limits for more information on this.

Based on my understanding if I upload before generating signedurl apigee will throw payload limit exceeds error ...

Correct. (try it!)

...so first we need to generate signed url and then using that url we need to upload the object, which could not be done using single request.

Yes. I think you've got it. The idea is that for larger files, you can use Apigee to "broker" the upload. The actual upload does not go through Apigee, but if you have a second system, like Google Cloud storage (GCS), that accepts signed URLs, then you can use Apigee to generate an authenticated time-limited unique URL, that the client app would use to upload its large data. Apigee can generate the signed URL, send it back to the client, then the client can use that URL (pointing to storage.googleapis.com), to upload its large data. Therefore there are at least 2 requests: one from the client to Apigee to request the signed URL, and the 2nd from the client to storage.googleapis.com, to upload the large payload. I say "at least 2" because in the general case, the client may use multiple requests to storage.googleapis.com to upload a very large file. So it is more correct to say "at least 2 requests".

It's not necessary to use GCS for the 2nd system. You can use AWS S3, or some other storage system that you own and operate. The key thing is that second system is not "behind" Apigee. It's got its own endpoint.

please clarify if there is a way to upload a big file through apigee in a single request

In case this is still not clear: there is not. There is a limit to the size of the payload you may pass through Apigee. This limit is documented. The "signed URL" pattern (reviewed here in a YT screencast video) is a way to workaround the limit, but using that pattern, (a) there are at least two requests, not one, and (b) the upload does not pass through Apigee.

[sfathima]

Dear Dchiesa,

Thank you so much

best regards,

Shirin