{ Community }
  • Academy
  • Docs
  • Developers
  • Resources
    • Community Articles
    • Apigee on GitHub
    • Code Samples
    • Videos & eBooks
    • Accelerator Methodology
  • Support
  • Ask a Question
  • Spaces
    • Product Announcements
    • General
    • Edge/API Management
    • Developer Portal (Drupal-based)
    • Developer Portal (Integrated)
    • API Design
    • APIM on Istio
    • Extensions
    • Business of APIs
    • Academy/Certification
    • Adapter for Envoy
    • Analytics
    • Events
    • Hybrid
    • Integration (AWS, PCF, Etc.)
    • Microgateway
    • Monetization
    • Private Cloud Deployment
    • 日本語コミュニティ
    • Insights
    • IoT Apigee Link
    • BaaS/Usergrid
    • BaaS Transition/Migration
    • Apigee-127
    • New Customers
    • Topics
    • Questions
    • Articles
    • Ideas
    • Leaderboard
    • Badges
  • Log in
  • Sign up

Get answers, ideas, and support from the Apigee Community

  • Home /
  • BaaS/Usergrid /
avatar image
2
Question by Toshihiro Shibamoto · May 15, 2015 at 01:38 PM · 1.2k Views baascachingquery

What is the best practice to write huge number of data into BaaS storage?

Our customer wants to store one million of user records at monthly batch process and need efficient way of storing data onto On-prem BaaS Collections entities. Is there any way like parallel query, bulk operation or SQL Loader like functions in BaaS with Apigee Edge? The source data is in JSON format converted from XML data. Please give some advice or suggestions for the customer.

Comment
Add comment
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Close

3 Answers

  • Sort: 
avatar image
2
Best Answer

Answer by Manish S · May 28, 2015 at 05:54 AM

@Toshihiro Shibamoto,

Regarding your query about number of entities in a POST call - I have tried 50000 entities in one requests ( using Node.JS) and it has worked fine. so I believe limit of entities in one POST call is certainly more than 50K . Even though I have not tried, but I feel if you have proper infra and large size machines, you should be able to POST upto 1 Lakh in single call as well.

In case you want to POST millions of data, you need to break it into batches of say 50K/1Lakh ( depending on your infra ) and make requests.

In PUT call, there can be only one entity at a time.

For PUT, number of BaaS calls = number of entities you need to update .

I hope this helps.

Comment
Add comment Show 3 · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Toshihiro Shibamoto ♦♦ · May 28, 2015 at 06:27 AM 0
Link

@ManishS, Thank you for the very useful information.Do you also have any idea on the supported concurrency of simultaneous request handled by BaaS? In the case of 1Lakh can we make 10 requests at a time or do we need to wait for the response to send the next one sequentially?

It may also depend on the infra, capacity of web server, etc. But if you have any comment, I’ll appreciate it.

avatar image Manish S ♦ Toshihiro Shibamoto ♦♦ · May 28, 2015 at 07:15 AM 0
Link

@Toshihiro Shibamoto,

BaaS is optimized for batch write operation. So simultaneous calls for POST can be made easily . I have tried 3 concurrent calls from Node.JS in batches of 50 thousand ( total 1.5 Lakh entities) and it did work perfectly ( it had one cassandra node only ) . So with a decent infra set up I believe you should be able to achieve lot more .

avatar image Toshihiro Shibamoto Manish S ♦ · May 28, 2015 at 07:31 AM 0
Link

@Manish S, Understood. Thank you so much for the answer.

avatar image
0

Answer by pbhogill · May 15, 2015 at 07:28 PM

Hi @Toshihiro Shibamoto there is another thread that discusses this topic somewhat. You can also use a scripting tool / language to invoke the API to push data into BaaS. Also you can post multiple records to a BaaS collection using one POST call.

Comment
Add comment Show 1 · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Toshihiro Shibamoto ♦♦ · May 16, 2015 at 12:26 AM 0
Link

Thank you so much Prithpal Bhogill. I saw the post earlier but thought it looks like a different thing if I understand correctly. We just need an efficient handling of very huge number of data without filtering them out. Does BaaS have any optimization mechanism internally for processing multiple entities in a single POST/PUT query? If it's not the case the time taken for handling a million records would be still longer than acceptable range and then we need some kind of implementation tricks as mentioned above. The customer is trying to get benchmark data measured with the scale if available. Will appreciate if any further comments.

avatar image
0

Answer by alan@apigee.com · May 19, 2015 at 08:30 PM

@Toshihiro Shibamoto

- ideally you won't need to do massive numbers of uploads, but per Prithpal, you can POST multiple entities at the the same time. Unfortunately, you can't PUT multiple items at a time.

Depending on the use case, it might be more efficient to "PUT" (Update) a small set of records instead of deleting all records and rePOSTing all records.

Comment
Add comment Show 1 · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Toshihiro Shibamoto ♦♦ · May 21, 2015 at 01:32 AM 0
Link

@alan@apigee.com Thanks a lot for the comments. What do you mean by '"PUT" (Update) a small set of records' while you said 'can't PUT multiple items at a time'? And is there a restriction in the numbers to POST multiple entities - is it 1000 also? Then what is the best practice to handle 1million of data to POST/PUT with using any ql statement or whatever to optimize it?

Follow this Question

Answers Answers and Comments

18 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

Add Cache Control headers to BAAS REST APIS 2 Answers

BAAS: SELECT * WHERE [element] CONTAINS '[value-with-hyphens]' 1 Answer

Problem with a query limit, it’s only returning 10 items. 1 Answer

Quering in BaaS. 1 Answer

Is BaaS good choice for extensive caching response 2 Answers

  • Products
    • Edge - APIs
    • Insights - Big Data
    • Plans
  • Developers
    • Overview
    • Documentation
  • Resources
    • Overview
    • Blog
    • Apigee Institute
    • Academy
    • Documentation
  • Company
    • Overview
    • Press
    • Customers
    • Partners
    • Team
    • Events
    • Careers
    • Contact Us
  • Support
    • Support Overview
    • Documentation
    • Status
    • Edge Support Portal
    • Privacy Policy
    • Terms & Conditions
© 2021 Apigee Corp. All rights reserved. - Apigee Community Terms of Use - Powered by AnswerHub
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Post an idea
  • Spaces
  • Product Announcements
  • General
  • Edge/API Management
  • Developer Portal (Drupal-based)
  • Developer Portal (Integrated)
  • API Design
  • APIM on Istio
  • Extensions
  • Business of APIs
  • Academy/Certification
  • Adapter for Envoy
  • Analytics
  • Events
  • Hybrid
  • Integration (AWS, PCF, Etc.)
  • Microgateway
  • Monetization
  • Private Cloud Deployment
  • 日本語コミュニティ
  • Insights
  • IoT Apigee Link
  • BaaS/Usergrid
  • BaaS Transition/Migration
  • Apigee-127
  • New Customers
  • Explore
  • Topics
  • Questions
  • Articles
  • Ideas
  • Badges