Hello,
We've been running a series of performance tests on an array of Edge Microgateway instances load balanced using Amazon ELB.
One very consistent error I've found causing the gateways to crash appears to be be caused by the number of open file connections. Here is the error we see in the logs:
Error: EMFILE: too many open files, uv_resident_set_memory at Error (native) at /home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:186:25 at Array.forEach (native) at Object.serializeLogRecord (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:149:36) at Object.logger.stats (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:111:61) at null._repeat (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:50:18) at wrapper [as _onTimeout] (timers.js:264:19) at Timer.unrefdHandle (timers.js:301:14) /home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:186 var mem = process.memoryUsage();
The issue can be expected when running a load test and monitoring a list of open files:
lsof -i -n -P | grep node
What we find is that something with in Egde Micro appears to increase the number of open file handles until the ulimit is reached. Then this error occurs crashing the server and restarting. It appears to happen consistently when testing over 100 TPS.
Is this something unique to my installation? What can we do to prevent this kind of issue in production?
I'll attempt to increase the users ulimit -a (open files) to a higher amount than the default 1024, however is seems this would count more as a work around than a solution.
Each http request proxied by Edge Micro will consume two fds, so a large number of concurrent or pending requests will quickly consume the default limit of 1024. Please increase the limit as @archendra says.
Apigee Edge Microgateway Vertical and Horizontal scaling for 1000 TPS and beyond. 3 Answers
Edgemicro intro plugin appending to response body instead of replacing 0 Answers
Transfer Edge Gateway proxys to Microgateway 2 Answers
Microgateway api key does not work, although OAuth does 1 Answer
Can we configure northbound mutual TLS for a specific Microgateway proxy? 2 Answers