Edge Microgateway Error during Load Testing

kkleva
Participant V

Hello,

We've been running a series of performance tests on an array of Edge Microgateway instances load balanced using Amazon ELB.

One very consistent error I've found causing the gateways to crash appears to be be caused by the number of open file connections. Here is the error we see in the logs:

Error: EMFILE: too many open files, uv_resident_set_memory
    at Error (native)
    at /home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:186:25
    at Array.forEach (native)
    at Object.serializeLogRecord (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:149:36)
    at Object.logger.stats (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:111:61)
    at null._repeat (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:50:18)
    at wrapper [as _onTimeout] (timers.js:264:19)
    at Timer.unrefdHandle (timers.js:301:14)
/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:186
      var mem = process.memoryUsage();

The issue can be expected when running a load test and monitoring a list of open files:

lsof -i -n -P | grep node

What we find is that something with in Egde Micro appears to increase the number of open file handles until the ulimit is reached. Then this error occurs crashing the server and restarting. It appears to happen consistently when testing over 100 TPS.

Is this something unique to my installation? What can we do to prevent this kind of issue in production?

I'll attempt to increase the users ulimit -a (open files) to a higher amount than the default 1024, however is seems this would count more as a work around than a solution.

Solved Solved
0 4 1,046
1 ACCEPTED SOLUTION

Not applicable

Hi there!

Looking at the error logs, Looks like log files are getting generated more frequent due to frequent logging(as your are running performance tests) and running into open file connections issue. Please increase the open file limit to a higher number something like 100k (ulimit -n 100000). Also, Make sure you setup log level to 'warn' during performance tests run. Hope this helps.

Thanks,

Archendra

View solution in original post

4 REPLIES 4

Not applicable

Hi there!

Looking at the error logs, Looks like log files are getting generated more frequent due to frequent logging(as your are running performance tests) and running into open file connections issue. Please increase the open file limit to a higher number something like 100k (ulimit -n 100000). Also, Make sure you setup log level to 'warn' during performance tests run. Hope this helps.

Thanks,

Archendra

I'll try lowering the log level specified in the agent/config/default.yaml to warn and re-running.

Would you consider a healthy ulimit of open files at this throughput? "Something like 100k" doesn't let me know what we should expect when testing around 100 TPS.

The number of fds consumed will vary depending on the target latency and the number of concurrent requests. For purposes of benchmarking you could just set this limit to unlimited -

ulimit -n unlimited

Not applicable

Each http request proxied by Edge Micro will consume two fds, so a large number of concurrent or pending requests will quickly consume the default limit of 1024. Please increase the limit as @archendra says.