getting java.lang.OutOfMemoryError: Compressedclass space. Safe usage of ObjectMapper in Java callouts?

[#1]

Hi,

We have a custom Java callout we use to validate an inbound call to a Swagger spec before going anywhere near the backend service. recently we've started seeing a lot of compressed class space errors on our message processors.

The code looks like this:

    public ValidationCallout() {
        this.objectMapper = new ObjectMapper();
        ResourceMapper resourceMapper = ValidationCallout.class.getClassLoader().getResource(LINK_RELATION_MAPPING_LOCATION) == null ?
                DefaultResourceMapper.empty() : DefaultResourceMapper.fromFile(LINK_RELATION_MAPPING_LOCATION);
        this.validator = new DefaultSwaggerValidator(new SwaggerParserYamlImpl().parse(CONTRACT_LOCATION),
                resourceMapper);
    }


    @Override
    public ExecutionResult execute(MessageContext messageContext, ExecutionContext executionContext) {
        if (!executionContext.isErrorFlow()) {
            if (executionContext.isRequestFlow()) {
                return handleRequest(messageContext, executionContext);
            } else {
                return handleResponse(messageContext, executionContext);
            }
        }
        return ExecutionResult.SUCCESS;
    }

Apigee is erroring at the part it tries to validate the request. If the policy is disabled, the call makes it to the backend and back.

We're running some additional class trace options now to see if ObjectMapper is the culprit, but does the usage of ObjectMapper look suspicious here? Will a new instance of the enclosing class be created each request?

Many thanks

[#2]

As an update on this, we are seeing no classes being unloaded whatsoever from our Java policy when looking in edge-message-processor.log with the TraceClassLoading and TraceClassUnloading JVM options on. Is this something to be concerned about?

[dchiesa1]

Seems like you're facing a standard effort at diagnosing Java memory usage. Couple comments for you

• The lifecycle for callouts does not create a new callout instance for each request. Assume one instance per org+env+proxy tuple. The instance execute() method will be called by multiple threads concurrently.
• you wrote "we've started seeing a lot of compressed class space errors on our message processors." I think that is "java.lang.OutOfMemoryError:Compressedclass space" . Is that right?
• I think you're referring to the ObjectMapper from the Jackson, library, is that right? if so, .... Java Callouts run within the Message Processor process. Jackson is included in the classpath for the MP, which means a Java callout that you write could load classes from the Jackson JAR. The problem is, which version is Apigee using? That's not documented, and therefore you cannot rely on it. So a best practice is to package the JARs your callout relies on (that will include ~3 jars just for jackson if i recall correctly), as resources in the Proxy or Environment. Have you done that? Have you uploaded the Jackson JARs as a resource, explicitly?
• Your use of ObjectMapper seems safe, but there is a much better way. The author of Jackson says you can declare an ObjectMapper instance as class static, and use it from multiple threads as long as you don't try to reconfigure the instance (set different options) from those different threads. Because ObjectMapper is apparently a bigger object, this will save you memory, and may avoid OOME. Since v2.0 of Jackson, you also have the option to use ObjectReader and ObjectWriter - they are threadsafe and lighter-weight than ObjectMapper. See the Stackoverflow post for more details.

  I suspect that converting to use a single static ObjectMapper would relieve some memory pressure. Easy to try.

  public class ValidationCallout implements Execution {
    private static final ObjectMapper objectMapper = new ObjectMapper();
    static {
      // configure objectMapper here  
    } 
    public ValidationCallout() {
          // this.objectMapper = new ObjectMapper();
          ResourceMapper resourceMapper = ValidationCallout.class.getClassLoader().getResource(LINK_RELATION_MAPPING_LOCATION) == null ?
                  DefaultResourceMapper.empty() : DefaultResourceMapper.fromFile(LINK_RELATION_MAPPING_LOCATION);
          this.validator = new DefaultSwaggerValidator(new SwaggerParserYamlImpl().parse(CONTRACT_LOCATION),
                  resourceMapper);
      }
    ...
  }
  

  I don't know about the DefaultSwaggerValidator class - whether that is thread-safe - apparently so, since you are seeing no errors. Also don't know whether it can be safely declared as static or not. I'll let you do that research.

• Finally, I feel we may be focusing too much on one particular thing - the use of ObjectMapper. You suggested that as a point of inquiry, but you didn't state why you chose to focus on that. Ask yourself, What else does that class do? What other dependencies are there? The objectMapper in your original code gets created once per instance of the class. But objects that get created once for each request will be much more impactful for memory usage. What object instances get created for every request - in handleRequest or handleResponse? The callout might benefit from a close code review.
• To help with the investigating memory usage, do you know anyone who is an expert at JVM tuning and memory usage diagnostics? Tons of good tips here.