java callout class remains instantiated after the proxy was undeployed and deleted

[dan54]

It seems that a java callout class can remain instantiated even though the proxy that used it was undeployed and deleted…

My environment: Apigee Edge Private Cloud is running in GCP. I have a java callout that uses Hikari database pooling to make queries to a GCP SQL postgres database. The proxy/java callout is the only client using the database.

Here is a sequence that shows what I am seeing:

1. deploy an Apigee proxy with a java callout that uses database connection pooling
2. call the REST API on the proxy 15-20 times
3. the postgres database reports 12 connections (max configured connections is 15)
4. update the .jar file several times, save and redeploy the proxy each time
5. call the REST API on the proxy 15-20 times
6. the postgres database now reports 28 connections (max configured connections of 15 implies that two or more instances of the java callout class are running)
7. undeploy the java-callout proxy
8. the number of database connections remains at 28
9. stop the database
10. delete the proxy
11. start the database
12. the number of database connections returns to 28
13. reboot Apigee edge private cloud
14. the number of database connections drops to zero

Thoughts on the sequence:

1. After stopping and starting the database, it's expected that the database pooling library would reclaim its connections (as long as the java callout class doing the connection pooling remains instantiated)
2. However, after undeploying and/or deleting a proxy with the java callout that holds those connections, I did not expect to see the connections remain in place. It seems the only way this can happen is if the java class associated with the proxy remained instantiated even after the proxy was deleted.
   
   To me, this behavior seems like a bug. Is that the case?

[dchiesa1]

---

@dan54 wrote:

after undeploying and/or deleting a proxy with the java callout that holds those connections, I did not expect to see the connections remain in place. It seems the only way this can happen is if the java class associated with the proxy remained instantiated even after the proxy was deleted.

---

That is literally the purpose of a connection pool.  It keeps the connections open and current, even when there is no user of the connection.  If I understand what the Hikari thing does, that is its purpose.  Not a bug. A feature.  I think that is the reason you use a connection pool. 

In light of that,  I ask you:  what is the problem here? 

Do you observe pool exhaustion when you de-deploy the API proxy that uses your Java callout?  If so, then maybe you need to be returning connections to the pool after use,  in the Java logic.  I believe the design intent of connection pools is that you use the connection, and then when you're finished, return it to the pool by calling .close() on the connection.  If your java code is not doing that (using try-with-resources or an explicit call to .close()) then subsequent users of the pool may experience pool exhaustion, in ability to connect.  The solution here is to modify your code to return connections when you're finished. 

If you ARE returning connections to the pool, then maybe the problem is that you are concerned with connections remaining open to the database after undeploy. Maybe you think  that might .. cause some database performance degradation?  If that's the case, then (a) have you measured it, and (b) maybe you should not use a connection pool?  

Maybe there is some other problem?  

If you define the problem as "there are connections remaining open" then -I'm sorry to be so direct but - I think the solution is  "don't use a pool that is designed to keep connections open". 

[dan54]

@dchiesa1 naturally, I expect the connections to remain open, as long as the proxy with the connection pooling is running, or even if it is restarted.

However, If you carefully read the sequence I described, you will see that the database connections remained open, even after the proxy with the db connection pooling class was undeployed and even after it was deleted. 

My surprise is that the class with the connection pool seems to continue to exist, even when the proxy it is running within is undeployed or deleted. Are you asserting that when I delete or undeploy a proxy, it is normal for the associated java classes to continue to be instantiated?

[dchiesa1]

Yep, I read what you wrote carefully.

I am not an expert in the Java GC behavior, nor how your connection pool is designed, whether it uses its own classloader, etc. 

It's possible that the KVM just hasn't reclaimed the memory associated to the pool because it has no need to do so (no pressure). It's possible the GC runs in multiple stages and the pool is marked for cleanup. It's possible there is some other thing preventing collection.

If the jar containing the connection pool is an org-scoped jar or an environment-scoped jar (which would make sense), then those things get their own org-scoped or env-scoped classloader. Which can have implications on how resources are maintained in JVM memory, how GC happens. Usually those classloaders stick around until the JVM restarts. 

Undeploy, for Apigee, just means "make it so the proxy is no longer accepting requests." Just how that undeploy is implemented internally, is not defined.  Does it mean the classloader for the Proxy gets destroyed?  I don't know that.  It's not part of the documented Apigee interface.  Does it have effects on the env-scoped classloader?  I don't know.   Similarly, deleting a proxy just means removing the definition from the organization store.  Again it does not mean that any prior-used classloader gets unloaded or destroyed. Those are implementation details. 

What problem are we solving?  If there is something that Apigee is doing that is incorrect and that is causing a disruption in your service, then the appropriate thing is to open a support case through your support contact, and describe the behavior, provide the reproduction and test case, and  ask that the behavior you observe be fixed, mitigated, or changed.  The support people might connect with engineering leaders who, if they accept that the behavior is a bug, might advise "don't run a connection pool in the Java callout; use an external Java app for that."  Which means "there is a workaround to the problem" , and THAT means the engineering team won't have a justification for urgently taking action to investigate, design, implement, test, and deliver a change. 

But right now it seems like your system is working.  So I don't know if the eng team would accept it as a bug anyway.  

[dan54]

Thanks @dchiesa1 for the information! If I understand correctly, it's plausible that classes loaded into the Apigee JVM might persist even after the parent proxy using the class is undeployed or deleted. I'm curious if there's a mechanism within a Java callout to identify this potential "orphan state" and initiate a cleanup process (i.e. disconnect from db etc.) . Any guidance on handling such scenarios?