Cache replication issues

[#1]

We are experiencing some cache replication issues.

We write to a cache in 1 DC, read in the other DC and don t get the the correct write. Even after waiting a long time(over a minute). The L1 cache is supposed to be using ehcache and use port 4528 to sync. We check the tcp connection on the MPs and we can see the connection of the MP within a DC on 4528 but not accross AZ....

Is the documentation wrong? Does cross DC replication use cassandra?

Do we have a problem? Where should the ehcache replication conf be coming from?

http://docs.apigee.com/sites/docs/files/ApigeeEdgePrivateCloud-Install-Config-Guide_v3.pdf

"A Message Processor must open port 4528 as its management port. If you have multiple Message Processors, they must all be able to access each other over port 4528 (indicated by the loop arrow in the diagram above for port 4528 on the Message Processor). If you have multiple Data Centers, the port must be accessible from all Message Processors in all Data Centers. "

[madhans]

Is this behavior also happening with KVM ? This looks very much like a behavior when the clocks are out of sync.

[#3]

Hi Madhan,

We checked our NTP and have a jitter <10ms.

According to your documentation, the cache should replicate across mps using RMI calls on 4528. So cassandra is out of the equation.

We don't see connection between dc on 4528.

Where can we adjust this cache configuration?

[cmbrown]

L1 cache goes across DC's and yes it needs all the MP's to communicate across dc's on 4528. Did you check your MP log files? DO you see any errors when trying to connect ?

[#5]

L1 cache do not replicate. Data is replicated when it is written to L2 (Cassandra).

4528 loop between MPs you see on the diagram is MP to MP communication for cache update events/eviction not data replication.

Validate time across different hosts and ensure your VMs are synchronizing time via hypervisor or NTP.

Validate Cassandra nodes communication is possible across DCs. Test using KVM as comparison.

See additional Cache details here:

http://docs.apigee.com/api-services/content/cache-internals