This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Log in to ask a question

Router & Message Processor High Availability

Posted on 10-30-2018 04:40 AM
balakumarm
New Member
Reply posted on --/--/---- --:-- AM

We are setting up APIGEE on private cloud (1 DC full HA on AWS). We have some hiccups while testing disaster scenarios.

Problem: trace session not working when we shutdown EC2 (router / MP node) abruptly.

We have two nodes both running Router and Message processor. When we shutdown one node with APIGEE stop script (/opt/apigee/apigee-service/bin/apigee-all stop), we can still trace session from Edge UI and other node serving traffic as normal.

Alternatively, when we shutdown EC2 of one node, trace session in Edge UI not working. Even we tried manually set not reachable for the node, trace session is not working. Getting below exception in Mgmt server.

1. Is there any additional configuration required to make all components work seamlessly (no manual intervention) when minimum required servers running?

2. We notice API traffic is working when EC2 is down. We are not sure is it by chance going to the running node. Is it possible Router on running node forward the traffic to MP on the node not running (EC2 shutdown scenario)?

2018-10-30 04:06:21,299 org:comp env:prod target:/v1/organizations/comp/environments/prod/apis/v-particulars/revisions/3/debugsessions contextId:- action:CREATE qtp1543146693-31 ERROR DISTRIBUTION - DebugSessionAPI.createDebugSession() : createDebugSession : Unable to connect to the server with UUID {}, skipping it1a1994a0-b81f-46a7-97a1-41011a16c60c
org.apache.http.conn.HttpHostConnectException: Connect to 10.1.5.13:8082 [/10.1.5.13] failed: Connection timed out (Connection timed out)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:115) ~[httpclient-4.3.5.jar:4.3.5]
at com.apigee.distribution.DebugSessionAPI.createDebugSession(DebugSessionAPI.java:242) ~[application-distribution-1.0.0.jar:na]
at sun.reflect.GeneratedMethodAccessor387.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173) [cxf-rt-core-2.4.7.jar:2.4.7]
at com.apigee.rest.framework.CustomJAXRSInvoker.performInvocation(CustomJAXRSInvoker.java:126) [rest-1.0.0.jar:na]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) [cxf-rt-core-2.4.7.jar:2.4.7]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:241) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) [cxf-rt-core-2.4.7.jar:2.4.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) [cxf-rt-core-2.4.7.jar:2.4.7]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106) [cxf-rt-core-2.4.7.jar:2.4.7]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263) [cxf-api-2.4.7.jar:2.4.7]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-rt-core-2.4.7.jar:2.4.7]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:232) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:209) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:152) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:114) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:108) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:830) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:543) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:463) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1584) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1228) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1130) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.Server.handle(Server.java:564) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:318) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [jetty-io-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112) [jetty-io-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_181]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_181]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_181]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:123) ~[httpclient-4.3.5.jar:4.3.5]
... 64 common frames omitted

@Anil Sagar @ Google

0 1 482
Topic Labels
0 Likes
1 REPLY 1

Posted on --/--/---- --:-- AM
Not applicable
Reply posted on --/--/---- --:-- AM

As you are stopping one rmp the traffic is flowing to the other one. that is the expected behavior. When one router is working it has the ability to connect to any of the message processors, which is up and serve the request. Regarding the trace not working we need to see but that is not related to rmps.

in the error below one is something not expected. I think this should be resolved as well.

Unable to connect to the server with UUID {}, skipping it1a1994a0-b81f-46a7-97a1-41011a16c60c
0 Likes