edge management server starts and dead after sometime

[vivekchikane]

Hi All,

I have done a single node edge installation and earlier it was working fine.

When I restarted all components, it starts successfully. All components are up and running but I am not able to login to edge. After a minute when Isee status, edge-management-server is DEAD. Tried this multiple times and faces same issue.

In the logs I found the below error -
com.datastax.driver.core.exceptions.UnsupportedProtocolVersionException: [/<Ip of my server>:9042] Host does not support protocol version V3 but V2

What could be the issue?

[plamenstoyanov]

Hi Vivek,

the usual reason why the stateless components (Router, Message Processor, Management Server) stop soon after you start them is unavailability of the datastore components - zookeeper, cassandra.

You mention single node Edge installation. I assume you've used All In One (aio) profile, which installs all the components in the same node. If that is the case, could you validate if the datastore components are up and running with:

apigee-all status

If apigee-cassandra or apigee-zookeeper are stopped, try to restart them and check the logs.

apigee-service apigee-cassandra start
apigee-service apigee-zookeeper start

logs are in /opt/apigee/var/log/<component>

[vivekchikane]

Hi Plamen, Thanks for your response.

Yes I already check status and all other components are up and running. I did apigee-all restart as well multiple times.

[deniska]

Did you try to encrypt cassandra natively or something? Updated yum update and cassandra were not excluded? Error states that MS cannot connect to cassandra as the client to pull out relevant info for a successful start.

Did it work before? What happened before the error started to occur, any modifications you done? What's your Apigee version?

[vivekchikane]

Hi Denis, Thanks for your response.
No I did not encrypt anything or updated any files.

Yes it was working before. I created organization and environment separately and when I login to edge UI, I was always getting an error stating that data store not added to the pod of the organisation. I was getting same error when I tried creating a default virtual host. To resolve this, I added all the data stores manually to respective pods using below command -

curl -X POST -u username:password http://my_ip/v1/servers -d "type=application-datastore®ion=dc-1&pod=central&internalIP=my_ip" (few letters are getting converted into special characters)


After all data stores association to the pods, I restarted all components using apigee-all restart then the problem started.

I am using latest private cloud version 4.19.06

[deniska]

Why you ran this command, is your REGION name in conf file different than 'dc-1'?

It seems you (maybe) did something on cassandra level and cassandra dies every time or declines requests.

You have backup folder, did you tried to restore cassandra conf before your actual configuration?

[vivekchikane]

@Denis Kalitviansky I ran this command because I was getting error that cassandra data store not added to the pod. But cassandra is not getting died, management server is.

I will check if I can restore cassandra conf.

[vivekchikane]

I am now trying a fresh installation using aio profile. I see below error while installation comes to management server -

apigee-configutil: edge-management-server: # OK error: No such attribute: EndpointToHostId -- StackTrace -- javax.management.AttributeNotFoundException: No such attribute: EndpointToHostId at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:81) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:903) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:273) at com.sun.proxy.$Proxy8.getEndpointToHostId(Unknown Source) at org.apache.cassandra.tools.NodeProbe.getHostIdMap(NodeProbe.java:514) at org.apache.cassandra.tools.NodeTool$Status.execute(NodeTool.java:2198) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206) Error: Unable to connect to <server_ip> Error Processing schema file: /opt/apigee/edge-management-server-4.19.06-0.0.20030/schema/cassandra/analytics-schema.txt Error: setup.sh: /opt/apigee/apigee-service/bin/apigee-service exited with unexpected status 1

-------------------------

When I see edge-management-server logs, I still see the same error mentioned in the question - "Host does not support protocol version V3 but V2"

I think the first time when we did installation, we might got the same error but we installed management server and then router using component profile.

Not sure why it is happening on our server (cent os).

[deniska]

Can you share your conf file? And exact centos version?

[vivekchikane]

conf-aio.txt Herewith attached conf file.

Centos version - centos-release-7-3.1611.el7.centos.x86_64