importing proxies from edge private cloud 4.15.04 into 4.16.09 fails with strange error

ccovney
Participant V

Hi everyone,

I've just installed a new instance of edge private cloud 4.16.09 and I'm in the process of uploading and deploying our API Proxy bundles from our other instance of private cloud which runs version 4.15.04. To my surprise, the API proxy deployments are failing! I get this red error message in the management ui:

The revision is deployed, but traffic cannot flow. ; and java.lang.NoClassDefFoundError: Could not initialize class com.google.protobuf.ByteString

I was able to find a few errors in various log files which seem to correspond to the aforementioned UI error.

Management server system.log:

2017-01-16 16:12:07,502 org:intralinks env:dev qtp1660837991-41 INFO c.a.d.DeploymentAPI - DeploymentAPI.deployRevision() : New deployment request for org intralinks of type paid on env dev for apiproxy DemoSIEM with rev 1 
2017-01-16 16:12:07,779 org:intralinks env:dev qtp1660837991-41 WARN CONFIG_MODEL - HttpProxyConnection.logInvalidBasePath() : HttpProxyConnection.logInvalidBasePath() - Invalid basepath '/v2/workspaces/*/batches' 
2017-01-16 16:12:07,874 org:intralinks env:dev qtp1660837991-41 WARN CONFIG_MODEL - HttpProxyConnection.logInvalidBasePath() : HttpProxyConnection.logInvalidBasePath() - Invalid basepath '/v2/workspaces/*/settings' 
2017-01-16 16:12:07,971 org:intralinks env:dev qtp1660837991-41 WARN CONFIG_MODEL - HttpProxyConnection.logInvalidBasePath() : HttpProxyConnection.logInvalidBasePath() - Invalid basepath '/via/v1/businessGroups/*/accounts' 
2017-01-16 16:12:07,991 org:intralinks env:dev qtp1660837991-41 INFO DISTRIBUTION - RemoteServicesDeploymentHandler.deploy() : RemoteServicesDeploymentHandler.deploy : Deployment request : DeployEvent{organization='intralinks', application='DemoSIEM', applicationRevision='1', deploymentSpec=basepath=/;env=dev;, deploymentID=null} 
2017-01-16 16:12:14,939 org:intralinks env:dev qtp1660837991-41 ERROR DISTRIBUTION - RemoteServicesDeploymentHandler.deployToServers() : RemoteServicesDeploymentHandler.deployToServers : Deployment exception for server with uuid c3f4f574-843d-4036-9d03-6581f39d71f8 : cause = java.lang.NoClassDefFoundError: Could not initialize class com.google.protobuf.ByteString
2017-01-16 15:51:50,313 org:intralinks env:dev qtp1660837991-460 ERROR DISTRIBUTION - RemoteServicesUnDeploymentHandler.unDeployFromServers() : RemoteServicesUnDeploymentHandler.unDeployFromServers : UnDeployment exception for server with uuid c3f4f574-843d-4036-9d03-6581f39d71f8 : cause = RPC Error 500: java.lang.NoClassDefFoundError: Could not initialize class com.google.protobuf.ByteString communication error = false com.apigee.rpc.RPCException: java.lang.NoClassDefFoundError: Could not initialize class com.google.protobuf.ByteString at com.apigee.rpc.impl.AbstractCallerImpl.handleError(AbstractCallerImpl.java:54) ~[rpc-1.0.0.jar:na]at com.apigee.rpc.impl.RPCMachineImpl$OutgoingCall.handleError(RPCMachineImpl.java:464) ~[rpc-1.0.0.jar:na]at com.apigee.rpc.impl.RPCMachineImpl.processFrame(RPCMachineImpl.java:356) ~[rpc-1.0.0.jar:na]at com.apigee.rpc.impl.ServerHandleImpl.inboundBufferUpdated(ServerHandleImpl.java:298) ~[rpc-1.0.0.jar:na]at com.apigee.rpc.impl.ServerHandleImpl.messageReceived(ServerHandleImpl.java:291) ~[rpc-1.0.0.jar:na]at io.netty.channel.ChannelHandlerUtil.handleInboundBufferUpdated(ChannelHandlerUtil.java:60) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.ChannelInboundMessageHandlerAdapter.inboundBufferUpdated(ChannelInboundMessageHandlerAdapter.java:82) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.invokeInboundBufferUpdated(DefaultChannelHandlerContext.java:896) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated0(DefaultChannelHandlerContext.java:864) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated(DefaultChannelHandlerContext.java:843) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.ChannelHandlerUtil.handleInboundBufferUpdated(ChannelHandlerUtil.java:69) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.ChannelInboundMessageHandlerAdapter.inboundBufferUpdated(ChannelInboundMessageHandlerAdapter.java:82) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.invokeInboundBufferUpdated(DefaultChannelHandlerContext.java:896) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated0(DefaultChannelHandlerContext.java:864) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated(DefaultChannelHandlerContext.java:843) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:154) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.handler.codec.ByteToMessageDecoder.inboundBufferUpdated(ByteToMessageDecoder.java:69) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.ChannelInboundByteHandlerAdapter.inboundBufferUpdated(ChannelInboundByteHandlerAdapter.java:51) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.invokeInboundBufferUpdated(DefaultChannelHandlerContext.java:896) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated0(DefaultChannelHandlerContext.java:864) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated(DefaultChannelHandlerContext.java:843) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.DefaultChannelPipeline.fireInboundBufferUpdated(DefaultChannelPipeline.java:1017) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:115) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:434) ~[netty-all-4.0.0.CR1.jar:na]at io.netty.channel

Management server scheduler.log:

2017-01-16 15:31:55,740 main ERROR SCHEDULER - SchedulerCassandraDataStore.validateKeySpaceForOrgSSLNotificationCFs() : ConnectionException while creating the column family: organization_ssl_notification_details in scheduler-datastore-config#dc-1#central# keyspace for cluster:{}, Exception: {} 
com.netflix.astyanax.connectionpool.exceptions.SchemaDisagreementException: SchemaDisagreementException: [host=None(0.0.0.0):0, latency=26(26), attempts=1]Can't change schema due to pending schema agreement 
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.precheckSchemaAgreement(ThriftKeyspaceImpl.java:668) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$100(ThriftKeyspaceImpl.java:79) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$14.internalExecute(ThriftKeyspaceImpl.java:794) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$14.internalExecute(ThriftKeyspaceImpl.java:791) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) ~[astyanax-core-1.56.43.jar:na] 
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:338) ~[astyanax-core-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeDdlOperation(ThriftKeyspaceImpl.java:511) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.internalCreateColumnFamily(ThriftKeyspaceImpl.java:790) ~[astyanax-thrift-1.56.43.jar:na] 
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.createColumnFamily(ThriftKeyspaceImpl.java:580) ~[astyanax-thrift-1.56.43.jar:na] 
at com.apigee.scheduler.dao.SchedulerCassandraDataStore.validateKeySpaceForOrgSSLNotificationCFs(SchedulerCassandraDataStore.java:173) [scheduler-1.0.0.jar:na] 
at com.apigee.scheduler.dao.SchedulerCassandraDataStore.initializeKeyspace(SchedulerCassandraDataStore.java:148) [scheduler-1.0.0.jar:na] 
at com.apigee.scheduler.service.SchedulerServiceImpl.start(SchedulerServiceImpl.java:47) [scheduler-1.0.0.jar:na] 
at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:168) [microkernel-1.0.0.jar:na] 
at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:71) [microkernel-1.0.0.jar:na] 
at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:178) [microkernel-1.0.0.jar:na] 
at com.apigee.kernel.MicroKernel.start(MicroKernel.java:139) [microkernel-1.0.0.jar:na] 
at com.apigee.kernel.MicroKernel.start(MicroKernel.java:135) [microkernel-1.0.0.jar:na] 
at com.apigee.kernel.MicroKernel.main(MicroKernel.java:84) [microkernel-1.0.0.jar:na]

Thanks in advance for all your help!

Best,

Chris

Solved Solved
0 1 399
1 ACCEPTED SOLUTION

ccovney
Participant V

The above errors turned out to be symptoms of the MP running out of java heap memory. The solution for which was increasing the heap memory:

-bash-4.2$ cat /opt/apigee/customer/application/message-processor.properties 
bin_setenv_min_mem=1024m
bin_setenv_max_mem=3072m
bin_setenv_max_permsize=512m

View solution in original post

1 REPLY 1

ccovney
Participant V

The above errors turned out to be symptoms of the MP running out of java heap memory. The solution for which was increasing the heap memory:

-bash-4.2$ cat /opt/apigee/customer/application/message-processor.properties 
bin_setenv_min_mem=1024m
bin_setenv_max_mem=3072m
bin_setenv_max_permsize=512m