com.apigee.rpc.RPCException: Unknown target RuntimeConfigurationServiceV1

[ccovney]

Hi everyone,

We just installed the Private Cloud or "OPDK" service pack 4.15.04.03-ws and it appears to have broken the environment. Call to APIs within all of our proxy bundles timeout, and if we try to undeploy and redeploy a proxy bundle, we get a red triangle icon next to the deployment status (instead of a green circle) and the UI throws a red error:

"The revision is deployed, but traffic cannot flow. Call timed out"

Whenever this error occurs, we see another error in one of the Message Processor system.log:

2015-09-23 17:18:19,660 ServerProtocolChildGroup-RPC-0 ERROR RPC - RPCMachineImpl.processFrame() : Unknown Target RuntimeConfigurationServiceV1 for the RPC request frame [type=REQUEST dataType=PROTO2RPC correlationId=19] 2015-09-23 17:18:19,662 ServerProtocolChildGroup-RPC-0 ERROR RPC - ServerHandleImpl.inboundBufferUpdated() : Error processing frame [[type=REQUEST dataType=PROTO2RPC correlationId=19], RPC Error 507: Unknown target RuntimeConfigurationServiceV1]: {} com.apigee.rpc.RPCException: Unknown target RuntimeConfigurationServiceV1 at com.apigee.rpc.impl.RPCMachineImpl.processFrame(RPCMachineImpl.java:267) ~[rpc-1.0.0.jar:na] at com.apigee.rpc.impl.ServerHandleImpl.inboundBufferUpdated(ServerHandleImpl.java:298) [rpc-1.0.0.jar:na] at com.apigee.rpc.impl.ServerHandleImpl.messageReceived(ServerHandleImpl.java:291) [rpc-1.0.0.jar:na] at io.netty.channel.ChannelHandlerUtil.handleInboundBufferUpdated(ChannelHandlerUtil.java:60) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.ChannelInboundMessageHandlerAdapter.inboundBufferUpdated(ChannelInboundMessageHandlerAdapter.java:82) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.invokeInboundBufferUpdated(DefaultChannelHandlerContext.java:896) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated0(DefaultChannelHandlerContext.java:864) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated(DefaultChannelHandlerContext.java:843) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.ChannelHandlerUtil.handleInboundBufferUpdated(ChannelHandlerUtil.java:69) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.ChannelInboundMessageHandlerAdapter.inboundBufferUpdated(ChannelInboundMessageHandlerAdapter.java:82) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.invokeInboundBufferUpdated(DefaultChannelHandlerContext.java:896) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated0(DefaultChannelHandlerContext.java:864) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated(DefaultChannelHandlerContext.java:843) [netty-all-4.0.0.CR1.jar:na] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:154) [netty-all-4.0.0.CR1.jar:na] at io.netty.handler.codec.ByteToMessageDecoder.inboundBufferUpdated(ByteToMessageDecoder.java:69) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.ChannelInboundByteHandlerAdapter.inboundBufferUpdated(ChannelInboundByteHandlerAdapter.java:51) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.invokeInboundBufferUpdated(DefaultChannelHandlerContext.java:896) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated0(DefaultChannelHandlerContext.java:864) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelHandlerContext.fireInboundBufferUpdated(DefaultChannelHandlerContext.java:843) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.DefaultChannelPipeline.fireInboundBufferUpdated(DefaultChannelPipeline.java:1017) [netty-all-4.0.0.CR1.jar:na] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:115) [netty-all-4.0.0.CR1.jar:na]

.....

Lastly, whenever we try to run the trace tool on one of the already deployed proxy bundles, we get a red error in the UI saying unable to create debug session and this error is written to the message proc system.log:

2015-09-23 17:22:25,048 qtp899933215-32 WARN REST - APIRegistry.locateSubResource() : No match found for request path /runtime/organizations/Intralinks/environments/dev/apis/oauth/revisions/5/debugsessions/ 2015-09-23 17:22:25,049 qtp899933215-32 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null

org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92)

org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

Any help or insight would be very very appreciated!

Best, Chris

[adas · Accepted solution]

@Chris Covney All the errors are related. It looks like the MPs didn't start with all the necessary services after the patch update. I am guessing you started seeing these errors after applying the 1504.03_ws patch. This patch is for 1504_ws OPDK release and not for the standard 1504 OPDK version. I hope your base version is 1504_ws and not 1504. Can you please confirm that and post the output of the http://mgmtIp:mgmtPort/v1/buildinfo here.

If your base version is indeed 1504_ws and you are still seeing this problem then I would suggest 2 things.

1. You restart all the message-processors once and see if that resolves the problem

2. If the problem still persists, please check if the RuntimeConfigurationServiceV1 is there in the message-processor.xml file under profiles. Also attach a copy of the file here.

It is likely that your base version was not websocket enabled which is OPDK 1504_ws and that's why you are seeing this issue. But once you tell us the buildinfo details we can look into it.

[adas · Accepted solution]

@Chris Covney All the errors are related. It looks like the MPs didn't start with all the necessary services after the patch update. I am guessing you started seeing these errors after applying the 1504.03_ws patch. This patch is for 1504_ws OPDK release and not for the standard 1504 OPDK version. I hope your base version is 1504_ws and not 1504. Can you please confirm that and post the output of the http://mgmtIp:mgmtPort/v1/buildinfo here.

If your base version is indeed 1504_ws and you are still seeing this problem then I would suggest 2 things.

1. You restart all the message-processors once and see if that resolves the problem

2. If the problem still persists, please check if the RuntimeConfigurationServiceV1 is there in the message-processor.xml file under profiles. Also attach a copy of the file here.

It is likely that your base version was not websocket enabled which is OPDK 1504_ws and that's why you are seeing this issue. But once you tell us the buildinfo details we can look into it.

[ccovney]

What is the best procedure for reverting the patch?

[adas]

@Chris Covney I think the patches come with a uninstall script. The README talks about that too. I don't remember the script name exactly, but I am sure the README mentions that, infact it does both - how to apply the patch and how to revert the patch.

Please refer to the README that comes with the patch installer and let me know if that is clear.

[vineet_vineetb]

@arghya das @Chris Covney Output of v1/buildinfo

SCM_BRANCH=origin/OPDK_1504
RELEASE_ID=1504_01
SCM_REVISION=3cc47579881c12409c5bfdf5885b561e15b4499c
RPM_NAME=apigee-rpm-1.0.0.810.3cc4757.1505041047-1504_01
BUILD_NUMBER=jenkins-4G_Release_Sanity-810
BUILD_TIMESTAMP=1430737396638

[adas]

@Vineet Bhatia Thanks for providing the info. @Chris Covney This tells me that your base version is OPDK_1504 and not OPDK_1504_WS. So you took the wrong patch, that patch is meant for OPDK_1504_WS only.

Here's the release notes for 4.15.04.03 patch release without the websockets support: http://apigee.com/docs/release-notes/content/41504...

I am sure you already have the ftp location for downloading the patches. The file name to look for would be apigee-edge-4.15.04.03.zip. Please rollback the previous patch and then apply this one, things should work after that.

[ccovney]

I'm certain I installed the contents of the 4.15.04.00-ws zip file from our FTP. We have not ever activated nor used the WS functionality. When looking around the file system, I see a bunch of websocks scripts and things of this nature, which would imply the installation of the WS version.

[ccovney]

Is there a better way to confirm which base installation was installed?

[adas]

@Chris Covney The ftp site had the websockets base version and patches as well as the regular GA release OPDK1504 and its patches. So you might have picked up the apigee-edge-4.15.04.00-ws zip file instead of the apigee-edge-4.15.04.03.zip.

Regarding your question about getting the base version, the best way is to make the following management api call:

curl -v http://{mgmtip}:8080/v1/buildinfo

The output would be same as what @Vineet Bhatia posted above. If you look carefully, there's SCM_Branch and BUILD_NUMBER and other details. Currently since the websockets feature is still in the works, it comes out of a separate branch called origin/OPDK_1504_WS, so that's one way of identifying. But soon we would merge the WS and the general code base for OPDK so it should not be a problem. Also all installers and patches for websockets would come with the _ws suffix.

Did reverting the _ws patch and installing the correct OPDK patch resolve your problem ?

[ccovney · Accepted solution]

reverting the patch worked. thanks arghya!