ActiveMQ JMS Integration to apigee

Not applicable

We are team working on apigee on-premise environment. Our requirement is to integrate the ActiveMQ JMS to apigee jms proxy. I'm unable to connect to activemq server and I'm setting the proxy as per the given apigee doc.

http://docs.apigee.com/api-services/content/jms-integration#creatingajmsproxyendpoint

In this given doc didn't explain about

1. how the communication happen between activemq jms server and apigee jms proxy?

2. if suppose communication happens how to test it?

Please can any one provide the sufficient docs or any sample API which is using any JMS server.

Solved Solved
0 12 3,201
1 ACCEPTED SOLUTION

adas
New Member

@sureshFirst of all, you need to make sure you are on the correct version of Apigee Edge on-premise installation. JMS feature is available in 1504 and 1504_ws.

Coming to the actual implementation and how it works, here you go:

The router behaves like a consumer of JMS messages and can subscribe to a queue for a JMS provider like ActiveMQ. For this to work, you need to have the activeMQ client libraries installed in the router. You need the following jars:

1. Go to the third-party lib folder:
cd /opt/apiged4/share/lib/thirdparty

2. Download JMS Client Libraries:
sudo wget http://repo1.maven.org/maven2/org/apache/activemq/activemq-client/5.8.0/activemq-client-5.8.0.jar

sudo wget http://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/gero...

sudo wget http://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1...

sudo wget http://repo1.maven.org/maven2/org/fusesource/hawtbuf/hawtbuf/1.2/hawtbuf-1.2.jar

3. Change the file owner and permissions:
sudo chown apigee geronimo-jms_1.1_spec-1.1.1.jar
sudo chown apigee geronimo-j2ee-management_1.1_spec-1.0.1.jar
sudo chown apigee hawtbuf-1.2.jar
sudo chown apigee activemq-client-5.8.0.jar
sudo chmod +x geronimo-jms_1.1_spec-1.1.1.jar
sudo chmod +x geronimo-j2ee-management_1.1_spec-1.0.1.jar
sudo chmod +x hawtbuf-1.2.jar
sudo chmod +x activemq-client-5.8.0.jar

Once this is done, you need to restart the router. This makes sure the router is now ready to act as a consumer of JMS messages. The router uses the client libraries to create a subscription to the destination queue using JNDI.

Next we need to create a jmsHost and create an api proxy that can be deployed to this jmsHost. Once the proxy is deployed, the router subscribes to the queue (mentioned in the jmsHost) and starts picking up messages from the queue. The router picks up these messages, converts them into an equivalent HTTP message (jms headers are converted into http headers, jms message is sent as an http message) and sends it to the message processor like any other http message. From here on in, the normal api proxy flow starts and you can apply all the policies and rules that you could apply for a standard http proxy. On the response flow, once the message processor has completed the response flow, it sends back the response to the router. The router now does a reverse operation and converts the http message into a jms message and writes back to a replyQueue, as specified in the jmsHost.

Now let's take an example:

Step 1: Create a jmsHost:

POST /v1/organizations/{org}/environments/{env}/jmshosts

<?xml version="1.0" encoding="UTF-8"?>
<JMSHost name="jmsHost">
   <Description>Sample JMS Host</Description>
   <ConnectionFactoryClass>org.apache.activemq.jndi.Active</ConnectionFactoryClass>
   <ConnectionFactory>org.apache.activemq.jndi.ActiveMQInitialContextFactory</ConnectionFactory>
   <ConnectionURL>{activemqConnectionURL}</ConnectionURL>
   <Context>ConnectionFactory</Context>
   <ContextUsername>somebody</ContextUsername>
   <ContextPassword>somepassword</ContextPassword>
   <ConnectionUsername>admin</ConnectionUsername>
   <ConnectionPassword>password</ConnectionPassword>
   <Connections>5</Connections>
</JMSHost>

Step 2: Create a jms proxy:
Refer to the attached jmsapi.zip. It has a httpProxy endpoint and multiple jmsProxy endpoint with different subscriptions - different destination queue or message selector etc. An example jmsProxy endpoint is below:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint proxyType="jmsConnection" name="jmsProxyEndpointDifferentQueue">
    <Description/>
    <FaultRules/>
    <Flows/>
    <JmsProxyConnection>
        <Destination>dynamicQueues/sourceAppsDataQ</Destination>
        <JmsHost>jmsHost</JmsHost>
        <MessageSelector>JMSType='apps'</MessageSelector>
        <MessageSelector>JMSType='data'</MessageSelector>
    </JmsProxyConnection>
    <RouteRule name="default">
        <TargetEndpoint>jmsTarget</TargetEndpoint>
    </RouteRule>
</ProxyEndpoint>


Note that MessageSelector is option, you can simply provide a JmsProxyConnection with the Destination and JmsHost entities.

Step 3: Deploy the proxy.

Once you deploy the proxy, you should see messages being consumed from your acitveMQ queues by this proxy. You would also see consumers created and listening on respective queues, depending on how your proxy is configured. You can use the activeMQ admin console to see the list of subscribers and the messages enqueued and dequeued.

I hope this helps you resolve your query. Kindly accept my answer if it does. You can find more details here

View solution in original post

12 REPLIES 12

adas
New Member

@sureshFirst of all, you need to make sure you are on the correct version of Apigee Edge on-premise installation. JMS feature is available in 1504 and 1504_ws.

Coming to the actual implementation and how it works, here you go:

The router behaves like a consumer of JMS messages and can subscribe to a queue for a JMS provider like ActiveMQ. For this to work, you need to have the activeMQ client libraries installed in the router. You need the following jars:

1. Go to the third-party lib folder:
cd /opt/apiged4/share/lib/thirdparty

2. Download JMS Client Libraries:
sudo wget http://repo1.maven.org/maven2/org/apache/activemq/activemq-client/5.8.0/activemq-client-5.8.0.jar

sudo wget http://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/gero...

sudo wget http://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1...

sudo wget http://repo1.maven.org/maven2/org/fusesource/hawtbuf/hawtbuf/1.2/hawtbuf-1.2.jar

3. Change the file owner and permissions:
sudo chown apigee geronimo-jms_1.1_spec-1.1.1.jar
sudo chown apigee geronimo-j2ee-management_1.1_spec-1.0.1.jar
sudo chown apigee hawtbuf-1.2.jar
sudo chown apigee activemq-client-5.8.0.jar
sudo chmod +x geronimo-jms_1.1_spec-1.1.1.jar
sudo chmod +x geronimo-j2ee-management_1.1_spec-1.0.1.jar
sudo chmod +x hawtbuf-1.2.jar
sudo chmod +x activemq-client-5.8.0.jar

Once this is done, you need to restart the router. This makes sure the router is now ready to act as a consumer of JMS messages. The router uses the client libraries to create a subscription to the destination queue using JNDI.

Next we need to create a jmsHost and create an api proxy that can be deployed to this jmsHost. Once the proxy is deployed, the router subscribes to the queue (mentioned in the jmsHost) and starts picking up messages from the queue. The router picks up these messages, converts them into an equivalent HTTP message (jms headers are converted into http headers, jms message is sent as an http message) and sends it to the message processor like any other http message. From here on in, the normal api proxy flow starts and you can apply all the policies and rules that you could apply for a standard http proxy. On the response flow, once the message processor has completed the response flow, it sends back the response to the router. The router now does a reverse operation and converts the http message into a jms message and writes back to a replyQueue, as specified in the jmsHost.

Now let's take an example:

Step 1: Create a jmsHost:

POST /v1/organizations/{org}/environments/{env}/jmshosts

<?xml version="1.0" encoding="UTF-8"?>
<JMSHost name="jmsHost">
   <Description>Sample JMS Host</Description>
   <ConnectionFactoryClass>org.apache.activemq.jndi.Active</ConnectionFactoryClass>
   <ConnectionFactory>org.apache.activemq.jndi.ActiveMQInitialContextFactory</ConnectionFactory>
   <ConnectionURL>{activemqConnectionURL}</ConnectionURL>
   <Context>ConnectionFactory</Context>
   <ContextUsername>somebody</ContextUsername>
   <ContextPassword>somepassword</ContextPassword>
   <ConnectionUsername>admin</ConnectionUsername>
   <ConnectionPassword>password</ConnectionPassword>
   <Connections>5</Connections>
</JMSHost>

Step 2: Create a jms proxy:
Refer to the attached jmsapi.zip. It has a httpProxy endpoint and multiple jmsProxy endpoint with different subscriptions - different destination queue or message selector etc. An example jmsProxy endpoint is below:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint proxyType="jmsConnection" name="jmsProxyEndpointDifferentQueue">
    <Description/>
    <FaultRules/>
    <Flows/>
    <JmsProxyConnection>
        <Destination>dynamicQueues/sourceAppsDataQ</Destination>
        <JmsHost>jmsHost</JmsHost>
        <MessageSelector>JMSType='apps'</MessageSelector>
        <MessageSelector>JMSType='data'</MessageSelector>
    </JmsProxyConnection>
    <RouteRule name="default">
        <TargetEndpoint>jmsTarget</TargetEndpoint>
    </RouteRule>
</ProxyEndpoint>


Note that MessageSelector is option, you can simply provide a JmsProxyConnection with the Destination and JmsHost entities.

Step 3: Deploy the proxy.

Once you deploy the proxy, you should see messages being consumed from your acitveMQ queues by this proxy. You would also see consumers created and listening on respective queues, depending on how your proxy is configured. You can use the activeMQ admin console to see the list of subscribers and the messages enqueued and dequeued.

I hope this helps you resolve your query. Kindly accept my answer if it does. You can find more details here

Thank you very much for providing the valuable data and spent time.

We have actually tried with given sample api proxy. It is successfully pushed the data into sourceQ. You are using the below code in the targets. Could you please explain what it does and how data is pushed form sourceQ to replyQ.

<HTTPTargetConnection>

<Properties/>

<URL>http://httpbin.org/post</URL>

</HTTPTargetConnection>

<HTTPTargetConnection>

<Properties/>

<URL>http://httpbin.org/get</URL>

</HTTPTargetConnection>

Actually, we want to do two way communication between apigee to activemq

1. Get message from activemq to apigee

2. Push message from apigee to activemq

Please correct me am i doing wrong with concept.

Please queues.pngfind activemq queues in attached image.queues.png

@suresh Apigee proxies can act as a subscriber and consume messages from a queue, you cannot use them as a producer and publish messages to a queue.

The target configuration in the example is actually making an http call to the target and getting a response back, just like any other http proxy. Once the message processor receives the http response it sends it back to the router and the http to jms message conversion happens. The router then sends the acknowledgment msg to the replyQ. You can hack around this piece to say that the router is actually publishing a message to the replyQ. But its nothing more than a hack, in true sense the jms integration in Apigee is meant to be used as a consumer and not a publisher.

Thanks for giving very quick reply.

We are actually pushing the message to source queue from standalone code. Now, we want to consumes the pushed message in apigee prory. We are stuck at consuming the message in apigee proxy from activemq queue.

I'm attaching the standalone code which we are used to receiving the message. Please help me out.

In this file we have two methods one for pushing the message to source queue, other one is for receiving the message.

test.zip

adas
New Member

@suresh what exactly is the problem. Looking at the publisher code will not help. Have you made sure you are on the correct opdk version that supports jms. You can retrieve the information by calling /v1/buildinfo on the management server endpoint.

Secondly, I am assuming you followed the instructions given to create a jmsHost, create a prroxy using that jmsHost and deployed the proxy. After the proxy is deployed, do you see consumers created in the activemq admin console ? Did you download and install the activemq client libraries in the thirdparty folder ? If you can attach the router logs from this period, it would be super useful.

1. The main problem is we want to consumes the message from activemq queue and want to do some changes on the message in apigee proxy. How to do this?

2. I want to check the error logs of run time in on-premise. how can i?

we created a jmsHost, jms proxy and deployed. After the proxy is deployed we are able to see consumers (Queues) in activemq admin console. We copied required what you mentioned client libraries in the thirdparty folder as well.

I have attached the router logs and queues createdqueues.png here...

logs.zip

adas
New Member

The sample I sent you shows how you can add additional headers to the jms message in the apiproxy layer. Please follow that.

For logs please go to /opt/apigee4/var/log/apigee/logs and look for system.log under the respective component.

You mentioned the proxy is picking up messages and you see consumers on the queue, then what is not working ? I don't understand the part where you say "not working". Could you please elaborate. I will take a look at the files you have sent.

adas
New Member

@suresh The image shows that your consumers are getting created correctly. Now if you send some messages to these queues, you should see those getting picked up by the consumers and the router would send those messages to message-processor and eventually you should se those messages published to a replyQ as acknowledgements.

I also wrote a comprehensive test framework using Java/TestNG and Excel feeders which covers all the JMS cases. It has utility classes to publish messages to a activeMQ queue, read JMX metrics from the queue to check consumer count, enqueue and dequeue count etc. I might have to clean up few things before I can share that project with you. That might give you a fair idea of how it works and how to test the system. Let me know, if you think that would help. If you have any issues, please feel free to contact Apigee support.

Thank you for great help.

we checked the system logs of router. we found that errors like

Router-ClientThread-0-3 ERROR c.a.j.a.Transformer - Transformer.transformHttpToJms() : Error in setting properties pair X-Apigee.Message-IDapigee_server.xxx_CVB5JB6T4_ID:01HW551229-54573-1461304328944-4:2:1:1:17_ID:01HW551229-54573-1461304328944-4:2:1:1:17 java.lang.NullPointerException: null

In activemq also, message is en-queued and dequeued from sourceQ. But I can't see those messages published to a replyQ as acknowledgements.

Pls check the attachments for the same.

If you share the test framework to test the jms use cases which would help us a lot.

queues.png system-log.txt

adas
New Member

Can you send me your proxy zip. Also I would suggest opening a support ticket to get help on this as this thread is becoming too long.

Please find the attached proxy zip. Please provide support on the same.

jmsexample-rev3-2016-05-02.zip

Not applicable

Hi, we need integrate apigee to consume message from mqseries. can you send me an example?