Pushing data from Google Cloud Pub/Sub into an Apigee proxy

[hsbhattacharjee]

I want to push data from Google Cloud Pub/Sub into an Apigee proxy.

I know Receiving message using push in google pubsub is the best option. Can anyone guide me?

[dchiesa1]

No, I think that pushing data from Pub/Sub into an Apigee Edge proxy is not the right idea.

Think of Pub/Sub as a message store. If there are subscribing apps, they can watch the message store for available messages. But Apigee Edge is not a pub/sub subscriber. A Proxy in Apigee Edge is an HTTP listener.

For your subscriber, for the thing that pulls messages from the pub/sub store, you need a longer-running asset, a different type of artifact. For example you can use a cloud function:

https://cloud.google.com/functions/docs/calling/pubsub

Or an app running in Google App Engine. Or something else.

That subscriber could then invoke an API endpoint, managed by an Apigee Edge API proxy.

But Apigee Edge won't "subscribe" to the pub/sub topic itself.

More here:

https://cloud.google.com/pubsub/docs/overview#endpoints

[jachansen]

There is a pattern out there with Apigee in the middle of a pub/sub, not exactly as you describe. Someone handed me an architecture as Datastore->Microservice -> Apigee -> Kafka. I questioned it as a valid pattern and found out it had already been deployed in prod at a rather large company. Use a null target with a service call out to hit Kafka. Apigee was used for analytics and transformation. I wouldn't say it's common, but it's deployed out there. Apigee can do a lot, but it doesn't mean it should. I ultimately argued against the architecture and they left Apigee out.

[ylesyuk · Accepted solution]

Great question! Using Edge to consume pubsub messages is easy to setup and showcases versatility of the tool as well as the power of integration of involved technologies.

Create proxy with domain verify endpoint

1. Go to the URL https://www.google.com/webmasters/verification of your gcp account.

https://dbcedge-eval-test.apigee.net/push-handlers

2. Notice html verification file and download it.

3. Keep this page open.

Setup listening proxy

1. Create push-handlers No Target proxy with proxy base path push-handlers. Passthrough. Virtual Host: Secure. Environment: test. Build and Deploy.

2. In DEVELOP add new flow with the name of the html verification file, in our case: [google8343a21f428165ca.html

3. Add Assign Message for the GET html flow to the PreFlow RESPONSE. Populate Payload with the contents of the verification file and contentType text/plain.

4. Save and Deploy the proxy.

5. At the Webmaster Central page you hopefully still have opened, click on I'm not a robot reCAPTCHA and VERIFY button.

You should receive the url is verified message

Configuring http endpoint

Create pubsub topic

1. Create topic books

2. Press CREATE

Add Domain verification entry

1. In the API & Services select Credentials page. Select Domain Verification tab. Click Add domain. Add the domain URL (no protocol).

dbcedge-eval-test.apigee.net/push-handlers/

Register endpoint

1. In the books topic's menu, select + New Subscription item.

2. Populate subscription name bookssubscription; Delivery Type: Push into an endpoint url; enter the url without protocol (https is the only expected option): dbcedge-eval-test.apigee.net/push-handlers/. Press Create button.

Your subscription is created.

Testing pubsub push message

1. In the Edge UI, start tracing session for the push-handlers proxy.

2. In the pubsub books topic, chose Publish message

3. Enter sample message, ie: Test message. and a sample key-value attribute: key, val. Press Publish button.

4. In the Tracing session page, you should see your messages request

5. You can see the key/val field as well as data property. As the message's data field is base64 encoded, we can use a base64 decoder to validate it: ,"data":"VGVzdCBtZXNzYWdlLg=="

[hsbhattacharjee]

@ylesyuk.Thanks awsome....

[vbalan]

Hi,

Thank  you so much for sharing the step-by-step method for this POC.

Quick question - The flow is enabled with request.verb = "GET". But request show on Trace as a POST. Just curious why that is..?

[dchiesa1]

The flow is enabled with request.verb = "GET". But request show on Trace as a POST. Just curious why that is..?

The question you are asking touches on a fundamental behavior of the Apigee API Proxy, and has nothing to do with Google Pubsub in particular.

The API proxy you have configured is designed to listen to inbound HTTP requests on the basepath of /push-handlers. The proxy will handle inbound requests with any verb. In your API proxy you have a conditional flow configured specifically to handle a GET on /googlee4ffxyz.html . And you have one step configured for that conditional flow, something called "Verify-Domain". You have no other conditional flows - in other words no other rulesets configured for Apigee to do anything special for any other particular request, any verb + path combination other than GET on /googlee4ffxyz.html .

Apparently, according to the Trace session you showed, your proxy has received a POST request on /push-handlers/ . And the API Proxy received it and handled it. And there is no special flow for that combination, so it ... did not run any special set of policies.

If you would like to reject such requests, in other words reject requests that are not "special cased" by other conditional flows in the API Proxy, then you would need to include a catchall flow, something with no Condition element. Any inbound call that does not match any of the other Conditions would get handled by that Flow. It would look like this:

<ProxyEndpoint name="endpoint1">

  <HTTPProxyConnection>
    <BasePath>/push-handlers</BasePath>
    <VirtualHost>https_secure_1</VirtualHost>
  </HTTPProxyConnection>

  <FaultRules/>
  <PreFlow name="PreFlow"> ...  </PreFlow>

  <PostFlow name="PostFlow"> ... </PostFlow>

  <Flows>
    <Flow name='t1'>
      <Request>... </Request>
      <Response>... </Response>
      <Condition>proxy.pathsuffix MatchesPath "/googleeffxyz.html" and request.verb = "GET"</Condition>
    </Flow>

    <Flow name='unknown request'>
      <Request>
        <Step>
          <!-- send back an error message, this proxy does not handle this request -->
          <Name>RF-Unknown-Request</Name>
        </Step>
      </Request>
    </Flow>

  </Flows>

  <RouteRule ...>... </RouteRule>

</ProxyEndpoint>

If you want to explicitly handle the POST, then you need to add a conditional flow for that.

  <Flows>
    <Flow name='t1'>
      <Request>... </Request>
      <Response>... </Response>
      <Condition>proxy.pathsuffix MatchesPath "/googleeffxyz.html" and request.verb = "GET"</Condition>
    </Flow>

    <Flow name='t2'>
      <Request>... </Request>
      <Response>... </Response>
      <Condition>proxy.pathsuffix MatchesPath "/whatever" and request.verb = "POST"</Condition>
    </Flow>


    <Flow name='unknown request'>
      <Request>
        <Step>
          <!-- send back an error message, this proxy does not handle this request -->
          <Name>RF-Unknown-Request</Name>
        </Step>
      </Request>
    </Flow>

  </Flows>

[vbalan]

Hi,

Thank you for this Push-handler POC. Is there a step-by step POC for pulling from a topic using an Apigee proxy?

regards,

Vishnu