Alternative after termination of IoTCore service using MQTT

[fuji1]

Hello.

I am currently sending data from an edge device to BigQuery with the following configuration.
  Edge device ➡ (MQTTS) ➡ IoTCore ➡ Pub/Sub ➡ BigQuery

However, I am looking for an alternative as IoTCore will be out of service in 2023/8.
I found this article but could not find a solution.

I have two questions.
(1) If I follow this article and build a standalone MQTT broker in Compute Engine or GKE, how does Pub/Sub connect to the MQTT broker?
(2) Are there any other ways to send data from an edge device to BigQuery using MQTT?

Thank you in advance for your help.

[pthiagar · Accepted solution]

Hello @fuji1 

There's a GCP provided Dataflow streaming pipeline template that reads MQTT topic messages and writes them to Pub/Sub. See documentation  here: https://cloud.google.com/dataflow/docs/guides/templates/provided/mqtt-to-pubsub

Alternatively, you could define your own batch or streaming data pipeline using Apache Beam and run the pipeline in Dataflow.

Hope this helps.

[fuji1]

Hello @pthiagar 

Thank you for your reply.

I did not know that template.
That template is very helpful.

Thank you very much.

[pthiagar]

You're welcome.  Glad I could help!