There are 10-minute latencies in the RDS-mysql-to-bigquery streams

[peter_huang]

I've established streams using a single private connectivity. As I added more streams, I noticed a higher frequency of 10-minute latency without any error messages.

1. Is this related to the CIDR of the private connectivity being /29, which only contains 8 hosts?
2. Can I create a private connectivity with a /24 CIDR, or should I create multiple ones with /29?

2024-02-20 Update: Change the subject from `How many datastream streams can I use with one private connectivity?` to `There are 10-minute latencies in the RDS-mysql-to-bigquery streams.`

[lsolatorio]

Hi @peter_huang, 

Welcome and thank you for reaching out to our community.

A quick answer to your question is, there is no fixed number of streams with a single private connectivity, it is all determined by your resources and data volume.

You can explore /24 CIDR with a higher host count to provide more IP addresses, easing network congestion and reducing latency. 

The best practice here would be closely monitoring your resources like CPU, memory, disks and network utilization, while running your streams as this can help you identify potential bottlenecks so you can adjust the number of your streams accordingly. 

Here are some resources that can be of help:

• Network connectivity options for Datastream
• Manage private connectivity configurations
• Private connectivity

[peter_huang]

Hi @lsolatorio,

Thanks for the explanation; your answer supports my assumption.

By the way, I have granted the permissions to create a case ticket. I'll return and provide the root cause here.

[peter_huang]

In conclusion,

1. The 10-minute latencies are not related to the private connectivity.
2. Google support provided a solution: The latency can typically increase when there is an increased volume of changes to process. If that is the case, you can try to reduce the latency by following the below-mentioned workarounds:
   1. Increasing the CDC concurrency limit. [1]
   2. Reducing the size limit of the binlog files. [2]
3. I have tried increasing the CDC concurrency limit, but it did not work properly. I think it cannot be resolved today, as there were no logs or errors when the latencies occurred.

[1]: https://cloud.google.com/datastream/docs/stream-concurrency-controls#maxcdctasks 
[2]: (Disclaimer: This is an external link, Google is not responsible for any information provided here.)
https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_max_binlog_size