Troubleshooting Connection Errors between Dataform, Cloud Composer, and GitHub

[Savuk]

Hey!

We started using Dataform with cloud composer recently we started having a lot of this errors where the Dataform trigger return an exception with the description

"UNKNOWN:Error received from peer ipv4:xxx.xxx.xxx.xxx {grpc_message:"The remote repository https://github.com/xxxx/xxxx.git closed connection during remote operation."

According to what we've found this is a problem when trying to connect to Github. We've tried adding exponential retries to the DAG but we are still facing this issue which if we cant find a solution will make Dataform not viable for us.

Has someone stumble upon this?

[ms4446]

The error message "UNKNOWN:Error received from peer ipv4:xxx.xxx.xxx.xxx {grpc_message:"The remote repository https://github.com/xxxx/xxxx.git closed connection during remote operation." indicates that the connection to your GitHub repository is being terminated unexpectedly during Dataform operations.

Potential Causes

• Network Issues: Transient network problems, firewall restrictions, or IP address blocks on either Google Cloud or GitHub's side could be interfering with connectivity.

• Rate Limiting: GitHub imposes rate limits on API requests, and it's possible that Dataform operations might be exceeding these limits.

• Authentication: There could be issues with GitHub access tokens (incorrect, expired, or revoked) used by Dataform.

• Incorrect Repository Configuration: The GitHub repository connection within Dataform might be misconfigured.

• Large Repository or Operations: Large repositories or complex operations could lead to timeouts.

Troubleshooting Steps

Check Network Connectivity:

• Ensure no firewall rules are blocking traffic between Google Cloud and GitHub.

• Utilize network diagnostic tools like ping and traceroute to test connectivity to GitHub, though remember these tools may not diagnose protocol-specific issues (HTTPS/SSH).

• Test connecting to GitHub from a Google Cloud Compute Engine instance in the same project as Cloud Composer to isolate network issues.

Verify GitHub Rate Limits:

• Consult GitHub's documentation on API rate limits to understand current limits.

• Consider spacing out Dataform operations or requesting an increase in rate limits if you're hitting these limits frequently.

Review Authentication:

• Check that the GitHub access token used in Dataform is valid, has not expired, and possesses the necessary permissions (e.g., repo access).

• If using SSH keys for authentication, ensure they are correctly configured and the public key is added to your GitHub account.

Double-Check Repository Settings:

• Confirm the GitHub repository URL in Dataform is correct and accessible.

• Ensure Dataform has the appropriate permissions to access the repository.

Reduce Load (If Applicable):

• Consider breaking down large repositories or optimizing complex Dataform queries to lessen the load on GitHub.

Cloud Composer and Dataform Logs:

• Review Cloud Composer logs for specific error messages or clues.

• Also, check any available Dataform logs or diagnostic information for errors that occur before or during the connection issues.

[Savuk]

We are running 10 DAGs triggering different pipelines, 9 are daily and 1 hourly. Meaning that on a day to day basis everything works but roughly once a week in any of the DAGs we get this connection error.In the Dataform logs we don't see anything and in Cloud Composer we only see the mentioned log.

[ms4446 · Accepted solution]

Given the intermittent nature of the connection errors you're experiencing with your DAGs in Cloud Composer, and the fact that these issues occur roughly once a week without specific patterns in terms of which DAG fails, it suggests the problem might be transient or related to specific conditions that are occasionally met. Here are some refined troubleshooting steps and considerations based on this additional context:

1. Review System Load and Scheduling Patterns

• Peak Times: Identify if the failures correspond to peak usage times either on your Google Cloud resources or on GitHub's side. GitHub's API rate limiting or system load could be more restrictive during high traffic periods.

• DAG Overlaps: Check if the hourly DAG execution might occasionally overlap with one or more of the daily DAGs in a way that could increase the load on your system or GitHub's API beyond typical levels.

2. Examine External Dependencies

• External Services: If your DAGs rely on external services (including GitHub), review their status or incident logs for any reported issues around the times your jobs fail. This could provide clues if the issue is external.

• Network Fluctuations: Given the error points to a connection issue, network stability or brief interruptions could be at play. Although harder to diagnose, monitoring network performance metrics during expected DAG execution times might reveal patterns.

3. Optimize Retry Logic

• Smart Retries: Since you've implemented exponential retries, consider adjusting the retry logic based on the time of day or system load. For instance, introducing a longer initial delay or a more gradual backoff might help during known peak times.

• Error-Specific Handling: Tailor your retry mechanisms to be more responsive to the specific UNKNOWN:Error received from peer error. This might involve parsing error messages to ensure retries are only attempted when there's a reasonable expectation of recovery.

4. Increase Logging Verbosity

• Dataform Debugging: Even though Dataform logs don't show errors, ensure that logging levels are set to capture detailed information that might otherwise be missed. There might be pre-failure conditions or warnings that are not logged at default levels.

• Cloud Composer Logging: Similarly, ensure Cloud Composer's logging is as verbose as possible around the times the DAGs are scheduled to run. This might capture additional context not seen at standard logging levels.

5. Infrastructure and Configuration Review

• Resource Allocation: Review the resource allocation for Cloud Composer and any associated services. Resource constraints can sometimes lead to intermittent issues.

• Configuration Consistency: Ensure that all DAGs, especially the hourly one, are configured consistently with respect to how they handle connections and dependencies. A small configuration difference might be leading to the intermittent issues.

6. Engage with Support

• Google Cloud Support: With detailed logs and a pattern of when these errors occur, Google Cloud Support might be able to provide more targeted insights or identify issues within Cloud Composer or the network.

• GitHub Support: If there's a suspicion that the issue is on GitHub's end (e.g., rate limiting or API behavior), reaching out to GitHub with specific error messages and timings can help clarify this.

7. Monitor and Adjust

• Monitoring Tools: Utilize Google Cloud's monitoring tools to track the performance and health of your Cloud Composer environment and any related services. Setting up custom alerts for specific error messages or performance metrics can help catch issues early.

• Adjust Scheduling: If possible, experiment with adjusting the scheduling of your DAGs to reduce concurrency or spread out resource usage. This might help identify if resource contention or rate limiting is a factor.