Jupyter Stopped Functioning After Auto-Upgrade

My Vertex AI instance will not start after an auto-upgrade that was preformed on it. It seems to forever be stuck on "Starting", "Initializing" or "Provisioning". Using the diagnostic tool it appears the Jupyter is "unhealthy":

google error.png

Using the Google Cloud console to attempt to restart the service is not helpful as I get the following error:

ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 22)
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

However, I have setup the firewall permissions to allow such connections as outlined in the documentation.

Running the subsequent troubleshooter that is recommended to resolve the issue results in the following error:

Starting ssh troubleshooting for instance https://compute.googleapis.com/compute/v1/projects/future-pager-399113/zones/us-east4-a/instances/PR... in zone us-east4-a
Start time: 2024-04-17 14:38:16.717230

---- Checking network connectivity ----
The Network Management API is needed to check the VM's network connectivity.

Is it OK to enable it and check the VM's network connectivity? (Y/n)? Y

Enabling service [networkmanagement.googleapis.com] on project [future-pager-399113]...
Your source IP address is 35.231.107.27

Network Connectivity Test Result: UNREACHABLE

To view complete details of this test, see https://console.cloud.google.com/net-intelligence/connectivity/tests/details/ssh-troubleshoot-v6a48?...

Help for connectivity tests:
https://cloud.google.com/network-intelligence-center/docs/connectivity-tests/concepts/overview

---- Checking user permissions ----
User permissions: 0 issue(s) found.

---- Checking VPC settings ----
VPC settings: 0 issue(s) found.

---- Checking VM status ----
The Monitoring API is needed to check the VM's Status.

Is it OK to enable it and check the VM's Status? (Y/n)? Y

Enabling service [monitoring.googleapis.com] on project [future-pager-399113]...
ERROR: (gcloud.compute.ssh) HTTPError 400: The resource 'projects/future-pager-399113/zones/us-east4-a/instances/PROJECT' is not ready

This all seems to be circular and initially was caused by Googles auto-upgrade service. What are the steps to resolve this?

10 0 123
0 REPLIES 0