asia-northeast1 bucket much slower write speed than us-east1

Hi team,

My setup is the following,   I first gscfuse a bucket in {location} onto a gcp compute engine in the same {location} and then mount it on docker. My server is running inside the docker and every 10min I append new entries to a csv file in that bucket. 

When location=us-east1, it takes 300milliseconds to append but when location=asia-northeast1, it takes 30 seconds, so 100x slower!

I am wondering why is that? Notice that I am always writing from us-east1 to us-east1 and asia-northeast1 to asia-northeast1 so the distance from us-east1 to asia-northeast1 should not matter.

Here is the observability for my asia bucket, it is pretty weird that

1.i am appending to a csv every 10min but no write request count is showing

2.the data egress rate (this should be equivalent to the write speed since it is spiking every 10min, corresponding to my write frequency) is pretty slow, only 16B/s. 

gradientopt_0-1715159118988.png

My guess is that the write request on asia bucket is first sent to the open internet and then to gcs instead of from my compute engine directly to the bucket within google's network, but not sure. 

Could someone help me with this? This has been bothering our team for a long time and we could not resolve it. Thanks a lot!

More context on the bucket setup: (standard storage class, Not public access, None protection)

gradientopt_0-1715159354539.png

 

3 0 41