Data at Google Cloud Next '24!
Answer this question in the comments to earn a special badge: In your journey, what's one overlooked data skil...
Forum Posts
Influence Looker’s Roadmap and join the BI Customer Council (BICC)
Would you like to influence Google Cloud’s BI roadmap more? I’d like to invite you to join the Google Cloud BI...
Google-Managed Encryption
Hello, I am unclear about the Google-Managed Encryption Process.- How often does google rotate encryption keys...
Log Explorer - Improper Log Parsing
I have a Flink job running in a dataproc cluster. It's exporting logs to Log Explorer but I don't think Log Ex...
Dataproc Alert policy
custom/cpu/utilizationcustom/disk/percent_usedhow are the above cloud dataproc batch metric useful when settin...
Dataproc Alert
I need help regarding creating Dataproc alerts.What all type of alert we can create for Dataproc.
SQL Spark connection in juptyerlab
Hi,I have a Dataproc cluster and using the web interface (Jupyterlab) inside the cluster, I am trying to read ...
Vertex AI - running pyspark in jupyter notebook
Hi,I am currently converting some Alteryx workflows into pyspark and wanted to test the codes for debugging an...
Serverless Batches CMEK upcoming change
I noticed this alert on the serverless batches page:"Starting March 31, 2024, special requirements apply to us...
Dataproc high resource usage and its optimization and analysis
Hi,I'm using Dataproc on VMs. Currently we are running in production two clusters: master VM typeworker VM typ...
Submit Dataproc serverless job using service account
Hi Folks,I'm trying to submit to submit a Dataproc serverless job using service account (have necessary permis...
GCP Dataproc class not found error
Hi, I have a pipeline set up, it creates a cluster perform an ingestion and delete a cluster. In success logs ...
Dataproc Metastore not syncing metadata to Data Catalog
Hi,we have created a metastore and it has a database and table, post editing the metastore and enabling the Da...
Can't use Dataproc 2.2
When trying to use Dataproc image version 2.2. the gcloud CLI raises a validation error.Trying the same comman...
Does GCP Data Lineage captures Lineage of Hive Tables
Hi,we're using hive tables with dataproc metastore synced to data Catalog, we could see that data catalog was ...
Running a job in dataproc cluster via livy
I have a dataproc cluster with livy installed and it is working.what is the command to run a 'hello world' scr...
running a job in dataproc cluster via compute engine
is it possible to trigger a job in dataproc cluster from the compute engine vm?the sample 'hello world' job fi...
Disable Dataproc cluster logs
Hi!I have a Dataproc Single Node Cluster that is used as a Spark History Server. However I notice in Cloud Log...
Can't connect to Dataproc Metastore from Dataproc Batch Jobs
Hi,While I'm trying to connect to hive-metastore(Dataproc Metastore) using thrift url within Spark Configurati...
which service to choose : hive metastore on cloud sql or on metastore service
Hi When we are using a dataproc cluster that needs a hive metastore,what benefits we have using the metastsore...
Dataproc Metastore couldn't start
I got an issue happened since 3h30 UTC with Dataproc MetastoreAs I look into Logs Explorer, it happened after ...
trigger spark job from Airflow bitnami on compute engine
HiI have successfuly deployed a job in airflow from google cloud composer.I'm trying now to do the same by pro...
Spark Streaming with PubSub Lite
I want to create a Spark Streaming application in DataProc that has a subscription to PubSub Lite or PubSub (I...
SQL to Big Query Data Fusion Pipeline failed
Im trying to build google data fusion pipeline to load data from MS SQL server to Big Query. The source (MS Sq...
Facing the error which connecting cloudSQL MSSQL server using JDBC in dataproc
when i am running the same pyspark job to read the cloud sql data using jdbc in pycharm on local machine where...
Cloud SQL MSSQL not connecting from dataproc & dataproc serverless using jDBC
The cloud sql MSSQL which i created , i am able to connect it from sql server management studio on my laptop (...
Dataproc serverless accepts only Spark 3
Hello Hi realy would like to test a series of spark 2.4 jobs on dataproc serverless but there is only spak 3.x...
Namespace not able to view and Connection refused error in Data Fusion Pipeline
Dear Experts,we have configured two names space in the cloud data fusion, but from yesterday onwards not able ...
disk quota exceed but can not see its used anywhere
my quota is 4096 GB and its reached after I triggered the serverless spark job ( which failed) but its still s...
How to install python packages during runtime in dataproc serverless
In Dataproc, we have metadata option to install python packages in a dataproc cluster created through Composer...
Selected software image version '2.1.24-s8s-spark' does not support control plane decommissioning
Hi,I have a dataproc serverless batch job using the 2.1 Runtime version.Looking to use the new autoscaling ver...
who sent the job to dataproc?
Hello community,I need to know who started a job in the dataproc cluster... I need to know if it was a service...