This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Hi!I have a Dataproc Single Node Cluster that is used as a Spark History
Server. However I notice in Cloud Logging that this is creating quite a
lot of logs. Looking at the documentation I can see that indeed there
are a number of components that wil...
Hi,I have a dataproc serverless batch job using the 2.1 Runtime
version.Looking to use the new autoscaling version on my job (i.e.
"spark.dataproc.scaling.version": "2") but when submitting the job it
fails instantly with: "Selected software image ve...
Thanks again for the quick reply and suggestions!it seems a new patch
version 2.1.27 was just recently released, this gets automatically
picked for the 2.1.And the control plane error I was seeing before has
disappeared I tried checking for the ver...
@ms4446 thanks for the explanation. Some follow up questions:If wanted
to go with another patch version, how do I achieve this? When trying to
do that e.g. 2.1.23 (instead of 2.1.24) I get this errorERROR:
(gcloud.dataproc.batches.submit.pyspark) INV...
Thanks for the quick reply @ms4446I can also add that I did a simple
change to degrade the spark runtime version to 2.0 (specifically
2.0.45)and that seem to have worked fine with
"spark.dataproc.scaling.version= 2"!So I am not sure its necessarily t...
