This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
When running a job with a fairly large image (few GBs) transition time
from SCHEDULED to RUNNING takes about 6 mins.When enabling container
runnable image streaming with `enableImageStreaming` I don't really
observe any difference in startup time. Wh...
Batch allows the `machine-type` to be specified, which is very helpful
and allows putting task on a specific machine type like
`n2-standard-8`.Without this flag Batch seems to schedule on machines
based on task cpu/mem requirements which is great.It ...
The maximum number of parallel tasks for a job is 1k.
https://cloud.google.com/batch/quotasI'm sure there are sound reasons
for this limit, there been any consideration to increasing it? Currently
we write scripts that launch multiple jobs but would ...
What backend does Cloud Composer use by default for xComs? Does it just
use the database or does it use a custom GCS based XComs Backend?What is
the max file size for sharing data between tasks via Xcoms / Taskflow?Is
adding a custom XComs backend su...
I am viewing logs for a batch job with 10 tasks. I have the following
query:```my-job-uuid
logName="projects/my-project/logs/batch_task_logs"resource.labels.task_id="task/my-job-uuid-group0-10/0/0"```
I am understanding the task_id with the suffix "1...
If the image streaming requirements are not met, will the submit return
an error, or will the job proceed without image streaming enabled?One
can tell based on the log format?
Ok great that makes sense.Do you foresee any issues with creating a
custom GCS backend like shown here and having XComs automatically
passing? Thus allowing passing things around with the following
semantics:@taskdef get_data():large_blob = download_...
So by default the metadata database is used?What is the recommended way
to use GCS for XComs? Write a custom backend or is there any prebuilt
support / off the shelf opinionated GCS backend for using with Composer?
@bolianyin It looks like `Storage Object Admin` is not entirely
sufficient.The issue is that in addition to storage object access, the
service account also
needs:`storage.buckets.get``storage.buckets.list`which are not included
in the Storage Object ...
