LinkedIn
Twitter
In this post, I introduce newly published Python scripts that demonstrate calling the Chronicle REST API (currently in v1alpha). These Python examples complement the Chronicle API reference documentation.
But first, a little background.
Chronicle API Samples Python GitHub Repo
The Chronicle API Samples Python project on GitHub has sample Python client code (“scripts”) and sample inputs for the old, and now the new, Chronicle API. The scripts, like lists/get_list.py, are executed from the command line as modules using the -m flag. For example, python3 -m lists.get_list.
There isn’t a Python package to install via pip—you simply run the samples from the directory where you’ve checked them out after installing the prerequisites defined in requirements.txt. See the ReadMe in the repo for more details.
To get “usage” for any one of the scripts, run it with the -h or –-help flag:
python3 -m lists.get_list -h
usage: get_list.py [-h] [-c CREDENTIALS_FILE]
[-r {asia-northeast1...,me-west1,us}]
-n NAME -f LIST_FILE
options:
-h, --help show this help message and exit
-c CREDENTIALS_FILE, --credentials_file CREDENTIALS_FILE
credentials file path (default: '$HOME/.chronicle_credentials.json')
-r {asia-northeast1,...,me-west1,us}
the region where the customer is located (default: us)
-n NAME, --name NAME unique name for the list
-f LIST_FILE, --list_file LIST_FILE
path of a file to write the list content to, or - for STDOUT
Out with the old; in with the new
We’ve recently published to that same GitHub repository new Python scripts that demonstrate the new Chronicle REST API. You will find the new scripts in “v1alpha” sub-directories like:
- ./detect/v1alpha
- ./ingestion/v1alpha
- ./lists/v1alpha
They are invoked as modules just like the old scripts, for example, python3 -m lists.v1alpha.get_list -h gives usage on the script that calls the v1alpha referenceLists.get API method.
The remainder of this blog post will focus on the new Python scripts and sample inputs for create, get, and patch updates to the v1alpha ReferenceList Resource. For the patch updates, I will also describe the client-side logic and feature flags that enable addition or removal of items for an existing Reference List.
Prerequisites
In the examples that follow, I’ll use these three environment variables:
PROJECT_ID =<GCP Project ID shown in blue box>
PROJECT_INSTANCE=<Customer ID shown in red box>
REGION=<Chronicle Region, see below for further details on this value>
The values for these were found on the settings/profile page of my Chronicle installation:
https://<cust-code><-region>.backstory.chronicle.security/settings/profile
The Chronicle project’s region is only required if it is not the default (“us”). At the time of writing, possible values for this are:
- ”asia-northeast1"
- "asia-south1"
- "asia-southeast1"
- "australia-southeast1"
- "eu"
- "europe-west2"
- "europe-west3"
- "europe-west6"
- "me-central2"
- "me-west1"
- "us"
My region is the default (“us”), so it isn’t present in my URL and I can (and sometimes do) omit the --project_region="us" argument to the script. If you don’t have a region in your URL and you are not in the “us” region, you may have to contact your Chronicle representative to learn what value to put here.
Lastly, to use these APIs, I have saved the JSON credentials for my service account, which has been granted the required IAM permissions, in the default location: $HOME/.chronicle_credentials.json. As seen in the examples below, this allows me to omit the --credentials_file argument (but you must have that file in place if you copy/paste my code).
For create, get, and patch on the referenceLists resource, the required IAM permissions are:
- chronicle.referenceLists.create
- chronicle.referenceLists.get
- chronicle.referenceLists.update
Create Reference List v1alpha
Now let’s dive into the more interesting stuff! The new script lists.v1alpha.create_list creates a Reference List via the new Chronicle API’s referenceLists resource. Below, I’m using one of the new example input files (from the GitHub repo), which has six file hashes from a recent phishing/malware campaign. The description for the Reference List includes a link to the blog post, where these file hashes were sourced.
#
# Create new Reference List
#
python -m lists.v1alpha.create_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--project_region=$REGION \
--list_file=./lists/example_input/coldriver_sha256.txt \
--name="COLDRIVER_SHA256" \
--syntax="REFERENCE_LIST_SYNTAX_TYPE_REGEX" \
--description='Hashes of observed lure documents “Encrypted” PDFs and SPICA backdoor[1].
[1] https://blog.google/threat-analysis-group/google-tag-coldriver-russian-phishing-malware/' \
--credentials_file=$HOME/.chronicle_credentials.json
New list created successfully, at 2024-02-07T19:48:41.354271Z
The screenshot below shows the new Reference List in the Chronicle web UI. Note that I’m searching for the name that I provided in the above command, COLDRIVER_SHA256.
Oops! I see that I set the wrong syntax_type. This should be a PLAIN_TEXT_STRING instead of REGEX. Let’s fix that.
Get Reference List v1alpha
First, I confirm my mistake with a RESTful GET of the existing list using the get_list script. Again, I use the name that I provided to search for it, COLDRIVER_SHA256.
#
# Get
#
python -m lists.v1alpha.get_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--name="COLDRIVER_SHA256"
{
"name": "projects/proj0001-403014/locations/us/instances/b76bb0ab-54f3-417c-b1c2-7e8dc889cd25/referenceLists/COLDRIVER_SHA256",
"displayName": "COLDRIVER_SHA256",
"revisionCreateTime": "2024-02-07T19:48:41.354271Z",
"description": "can be updated in a replace operation",
"entries": [
{
"value": "0f6b9d2ada67cebc8c0f03786c442c61c05cef5b92641ec4c1bdd8f5baeb2ee1"
},
...
{
"value": "C97acea1a6ef59d58a498f1e1f0e0648d6979c4325de3ee726038df1fc2e831d"
}
],
"syntaxType": "REFERENCE_LIST_SYNTAX_TYPE_REGEX"
}
Patch Reference List v1alpha
Next, I use patch_list to attempt to overwrite the existing list content and to update the syntax type but the script balks with the message “Patch would not change list. Exiting.”
#
# Patch replace with same content balks.
#
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--list_file=./lists/example_input/coldriver_sha256.txt \
--name="COLDRIVER_SHA256" \
--syntax_type="REFERENCE_LIST_SYNTAX_TYPE_PLAIN_TEXT_STRING"
Patch would not change list. Exiting.
This showcases a client-side logic feature in this script: before attempting the update, the script does a GET and compares the current list to the update we are attempting. If there is no change in the list, there is no need to update. The script is smart that way but not smart enough to know that we just want to update the syntax_type. Fortunately, we can override this behavior with the --force flag.
#
# Patch with force replaces, prints success, returns content
# description and syntaxType are updated only if provided
#
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--name="COLDRIVER_SHA256" \
--list_file=./lists/example_input/coldriver_sha256.txt \
--syntax_type="REFERENCE_LIST_SYNTAX_TYPE_PLAIN_TEXT_STRING" \
--force
Patch success.
{
"name": "projects/proj0001-403014/locations/us/instances/b76bb0ab-54f3-417c-b1c2-7e8dc889cd25/referenceLists/COLDRIVER_SHA256",
"displayName": "COLDRIVER_SHA256",
"": "2024-02-07T20:22:44.959215Z",
"description": "can be updated in a replace operation",
"entries": [
{
"value": "0f6b9d2ada67cebc8c0f03786c442c61c05cef5b92641ec4c1bdd8f5baeb2ee1"
},
...
{
"value": "C97acea1a6ef59d58a498f1e1f0e0648d6979c4325de3ee726038df1fc2e831d"
}
],
"syntaxType": "REFERENCE_LIST_SYNTAX_TYPE_PLAIN_TEXT_STRING"
}
We’ve now seen the create and replace methods in action, which were also both supported by the old API. The next feature was not: we are going to add to an existing Reference List!
Patch Reference List v1alpha –-add
The v1alpha Reference List API doesn't inherently support adding to (or removing items from) an existing Reference List. It only supports replacement. This script introduces that functionality with client-side logic. Here's how it works; you provide a list of additions, and the script:
- Retrieves the current list.
- Appends the new items to the retrieved list.
- Replaces the existing list with the updated list.
From your perspective, it appears as if you've added items to the list. However, behind the scenes, the script still performs a replacement operation.
In the following example, the other example input file from GitHub (foo.txt) is used to add 3 new entries to the existing list. There are duplicate entries in the input file but they are de-duplicated by the script (with order preservation!) before the update.
Note that I’m also using the --quiet flag, so that the only output is the updated JSON with the updated revisionCreateTime. I can then pipe the results into jq (a command-line JSON processor). My jq command filters and then prints only the contents of the updated list, which makes it easier to see the three new additions at the end.
#
# Add
# The content is deduplicated before submission. Order is preserved.
# Adding --quiet silences print statements and returns only the JSON.
#
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--list_file=./lists/example_input/foo.txt \
--name="COLDRIVER_SHA256" \
--credentials_file=$HOME/.chronicle_credentials.json \
--add --quiet | jq '.entries | map(.value)'
[
"0f6b9d2ada67cebc8c0f03786c442c61c05cef5b92641ec4c1bdd8f5baeb2ee1",
...
"C97acea1a6ef59d58a498f1e1f0e0648d6979c4325de3ee726038df1fc2e831d",
"foo",
"bar",
"baz"
]
As we saw earlier, if I attempt that same exact add operation a second time, the script balks because there would be no change to the Reference List.
#
# Add the foo.txt content a second time: balks
#
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--list_file=./lists/example_input/foo.txt \
--name="COLDRIVER_SHA256" \
--add
Patch add would not change list. Exiting.
If you override that behavior with the force flag, it won’t change the list contents, but it will update the revisionCreateTime and you have the opportunity to update the description and syntaxType as we saw earlier.
#
# Add the foo.txt content a second time with --force
# content is deduplicated before submission but not deduplicated with existing
#
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--list_file=./lists/example_input/foo.txt \
--name="COLDRIVER_SHA256" \
--add --force --quiet | jq '.entries | map(.value)'
[
"0f6b9d2ada67cebc8c0f03786c442c61c05cef5b92641ec4c1bdd8f5baeb2ee1",
...
"C97acea1a6ef59d58a498f1e1f0e0648d6979c4325de3ee726038df1fc2e831d",
"foo",
"bar",
"baz"
]
The --remove option removes *all* instances of the items using the list it is provided. After running the example below, there are no more foo/bar/baz entries.
#
# Remove
#
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--list_file=./lists/example_input/foo.txt \
--name="COLDRIVER_SHA256_6" \
--remove --quiet | jq '.entries | map(.value)'
[
"0f6b9d2ada67cebc8c0f03786c442c61c05cef5b92641ec4c1bdd8f5baeb2ee1",
...
"C97acea1a6ef59d58a498f1e1f0e0648d6979c4325de3ee726038df1fc2e831d",
]
Try, try again
There is also a retry mechanism that verifies the changes have been made. In the example below, I’ve simulated failures by randomly sending either a subset or the complete list for update and forcing failure in the revisionCreateTime check. I made this little change to the script.
diff --git a/lists/v1alpha/patch_list.py b/lists/v1alpha/patch_list.py
--- a/lists/v1alpha/patch_list.py
+++ b/lists/v1alpha/patch_list.py
@@ -251 +251,2 @@ def main():
- content_lines,
+ content_lines[0:random.randint(0,len(content_lines))],
@@ -257 +258,2 @@ def main():
- success = ts == patched_json["revisionCreateTime"]
+ success = False
In the output below, you can see that the script checks the results after each update and finds in the first 4 times that the additions we are trying to make are not present, so it adds a little delay and some jitter and then tries again. On the 4th attempt, it finds that all of the additions (foo, bar, baz) are present and it exits.
(chronicle_cli) dandye-macbookpro:chronicle-api-samples-python dandye$ python -m lists.v1alpha.patch_list --project_id=$PROJECT_ID --project_instance=$PROJECT_INSTANCE --name="COLDRIVER_SHA256" --add --list_file ./lists/example_input/foo.txt
Attempt 1 of 6 failed, retrying in 1.85 seconds...
Attempt 2 of 6 failed, retrying in 2.91 seconds...
Attempt 3 of 6 failed, retrying in 4.34 seconds...
Attempt 4 of 6 failed, retrying in 6.18 seconds...
Patch add success.
{
"name": "projects/proj0001-403014/locations/us/instances/b76bb0ab-54f3-417c-b1c2-7e8dc889cd25/referenceLists/COLDRIVER_SHA256",
"displayName": "COLDRIVER_SHA256",
"revisionCreateTime": "2024-02-08T20:00:58.094719Z",
"description": "can be updated in a replace operation",
"entries": [
{
"value": "0f6b9d2ada67cebc8c0f03786c442c61c05cef5b92641ec4c1bdd8f5baeb2ee1"
},
{
"value": "foo"
},
{
"value": "bar"
},
{
"value": "baz"
}
],
"syntaxType": "REFERENCE_LIST_SYNTAX_TYPE_PLAIN_TEXT_STRING"
}
This post-verification logic helps to mitigate conflicts between concurrent updaters. If one updater retrieves the list before my PATCH and applies its changes after my PATCH operation, my changes are overwritten. Post-verification detects the conflict with the revisionCreateTime and then initiates new GET/PATCH cycles to ensure the content is changed. With exponential backoff and jitter (to avoid a “live lock”), both updaters should eventually succeed in applying their respective updates.
Should you need to disable this post-verification behavior, simply set --max_attempts=1.
Recap
In this blog post, I introduced the new REST API for Chronicle and a collection of new Python scripts that can interact with the API. I demonstrated how to create and get Chronicle Reference Lists with the new Python scripts and by using Python client-side logic in the patch list script, how to add and remove entries to/from existing Reference Lists.
Now that we’ve covered the features driven by the add, remove, quiet, and force flags in the patch list script, the full usage for the new script to patch Reference Lists should be less daunting:
python -m lists.v1alpha.patch_list --help
usage: patch_list.py [-h] [-c CREDENTIALS_FILE] -i PROJECT_INSTANCE -p PROJECT_ID
[-r {asia-northeast1,asia-south1,asia-southeast1,australia-southeast1,europe,europe-west2,europe-west3,europe-west6,me-central2,me-west1,us}] -n NAME -f LIST_FILE [-d DESCRIPTION]
[-t {REFERENCE_LIST_SYNTAX_TYPE_UNSPECIFIED,REFERENCE_LIST_SYNTAX_TYPE_PLAIN_TEXT_STRING,REFERENCE_LIST_SYNTAX_TYPE_REGEX,REFERENCE_LIST_SYNTAX_TYPE_CIDR}] [--add | --remove] [--force]
[--max_attempts MAX_ATTEMPTS] [--quiet]
Executable and reusable sample for patching a Reference List.
Command supports add, remove, and replace via [--add, --remove, <no-flag>].
Sample Commands (run from api_samples_python dir):
# Add
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--name="COLDRIVER_SHA256" \
--list_file=./lists/example_input/foo.txt \
--add
# Remove
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--name="COLDRIVER_SHA256"
--list_file=./lists/example_input/foo.txt \
--remove
# Replace (when no --add or --remove flags are provided)
python -m lists.v1alpha.patch_list \
--project_id=$PROJECT_ID \
--project_instance=$PROJECT_INSTANCE \
--name="COLDRIVER_SHA256"
--list_file=./lists/example_input/coldriver_sha256.txt
API reference:
https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/projects.locations.instances.referenceLists/patch
https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/projects.locations.instances.referenceLists#ReferenceList
https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/projects.locations.instances.referenceLists#resource:-referencelist
options:
-h, --help show this help message and exit
-c CREDENTIALS_FILE, --credentials_file CREDENTIALS_FILE
credentials file path (default: '/Users/dandye/.chronicle_credentials.json')
-i PROJECT_INSTANCE, --project_instance PROJECT_INSTANCE
Customer ID for Chronicle instance
-p PROJECT_ID, --project_id PROJECT_ID
Your BYOP, project id
-r {asia-northeast1,asia-south1,asia-southeast1,australia-southeast1,europe,europe-west2,europe-west3,europe-west6,me-central2,me-west1,us}, --region {asia-northeast1,asia-south1,asia-southeast1,australia-southeast1,europe,europe-west2,europe-west3,europe-west6,me-central2,me-west1,us}
the region where the customer is located (default: us)
-n NAME, --name NAME unique name for the list
-f LIST_FILE, --list_file LIST_FILE
path of a file containing the list content
-d DESCRIPTION, --description DESCRIPTION
description of the list
-t {REFERENCE_LIST_SYNTAX_TYPE_UNSPECIFIED,REFERENCE_LIST_SYNTAX_TYPE_PLAIN_TEXT_STRING,REFERENCE_LIST_SYNTAX_TYPE_REGEX,REFERENCE_LIST_SYNTAX_TYPE_CIDR}, --syntax_type {REFERENCE_LIST_SYNTAX_TYPE_UNSPECIFIED,REFERENCE_LIST_SYNTAX_TYPE_PLAIN_TEXT_STRING,REFERENCE_LIST_SYNTAX_TYPE_REGEX,REFERENCE_LIST_SYNTAX_TYPE_CIDR}
syntax type of the list, used for validation
--add only append to the existing list
--remove only remove from the existing list
--force patch regardless of pre-check on changes to list
--max_attempts MAX_ATTEMPTS
how many times to attempt the patch operation
--quiet only print the updated list
Please let me know if you find these scripts useful, have any questions, and/or feature requests!