MISP Ingestion Integration

[samnewton]

Hi Everyone,

I've set up the MISP Ingestion Integration but I'm having no luck getting events from MISP. Every time it downloads, there are 0 events. No errors, just no events.

Has anyone done this successfully? I'm pulling my hair out.

Thank you 

Sam

[cmmartin_google · Accepted solution]

You can use the CSV Extended format with something like PyMISP, e.g.,  for each indicator type you want to bring in run a scheduled job:

/python3 /home/user/misp_api/get_csv.py -f /home/user/misp_logs/ip-dst.log -t "ip-dst" -l 1d -c

And then use a Chronicle Forwarder, NXLog, or other agent to ingest into Chronicle SIEM.  This will work with the default MISP_IOC parser.

I wrote this enhanced version of the Ingestion Script (from the SIEM GitHub repo) which is a bit more involved to setup, but is more flexible and extracts more fields:

https://github.com/goog-cmmartin/thatsiemguy/tree/main/misp

[cmmartin_google · Accepted solution]

You can use the CSV Extended format with something like PyMISP, e.g.,  for each indicator type you want to bring in run a scheduled job:

/python3 /home/user/misp_api/get_csv.py -f /home/user/misp_logs/ip-dst.log -t "ip-dst" -l 1d -c

And then use a Chronicle Forwarder, NXLog, or other agent to ingest into Chronicle SIEM.  This will work with the default MISP_IOC parser.

I wrote this enhanced version of the Ingestion Script (from the SIEM GitHub repo) which is a bit more involved to setup, but is more flexible and extracts more fields:

https://github.com/goog-cmmartin/thatsiemguy/tree/main/misp

[besterminator]

Interesting solution, I just have one doubt: if an IoC is no longer valid, how is it handled?
Do you need to create a rule for example that takes IoCs with the collected timestamp in the last month?

[cmmartin_google]

If you've implemented a decay model then you can add support for that value, e.g., collect IOCs above 60, but only alert above 80.  That ensures you will get IOC updates when the score changes, and can stop alerting when not above a given threshold.

Alternatively, you can use labels, e.g., tag with false positive, and then in your YARA-L rules add exclusion to not alert on a given tag

  events:
    (
        $event.metadata.event_type = "NETWORK_CONNECTION" or
        $event.metadata.event_type = "NETWORK_HTTP" or
        $event.metadata.event_type = "NETWORK_FTP" or
        $event.metadata.event_type = "NETWORK_SMTP" or
        $event.metadata.event_type = "NETWORK_FLOW" or        
        $event.metadata.event_type = "NETWORK_UNCATEGORIZED"
    )
    $event.target.ip != ""
    $event.target.ip = $ip
    $asset = strings.coalesce($event.principal.hostname, $event.principal.ip)

    $misp.graph.metadata.vendor_name = "misp-project.org"
    $misp.graph.metadata.product_name = "MISP Threat Sharing"
    $misp.graph.metadata.entity_type = "IP_ADDRESS"
    $misp.graph.entity.ip = $ip
    // used to not alert on tagged false positives in MISP TIP
    not any $misp.graph.metadata.threat.category_details = "false_positive"

[samnewton · Accepted solution]

Thank you @cmmartin_google that was extremely helpful, it took a bit of hacking but using your github link managed to get it working. Much appreciated 👍🏻