Dialogflow CX - Multiple Files Returned Despite filterSpecs.filter

I'm facing an issue with filter control in Dialogflow CX. I'm trying to filter the response data retrieved from Datastore using search configuration, but my response consistently includes multiple files, not just the one specified by the filter.

I've confirmed that I don't have any fulfillment code involved in processing the response.

Expected Behavior:

The response should only include the file matching the specified filter criteria.

Current Behavior:

The response contains multiple files, seemingly ignoring the filter.

Additional Information:

  • Datastore Entity Structure: unstructured files with metadata
  • Request details: 

 

 

 

 

Post https://dialogflow.googleapis.com/v3/projects/X/locations/global/agents/Y/sessions/Z:detectIntent
{
"queryInput": {
"text": {
"text": "What's The impact of GenAI?"
},
"languageCode": "en"
},
"queryParams": {
"searchConfig": {
"filterSpecs": [{
"dataStores": ["conversation-datastore_1708980072466"],
"filter": "fileId: = \"1--msAKP3HOLXFRzBG1gLuV4alMUHcUTY\""
}]
}
}
}

 

 

 

 

Has anyone else encountered this issue? Any suggestions on how to troubleshoot or resolve it would be greatly appreciated.

0 11 253
11 REPLIES 11

I think the filter syntax is wrong and Dialogflow CX is not using it. Please take a look at the syntax documentation: https://cloud.google.com/generative-ai-app-builder/docs/filter-search-metadata#filter-expression-syn...

Here you can have an example in NodeJS as well: https://cloud.google.com/dialogflow/cx/docs/how/filter

Best,

Xavi

Hello @xavidop 

Apologies, I mistakenly pasted the wrong payload. I've tested it with the corrected syntax:

 

"filter": "fileId: ANY(\"1--msAKP3HOLXFRzBG1gLuV4alMUHcUTY\")"

 

However, I'm still encountering the same result.
Thank you.

Hi, 

did you check that this field exists? 

fileId

 

Yes, it does

adilpresence_1-1709564468844.png

 

okay so probably the character: -- it is used for another thing, can you try to use another field like fileName?

keep me in the loop!

Same result.

Here is my updated query : 

"filter": "fileName: ANY(\"Forecast_Analysis_Application services ww_798019_ndx.pdf.pdf\")"

 

And here is the API response (As you can see, multiple docs are returned)

{
    "responseId": "a2bd5695-2549-4951-be8e-46f82c943f99",
    "queryResult": {
        "text": "What's The impact of GenAI?",
        "languageCode": "en",
        "responseMessages": [
            {
                "text": {
                    "text": [
                        "Generative AI (GenAI) will boost labor productivity, potentially amplifying global GDP while exacerbating economic inequality."
                    ]
                }
            },
            {
                "payload": {
                    "richContent": [
                        [
                            {
                                "title": "11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "type": "info",
                                "subtitle": "Overview Impacts Recommendations Technology innovation and IT leaders looking to prepare for the digital future must:…",
                                "actionLink": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=1"
                            }
                        ]
                    ]
                }
            }
        ],
        "currentPage": {
            "name": "projects/presencedocs-ia/locations/global/agents/c2dfb98c-e9ea-41c5-9dc1-e5234d7d92e8/flows/00000000-0000-0000-0000-000000000000/pages/START_PAGE",
            "displayName": "Start Page"
        },
        "intentDetectionConfidence": 1,
        "diagnosticInfo": {
            "Execution Sequence": [
                {
                    "Step 1": {
                        "Type": "INITIAL_STATE",
                        "InitialState": {
                            "Event": "",
                            "FlowState": {
                                "Name": "Default Start Flow",
                                "FlowId": "00000000-0000-0000-0000-000000000000",
                                "Version": 0,
                                "PageState": {
                                    "Name": "Start Page",
                                    "Status": "ENTERING_PAGE",
                                    "PageId": "START_PAGE"
                                }
                            }
                        }
                    }
                },
                {
                    "Step 2": {
                        "Type": "STATE_MACHINE",
                        "StateMachine": {
                            "FlowState": {
                                "PageState": {
                                    "PageId": "START_PAGE",
                                    "Status": "TRANSITION_ROUTING",
                                    "Name": "Start Page"
                                },
                                "Version": 0,
                                "Name": "Default Start Flow",
                                "FlowId": "00000000-0000-0000-0000-000000000000"
                            },
                            "TriggeredTransitionRouteId": "kc-driven-transition"
                        },
                        "FunctionExecution": {
                            "Responses": [
                                {
                                    "source": "VIRTUAL_AGENT",
                                    "text": {
                                        "containAttemptedDataStoreInteractions": true,
                                        "containAiGeneratedContent": true,
                                        "containDataStoreContent": true,
                                        "text": [
                                            "Generative AI (GenAI) will boost labor productivity, potentially amplifying global GDP while exacerbating economic inequality."
                                        ]
                                    },
                                    "responseType": "HANDLER_PROMPT"
                                },
                                {
                                    "responseType": "HANDLER_PROMPT",
                                    "source": "VIRTUAL_AGENT",
                                    "payload": {
                                        "richContent": [
                                            [
                                                {
                                                    "subtitle": "Overview Impacts Recommendations Technology innovation and IT leaders looking to prepare for the digital future must:…",
                                                    "actionLink": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=1",
                                                    "title": "11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                                    "type": "info"
                                                }
                                            ]
                                        ]
                                    }
                                }
                            ]
                        }
                    }
                },
                {
                    "Step 3": {
                        "Type": "STATE_MACHINE",
                        "StateMachine": {
                            "FlowState": {
                                "Version": 0,
                                "FlowId": "00000000-0000-0000-0000-000000000000",
                                "Name": "Default Start Flow",
                                "PageState": {
                                    "Name": "Start Page",
                                    "PageId": "START_PAGE",
                                    "Status": "TRANSITION_ROUTING"
                                }
                            }
                        }
                    }
                }
            ],
            "Response Id": "a2bd5695-2549-4951-be8e-46f82c943f99",
            "Session Id": "test-session-1259",
            "Triggered Transition Names": [
                "kc-driven-transition"
            ],
            "DataStore Execution Sequence": {
                "executionResult": {
                    "ucs_fallback": false,
                    "language": "en",
                    "response_type": "GENERATIVE",
                    "response_reason": "GENERATIVE",
                    "unstructured_citation": true,
                    "latency": 0,
                    "faq_citation": false,
                    "banned_phrase": "",
                    "banned_phrase_check_type": "BANNED_PHRASE_CHECK_TYPE_UNSPECIFIED",
                    "website_citation": false
                },
                "steps": [
                    {
                        "name": "Call Search with original query [unstructured documents]",
                        "info": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465",
                        "status": {
                            "code": "OK"
                        },
                        "responses": [
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "debugId": "4882763e-6759-41ac-9d56-a2ca34d89612",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments",
                                "debugId": "5b3e5817-2751-4c98-9bc5-4dcdcd481084"
                            },
                            {
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments",
                                "debugId": "d3a904e2-d6e9-4b35-a01f-7d3690da90ac",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/188zAgwmL50Z69B5ShvXxx3jTxwLHQNbyoPWbsJpXPcI.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/5c283938d6f7789979dd34b6112ef232"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1cSrM1LaIrxcqAVm3eG_ALUmRrjhsuw7d.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1cSrM1LaIrxcqAVm3eG_ALUmRrjhsuw7d",
                                "debugId": "f1dc415e-1767-4ad5-bf6e-74dbb59fd129",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments"
                            },
                            {
                                "debugId": "be70a510-262c-4107-9fc2-6b95b9b3b006",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1TFBx0W2t2SC-Sn0rurJXXn5iM8c3r0JE.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1TFBx0W2t2SC-Sn0rurJXXn5iM8c3r0JE",
                                "text": "Ucs content search result with 3 answers, 8 passages, 0 segments"
                            },
                            {
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/a325cbef29e86dcfbe5775527b063228",
                                "debugId": "6f5b82be-ecf8-469d-83d6-5dd397273aa5",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1fNrneiXF-BZX9TpnBzT1GizmvfOh-cvkb6Iif8bMv28.pdf"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/7d81df7ddf1a80a4ef5eea1adcd7f9b8",
                                "debugId": "5f608b29-c924-4236-b87e-d7155c711a87",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1rvds4ql6NjhiMf0Wuut98pRzzhALi4s1b9CJrlk9FVQ.pdf",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1J1wxXYiIzeNYgBo6cJqfBWT_bIj97_Hm.pdf",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1J1wxXYiIzeNYgBo6cJqfBWT_bIj97_Hm",
                                "debugId": "cc8ab4af-a54f-49f7-b212-41f00eac5ae2"
                            },
                            {
                                "debugId": "b79bfead-ac57-487c-ac97-e6891139b1ac",
                                "text": "Ucs content search result with 3 answers, 7 passages, 0 segments",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/10wJa0rvMbtqGcHXgeLZaRWfd445wWatl.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/10wJa0rvMbtqGcHXgeLZaRWfd445wWatl"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1yWWHfi-6XC4ssR8NShCRtqwO88Ivrkni.pdf",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments",
                                "debugId": "606e4056-b507-4964-9e52-6ec8b6cf5352",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1yWWHfi-6XC4ssR8NShCRtqwO88Ivrkni"
                            },
                            {
                                "text": "Ucs content search result with 3 answers, 9 passages, 0 segments",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1dST8Scmhe4-xKzWtexNWha5kLwB6dlFs.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1dST8Scmhe4-xKzWtexNWha5kLwB6dlFs",
                                "debugId": "05bfb0d4-c7e5-4a69-b98f-56c85b2ef3c4"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1--msAKP3HOLXFRzBG1gLuV4alMUHcUTY.pdf",
                                "debugId": "ddb54abc-9d10-401a-8879-b52ae65f8d73",
                                "text": "Ucs content search result with 3 answers, 4 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1--msAKP3HOLXFRzBG1gLuV4alMUHcUTY"
                            },
                            {
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1ckl6iIqkyv4f_SMfQgUQxF6groHHewWq",
                                "debugId": "b9506c14-99c7-4dae-a765-d71fd468ba15",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1ckl6iIqkyv4f_SMfQgUQxF6groHHewWq.pdf"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1xCEKOrRdVXr1B2OQuxgOdRVzwsiZ6PfE.pdf",
                                "debugId": "de8aa529-fc31-4f28-8295-07a7c2e55262",
                                "text": "Ucs content search result with 1 answers, 1 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1xCEKOrRdVXr1B2OQuxgOdRVzwsiZ6PfE"
                            },
                            {
                                "text": "Ucs content search result with 1 answers, 2 passages, 0 segments",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1pUcAfRCrtc3v9KfxbPt0x_9MAmGXc8Vx.pdf",
                                "debugId": "48b2b8f5-7824-402b-b5ba-517a7edf6bbd",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/030a53cd2498c5bee68a2aaae6c67cfb"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/14eqVwYS8cz0jRKj5ibnpNgi4l8iaW60k.pdf",
                                "text": "Ucs content search result with 1 answers, 2 passages, 0 segments",
                                "debugId": "14705ecb-fd3b-4226-b6f0-99017e10988f",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/14eqVwYS8cz0jRKj5ibnpNgi4l8iaW60k"
                            },
                            {
                                "text": "Ucs content search result with 1 answers, 2 passages, 0 segments",
                                "debugId": "24ba6418-680c-4957-b4d6-c67a864b6c28",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1BSq7WkxladcChjAdhbZnCi6MW5x76w95",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1BSq7WkxladcChjAdhbZnCi6MW5x76w95.pdf"
                            },
                            {
                                "text": "Ucs content search result with 2 answers, 5 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/18SW-MjoZbl13EGq-O9eYRmKiDuT-NYL_",
                                "debugId": "749118bf-90c0-493a-b415-855a6a91695b",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/18SW-MjoZbl13EGq-O9eYRmKiDuT-NYL_.pdf"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1ELXskoRTZKyVx_MCRsSCEucvPYa7Sf_a",
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1ELXskoRTZKyVx_MCRsSCEucvPYa7Sf_a.pdf",
                                "debugId": "713c4d01-3021-4afe-b6d8-ef944f910bc1"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1DPFSp1UY7C0xcdz8vl9DCEfL1vDW5Utw.pdf",
                                "debugId": "a11ad1f9-e3c9-44b5-83ad-19075fb4419c",
                                "text": "Ucs content search result with 1 answers, 2 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1DPFSp1UY7C0xcdz8vl9DCEfL1vDW5Utw"
                            }
                        ]
                    },
                    {
                        "responses": [
                            {
                                "text": "Overview Impacts Recommendations Technology innovation and IT leaders looking to prepare for the digital future must: Generative AI (GenAI) will boost labor productivity, potentially amplifying global GDP while exacerbating economic inequality.",
                                "debugId": "375c8b9e-0442-4edc-b196-b3031afbf7c8",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=1",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "debugId": "fb0a45de-9f1f-40cd-8d31-53b4e88b397d",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=16",
                                "text": "Immediate impacts will drive productivity increases, but the impact might lead to worker displacement. GDP Acceleration In the short term, GenAI will drive immediate productivity gains."
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=8",
                                "text": "A GenAI-driven shift in business models will potentially have microeconomic implications on firms and their pricing strategies, along with how they interact with ecosystem partners or their customers.",
                                "debugId": "ffa5723f-9bbc-4c3e-9c2b-329e2607cdd5",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf#page=28",
                                "text": "Employees should be trained on the appropriate use of the GenAI system and the relevant policies and regulations governing its use. Violations of GenAI usage policies may result in disciplinary action, up to and including termination of employment.",
                                "debugId": "49a6ed7c-7508-4eb3-84ca-75fc5ffe186b",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk"
                            },
                            {
                                "debugId": "60eb5451-24e8-4179-bbce-b3d71b4a5a0a",
                                "text": "For example, organizations could decide that while data is being sent to a third-party GenAI SaaS platform, opting out of the user prompt information being used to train future models, and accepting the data retention policy of 30 days (as is in the case of OpenAI, for both), is secure enough for their needs and meets their risk appetite.",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf#page=11"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=1",
                                "debugId": "07d952c8-2d05-47d4-92f8-2e5cbee0c807",
                                "text": "Overview Impacts Recommendations Technology innovation and IT leaders looking to prepare for the digital future must: Generative AI (GenAI) will boost labor productivity, potentially amplifying global GDP while exacerbating economic inequality.",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=16",
                                "debugId": "3366df1e-6110-4ab3-a1ba-28f52ace3cad",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "text": "Immediate impacts will drive productivity increases, but the impact might lead to worker displacement. GDP Acceleration In the short term, GenAI will drive immediate productivity gains."
                            },
                            {
                                "text": "A GenAI-driven shift in business models will potentially have microeconomic implications on firms and their pricing strategies, along with how they interact with ecosystem partners or their customers.",
                                "debugId": "b806924d-3898-47c0-99c9-02937a792b50",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=8",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8"
                            },
                            {
                                "debugId": "388f6b27-39fc-463a-90e2-eff46c2e1a65",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf#page=28",
                                "text": "Employees should be trained on the appropriate use of the GenAI system and the relevant policies and regulations governing its use. Violations of GenAI usage policies may result in disciplinary action, up to and including termination of employment."
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "text": "For example, organizations could decide that while data is being sent to a third-party GenAI SaaS platform, opting out of the user prompt information being used to train future models, and accepting the data retention policy of 30 days (as is in the case of OpenAI, for both), is secure enough for their needs and meets their risk appetite.",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf#page=11",
                                "debugId": "6aad29d5-9233-4423-a03c-d2f59fe45bd3"
                            }
                        ],
                        "status": {
                            "code": "OK"
                        },
                        "name": "Convert UCS results for original query"
                    },
                    {
                        "name": "Query rewrite",
                        "responses": [
                            {
                                "text": "I should search for information about the impact of GenAI.\nAction: Search\nAction Input: What's the impact of GenAI?"
                            }
                        ],
                        "status": {
                            "code": "OK"
                        }
                    },
                    {
                        "status": {
                            "code": "OK"
                        },
                        "name": "Responsible AI (with original query)"
                    },
                    {
                        "status": {
                            "code": "OK"
                        },
                        "info": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465",
                        "responses": [
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments",
                                "debugId": "2587302e-eae1-44cc-b138-310dee0885e7"
                            },
                            {
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments",
                                "debugId": "1f8714b1-143f-4a55-a4e9-2fac4790234f",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf"
                            },
                            {
                                "debugId": "b87bc313-9c9e-4afc-8f10-bb9456b10fca",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/5c283938d6f7789979dd34b6112ef232",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/188zAgwmL50Z69B5ShvXxx3jTxwLHQNbyoPWbsJpXPcI.pdf",
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments"
                            },
                            {
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments",
                                "debugId": "385f42fb-f0d6-4b7f-a701-2d0cebc98072",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1cSrM1LaIrxcqAVm3eG_ALUmRrjhsuw7d",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1cSrM1LaIrxcqAVm3eG_ALUmRrjhsuw7d.pdf"
                            },
                            {
                                "text": "Ucs content search result with 3 answers, 8 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1TFBx0W2t2SC-Sn0rurJXXn5iM8c3r0JE",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1TFBx0W2t2SC-Sn0rurJXXn5iM8c3r0JE.pdf",
                                "debugId": "00bf67f8-1373-49e1-88d8-38d3352a017d"
                            },
                            {
                                "debugId": "7e57e108-2f42-437f-9245-d578fea561ac",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1fNrneiXF-BZX9TpnBzT1GizmvfOh-cvkb6Iif8bMv28.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/a325cbef29e86dcfbe5775527b063228",
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/7d81df7ddf1a80a4ef5eea1adcd7f9b8",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments",
                                "debugId": "57243573-a6f1-4716-99d9-1570f4a55315",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1rvds4ql6NjhiMf0Wuut98pRzzhALi4s1b9CJrlk9FVQ.pdf"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1J1wxXYiIzeNYgBo6cJqfBWT_bIj97_Hm",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1J1wxXYiIzeNYgBo6cJqfBWT_bIj97_Hm.pdf",
                                "debugId": "ec580c9c-a564-41e1-b288-2a399e4722ed",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments"
                            },
                            {
                                "debugId": "b87dc8f3-6c2d-4d11-9765-55fde7f1fab9",
                                "text": "Ucs content search result with 3 answers, 7 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/10wJa0rvMbtqGcHXgeLZaRWfd445wWatl",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/10wJa0rvMbtqGcHXgeLZaRWfd445wWatl.pdf"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1yWWHfi-6XC4ssR8NShCRtqwO88Ivrkni",
                                "debugId": "b32329e0-f7fe-4891-b9be-1b1b57d48fbc",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1yWWHfi-6XC4ssR8NShCRtqwO88Ivrkni.pdf",
                                "text": "Ucs content search result with 3 answers, 6 passages, 0 segments"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1dST8Scmhe4-xKzWtexNWha5kLwB6dlFs",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1dST8Scmhe4-xKzWtexNWha5kLwB6dlFs.pdf",
                                "text": "Ucs content search result with 3 answers, 9 passages, 0 segments",
                                "debugId": "85b61ac2-7767-493d-bb03-8e3f4622fa92"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1--msAKP3HOLXFRzBG1gLuV4alMUHcUTY",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1--msAKP3HOLXFRzBG1gLuV4alMUHcUTY.pdf",
                                "text": "Ucs content search result with 3 answers, 4 passages, 0 segments",
                                "debugId": "88e089f0-9f25-4919-8c25-2d83c6277a03"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1ckl6iIqkyv4f_SMfQgUQxF6groHHewWq",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1ckl6iIqkyv4f_SMfQgUQxF6groHHewWq.pdf",
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments",
                                "debugId": "49ca5710-8356-43ef-893b-cd6dcc97b043"
                            },
                            {
                                "text": "Ucs content search result with 1 answers, 1 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1xCEKOrRdVXr1B2OQuxgOdRVzwsiZ6PfE",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1xCEKOrRdVXr1B2OQuxgOdRVzwsiZ6PfE.pdf",
                                "debugId": "d736c040-c11c-47ac-ab43-70159bdf8ce9"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1pUcAfRCrtc3v9KfxbPt0x_9MAmGXc8Vx.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/030a53cd2498c5bee68a2aaae6c67cfb",
                                "debugId": "370f2bb9-ddc4-41fa-b832-a410184da8f4",
                                "text": "Ucs content search result with 1 answers, 2 passages, 0 segments"
                            },
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/14eqVwYS8cz0jRKj5ibnpNgi4l8iaW60k",
                                "text": "Ucs content search result with 1 answers, 2 passages, 0 segments",
                                "debugId": "debbe015-d68d-4b9b-a6a7-4fee30bb1a5c",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/14eqVwYS8cz0jRKj5ibnpNgi4l8iaW60k.pdf"
                            },
                            {
                                "text": "Ucs content search result with 1 answers, 2 passages, 0 segments",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1BSq7WkxladcChjAdhbZnCi6MW5x76w95.pdf",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1BSq7WkxladcChjAdhbZnCi6MW5x76w95",
                                "debugId": "2603de70-28f8-4f4c-bf5d-141785aea87e"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/18SW-MjoZbl13EGq-O9eYRmKiDuT-NYL_.pdf",
                                "debugId": "13e7aab0-29ee-4524-86b4-b5f6534a9df0",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/18SW-MjoZbl13EGq-O9eYRmKiDuT-NYL_",
                                "text": "Ucs content search result with 2 answers, 5 passages, 0 segments"
                            },
                            {
                                "debugId": "10ddd662-4503-4895-adb3-a2596dac5302",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1ELXskoRTZKyVx_MCRsSCEucvPYa7Sf_a.pdf",
                                "text": "Ucs content search result with 3 answers, 10 passages, 0 segments",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1ELXskoRTZKyVx_MCRsSCEucvPYa7Sf_a"
                            },
                            {
                                "text": "Ucs content search result with 1 answers, 2 passages, 0 segments",
                                "debugId": "72144252-68b4-4b5e-9a5e-e864a3445080",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1DPFSp1UY7C0xcdz8vl9DCEfL1vDW5Utw",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1DPFSp1UY7C0xcdz8vl9DCEfL1vDW5Utw.pdf"
                            }
                        ],
                        "name": "Call Search with rewritten query [unstructured documents]"
                    },
                    {
                        "name": "Convert UCS results for rewritten query",
                        "responses": [
                            {
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "debugId": "57a2486b-f041-437b-baf1-99ac34951541",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=1",
                                "text": "Overview Impacts Recommendations Technology innovation and IT leaders looking to prepare for the digital future must: Generative AI (GenAI) will boost labor productivity, potentially amplifying global GDP while exacerbating economic inequality."
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=16",
                                "debugId": "f65d73db-552c-4389-b9c6-7e4d969b1b4f",
                                "text": "Immediate impacts will drive productivity increases, but the impact might lead to worker displacement. GDP Acceleration In the short term, GenAI will drive immediate productivity gains.",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=8",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "text": "A GenAI-driven shift in business models will potentially have microeconomic implications on firms and their pricing strategies, along with how they interact with ecosystem partners or their customers.",
                                "debugId": "97b0eb95-b9d3-4818-8fe1-677cf21b5a1f"
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf#page=28",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "debugId": "372d1c99-3298-41c1-b96c-0ae7b972344f",
                                "text": "Employees should be trained on the appropriate use of the GenAI system and the relevant policies and regulations governing its use. Violations of GenAI usage policies may result in disciplinary action, up to and including termination of employment."
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf#page=11",
                                "text": "For example, organizations could decide that while data is being sent to a third-party GenAI SaaS platform, opting out of the user prompt information being used to train future models, and accepting the data retention policy of 30 days (as is in the case of OpenAI, for both), is secure enough for their needs and meets their risk appetite.",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "debugId": "1a3dab88-eb9e-4086-ac5f-71803c9a1bf3"
                            },
                            {
                                "debugId": "a1d0fd6f-cf5f-4af6-bcb4-508f0a21a7e6",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=1",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "text": "Overview Impacts Recommendations Technology innovation and IT leaders looking to prepare for the digital future must: Generative AI (GenAI) will boost labor productivity, potentially amplifying global GDP while exacerbating economic inequality."
                            },
                            {
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=16",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "text": "Immediate impacts will drive productivity increases, but the impact might lead to worker displacement. GDP Acceleration In the short term, GenAI will drive immediate productivity gains.",
                                "debugId": "90a5343f-12e8-4ba6-afde-870abf6f8421"
                            },
                            {
                                "text": "A GenAI-driven shift in business models will potentially have microeconomic implications on firms and their pricing strategies, along with how they interact with ecosystem partners or their customers.",
                                "debugId": "124a5462-1fa2-4269-930a-038d7ddbe84a",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/11s_vYNdi9tYG6CYCZG1UWxax5ueEFEG8.pdf#page=8"
                            },
                            {
                                "text": "Employees should be trained on the appropriate use of the GenAI system and the relevant policies and regulations governing its use. Violations of GenAI usage policies may result in disciplinary action, up to and including termination of employment.",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf#page=28",
                                "debugId": "2a25491e-ec70-48b7-9e15-e97061b6d92e"
                            },
                            {
                                "text": "For example, organizations could decide that while data is being sent to a third-party GenAI SaaS platform, opting out of the user prompt information being used to train future models, and accepting the data retention policy of 30 days (as is in the case of OpenAI, for both), is secure enough for their needs and meets their risk appetite.",
                                "document": "projects/1019326759656/locations/global/collections/default_collection/dataStores/conversation-datastore_1708980072465/branches/0/documents/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk",
                                "debugId": "e3265417-68b2-4342-91c8-3fec17fe85e6",
                                "url": "https://storage.cloud.google.com/my-ia-bucket/1t9Y_YGMJ-z8L7i2_MYTBZpqEicQefpvk.pdf#page=11"
                            }
                        ],
                        "status": {
                            "code": "OK"
                        }
                    },
                    {
                        "name": "ReAct turn",
                        "responses": [
                            {
                                "text": "The search results are relevant and [a] contains all information. I will follow answering guideline 3 [a].\nSource: [a]\nFinal Answer: Generative AI (GenAI) will boost labor productivity, potentially amplifying global GDP while exacerbating economic inequality."
                            }
                        ],
                        "status": {
                            "code": "OK"
                        }
                    },
                    {
                        "info": "Source: 57a2486b-f041-437b-baf1-99ac34951541",
                        "status": {
                            "code": "OK"
                        },
                        "name": "Parse ReAct Answer"
                    },
                    {
                        "status": {
                            "code": "OK"
                        },
                        "responses": [
                            {
                                "text": "Grounding accepted"
                            }
                        ],
                        "name": "Grounding"
                    },
                    {
                        "status": {
                            "code": "OK"
                        },
                        "name": "Responsible AI (with result before grounding)"
                    },
                    {
                        "name": "Responsible AI (with original query and result)",
                        "status": {
                            "code": "OK"
                        }
                    }
                ],
                "additionalInfo": {
                    "agent_project_number": "1019326759656",
                    "ucs_project_number": "1019326759656",
                    "rewritten_query": "What's the impact of GenAI?",
                    "tracking_id": "623c7e0b-4a2a-429a-8ed8-134439313073",
                    "search_results_used_in_main_prompt": "5/5",
                    "user_query": "What's The impact of GenAI?"
                }
            }
        },
        "match": {
            "confidence": 1
        },
        "advancedSettings": {
            "loggingSettings": {}
        }
    },
    "responseType": "FINAL"
}

 

can you try:

"filter": "fileName: ANY(\"ww_798019_ndx\")"

I get the same response as above..

I am not sure then. What I would suggest is to create a Search app using the same datastore and play with the API until you have the filter working

The filter is working just fine using a Search app...

Then I will recommend to open an issue here: https://issuetracker.google.com/savedsearches/6123401