Traffic callout extension erroring out on gRPC backend on the loadbalancer

[varkey98]

I have a load balancer which has a grpc backend. When configuring traffic callout extension on that lb, the response data sent to my envoy (ext_proc_v3) server doesnt have `end_of_stream` set in the response data. This is what my server got for a single request:

2024/01/24 11:39:07 Started process
2024/01/24 11:39:07 Received stream
2024/01/24 11:39:07 request_headers:{headers:{headers:{key:":scheme"  raw_value:"https"}  headers:{key:":path"  raw_value:"/helloworld.Greeter/SayHello"}  headers:{key:":method"  raw_value:"POST"}  headers:{key:":authority"  raw_value:"35.212.139.253"}  headers:{key:"grpc-accept-encoding"  raw_value:"identity,deflate,gzip"}  headers:{key:"accept-encoding"  raw_value:"identity"}  headers:{key:"user-agent"  raw_value:"grpc-node-js/1.8.10"}  headers:{key:"content-type"  raw_value:"application/grpc"}  headers:{key:"x-forwarded-proto"  raw_value:"https"}  headers:{key:"via"  raw_value:"1.1 google"}  headers:{key:"x-forwarded-for"  raw_value:"49.207.211.218,35.212.139.253"}}}
2024/01/24 11:39:07 Received stream
2024/01/24 11:39:07 request_body:{body:"\x00\x00\x00\x00\x08\n\x06varkey"  end_of_stream:true}
2024/01/24 11:39:07 Received stream
2024/01/24 11:39:07 response_headers:{headers:{headers:{key:":status"  raw_value:"200"}  headers:{key:"content-type"  raw_value:"application/grpc"}}}
2024/01/24 11:39:07 Received stream
2024/01/24 11:39:07 response_body:{body:"\x00\x00\x00\x00\x0e\n\x0chello varkey"}

Ideally there should be `end_of_stream` set in response_body so that we can infer the request data has been completely sent over. This works fine for HTTP/1 backends, issue is with gRPC backend application

[Marvin_Lucero]

Hi @varkey98 ,

Based on your logs, it seems the flag is properly set for the request, but not for the response. This could be a problem where the traffic callout extension or the gRPC backend might not be configured correctly or might not work well together.

I would suggest checking the version of the traffic callout extension compatible with the gRPC backend, as well as the version of the Envoy Proxy.

You can also try enabling debugging or verbose logging on the traffic callout extension and Envoy Proxy for more clear information regarding how traffic is being processed. 

Let me know if this helps.