HTTP Health monitor not respecting TLS settings

[olofhaglund]

We are trying to set up some load balancing between two targets. However, in the test environment, the targets have self-signed certs so I set the value of "ignoreValidationErrors" to true with makes so we can call the target servers without issues but the HTTPMonitor doesn't seem to care about this value since I in the logs can see that it complains about it couldn't validate the certs. I have turned the validation of both in the target server and in HTTPTargetConnection in the HTTP block just to be sure. I feel like Apigee should use the same TLS settings when performing health checks as the proxy does when routing the traffic.

Any ideas on how to move forward from here?

This is the definition of my target server

{
    "host": "target.example.com",
    "isEnabled": true,
    "name": "FMC-CANARY-EXAMPLE",
    "port": 443,
    "sSLInfo": {
        "ciphers": [],
        "clientAuthEnabled": "false",
        "enabled": "true",
        "ignoreValidationErrors": true,
        "protocols": []
    }
}

And the settings of my

    <HTTPTargetConnection>
        <SSLInfo>
            <Enabled>true</Enabled>
            <IgnoreValidationErrors>true</IgnoreValidationErrors>
        </SSLInfo>
        <LoadBalancer>
            <MaxFailures>5</MaxFailures>
            <ServerUnhealthyResponse>
                <ResponseCode>500</ResponseCode>
                <ResponseCode>502</ResponseCode>
                <ResponseCode>503</ResponseCode>
            </ServerUnhealthyResponse>
            <Algorithm>Weighted</Algorithm>
            <Retry>true</Retry>
            <Server name="FMC-CANARY-EXAMPLE">
                <Weighted>5</Weighted>
            </Server>
            <Server name="FMC-CANARY-EXAMPLE-CANARY">
                <Weighted>1</Weighted>
            </Server>
        </LoadBalancer>
        <Path>/</Path>
        <HealthMonitor>
            <IsEnabled>true</IsEnabled>
            <IntervalInSec>5</IntervalInSec>
            <HTTPMonitor>
                <Request>
                    <ConnectTimeoutInSec>10</ConnectTimeoutInSec>
                    <SocketReadTimeoutInSec>30</SocketReadTimeoutInSec>
                    <Port>443</Port>
                    <Verb>GET</Verb>
                    <Path>/health</Path>
                </Request>
                <SuccessResponse>
                    <ResponseCode>200</ResponseCode>
                </SuccessResponse>
            </HTTPMonitor>
        </HealthMonitor>
    </HTTPTargetConnection>

We are running on-prem on version 4.19.06.00

[#2]

it looks like you are using one way SSL only. And your backend may be expecting two way TLS or Mutual TLS. In that case you have to enable clientauth and configure keystore and truststore information for target server.

[olofhaglund]

The backend only uses one way TLS.

[#4]

then try removing the below code from the HTTPTargetConnection.

<SSLInfo><Enabled>true</Enabled><IgnoreValidationErrors>true</IgnoreValidationErrors></SSLInfo>

Also try removing retry.

Then see if the simple curl to your backend from Message processor is working fine.

[olofhaglund]

Same issue

[#6]

What is the curl request from your message processor to backend is giving ?

[olofhaglund]

{
    "fault": {
        "faultstring": "The Service is temporarily unavailable",
        "detail": {
            "errorcode": "messaging.adaptors.http.flow.NoActiveTargets"
        }
    }
}

[#8]

so, the service is not connecting. Then HTTPS will fail.

[olofhaglund]

No, NoActiveTargets is the response Apigee gives when there are no active targets in the load balancer. Since the HTTPS health checks fail Apigee removes the targets from the load balancer and hence the error. Just as described in the Apigee documentation.

https://docs.apigee.com/api-platform/troubleshoot/runtime/503-service-unavailable-health-check-failu...

I know this as the traffic works until 5 uninterrupted health checks have been performed. If I deactivate the health checks the proxy works perfectly fine.

And please continue with comments instead creating new answers until you know the answer.

[claudiuavat1]

Hello,

This describes exactly something we are facing when trying to implement HTTP health monitoring. 

Does anybody know of a workaround when using self signed certificates on the backend exposing the health endpoint?

[dchiesa1]

Does anybody know of a workaround when using self signed certificates on the backend exposing the health endpoint?

No. I don't think there is a workaround, beyond "Use certs signed by a 'trusted CA'". As of today, there is no way to set a TrustStore for the health monitor requests. The TrustStore used is the builtin Java cacerts file, which has the trusted Mozilla CAs within it. It's not possible for you to modify that cacerts file.

There is an outstanding bug to allow this. The internal reference is b/213644774 .