Java to cloud sql connection with setVerifyServerCertificate set to true - error -Signature does not match.

[vbalan]

Hey guys,

We're trying to verify the server cert while connecting to cloud sql via our java callout.

Works perfectly fine in non-prod cloud sql verification.

But throws the error "java.security.SignatureException: Signature does not match." while trying to connect to Prod with setVerifyServerCertificate(true).

If we disable verifyServerCert, then it works fine in prod.

The server-ca.pem was downloaded from google cloud sql console and added to a truststore as

keytool -importcert -alias MySQL_Prod_CACert -file server-ca.pem -keystore apigee-cloudsql-prd-truststore.jks -storepass <SomePassword>-noprompt

Any pointers on how to debug?

Thanks in advance,

Vishnu

[vbalan]

Also tried the keytool command using -trustcacerts switch

[ylesyuk · Accepted solution]

I do not believe your cloud sql driver call would automatically see apigee truststores or keystores.

You would need to pass your ca pem into your java callout via context/callout parameters and use it to populate the ca root certificate.

https://developers.google.com/resources/api-libraries/documentation/sqladmin/v1beta4/java/latest/com...

same for the mTLS connection with Cloud SQL (ie, use setClientCertificate() and setClientKey() methods)

[vbalan]

My apologies if i haven't made that clear in my question. The truststore that i created is passed to the connection object. that's how i mentioned it works for the non-prod env.

These are the settings for Prod and non-prod-

URL trustStore = null;

if("prd".equals(csp.environment.toLowerCase())) {

trustStore = this.getClass().getResource("apigee-cloudsql-prd-truststore.jks"); }

else { trustStore = this.getClass().getResource("apigee-cloudsql-np-truststore.jks");

}

msds.setTrustCertificateKeyStoreType("jks");

msds.setTrustCertificateKeyStoreUrl("file:" + trustStore.getFile()); msds.setTrustCertificateKeyStorePassword(csp.jksStorePassword);

[ylesyuk]

OK. I'm with you now.

The error is pretty straightforward, your CA certificate does not validate your server certificate.

The easiest way to troubleshoot ssl/tls issues is to run your connection code as a standalone java code outside of apigee with flag

-Djavax.net.debug=all

Of course, if you have an OPDK, you can restart your MP with same flag. But for SaaS, I'm afraid that's the *easy* option.

See here for details,

https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

[vbalan]

Tu es un génie!

I ran it and got the attached error.

signature-check-failed-error.txt

Here's the ca cert in my truststore that i had downloaded from prod google cloud sql console.

server-ca.txt

Any pointers on why it'd throw certificate unknown error?

Invalidated: [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] main, SEND TLSv1.1 ALERT: fatal, description = certificate_unknown

[ylesyuk]

Another observation, I would expect you do use command like this:

keytool -import -v -trustcacerts -alias mockserver -file mockserversscert.crt -keystore edgeuitruststoresscert.jks<br>

to populate your truststore and obviously not to use a password.

[ylesyuk]

the trustchain looks structurally all right.

there is one root trusted certificate added to the runtime:

***
adding as trusted cert:
  Subject: C=US, O="Google, Inc", CN=Google Cloud SQL Server CA
  Issuer:  C=US, O="Google, Inc", CN=Google Cloud SQL Server CA
  Algorithm: RSA; Serial number: 0x0
  Valid from Mon Mar 11 15:14:15 PDT 2019 until Thu Mar 08 14:15:15 PST 2029

And this is not the one that is in your .pem file

 Issuer: CN=Google Cloud SQL Server CA, O=Google, Inc, C=US
        Validity
            Not Before: Feb  1 18:18:41 2019 GMT
            Not After : Jan 29 18:19:41 2029 GMT
        Subject: CN=Google Cloud SQL Server CA, O=Google, Inc, C=US

You can fetch your server certificate using command like this:

openssl s_client -showcerts -connect www.example.com:443 </dev/null

Can you validate your server certificate using openssl like this against your chain:

# no intermediate
openssl verify -CAfile ca-certs/rootca-crt.pem server-certs/traininglab-crt.pem 

# with intermediate
openssl verify -CAfile ~/projects/sscerts/root/ca/intermediate/certs/ca-chain.cert.pem \
      mockserversscert.crt

One more question, if it's 1way, then you just need a single simple truststore with a single root certificate. I can see a key populated in your keystore. are you trying to set up a mtls? if not, what's the key cert you've added to the keystore?

[vbalan]

Thank you so much, génie!

I was finally able to resolve it with your help.

I went back and checked the server-ca.pem file that the prod cloud sql gave me. It had 2 certs in there.

One with an expiry of Thu Mar 08 14:15:15 PST 2029

and the 2nd with an expiry of Jan 29 18:19:41 2029 GMT.

When i verified the server cert that was coming in via the debug cmd that you shared -Djavax.net.debug=all

I found that it was issued by the 2nd cert in the pem file with the expiry date of Jan 29 18:19:41 2029 GMT

So i created my truststore jks just with that 2nd server-ca file and viola!

It worked! 🙂

Thank you so much once again for your time, patience and willingness to help and share your knowledge!

Much appreciated.

regards,

Vishnu

[ylesyuk]

Excellent!

You are most welcome!