Certificate chain troubleshooting for a MITM proxy
This article describes how to check the certificates that are being presented from a Man-in-the-Middle (MITM) proxy that sits between an endpoint and a server, and how to resolve any resulting issues.
The examples shown here are based on Cisco endpoints and Enhanced Room Management products
Specifically, this guide uses the ERM Registry (update) and Activation (license) portals as examples.
Both services use a common certificate (issued by the Let’s Encrypt CA) with the FQDNs added to the SAN. If there is no intermediate proxy, to show the certificates presented to the client from the server, run:
openssl s_client -connect erm-registry.pexip.io:443 -showcerts
openssl s_client -connect erm-activation.pexip.io:443 -showcerts
If there is an intermediate proxy then you need to add a -proxy switch, in the style:
openssl s_client -proxy ip:port -connect erm-activation.pexip.io:443 -showcerts
where ip:port is the IP address or FQDN of the proxy and the corresponding port number.
Good example output that you might see when running this from the CLI of an ERM VM or an ERM Proxy
There are three certificates presented from erm-registry.pexip.io, which include:
- the host TLS (leaf) certificate
- an Intermediate CA certificate (R3)
- a cross-signed ISRG X1 root CA certificate
The cross-signed root certificate is presented as part of the chain to aid backwards compatibility with older devices (such as old Android devices). Technically, this is not required by the ERM server or an endpoint and they will generally terminate the chain after reading the R3 intermediate, matching the issuer to the ISRG X1 root CA that would be stored in the client device's own root CA store.
admin@erm:~$ openssl s_client -connect erm-registry.pexip.io:443 -showcerts CONNECTED(00000003) depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = erm-registry.pexip.io verify return:1 --- Certificate chain 0 s:CN = erm-registry.pexip.io i:C = US, O = Let's Encrypt, CN = R3 -----BEGIN CERTIFICATE----- MIIFSDCCBDCgAwIBAgISBLitDsEkoUqzZ8+ZX1+PtZOhMA0GCSqGSIb3DQEBCwUA MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD etc... -----END CERTIFICATE----- 1 s:C = US, O = Let's Encrypt, CN = R3 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 -----BEGIN CERTIFICATE----- MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh etc... -----END CERTIFICATE----- 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1 i:O = Digital Signature Trust Co., CN = DST Root CA X3 -----BEGIN CERTIFICATE----- MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT etc... -----END CERTIFICATE----- --- Server certificate subject=CN = erm-registry.pexip.io issuer=C = US, O = Let's Encrypt, CN = R3 --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 4542 bytes and written 393 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) ---
Bad example output that you might see when running this from the CLI of an ERM VM or an ERM Proxy
Should a MITM proxy present an erroneous certificate, there are some things that may resolve the issue:
- Fix the MITM proxy (Zscaler) to simply pass the HTTP connection without terminating.
- Add the root CA of the Zscaler to the ERM server.
- Create a proper HTTP proxy and define that within ERM via the CLI.
In this case the three certificates are issued from Zscaler, which is acting as a MITM proxy. As the Zscaler root CA certificate was not installed into the ERM trusted root store, the chain could not be terminated and thus remain untrusted.
admin@PexipERMLBTV:~$ openssl s_client -connect erm-registry.pexip.io:443 -showcerts CONNECTED(00000003) depth=2 C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = Zscaler Intermediate Root CA (zscalerthree.net), emailAddress = support@zscaler.com verify error:num=20:unable to get local issuer certificate verify return:1 depth=1 C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = "Zscaler Intermediate Root CA (zscalerthree.net) (t) " verify return:1 depth=0 CN = erm-registry.pexip.io verify return:1 --- Certificate chain 0 s:CN = erm-registry.pexip.io i:C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = "Zscaler Intermediate Root CA (zscalerthree.net) (t) " -----BEGIN CERTIFICATE----- MIIFCTCCA/GgAwIBAgISb9+/81O6EeHXRt4lfc+sUBb7MA0GCSqGSIb3DQEBCwUA MIGPMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEVMBMGA1UEChMM etc... -----END CERTIFICATE----- 1 s:C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = "Zscaler Intermediate Root CA (zscalerthree.net) (t) " i:C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = Zscaler Intermediate Root CA (zscalerthree.net), emailAddress = support@zscaler.com -----BEGIN CERTIFICATE----- MIIERjCCAy6gAwIBAgIEYtwrATANBgkqhkiG9w0BAQsFADCBrjELMAkGA1UEBhMC VVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFTATBgNVBAoTDFpzY2FsZXIgSW5jLjEV etc... -----END CERTIFICATE----- 2 s:C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = Zscaler Intermediate Root CA (zscalerthree.net), emailAddress = support@zscaler.com i:C = US, ST = California, L = San Jose, O = Zscaler Inc., OU = Zscaler Inc., CN = Zscaler Root CA, emailAddress = support@zscaler.com -----BEGIN CERTIFICATE----- MIIESzCCAzOgAwIBAgICAQEwDQYJKoZIhvcNAQELBQAwgaExCzAJBgNVBAYTAlVT MRMwEQYDVQQIEwpDYWxpZm9ybmlhMREwDwYDVQQHEwhTYW4gSm9zZTEVMBMGA1UE etc... -----END CERTIFICATE----- --- Server certificate subject=CN = erm-registry.pexip.io issuer=C = US, ST = California, O = Zscaler Inc., OU = Zscaler Inc., CN = "Zscaler Intermediate Root CA (zscalerthree.net) (t) " --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 4186 bytes and written 739 bytes Verification error: unable to get local issuer certificate --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 20 (unable to get local issuer certificate) ---
Example output that you might see when running this from a Cisco endpoint
Should a MITM proxy present an erroneous certificate towards an endpoint, there are some things that may resolve the issue:
- Fix the MITM proxy to simply pass the HTTP connection without terminating
- Add the root CA of the proxy to the Custom CA store of the Cisco endpoint.
- Configure the Cisco endpoint to use an HTTP proxy in its configuration settings via .
- Disable TlsVerify for the different connection methods (such as Provisioning, HTTPFeedback, PhoneBook etc). Note that this is not recommended, but can be used as a useful test.
Endpoints may experience the same issue if they connect to an ERM instance via some other infrastructure, such as a load balance or reverse proxy. However, it is difficult to understand this when looking at a Cisco endpoint as the warnings and logs are not clear. Further, you cannot run an openssl command from the CLI of a Cisco endpoint as the OS commands are hidden, and if you tried to take a packet capture, the endpoint (if on a reasonably modern firmware) may well use TLS 1.3 thus the packet containing the server certificate sent to the endpoint will be encrypted.
Looking at the endpoint web GUI, you may see something like this:
And clicking reveals a terse error regarding the HTTP Feedback slot (SSL peer certificate or SSH remote key was not OK):
The application.log on a Cisco endpoint may contain a useful indicator that the certificate chain cannot be terminated due to a self-signed certificate in the chain (although, by default, certificate details themselves are not logged, even with extended logging enabled), for example:
2022-09-28T01:37:37.530+07:00 appl[2579]: HttpClient W: (2) Failed to connect to 'erm-requests.fab-sas.co.uk': SSL certificate problem: self signed certificate in certificate chain
For reference, here is some output from the application.log on a Cisco endpoint when things work as expected (extended logging enabled):
2022-09-27T23:28:29.495+07:00 appl[2555]: HttpClient[1]: (61) HTTP: Outgoing => POST https://erm-requests.fab-sas.co.uk/tms/4ud67ba/b0bae3543148445e8d71e3143c504348/ 2022-09-27T23:28:29.495+07:00 appl[2555]: HttpClient[1]: (61) AuthMethod : ANY 2022-09-27T23:28:29.495+07:00 appl[2555]: HttpClient[1]: (61) Flags : bypass-proxy, curl-handles-redirects, ro-cookies, tls-verify 2022-09-27T23:28:29.495+07:00 appl[2555]: HttpClient[1]: (61) BODY : 764 bytes 2022-09-27T23:28:29.749+07:00 appl[2555]: HttpClient[1]: (61) HTTP: Incoming <= 200, 'No error'
As mentioned above, by default the details of a server's certificate are not logged, however, Cisco endpoints let you increase the debug log level of various logging contexts. You can run the following command on the CLI of an endpoint to increase the verbosity of the HTTP events:
log ctx HttpClient debug 9
If you recreate the event (for example, if this is an issue with provisioning, then in the endpoint option, set the Mode to OFF and , then reset to TMS and ), you should then see more detail in the application.log file, such as this good connection:
2022-10-04T20:10:25.390+07:00 appl[2503]: HttpClient[4]: (61) [curl] Connected to erm-requests.fab-sas.co.uk (192.168.199.165) port 443 (#27) 2022-10-04T20:10:25.391+07:00 appl[2503]: HttpClient[4]: (61) [curl] ALPN, offering h2 2022-10-04T20:10:25.391+07:00 appl[2503]: HttpClient[4]: (61) [curl] ALPN, offering http/1.1 2022-10-04T20:10:25.393+07:00 appl[2503]: HttpClient[4]: (61) [curl] CAfile: /config/certs/ca/default.pem 2022-10-04T20:10:25.393+07:00 appl[2503]: HttpClient[4]: (61) [curl] CApath: none 2022-10-04T20:10:25.394+07:00 appl[2503]: HttpClient[4]: (61) [curl] TLSv1.3 (OUT), TLS handshake, Client hello (1): 2022-10-04T20:10:25.394+07:00 appl[2503]: HttpClient[8]: Engine::processCompletedTasks 2022-10-04T20:10:25.395+07:00 appl[2503]: HttpClient[8]: Engine::processPendingRequests 2022-10-04T20:10:25.395+07:00 appl[2503]: HttpClient[8]: Engine::doCurlProcessing 2022-10-04T20:10:25.402+07:00 appl[2503]: HttpClient[4]: (61) [curl] TLSv1.3 (IN), TLS handshake, Server hello (2): 2022-10-04T20:10:25.404+07:00 appl[2503]: HttpClient[4]: (61) [curl] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): 2022-10-04T20:10:25.404+07:00 appl[2503]: HttpClient[4]: (61) [curl] TLSv1.3 (IN), TLS handshake, Certificate (11): 2022-10-04T20:10:25.407+07:00 appl[2503]: HttpClient[4]: (61) [curl] TLSv1.3 (IN), TLS handshake, CERT verify (15): 2022-10-04T20:10:25.408+07:00 appl[2503]: HttpClient[4]: (61) [curl] TLSv1.3 (IN), TLS handshake, Finished (20): 2022-10-04T20:10:25.409+07:00 appl[2503]: HttpClient[4]: (61) [curl] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): 2022-10-04T20:10:25.409+07:00 appl[2503]: HttpClient[4]: (61) [curl] TLSv1.3 (OUT), TLS handshake, Finished (20): 2022-10-04T20:10:25.410+07:00 appl[2503]: HttpClient[4]: (61) [curl] SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 2022-10-04T20:10:25.410+07:00 appl[2503]: HttpClient[4]: (61) [curl] ALPN, server accepted to use h2 2022-10-04T20:10:25.410+07:00 appl[2503]: HttpClient[4]: (61) [curl] Server certificate: 2022-10-04T20:10:25.410+07:00 appl[2503]: HttpClient[4]: (61) [curl] subject: CN=*.fab-sas.co.uk 2022-10-04T20:10:25.410+07:00 appl[2503]: HttpClient[4]: (61) [curl] start date: Sep 28 18:08:53 2022 GMT 2022-10-04T20:10:25.410+07:00 appl[2503]: HttpClient[4]: (61) [curl] expire date: Dec 27 18:08:52 2022 GMT 2022-10-04T20:10:25.410+07:00 appl[2503]: HttpClient[4]: (61) [curl] subjectAltName: host "erm-requests.fab-sas.co.uk" matched cert's "*.fab-sas.co.uk" 2022-10-04T20:10:25.410+07:00 appl[2503]: DEC_FSM-0[5]: Repeating frame 2022-10-04T20:10:25.412+07:00 appl[2503]: HttpClient[4]: (61) [curl] issuer: C=US; O=Let's Encrypt; CN=R3 2022-10-04T20:10:25.412+07:00 appl[2503]: HttpClient[4]: (61) [curl] SSL certificate verify ok.
We can see the Issuer and SAN entries for the server's certificate, and while the full chain is not logged, there should be enough information here to determine the cause of the issue.
It might also be beneficial to check the route a connection may take to a service, for example:
systemtools network traceroute otj.pexip.io