Troubleshooting Salesforce.com SSL Chaining Issue (IO Exception: sun.security.validator.ValidatorException: PKIX path building failed:)

In my recent project we were making outbound web service call-out to external system in 2 way secure environment. Everything works well till downstream environment certificate were updated as they are about to expire in couple of days. Once certificate were updated in downstream server and salesforce certificate was configured, we were getting error something like below. IO Exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target All other application which were connected to downstream environment was working fine but just salesforce had issue. We initially were in impression that it is salesforce.com side fault but it turnout as configuration issue at external system. I searched all around but didn't find much information how to resolve issue, all post I find was telling that it is configuration issue at server side which were right but how to diagnose it and be 100% sure. I discussed this issue with Dan Guggenheim (Our Technically Architect) and imagine he has already faced and figured out steps. Here are steps that we followed to diagnose problem. Quick On-line Utility : Their is online utility at https://www.sslshopper.com/ssl-checker.html that will list certificate chain and tell if their is any issue if that doesn't work then try next. Command Line tool :- Download the SSLPoke utility class that can be found at https://confluence.atlassian.com/download/attachments/218272870/SSLPoke.class?version=1&modificationDate=1275292817310&api=v2  and store it in your java classpath. From the command line, run:   java SSLPoke sapeservicesuat.domainname.com 443 If the ssl configuration is correct, you should see:  Successfully connected If there are problems, you will probably see the unable to find valid certification path to requested target exception, or something similar To get more details, add another argument to the SSLPoke command:  java -Djavax.net.debug=ssl SSLPoke sapeservicesuat.domainname.com 443.  You may need to redirect the output to a text file so you can search it later ( add > fileName.txt>. As you review the output, look for the certificate chain.  It looks something like this (notice chain[0],chain[1]: chain [0] = [ [  Version: V3  Subject: CN=sapeservicesuat.domainname.com, OU=IT, O=Sapient Corporation, L=Boston, ST=Massachusetts, C=US  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5  Key:  Sun RSA public key, 2048 bits  public exponent: 65537  Validity: [From: Thu Jan 30 05:30:00 IST 2014,               To: Fri Feb 20 05:29:59 IST 2015]  Issuer: CN=Thawte SSL CA, O="Thawte, Inc.", C=US  SerialNumber: [    7391c10b 37f0d22e 15dcabf3 355cabde] Certificate Extensions: 8 [1]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false AuthorityInfoAccess [  [   accessMethod: ocsp   accessLocation: URIName: http://ocsp.thawte.com ,   accessMethod: caIssuers   accessLocation: URIName: http://svr-ov-aia.thawte.com/ThawteOV.cer ] ] [2]: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: A7 A2 83 BB 34 45 40 3D   FC D5 30 4F 12 B9 3E A1  ....4E@=..0O..>. 0010: 01 9F F6 DB                                        .... ] ] [3]: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[  CA:false  PathLen: undefined ] [4]: ObjectId: 2.5.29.31 Criticality=false CRLDistributionPoints [  [DistributionPoint:     [URIName: http://svr-ov-crl.thawte.com/ThawteOV.crl] ]] [5]: ObjectId: 2.5.29.32 Criticality=false CertificatePolicies [  [CertificatePolicyId: [2.16.840.1.113733.1.7.54] [PolicyQualifierInfo: [  qualifierID: 1.3.6.1.5.5.7.2.1  qualifier: 0000: 16 1B 68 74 74 70 73 3A   2F 2F 77 77 77 2E 74 68  ..https://www.th 0010: 61 77 74 65 2E 63 6F 6D   2F 63 70 73 2F          awte.com/cps/ ]]  ] ] [6]: ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [  serverAuth  clientAuth ] [7]: ObjectId: 2.5.29.15 Criticality=true KeyUsage [  DigitalSignature  Key_Encipherment ] [8]: ObjectId: 2.5.29.17 Criticality=false SubjectAlternativeName [  DNSName: sapeservicesuat.domainname.com ] ] ] chain [1] = [ [  Version: V3  Subject: CN=Thawte SSL CA, O="Thawte, Inc.", C=US  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5  public exponent: 65537  Validity: [From: Mon Feb 08 05:30:00 IST 2010,               To: Sat Feb 08 05:29:59 IST 2020]  Issuer: CN=thawte Primary Root CA, OU="(c) 2006 thawte, Inc. - For authorized use only", OU=Certification Services Division, O="thawte, Inc.", C=US  SerialNumber: [    4d5f2c34 08b24c20 cd6d507e 244dc9ec] Certificate Extensions: 7 [1]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false AuthorityInfoAccess [  [   accessMethod: ocsp   accessLocation: URIName: http://ocsp.thawte.com ] ] [2]: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 7B 5B 45 CF AF CE CB 7A   FD 31 92 1A 6A B6 F3 46  .[E....z.1..j..F 0010: EB 57 48 50                                        .WHP ] ] [3]: ObjectId: 2.5.29.19 Criticality=true BasicConstraints:[  CA:true  PathLen:0 ] [4]: ObjectId: 2.5.29.31 Criticality=false CRLDistributionPoints [  [DistributionPoint:     [URIName: http://crl.thawte.com/ThawtePCA.crl] ]] [5]: ObjectId: 2.5.29.15 Criticality=true KeyUsage [  Key_CertSign  Crl_Sign ] [6]: ObjectId: 2.5.29.17 Criticality=false SubjectAlternativeName [  CN=VeriSignMPKI-2-9 ] [7]: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: A7 A2 83 BB 34 45 40 3D   FC D5 30 4F 12 B9 3E A1  ....4E@=..0O..>. 0010: 01 9F F6 DB                                        .... ] ] ] ] chain [2] = [ [  Version: V3  Subject: CN=thawte Primary Root CA, OU="(c) 2006 thawte, Inc. - For authorized use only", OU=Certification Services Division, O="thawte, Inc.", C=US  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 ] *** ] If you only see one entry in the chain, that is usually a pretty good indication that the server is not presenting the correct sequence of certs. Salesforce.com is very particular about certificate chaining and server must must send intermediate certificate in the correct order.The correct order is: Server certificate. Intermediate certificate that signed the server certificate if the server certificate was not signed directly by a root certificate. Intermediate certificate that signed the certificate in step 2. Any remaining intermediate certificates. Do not include the root certificate authority certificate. The root certificate is not sent by your server. in my case, we just compare what order server was presenting before updating certification what is wrong after it is updated and got issue in second approach.