When configuring SSL services, generally you need to set a certificate, a private key and the CA bundle containing the intermediate certificate(s), which is often a bundle of several different certificates.

For example, https://www.jethrocarr.com‘s configuration looks like:

SSLEngine on
SSLCertificateFile jethrocarr.com.crt
SSLCertificateKeyFile jethrocarr.com.key
SSLCertificateChainFile startssl.intermediate.ca.crt

When your browser connects, it doesn’t trust jethrocarr.com.crt, but it checks it against the certificates that have signed it in startssl.intermediate.ca.crt – and those certificates are signed by the CA that your browser trusts.

This means that the CAs can protect their root certificates which are trusted by the browser much more securely and sign their certificates with intermediates than can be revoked and easily(ish) replaced should the need arise.

Generally this works fine from the end user perspective, although there are sometimes issues when an sysadmin forgets to add the intermediate CA bundle and doesn’t immediately notice as sometimes some browsers work fine whilst others fail depending whether or not they already trust the intermediates.

 

Today I ran into a different new issue where Amazon Web Services is fussy about the order of the certificates in that bundle when adding a certificate to an Elastic Load Balancer for SSL termination.

Any attempt to upload my certificate was met with “Invalid Public Key Certificate”, which didn’t make a lot of sense as I was certain that my certificates were OK. It was easy to verify and prove this, using OpenSSL:

$ openssl rsa -noout -modulus -in example.com.key | openssl md5
(stdin)= 30e1b6cb4168117b7923392ca536c701

$ openssl x509 -noout -modulus -in example.com.crt | openssl md5
(stdin)= 30e1b6cb4168117b7923392ca536c701

$ openssl verify -verbose -CAfile cabundle.crt example.com.crt 
example.com.crt: OK

This proved that my certificates were all correct so the fault was Amazon-side. A post on their forums helped me “fix” the issue, by adjusting the order of my CA bundle, which subsequently fixed the error.

So is this a bug with Amazon? It’s tricky to say – there are several posts online which state that the order is important for some systems, but not for all. Clearly anything based around OpenSSL doesn’t care, as it was able to verify my out-of-order CA bundle happily enough.

As one does with issues like this, I dug into RFC 3280 which details how the certificate path validation should occur. Section 6.1 (Basic Path Validation) details that the path validation process is actually outside the specification, but then goes on and defines how the validation could occur, with the order of the certificates being implied, but not stated outright.

The primary goal of path validation is to verify the binding between
a subject distinguished name or a subject alternative name and
subject public key, as represented in the end entity certificate,
based on the public key of the trust anchor.  This requires obtaining
a sequence of certificates that support that binding.  The procedure
performed to obtain this sequence of certificates is outside the
scope of this specification.

To meet this goal, the path validation process verifies, among other
things, that a prospective certification path (a sequence of n
certificates) satisfies the following conditions:

   (a)  for all x in {1, ..., n-1}, the subject of certificate x is
   the issuer of certificate x+1;

   (b)  certificate 1 is issued by the trust anchor;

   (c)  certificate n is the certificate to be validated; and

   (d)  for all x in {1, ..., n}, the certificate was valid at the
   time in question.

Following the above, the specification goes on into detail different ways the path can be validated, which also imply that the certificates should be read in and then sorted by software, but it doesn’t actually state exactly.

Sadly with the way that this specification is written it’s not clear, which means the only 100% certain way to ensure nothing is unhappy is to order the CA bundles file in the correct order, which is something I would expect the SSL provider to do when they provide you with the files.