How do I troubleshoot HTTP 403 Forbidden errors from an API Gateway custom domain name that requires mutual TLS?
My Amazon API Gateway custom domain name that has mutual Transport Layer Security (TLS) authentication activated returns HTTP 403 Forbidden errors. Why is this happening, and how do I troubleshoot the issue?
Short description
Note:
API Gateway can return 403 Forbidden errors for a variety of reasons. This article addresses 403 Forbidden errors related to mutual TLS only. For information on troubleshooting other types of 403 Forbidden errors, see How do I troubleshoot HTTP 403 Forbidden errors from API Gateway?
To invoke an API Gateway API using a custom domain name that requires mutual TLS, clients must present a trusted certificate in the API request. When a client attempts to invoke the API, API Gateway looks for the client certificate's issuer in your truststore.
If any of the following conditions occur, then API Gateway fails the TLS connection and returns a 403 status code:
-
API Gateway can't find the issuer of the client certificate in your truststore.
-
The client certificate is using an insecure signature algorithm.
-
The client certificate is self signed.
If Amazon CloudWatch logging is activated for your API, then an error message that indicates the cause of the error appears in your execution logs.
Important:
If the API request doesn't produce any CloudWatch Logs after logging is activated, then the 403 Forbidden error isn't related to mutual TLS.
For REST APIs
If you set up Amazon CloudWatch logging for your REST API, then one of the following error messages also appears in your execution logs:
-
Access denied. Reason: Could not find issuer for certificate
-
Access denied. Reason: Client cert using an insecure Signature Algorithm
-
Access denied. Reason: self signed certificate
For HTTP APIs
HTTP APIs don't support execution logging. To troubleshoot 403 Forbidden errors returned by a custom domain name that requires mutual TLS and invokes an HTTP API, you must do the following:
1. Create a new API mapping for your custom domain name that invokes a REST API for testing only.
Note:
If you don't have a REST API available for testing, then use the example PetStore REST API. Then, deploy the example API to a new stage and create a new API mapping for it that uses your custom domain name.
2. Follow the instructions in the
Resolution
section of this article using the new API mapping that you created to your REST API.
3. After the error is identified and resolved, reroute the API mapping for your custom domain name back to your HTTP API.
Resolution
Confirm the cause of the error
1. If you haven't already, turn on CloudWatch logging for your REST API. Make sure that you configure execution and access logging.
Note:
When configuring access logging for this use case, it's a best practice to use the following $context variables. These variables do two things:
-
They tell API Gateway to generate CloudWatch Logs when your custom domain name that requires mutual TLS returns a 403 Forbidden error.
-
They make it easier to identify the caller that tried to invoke your API when you review your CloudWatch Logs.
Recommended $context variables for CloudWatch access logging that will allow API Gateway to generate execution and access logs
{ "accountId":"$context.accountId", "apiId":"$context.apiId", "domainName":"$context.domainName", "domainPrefix":"$context.domainPrefix", "error.message":"$context.error.message", "error.responseType":"$context.error.responseType", "extendedRequestId":"$context.extendedRequestId", "httpMethod":"$context.httpMethod", "identity.sourceIp":"$context.identity.sourceIp", "identity.clientCert.clientCertPem":"$context.identity.clientCert.clientCertPem", "identity.clientCert.subjectDN":"$context.identity.clientCert.subjectDN", "identity.clientCert.issuerDN":"$context.identity.clientCert.issuerDN", "identity.clientCert.serialNumber":"$context.identity.clientCert.serialNumber", "identity.clientCert.validity.notBefore":"$context.identity.clientCert.validity.notBefore", "identity.clientCert.validity.notAfter":"$context.identity.clientCert.validity.notAfter", "identity.userAgent":"$context.identity.userAgent", "path":"$context.path", "protocol":"$context.protocol", "requestId":"$context.requestId", "requestTime":"$context.requestTime", "requestTimeEpoch":"$context.requestTimeEpoch", "resourceId":"$context.resourceId", "resourcePath":"$context.resourcePath", "stage":"$context.stage", "responseLatency":"$context.responseLatency", "responseLength":"$context.responseLength", "status":"$context.status" }
2. Identify what's causing the 403 Forbidden errors by viewing your REST API's execution logs in CloudWatch. If a 403 Forbidden error related to mutual TLS is logged, then an error message similar to the following example appears.
Example CloudWatch Logs error message for when a custom domain name that requires mutual TLS returns a 403 Forbidden error
Extended Request Id: {extendedRequestId}
Access denied. Reason: {reason}
ForbiddenException Forbidden: {requestId}
To resolve "Access denied. Reason: Could not find issuer for certificate" errors
Verify that the issuer of the client certificate in the API request is included in the custom domain name's truststore
The issuer of the client certificate (
client.pem
) in the API request must be included in your custom domain name's truststore. The issuer must also be included as part of the certificate bundle (
bundle.pem
) in Amazon Simple Storage Service (Amazon S3).
To verify if the issuer of the client certificate is included in the required truststore, run the following OpenSSL command:
$ openssl verify -CAfile bundle.pem client.pem
-or-
If the certificate bundle contains intermediate certificate authorities, then run the following OpenSSL command:
$ openssl verify -CAfile rootCA.pem -untrusted intCA.pem client.pem
The command returns an
OK
response if the issuer of the client certificate in the API request is included in the required truststore.
The command returns the following error if the issuer of the client certificate isn't included in the required truststore: "error X at Y depth lookup: unable to get local issuer certificate"
Verify that all of the client certificates in your custom domain name's truststore are valid
If one of the client certificate's in your custom domain name's truststore isn't valid, then some clients might not be able to access your API.
To verify whether all of the client certificates in your truststore are valid, do the following:
1. Open the API Gateway console.
2. In the left navigation pane, choose
Custom domain names
. Then, choose your custom domain name that requires mutual TLS.
3. In the
Details
section, see whether there's the following error message:
There is an invalid certificate in your truststore bundle
.
4. If you see the error message, then you must decode the certificates in your truststore to identify which certificate produced the warning.
Note:
The following OpenSSL command displays the contents of a certificate, including its subject:
$ openssl x509 -in certificate.crt -text -noout
5. Update or remove the certificates that produced the warning. Then, upload a new truststore to Amazon S3.
For more information, see Troubleshooting certificate warnings.
Note:
API Gateway accepts client certificates signed directly by the root certificate authority or any other intermediate if their certificate chain is preserved. To validate client certificates signed by the last intermediate certificate authority only, use a request parameter-based AWS Lambda authorizer. You can use your custom validation algorithm at the Lambda function level by accepting the client certificate as input from the API request.
To resolve "Access denied. Reason: Client cert using an insecure Signature Algorithm" errors
Verify that your truststore text file uses a supported hashing algorithm
API Gateway supports the following hashing algorithms in the truststore:
-
SHA-256 or stronger
-
RSA-2048 or stronger
-
ECDSA-256 or stronger
To verify if your truststore text file uses a supported hashing algorithm or not, then run the following OpenSSL command:
$ openssl x509 -in client.crt -text -noout | grep 'Signature Algorithm'
The command response returns your truststore's signature algorithm.
For more information, see Configuring your truststore.
To resolve "Access denied. Reason: self signed certificate" errors
Verify that the self-signed client certificate in the API request isn't altered or corrupted
The following must match exactly:
-
The modulus of the private key (
private.key
) used to sign the self-signed certificate within the truststore in S3 (
bundle.crt
or
bundle.pem
).
-
The modulus from the client's certificate passed in the API request (
client.crt
).
To compare the two modulus, run the following OpenSSL commands:
$ openssl rsa -noout -modulus -in private.key
$ openssl x509 -noout -modulus -in bundle.crt
$ openssl x509 -noout -modulus -in client.crt
Note:
To produce a shorter hash value for easier comparison, you can
PIPE-
ing
the output modulus into a cryptographic hash function. For example:
openssl sha1
.
$ openssl [operation] -noout -modulus -in [data] | openssl sha1
Valid command output examples
2143831a73a8bb28467860df18550c696c03fbcb
2143831a73a8bb28467860df18550c696c03fbcb
2143831a73a8bb28467860df18550c696c03fbcb
To confirm data integrity, verify that there wasn't any data modification at the content level by running the following
diff
command:
$ diff client.crt bundle.crt
For more information, see Configuring your truststore.
Submit feedback
Do you need billing or technical support?
Contact AWS Support