Blog
Token
About Us
AiDOOS Wiki
Project Cost Estimation

Welcome to Knowledge Base!

KB at your finger tips

This is one stop global knowledge base where you can learn about all the products, solutions and support features.

Categories
All
CRM
Salesforce
ERP
SAP
Storage and Backups
Purestorage Netapp Nutanix
Cloud
AWS
Computing
IBM
Database
MongoDB MySQL PostgreSQL
DevOps
Docker Kubernetes Jenkins GitLab
Mobile
React Native Android
Web
React Angular PHP
Programming
Python
Products
CNC Software (Mastercam) 3D perception 7shifts

Cloud-AWS

Troubleshoot 504 errors from an API Gateway REST API

How do I troubleshoot HTTP 504 errors from an API Gateway REST API with a Lambda backend?

Last updated: 2022-12-14

I receive an HTTP 504 error status code when I invoke a REST API using Amazon API Gateway with an AWS Lambda backend. How do I troubleshoot this error?

Resolution

When your REST API requests return an HTTP 504 error status code, you must run various checks.

Check for IntegrationLatency spikes

Check if there's an IntegrationLatency spike in API Gateway by reviewing the integration latency duration time. To see the IntegrationLatency duration time, configure the access logging variable $context.integration.latency for HTTP API logging.

For more information, see Set up Amazon CloudWatch API logging using the API Gateway console.

An IntegrationLatency spike in API Gateway indicates that the request spent most of its time in Lambda. Check the Lambda function's Duration performance metric to confirm this.

For more information, see Working with Lambda function metrics.

Review requests with CloudWatch Logs Insights

Use Amazon CloudWatch Logs Insights to review requests that resulted in 504 errors. To review the requests, on the CloudWatch console, in the navigation pane, choose Logs , Log Insights . Select your API Gateway log group. Then, set the relative time with one of the following queries: 

parse @message '(*) *' as reqId, message
| filter message like /Method completed with status: \d\d\d/
| parse message 'Method completed with status: *' as status
| filter status = 504
| sort @timestamp desc
| limit 20

-or- 

fields @timestamp, @message
| filter message like /Method completed with status: 504/
| sort @timestamp desc
| limit 20

Instrument X-Ray traces

If 504 errors keep happening continuously, determine where the Lambda function is spending its time. Instrument AWS X-Ray traces for Lambda functions based on the Lambda function runtime.

For Python: 

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()

For Node.js: 

const AWSXRay = require('aws-xray-sdk-core')
const AWS = AWSXRay.captureAWS(require('aws-sdk'))

Note: You need to build a new deployment package after instrumenting X-Ray traces on a Lambda function.

Implement API retries

When a 504 error occurs and the request isn't found in Lambda, implement API retries on the client. The error might have resulted from a temporary network failure in API Gateway.

Check the Lambda function configuration

Make sure that your Lambda function has only API Gateway event-specific processing logic. This way, the Lambda function takes less time to run and can keep up with incoming events.

REST APIs have a default maximum integration timeout of 29 seconds. Therefore, make sure that the Lambda function's running duration is under 29 seconds.

If you have a use case where the application is time sensitive, see Set up asynchronous invocation of the backend Lambda function.

API Gateway dimensions and metrics

Setting up CloudWatch logging for a REST API in API Gateway

View API Gateway log events in the CloudWatch console

Tracing user requests to REST APIs using X-Ray

Using Lambda with X-Ray

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support

Invoke a Lambda function asynchronously from API Gateway

How can I invoke a Lambda function asynchronously from my Amazon API Gateway API?

Last updated: 2022-12-12

I want to invoke an AWS Lambda function asynchronously instead of synchronously for my Amazon API Gateway API.

Resolution

REST APIs

In Lambda non-proxy integration, the backend Lambda function is invoked synchronously by default. You can configure the Lambda function for a Lambda non-proxy integration to be invoked asynchronously by specifying 'Event' as the Lambda invocation type.

1.    Open the API Gateway console, choose APIs , and then choose your REST API.

2.    In Resources , choose GET , and then choose Integration Request .

3.    In Integration type , choose Lambda Function .

4.    Expand HTTP Headers , and then choose Add header .

5.    For Name , enter X-Amz-Invocation-Type .

6.    For Mapped from , enter 'Event' .

7.    Redeploy the REST API.

To invoke the Lambda function with the option for either asynchronous or synchronous, add an InvocationType header.

1.    Open the API Gateway console, choose APIs , and then choose your REST API.

2.    In Resources , choose GET , and then choose Method Request .

3.    In Request Validator , choose the edit icon, choose the dropdown list, and then choose Validate query string parameters and headers .

4.    Choose the update icon to save changes.

5.    Expand HTTP Headers , and then choose Add header .

6.    For Name , enter InvocationType , and then choose Required .

7.    In Integration Request , Expand HTTP Headers , and then choose Add header .

8.    For Name , enter X-Amz-Invocation-Type .

9.     For Mapped from , enter method.request.header.InvocationType .

10.    Redeploy the REST API.

Clients can include the InvocationType: Event header in API requests for asynchronous invocations or InvocationType: RequestResponse for synchronous invocations.

For more information, see Set up asynchronous invocation of the backend Lambda function.

HTTP APIs

HTTP APIs only support proxy integrations for Lambda. You can't set the X-Amz-Invocation-Type header in the API Gateway integration for HTTP APIs. You can use two Lambda functions with one acting as proxy.

Example configuration: 

HTTP API --> Invoke Lambda1 synchronously --> Invoke Lambda2 asynchronously

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support
Read article

Troubleshoot high latency in API Gateway requests

How do I troubleshoot high latency in my API Gateway requests that are integrated with Lambda?

Last updated: 2022-12-08

Response times are slow when I make requests to an Amazon API Gateway API that’s integrated with an AWS Lambda function. How do I determine the cause of high latency?

Resolution

High latency must be addressed when an API endpoint that’s integrated with a Lambda function takes too long to send responses to a client. Review the metrics for API Gateway to identify the section of the request/response flow that’s causing high latency. After you determine the cause of high latency, you can work to reduce the delays.

Filter CloudWatch metrics to review latency metrics on the API

To identify the section of the request/response flow that’s causing high latency, first perform the following steps:

  1. Observe the latency of the client after sending a request to the API.
  2. After you note the overall latency, open the Amazon CloudWatch console. In the left navigation pane, choose Metrics , All metrics . In the metrics search box, enter APIGateway . From the search results, choose API Gateway , ApiId .
  3. In the list of APIs, filter for the specified API by using the API ID or the API name. After filtering, check the IntegrationLatency and Latency check boxes.
    Note: The API ID and API name are available from the API Gateway console.
  4. Open the Graphed metrics tab. For Statistic , choose Maximum . For Period , choose 1 minute . Above the graph, select the Custom time period. Choose the time frame during which the client experienced high latency.
  5. Review both the IntegrationLatency and Latency metrics. Note the values and timestamps when these metrics have high values. The values can explain the cause for high latency.

Compare metrics to identify the cause of high latency

Continue to review the metrics related to the request/response flow to find the cause of high latency:

  1. Compare the API Gateway Latency metric to the overall latency value observed at the client.
    For example, an API has a Latency metric with a Maximum value that’s approximately equal to the Max Latency value at the client. These values suggest that the maximum delay in the request/response flow is the time taken by API Gateway to process requests. API Gateway processing time includes the time taken to send requests to Lambda, wait for responses from Lambda, and send responses to a client.
  2. Compare the IntegrationLatency metric with the Latency metric for the API.
    For example, the IntegrationLatency metric is approximately equal to the Latency metric. These values indicate that latency at the API is primarily caused by backend requests sent to Lambda that are taking longer to respond. The IntegrationLatency metric includes the time between API Gateway sending a request and API Gateway receiving a response from the backend
  3. When the IntegrationLatency metric is low when compared to the Latency metric for the API, the backend response times are low. In this scenario, it takes longer to process the API requests or responses.
    For example, mapping templates configured in the API or an API Gateway Lambda authorizer both might create delays.
  4. When the Latency metric for the API is much lower than the latency observed at the client, the route might be causing delays. Review the route between the client and API Gateway to confirm whether there are any intermediate endpoints that are adding delays.
    For example, private VPN connections or proxies might create delays.

View Lambda metrics to identify the cause of high IntegrationLatency

Finally, focus on the Lambda metrics related to the request/response flow to find the cause of high IntegrationLatency :

  1. Check the Lambda function Duration metric to confirm whether the Lambda function’s execution time is longer. If the Lambda function’s execution time has increased, review the CloudWatch log to find the section of code that’s causing high latency. By default, Lambda functions log the START, END, and REPORT statements in CloudWatch logs. Add custom log statements at each logical section of the Lambda function code to get verbose CloudWatch logs.
  2. If the Duration metric didn't change during the period of high latency at the client, determine if the initialization time increased. Initialization time in a Lambda function is the time taken to set up the execution environment to process a request. Requests that come from API Gateway might require a new environment for processing. This is set up through Lambda. Typically, the code that’s present outside the Lambda function handler runs during the initialization time. Code that takes longer to complete can cause delays for the overall response times to the client.
    Note : Initialization time is known as INIT or cold start.
  3. Confirm whether there’s any increase in the initialization time’s Duration by verifying the report statements in the Lambda function logs. Initialization time that's high for some requests can cause an increase in the IntegrationLatency metric for API Gateway.

Working with metrics for HTTP APIs

Amazon API Gateway dimensions and metrics

Monitoring WebSocket API execution with CloudWatch metrics

Viewing metrics on the CloudWatch console

Lambda runtime environment lifecycle

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support
Read article

Troubleshoot API Gateway logs

How can I use Amazon API Gateway logs for troubleshooting?

Last updated: 2022-12-07

I want to use Amazon API Gateway logs for troubleshooting API issues.

Short description

To troubleshoot an API Gateway REST API or WebSocket API, turn on execution logging and access logging using Amazon CloudWatch Logs. Make sure that you have attached all the required permissions to the API Gateway AWS Identity and Access Management (IAM) role. For instructions, see How do I turn on CloudWatch Logs for troubleshooting my API Gateway REST API or WebSocket API?

Note: HTTP APIs currently support access logging only, and logging setup is different for these APIs. For more information, see Configuring logging for an HTTP API.

Resolution

After you turn on logging, API Gateway automatically creates a CloudWatch log group named API-Gateway-Execution-Logs/{rest-api-id}/{stage-name} . For more information, see CloudWatch log formats for API Gateway.

AWS Lambda authorizers

If you have Lambda authorizers turned on for your API, the logs indicate if the request is authorized and include the authorizer and request IDs.

Example authorized logs: 

Starting authorizer: 62mgrc for request: 3323ffe5-6ddf-45c8-896a-b45ec2284cc0
Successfully completed authorizer execution

Example unauthorized logs: 

Starting authorizer: 62mgrc for request: fd443056-60d0-425f-874e-e93e5d61351e
Unauthorized request: fd443056-60d0-425f-874e-e93e5d61351e

Usage plans

After the Lambda authorizer logs, API Gateway verifies the usage plan for the API request. If you haven't configured a usage plan, then API requests aren't counted for throttling limits.

Example usage plan logs: 

Verifying Usage Plan for request: 3323ffe5-6ddf-45c8-896a-b45ec2284cc0. API Key: API Stage: <api_id>/<stage>
API Key authorized because method 'GET /' does not require API Key. Request will not contribute to throttle or quota limits
Usage Plan check succeeded for API Key and API Stage <api_id>/<stage>

If you have configured APIs with API keys but the X-API-Key header is missing, then the logs return an error similar to the following: 

API Key not authorized because method 'GET /' requires API Key and API 
Key is not associated with a Usage Plan for API Stage 
<api_id>/<stage>: API Key was required but not present

If you added the correct API key in the header but the request is throttled, then the logs return an error similar to the following: 

API Key **********************************FGQes7 exceeded throttle limit for API Stage <api_id>/<stage>: Key throttle limit exceeded for Usage Plan ID iwuzkt. Limit: 5.00 Burst: 0
Method completed with status: 429

Request log types

Note: All request log types require turning on Log full requests/responses data in the execution logs.

Method request logs

Method request logs includes detailed information received by an API request including the request path, query parameter, request header, and request body.

Endpoint request logs

Endpoint request logs include detailed information received by an API request including the URI, request headers, and request body.

The data included in endpoint requests are sent to your backend integration for further processing.

Endpoint response logs

Endpoint response logs include the data returned by the backend integration including the status code, headers body, and Integration latency.

Method response logs

Method response logs includes the headers and body sent to the client after the transformation.

Important: API Gateway limits log events to 1024 bytes. Log events larger than 1024 bytes, such as request and response bodies, are truncated by API Gateway before submission to CloudWatch logs.

How can I troubleshoot missing CloudWatch logs for API Gateway REST APIs?

Setting up CloudWatch logging for a REST API in API Gateway

Monitoring REST API execution with Amazon CloudWatch metrics

Monitoring WebSocket API execution with CloudWatch metrics

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support
Read article

Troubleshoot API Gateway SSL errors

How can I troubleshoot the API Gateway error “Execution failed due to configuration error: General SSLEngine problem”?

Last updated: 2022-12-01

Amazon API Gateway returned an error for my API request similar to the following:

"Execution failed due to configuration error: General SSLEngine problem"

Resolution

API Gateway API requests perform an SSL handshake on the backend. Successful API Gateway SSL handshakes must include the following requirements:

A supported CA

The CA must be supported by API Gateway for HTTP, HTTP proxy, and private integrations. To check the CA fingerprint, run the following OpenSSL command for your operating system:

Linux 

openssl x509 -in cert.pem -fingerprint -sha256 -noout

openssl x509 -in cert.pem -fingerprint -sha1 -noout

Windows 

openssl x509 -noout -fingerprint -sha256 -inform pem -in [certificate-file.crt]

openssl x509 -noout -fingerprint -sha1 -inform pem -in [certificate-file.crt]

A valid ACM certificate that isn't expired

To check the expiration date of the certificate, run the following OpenSSL command: 

openssl x509 -in certificate.crt -text -noout

In the output, check for the validity timestamp: 

Validity
            Not Before: Apr 29 12:49:02 2020 GMT
            Not After : Apr 29 12:49:02 2021 GMT

In this example output, the certificate is valid on April 29, 2020 and expires after April 29, 2021.

A valid CA certificate

Check the CA certificate configuration by running the following OpenSSL command: 

openssl s_client -connect example.com:443 -showcerts

Validate that:

  • The subject of the intermediate and certificate matches the issuer of the entity certificate.
  • The subject of the root certificate matches the issuers of the intermediate certificate.
  • The subject and issuer are the same in the root certificate.

A certificate that doesn't exceed 2048 bits

Check the size of the certificate by running the following OpenSSL command: 

openssl x509 -in badssl-com.pem -text -noout | grep -E '(Public-Key):'

Note: The certificate size can't exceed 2048 bits.

If your certificate doesn't meet any of these requirements, first update your private CA. Then, reissue a new certificate using AWS Certificate Manager (ACM).

Set up API Gateway private integrations

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support
Read article

Forward host headers for an API Gateway REST API

How can I forward the host header with private integration for an API Gateway REST API?

Last updated: 2022-11-29

I want to use the host header value for my Amazon API Gateway endpoint or custom domain. How can I forward the host header with private integration for an API Gateway REST API?

Short description

Amazon API Gateway overwrites the host value that it receives in the original request to the integration endpoint. Amazon API Gateway assigns the host value depending on the integration request endpoint URL.

To use the host header value of an API Gateway endpoint in the backend, use the custom header. You can use the custom header in the integration request that contains the value of the request host header.

Resolution

Create an API Gateway REST API

1.    Open the API Gateway console.

2.    Choose Create API .

-or-

(If this is your first time using API Gateway) A page that introduces you to the features of the service appears. In REST API , choose Build . When the Create Example API popup appears, choose OK .

3.    For Choose an API type , in the REST API pane, choose Build .

4.    In Create new API , choose New API .

5.    In Settings , do the following:
For API name , enter a name that describes your API's purpose. For example: ForwardHostHeader .
(Optional) For Description , enter a short description of your API's purpose. for example, Forward host header for private integration .
For Endpoint Type , choose either Regional , Edge optimized , or Private .

6.    Choose Create API .

Configure your API's method request and Integration request

1.    Open the API Gateway console.

2.    Follow the instructions to create an API with private integration using the API Gateway console.

3.    In Method Request , add HTTP Request Headers as host.

4.    In Integration Request , expand HTTP Headers , and choose Add header .

5.    Enter a name for the custom header. For example, my_host .

6.    In Mapped from , enter the method request. For example, method.request.header.host , and then choose the check mark icon.

Deploy your API to a new stage

1.    Open the API Gateway console.

2.    Follow the instructions to Deploy a REST API to a stage. For more information, see Setting up a stage using the API Gateway console.

3.    In the Stage Editor pane, copy the Invoke URL to your clipboard. You can use this URL to invoke your API.

REST API invoke URL example: 

https://1a2bc3d456.execute-api.us-east-1.amazonaws.com/stage

4.    If you use an Application Load Balancer for your backend, you can test this configuration by adding a rule. Choose "http header" as a condition and specify "my_host" as the same header and value as the API Gateway host value.

5.    Paste the API Gateway URL into a browser window to invoke your API. The "my_host" header configuration in API Gateway forwards the API Gateway host value to the Application Load Balancer. The Application Load Balancer matches the condition set and then returns the response as configured.

Set up request and response data mappings using the API Gateway console

Set up an API using the API Gateway console

Amazon API Gateway important notes

How do I integrate an API Gateway REST API with an Application Load Balancer?

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support
Read article