AWS SAM local (WSGI) drops headers with underscores

2025-03-30 (Sunday) | 600 words (~3 minutes reading)

TLDR: SAM local drops headers that contain an underscore, due to using Flask and WSGI internally, which differs from the behaviour of a deployed API Gateway.

I’m working on an existing project that uses AWS CDK to provision API Gateway in front of some Lambda functions. The team would like to be able to run this locally for faster development iterations, so I set that up with the sam local start-api command, using the template emitted by cdk synth. So far, so good.

However, the application happens to take a custom HTTP header, which we can call foobar_custom in this post. The important thing is that this custom header contains an underscore character (and that it’s the only custom header used, which made the exact issue a little more difficult to indentify).

The local stack starts up OK and prints this to the console:

...
Mounting lambdaFoobar at http://127.0.0.1:3000/foobar/something [POST, OPTIONS]                                                                    
You can now browse to the above endpoints to invoke your functions.
...
2025-03-30 11:57:23 WARNING: This is a development server. Do not use it in a
production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:3000
2025-03-30 11:57:23 Press CTRL+C to quit

I made a test request to the local stack like this:

curl -X POST http://127.0.0.1:3000/foobar/something \
  -H "Content-Type: application/json" \
  -H "foobar_custom: foobar1" \
  -d '{ "input": { "foo": "bar" } }'

The Lambda function received an API Gateway integration event for this request, but the custom header was missing:

{
  "body": { "input": { "foo": "bar" } },
  "headers": {
    "Content-Type": "application/json",
    "Host": "127.0.0.1:3000",
    "User-Agent": "curl/8.9.1",
    "X-Forwarded-Port": "3000",
    "X-Forwarded-Proto": "http"
  },
  "requestContext": {
    "domainName": "localhost",
    "http": {
      "method": "POST",
      "path": "/foobar/something",
      "protocol": "HTTP/1.1",
      "sourceIp": "127.0.0.1"
    },
    "routeKey": "POST /foobar/something"
  }
}

At first, I thought that sam local was dropping all the headers on incoming requests, or that the configuration of the API Gateway -> Lambda integration was different when running via sam local compared to when deployed via CDK.

After investigating that angle for a while, I tried specifying some other custom headers on the request to see if it was something specific about the foobar_custom header:

curl -X POST http://127.0.0.1:3000/foobar/something \
  -H "Content-Type: application/json" \
  -H "foobar_custom: foobar1" \
  -H "foobar-custom: foobar2" \
  -H "Foobar-Custom: foobar3" \
  -H "FoobarCustom: foobar4" \
  -H "Foobar_Custom: foobar5" \
  -d '{ "input": { "foo": "bar" } }'

With those request headers, the event received by the Lambda has these headers:

{
  "Content-Type": "application/json",
  "Foobar-Custom": "foobar2,foobar3",
  "Foobarcustom": "foobar4",
  "Host": "127.0.0.1:3000",
  "User-Agent": "curl/8.9.1",
  "X-Forwarded-Port": "3000",
  "X-Forwarded-Proto": "http"
}

The foobar_custom and Foobar_Custom headers were dropped completely. The FoobarCustom header was converted to Foobarcustom. It looks like the foobar-custom header was converted to Foobar-Custom, duplicating the other request header with the same name, and the values of both were concatenated together as foobar2,foobar3.

Now the cause of the problem is clear: incoming request headers that contain an underscore are dropped completely. This is the standard behaviour in WSGI, which sam local uses internally to approximately emulate some behaviours of API Gateway. Importantly, the real API Gateway in AWS does not have this behaviour, which is why this application was able to function like this before we tried to run it with sam local.


Tech mentioned