withastro/astro

"Headers Overflow Error" when fetching url

Closed this issue · 6 comments

ambrt commented

Astro Info

Astro                    v4.16.16
Node                     v23.3.0
System                   Linux (x64)
Package Manager          npm
Output                   server
Adapter                  @astrojs/cloudflare
Integrations             none

If this issue only occurs in one browser, which browser is a problem?

No response

Describe the Bug

This simple fetch:

---
await fetch("https://www.walmart.com/ip/LEGO-Marvel-Endgame-Final-Battle-76266-Avengers-Model-Build-Display-Collectible-Playset-6-Minifigures-Including-Captain-Marvel-Shuri-Wanda-Maximoff-F/1488091928?classType=REGULAR&athbdg=L1800")
---

Is throwing Headers Overflow Error:

  Stack trace:
    at node:internal/deps/undici/undici:13484:13
    [...] See full stack trace in the browser, or rerun with --verbose.
  Caused by:
  Headers Overflow Error
    at Parser.trackHeader (node:internal/deps/undici/undici:6055:37)
    at wasm_on_header_value (node:internal/deps/undici/undici:5845:34)
    at wasm://wasm/0002f79e:wasm-function[20]:0x647a
    at Parser.readMore (node:internal/deps/undici/undici:5950:16)
    at TLSSocket.emit (node:events:513:28)

If i'll just run node test.js:

fetch("https://www.walmart.com/ip/LEGO-Marvel-Endgame-Final-Battle-76266-Avengers-Model-Build-Display-Collectible-Playset-6-Minifigures-Including-Captain-Marvel-Shuri-Wanda-Maximoff-F/1488091928?classType=REGULAR&athbdg=L1800").then(resp=>resp.text()).then(text=>console.log(text)).catch(err=>console.log(err))

then it works.

I've set alias node="node --max-http-header-size=36384" in ~/.bashrc but it didn't solve the problem in astro.

What's the expected result?

Should fetch this url and its content.

Link to Minimal Reproducible Example

https://stackblitz.com/edit/github-i8a6hf?file=src%2Fpages%2Findex.astro,src%2Fpages%2Fa.js&on=stackblitz

Participation

  • I am willing to submit a pull request for this issue.
ematipico commented

@ambrt I can't access the reproduction

ematipico commented

Can you use stackblitz to share a reproduction with us?

github-actions commented

Hello @ambrt. Please provide a minimal reproduction using a GitHub repository or StackBlitz. Issues marked with needs repro will be closed if they have no activity within 3 days.

ambrt commented

I copied the headers from browser and they are close to 28KB.

Here's a reproduction with fake 16300 Bytes header that throws the same error:
https://stackblitz.com/edit/github-i8a6hf?file=src%2Fpages%2Findex.astro,src%2Fpages%2Fa.js&on=stackblitz

ambrt commented

Ok, I'm using NVM and i think that's the reason alias node="node --max-http-header-size=36384" in bashrc didn't work.

Reproduction will still throw error because stackblitz has default 16K max node headers size.
It's undici error so not related to Astro.

ambrt commented

To make it work
NODE_OPTIONS="--max-http-header-size=40000" npm run dev
or

export NODE_OPTIONS="--max-http-header-size=40000"
npm run dev