[Help] ESP32-Cam http server header limit

[Relations]

I could use some help from anyone who is familiar with the http server build for the ESP32 arduino core.

I keep running into the problem where the server responds to a GET request with 'Header fields are too long for the server to interpret'

 

A detailed explanation of my issue can be found on github.

https://github.com/espressif/arduino-esp32/issues/2983

 

If someone prefers to view it here on the forums instead, here is the copy paste of it 

Spoiler

What I'm trying to do:

Access ESP32 Camera Http Server Example (included in examples) over the internet, on an Android smartphone running chrome

What is failing: (what I'm failing at)

• Header field in GET request exceeds ESPs limit
• No idea how to change the limit

Hardware:

Board: ESP32-Cam module
Link to Board : https://www.aliexpress.com/item/32992663411.html?spm=a2g0s.9042311.0.0.15764c4dzmz0WC
Core Installation/update date: v1.0.2 in Boards Manager
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: Yes
Upload Speed: 921600
Computer OS: Windows 10
Phone: Oneplus 6T running Andorid 9

Description:

• Based on the Camera Example found in the ESP32 category
• I changed the port of the index page and the MJPEG stream service (config.server_port and ctrl_port in startCameraServer())
• I set the port forwarding rules for my router to point those 2 ports to my ESP32-Cam
• I get the error "Header fields are too long for the server to interpret" when I attempt to load the webpage from my Android Smartphone running Chrome.
• Accessing the ESP via my external IP in INCOGNITO mode works fine
• Accessing the ESP via my external IP without incognito HARDLY WORKS, but randomly works now and then
• I don't know how to see the headers my phone is sending

What I've tried

• I have tried changing the definition of the limits in my sketch and in app_httpd.cpp

#include "sdkconfig.h"
#include "esp_http_server.h"
#undef CONFIG_HTTPD_MAX_REQ_HDR_LEN
#define CONFIG_HTTPD_MAX_REQ_HDR_LEN 1024
#undef HTTPD_MAX_REQ_HDR_LEN
#define HTTPD_MAX_REQ_HDR_LEN 1024

• Changing the max_resp_headers
  config.max_resp_headers = 50;

Found this other person asking for help on the espressive forums, https://esp32.com/viewtopic.php?t=7685but I have no idea what to do to change the limit.
How do I use the Kconfig file thing? It's right there but I have no idea how do I make the compiler use my values instead.
Read through this as well but I think it's meant for building the sdkconfig.h file https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/kconfig.html

Debug Messages:

Nothing of note. Posted either way

[D][esp32-hal-psram.c:47] psramInit(): PSRAM enabled

[D][WiFiGeneric.cpp:336] _eventCallback(): Event: 0 - WIFI_READY
[D][WiFiGeneric.cpp:336] _eventCallback(): Event: 2 - STA_START
.[D][WiFiGeneric.cpp:336] _eventCallback(): Event: 4 - STA_CONNECTED
[D][WiFiGeneric.cpp:336] _eventCallback(): Event: 7 - STA_GOT_IP
[D][WiFiGeneric.cpp:379] _eventCallback(): STA IP: 192.xxx.xxx.xxx, MASK: 255.255.255.0, GW: 192.xxx.xxx.xxx
.
WiFi connected
Starting web server on port: 'xxxx'
Starting stream server on port: 'xxxx'
Camera Ready! Use 'http://192.xxx.xxx.xxx' to connect

IP redacted

Apologies in Advance

I've got no formal education in coding, I've only picked up what I can from youtube tutorials and Arduino projects. I'm also not a native to the English language. Bare with me while I attempt to learn something in the process!

 

[zhick]

Kconfig is a tool used to change compile-time definitions for a piece of software, like in this case the size of a buffer used to store a http request header. It basically works by generating a header (sdkconfig.h) which has all the defines and injecting that into the build process.

Your

#undef CONFIG_HTTPD_MAX_REQ_HDR_LEN
#define CONFIG_HTTPD_MAX_REQ_HDR_LEN 1024

would in theory work, if it was at the correct place, which is right before where the relevant buffer is declared (somewhere in esp-idf, possibly esp_http_server.h, but probably in some .c file).

 

What you'd need to do is to change the value defined in sdkconfig.h and recompile the relevant files. Normally you'd do that using

make menuconfig

, changing the relevant options and recompiling your firmware. The link to espressifs documentation you found has all the details.

However apparently esp32-arduino ships a precompiled esp-idf (according to this forum post I found at least: https://esp32.com/viewtopic.php?t=5225 , I've never used arduino myself), so as is pointed out there, there's no way for you to change the definitions easily. You'd need to compile your own esp-idf. The above forum post has a link to https://github.com/espressif/arduino-esp32/blob/master/docs/esp-idf_component.md which has more details on how you'd go about that, but depending on your level of experience this is probably not going to be trivial task. I'm assuming you're running windows, so you'll first need to follow this: https://docs.espressif.com/projects/esp-idf/en/latest/get-started/windows-setup.html         ,     to setup a suitable development environment.

[colonel_mortis]

Is it possible that you have a bunch of cookies for the site that you're using? I've had issues in the past (not with an arduino) where I have had so many cookies set (due to a bug with the site) that I received header too long errors. That would also explain why it works in incognito. You should be able to clear your cookies in Chrome's settings, but I don't think there's a way on mobile chrome to clear them for just one site.

 

If you're just accessing it on 192.x.x.x (which is a local IP, so it doesn't matter if you post the whole thing, but it won't help here), it seems slightly unlikely, but it might be worth a try. Otherwise, are you able to try accessing it from a computer? That way, you can use the network inspector in the developer tools to actually see exactly what is being sent.

 

You might also have more luck changing the header size somewhere else - one of the examples that I found uses a separate file, sdkconfig.defaults, to define the constants https://github.com/espressif/esp-idf/blob/master/examples/protocols/http_server/file_serving/sdkconfig.defaults. I don't know anything about that file, and it may well be specific to a particular toolchain, but it might be worth a look. (The answer above mine looks like it might do that.)

[Relations]

11 hours ago, zhick said:

Kconfig is a tool used to change compile-time definitions for a piece of software, like in this case the size of a buffer used to store a http request header. It basically works by generating a header (sdkconfig.h) which has all the defines and injecting that into the build process.

Your

#undef CONFIG_HTTPD_MAX_REQ_HDR_LEN
#define CONFIG_HTTPD_MAX_REQ_HDR_LEN 1024

would in theory work, if it was at the correct place, which is right before where the relevant buffer is declared (somewhere in esp-idf, possibly esp_http_server.h, but probably in some .c file).

 

What you'd need to do is to change the value defined in sdkconfig.h and recompile the relevant files. Normally you'd do that using

make menuconfig

, changing the relevant options and recompiling your firmware. The link to espressifs documentation you found has all the details.

However apparently esp32-arduino ships a precompiled esp-idf (according to this forum post I found at least: https://esp32.com/viewtopic.php?t=5225 , I've never used arduino myself), so as is pointed out there, there's no way for you to change the definitions easily. You'd need to compile your own esp-idf. The above forum post has a link to https://github.com/espressif/arduino-esp32/blob/master/docs/esp-idf_component.md which has more details on how you'd go about that, but depending on your level of experience this is probably not going to be trivial task. I'm assuming you're running windows, so you'll first need to follow this: https://docs.espressif.com/projects/esp-idf/en/latest/get-started/windows-setup.html         ,     to setup a suitable development environment.

Thanks for your help. Every time I compile the code for upload, the compiler rebuilds everything, so technically I could just edit the sdkconfig.h file with the macro definitions. But right at the top of the file it says :

/*
 *
 * Automatically generated file; DO NOT EDIT.
 * Espressif IoT Development Framework Configuration
 *
 */

Sooo Yea... I'm not sure how I should do the overriding. I guess I could just edit the file and recompile and it should work. But I kindda want to know how to do it properly next time.

Edited July 17, 2019 by colonel_mortis
Clean up quote

[Relations]

 every time i try to quote someone on the forum this happens, every edit just makes it longer and longer.... What did I do wrong...

[zhick]

I'm not sure there is a proper way to achieve this with arduino.

And are you sure the esp-idf is recompiled as well?

You could check this by looking for a file named httpd_main.c in somewhere in your installation and adding a line

Quote

#error This should not compile

.

If building now fails (with error "This should not compile"), you should indeed be able to just change the variable in sdkconfig.h.