How to stop Apache stripping the Content-Length header created by a cgi-bin script ?
by Mike Davies from LinuxQuestions.org on (#6NYHS)
I am trying to get Apache to handle OTA updates for some ESP8266 based systems.
This used to work OK and then someone wondered why their ESP systems were not working like they should and we discovered that the OTA updates no longer worked, quite possibly for years.
The OTA update is handled by a Perl script. Once a day the ESPs fire off a HTTP request to our server to run a cgi-bin script. That talks to Apache like this...
Code:print "Status: 200 OK\n";
print "Content-Type: application/octet-stream\n";
print "Content-Disposition: attachment; filename=$firmware\n";
my $file_size= -s $firmware;
print "Content-Length: $file_size\n";
print "x-MD5: $new_md5\n\n";
open my $fh, '<:raw', $firmware or die "Cannot open $firmware - $!\n";
my $buffer;
print $buffer while
(read ($fh,$buffer,65536));
close ($fh);Obviously, there's a bit more to it than that, validating that the caller is allowed to run the script, checking if an update is required, and deciding which filename to put in $firmware, and the corresponding MD5, but these few lines of code used to work OK if an update was required.
What happens now if we run this code, is that Apache strips out the 'Content-Length', so at the receiving end, the ESP does not know how big the file is, and decides to fail the update because it doesn't know how much memory it needs to allocate for the update. The receive goes like this ...
Code:15:55:06.353 -> -----
15:55:06.353 -> [HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
15:55:06.353 -> [HTTP-Client][handleHeaderResponse] RX: 'Date: Tue, 02 Jul 2024 14:55:06 GMT'
15:55:06.353 -> [HTTP-Client][handleHeaderResponse] RX: 'Server: Apache/2.4.59 (Debian)'
15:55:06.386 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Disposition: attachment; filename=/var/www/firmware/latest-16126456.bin'
15:55:06.386 -> [HTTP-Client][handleHeaderResponse] RX: 'x-MD5: 891389faa5b64affacbd1a80e4417165'
15:55:06.386 -> [HTTP-Client][handleHeaderResponse] RX: 'Connection: close'
15:55:06.386 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/octet-stream'
15:55:06.420 -> [HTTP-Client][handleHeaderResponse] RX: ''
15:55:06.420 -> [HTTP-Client][handleHeaderResponse] code: 200
15:55:06.420 -> [httpUpdate] Header read fin.
15:55:06.420 -> [httpUpdate] Server header:
15:55:06.420 -> [httpUpdate] - code: 200
15:55:06.420 -> [httpUpdate] - len: -1
15:55:06.420 -> [httpUpdate] - MD5: 891389faa5b64affacbd1a80e4417165
15:55:06.420 -> [httpUpdate] ESP8266 info:
15:55:06.420 -> [httpUpdate] - free Space: 1519616
15:55:06.420 -> [httpUpdate] - current Sketch Size: 577344
15:55:06.453 -> [httpUpdate] - current version: Unknown
15:55:06.453 -> [httpUpdate] Content-Length was 0 or wasn't set by Server?!
15:55:06.453 -> [HTTP-Client][end] tcp stop
15:55:06.453 -> [OTA update] Update failed.As I said, this all used to work OK with a custom build of Apache 2.4.48, and we're now running a Debian based server with Apache 2.4.59. We do not know if that is related to the problem.
The WWW appears to have very little information about setting up an Apache server to do this task, and the fellow that set all this up died during Covid, so we are at a loss as to what to try next.
I have read that 'Content-Length' gets stripped for chunked transfers (which makes sense), but it cannot be doing a chuncked transfer because I print out all the headers I receive on the ESP.
I tried various tips like adding a "Transfer-Encoding: identity" but that gets stripped out too.
How do I stop Apache stripping out the 'Content-Length' header ?
This used to work OK and then someone wondered why their ESP systems were not working like they should and we discovered that the OTA updates no longer worked, quite possibly for years.
The OTA update is handled by a Perl script. Once a day the ESPs fire off a HTTP request to our server to run a cgi-bin script. That talks to Apache like this...
Code:print "Status: 200 OK\n";
print "Content-Type: application/octet-stream\n";
print "Content-Disposition: attachment; filename=$firmware\n";
my $file_size= -s $firmware;
print "Content-Length: $file_size\n";
print "x-MD5: $new_md5\n\n";
open my $fh, '<:raw', $firmware or die "Cannot open $firmware - $!\n";
my $buffer;
print $buffer while
(read ($fh,$buffer,65536));
close ($fh);Obviously, there's a bit more to it than that, validating that the caller is allowed to run the script, checking if an update is required, and deciding which filename to put in $firmware, and the corresponding MD5, but these few lines of code used to work OK if an update was required.
What happens now if we run this code, is that Apache strips out the 'Content-Length', so at the receiving end, the ESP does not know how big the file is, and decides to fail the update because it doesn't know how much memory it needs to allocate for the update. The receive goes like this ...
Code:15:55:06.353 -> -----
15:55:06.353 -> [HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
15:55:06.353 -> [HTTP-Client][handleHeaderResponse] RX: 'Date: Tue, 02 Jul 2024 14:55:06 GMT'
15:55:06.353 -> [HTTP-Client][handleHeaderResponse] RX: 'Server: Apache/2.4.59 (Debian)'
15:55:06.386 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Disposition: attachment; filename=/var/www/firmware/latest-16126456.bin'
15:55:06.386 -> [HTTP-Client][handleHeaderResponse] RX: 'x-MD5: 891389faa5b64affacbd1a80e4417165'
15:55:06.386 -> [HTTP-Client][handleHeaderResponse] RX: 'Connection: close'
15:55:06.386 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/octet-stream'
15:55:06.420 -> [HTTP-Client][handleHeaderResponse] RX: ''
15:55:06.420 -> [HTTP-Client][handleHeaderResponse] code: 200
15:55:06.420 -> [httpUpdate] Header read fin.
15:55:06.420 -> [httpUpdate] Server header:
15:55:06.420 -> [httpUpdate] - code: 200
15:55:06.420 -> [httpUpdate] - len: -1
15:55:06.420 -> [httpUpdate] - MD5: 891389faa5b64affacbd1a80e4417165
15:55:06.420 -> [httpUpdate] ESP8266 info:
15:55:06.420 -> [httpUpdate] - free Space: 1519616
15:55:06.420 -> [httpUpdate] - current Sketch Size: 577344
15:55:06.453 -> [httpUpdate] - current version: Unknown
15:55:06.453 -> [httpUpdate] Content-Length was 0 or wasn't set by Server?!
15:55:06.453 -> [HTTP-Client][end] tcp stop
15:55:06.453 -> [OTA update] Update failed.As I said, this all used to work OK with a custom build of Apache 2.4.48, and we're now running a Debian based server with Apache 2.4.59. We do not know if that is related to the problem.
The WWW appears to have very little information about setting up an Apache server to do this task, and the fellow that set all this up died during Covid, so we are at a loss as to what to try next.
I have read that 'Content-Length' gets stripped for chunked transfers (which makes sense), but it cannot be doing a chuncked transfer because I print out all the headers I receive on the ESP.
I tried various tips like adding a "Transfer-Encoding: identity" but that gets stripped out too.
How do I stop Apache stripping out the 'Content-Length' header ?