[Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]

[Martijn Pieters]

On Tue, Jul 21, 2009 at 18:40, Tres Seaver <tseaver at palladion.com> wrote:
>   File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
>     line 332, in testMultipleRangesBigFileOutOfOrder
>     (70000, 80001)])
>   File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
>     line 209, in expectMultipleRanges
>     str(len(body)), rsp.getHeader('content-length')))
>   File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in
>     failIf
>     if expr: raise self.failureException, msg
>  AssertionError: Incorrect Content-Length is set! Expected 20425,
>     got 20426.
>
> I don't grok the range support code at all:  probably Martijn Pieters is
> the only person in the world who does.  The tests are quite obscure to
> me, so I can't diagnose even whether the failure is a testing artifact
> or a real bug.

Here's my response to Tres; for some reason my mail reader didn't
include zope-dev. I can add now that I'll be at a sprint this weekend
where I'll probably have time to take a look myself:

----8<------------------

Sorry you find them obscure; I followed unit test expectations as best
as I could.

The test that fails does the following:

1) Create a big file (uploadBigFile, enough random data to be more
than one pdata chunk)
2) Request for multiple HTTP ranges (slices) of the file to be sent.
These slices should be returned a) in the same order, b) of the
correct length, and of course c) the correct slice of the big file
created in 1.
In this case line 330 specifies that bytes 10-15 (inclusive), the last
10000 bytes, and bytes 70000-80000 (inclusive) are are to be returned.
The call to expectMultipleRanges includes the expected slices (python
syntax, so end value is exclusive).
3) On line 209 of expectMultipleRanges it then tests if the resulting
response has the correct content length header for the whole response
set. The failure reported is that the Content-Length header reports 1
byte more than what was actually sent, so the body was 20425 bytes
long, while the content-header claims it would be 20426 bytes.

Because this is a multi-part range request, Image.py line 277 and
onwards handle this request. The response ends up looking something
like this:

  Content-length: size we pre-calculate
  Content-type: multipart/byteranges; boundary="boundarystring"
  More headers

  --boundarystring
  Content-type: image/whatever
  Content-range: bytes start-end/total-file-size

  .... data ...

  --boundarystring
  ... Another section
  ... etc.

  --boundarystring--

There are 2 ways the failure could be caused.

Something could go wrong with the size calculation on lines 280-287;
the total body length is the length of the range data plus the length
of the headers per multi-part plus the length of the MIME boundaries
between them. There is thus a length per part plus the length of the
last boundary.

Or something goes wrong when writing the range data in lines 309-358.
Because there is only one byte difference between what was calculated
on lines 280-287 and what is produced here this is probably a newline
somewhere.

I do find it puzzling this only fails on 64 bit platforms. It could be
that the pdata chunk handling is borken and has a off-by-one error
when running on 64-bit platforms.

Hope this clarifies things a little. I have little time right now to
chase this myself right now, sorry!

----8<------------------

-- 
Martijn Pieters