Issue 4935: Segmentation fault in bytearray tests

This happens on a 32-bit build on a 64-bit system, which happens to have
some interesting properties: for example, malloc() will happily allocate
memory larger than Py_SSIZE_T_MAX.

The crash is exactly triggered by the following snippet:

        if sys.maxsize < (1 << 32) and struct.calcsize('P') == 4:
            self.assertRaises(OverflowError,
                              self.marshal(b'\ta\n\tb').expandtabs,
sys.maxsize)

Here is a patch. The idea is to use unsigned arithmetic instead of
signed, and to check for overflows by comparing with PY_SSIZE_T_MAX.

(the exact reason of the crash is unknown, it looks like either a
compiler bug or a mis-optimization as (i < 0) returns 0 while i is
-0x7FFFFFFF)

What's the purpose of old_i?  It looks like it's never used for anything.

Other than than, the patch looks good to me.

I'd guess that the "if (i < 0)" was simply optimized away.  This isn't 
necessarily a compiler bug: if I understand correctly, a strict reading of 
the C standards says it's legitimate for a compiler to assume that code is 
written in such a way that signed-arithmetic overflow never happens, and 
gcc (for one) is known to take advantage of this.

Also, it would be nice to cleanup the whitespace in this function while 
you're fixing it;  at the moment it's showing me a mixture of tabs and 
spaces.

Actually, what's the purpose of old_j?  It looks like that's not needed 
any more, either!

You are right, old_i and old_j are unused, they were part of another
approach I tried and which failed.
Attaching new patch with these 2 variables removed, and the function
cleanly reindented (of course the patch is messier because of this).

Looks fine;  I think this should be applied.

It seems as though your reindentation left trailing whitespace on
the blank lines in the function:  the svn commit hook might
complain about this...

Committed in trunk, py3k, 2.6 and 3.0. Thanks!