Overflow error (was Vol 67, Issue 192)

[Dave Angel]

Ryniek90 wrote:
> <div class="moz-text-flowed" style="font-family: -moz-fixed">Chris 
> Rebert pisze:
>> On Sun, Apr 12, 2009 at 3:38 PM, Ryniek90 <ryniek90 at gmail.com> wrote:
>>>> Paul Rubin <http://phr.cx@NOSPAM.invalid>
>>>> Ryniek90 <ryniek90 at gmail.com> writes:
>>>>
>>>>> When i wanted to send an .iso file of 4GB length, i had traceback:
>>>>> "OverflowError: requested number of bytes is more than a Python 
>>>>> string
>>>>> can hold"
>>>>>
>>>> You're not supposed to put the 4GB all in one string. Open the
>>>> socket and send smaller packets through it.
>>>>
>>>> ------------------------------------------------------------------------ 
>>>>
>>> Ok, so i will split that data for smaller files. But i've still 
>>> haven't got
>>> answer for question: "What's the max. length of string bytes which 
>>> Python
>>> can hold?
>>
>> sys.maxsize
>> The largest positive integer supported by the platform’s
>> Py_ssize_t type, and thus the maximum size lists, strings, dicts, and
>> many other containers can have.
>>
>> Cheers,
>> Chris
>>
>
> Thanks. I've wanted to check very carefully what's up, and i found 
> this: "strings (currently restricted to 2GiB)".
> It's here, in PEP #353 (PEP 0353 
> <http://www.python.org/dev/peps/pep-0353/>). Besides of this, i've 
> found in sys module's docstring this:
>
> maxint = 2147483647
> maxunicode = 1114111
>
> Which when added gives us 2148597758.0 bytes, which are equal to 
> 2049.0624980926514 MiB's.
> So files larger than 2049.06 MiB's should be splitted into smaller 
> ones and sent partially.
> Correct me if i'm wrong.
>
>
> </div>
>
How much RAM is in your system? Unless it's at least 50 gb, in a 64bit 
OS, I'd keep my max chunk size to much smaller than 2gb. For a typical 
32bit system with 2 to 4gb of RAM, I'd probably chunk the file a meg or 
so at a time. Using large sizes is almost always a huge waste of resources.