mach_msg fails to copy out-of-line data when length >= 512MB

[Sergio López]

Hi,

Sending messages with out-of-line data length equal or greater than
512 MB fails silently. The message arrives to the receiver, dataCnt
indicates the amount pointed by the sender, but the buffer is not
mapped into the receiver's space. This usually results in a unsolved
page fault in the receiver when trying to access to addresses that
should be covered by that buffer.

The problem is located in ipc_kmsg.c:1422 and ipc_kmsg.c:2416, which
both read as this:

length = ((number * size) + 7) >> 3;

length is a local variable, number is data length, and size is the
size for each element (?), 8 in our case. All those are 32 bits vars,
so an integer overflow arises for values equal or greater than
512*1024*1024.

This can be fixed by changing length to be an unsigned long long, and
casting number to that type for this operation. But an interesting
related question is if users should be allowed to send an arbitrary
amount of OOL data with each message. Perhaps an static limit should
be imposed, returning some error for mach_msg, and glibc changed to
cope with this situations.

What do you think?