string functions overrun supplied buffers

[alexfru]

It looks like a number of functions from <string.h> and similar originally had the length/size/count parameters of type int (or ssize_t) instead of unsigned (or size_t). Perhaps, the original code was written using careless K&R syntax omitting explicit parameter type declarations and defaulting all such parameters to int. But then the code was changed to explicitly giving the parameters type size_t without checking the logic.

So, for example, this is what we have now in memcpy():

void *
memcpy (vt, vf, n)
    void *vt;
    const void *vf;
    register size_t n;
{
    register char *t = vt;
    register const char *f = vf;

    while (--n >= 0)
        *t++ = *f++;

    return vt;
}

Obviously, the condition in while is always going to be evaluated to 1 (since n and --n is unsigned and greater than or equal to zero always by definition), and thus the loop turns into an infinite loop overwriting memory and crashing the program (or the system).

Now, we probably haven't seen this happen because gcc recognizes calls to memset() and the like and normally chooses to generate function code in place instead of generating simple function calls, so whatever memset() ends up in libc.a, it typically has no effect whatsoever on the program compiled with gcc and using the function.

Here is a short list of the functions suffering from this bug:

src/libc/compat:
memccpy()
memchr()
memcmp()
memcpy()
memset()

src/libc/gen:
strncasecmp()
strncat()

Also, there are some functions that improperly mix and match signed and unsigned integer types and should ideally be fixed, but are probably OK to be left untouched for now since the sizes/counts are limited by the amount of RAM of our system:

src/libc/gen:
strncpy()
qsort()

There may be other instances of this bug. I haven't dug further, just looked around the obvious places once discovered that memset(ptr, 0, 1); crashes the program for no apparent reason.

This must be fixed for Smaller C and calls from manually written assembly code.

[sergev]

Good catch. It may also explain all these issues with gcc-4.7.2.

[alexfru]

I doubt it. This code has been broken for a very long time and had no ill effects on builds with 4.7.2.

[sergev]

Fixed in bca48c9