Ensure operations on signed integers do not overflow

[yesmar]

There is integer overflow potential in the json_serialization_size and json_serialization_size_pretty functions. On lines 1501 and 1561 the result of the expression res + 1 is cast to a size_t before being returned by the function. Adding 1 to an int with a value of INT_MAX results in undefined behavior. Up casting the result of the expression doesn't mitigate the integer overflow that just happened. It merely promotes the erroneous result to the larger type. Because this is undefined behavior, the C Standard permits compiler implementers to do whatever they want. On some systems INT_MAX + 1 may wrap to 0, but this is not guaranteed behavior (unlike unsigned values, which never overflow; they simply wrap).

The value of res results from a call to json_serialize_to_buffer_r, which returns -1 if an error occurred or, assuming successful serialization, a positive value representing the number of bytes written. It seems possible that a buffer of INT_MAX bytes could occur, therefore the expression should be guarded against overflow.

The simple fix in this case is to cast the res operand to a size_t, which automatically promotes the 1 operand to a size_t. Instead of casting the result, just return the computed value, which will implicitly be a size_t. Even if res was INT_MAX, because it was promoted to a size_t, adding 1 would not cause integer wrap.

See https://wiki.sei.cmu.edu/confluence/display/c/INT32-C.+Ensure+that+operations+on+signed+integers+do+not+result+in+overflow for additional details.