How to handle malloc error?

[xor256]

What happened when the malloc return NULL (no memory)

/* GMP::Integer.allocate */
VALUE integer_c_alloc(VALUE self)
{
	mpz_t* data = malloc(sizeof(mpz_t));
	/* GMP initialization */
	mpz_init(*data);

	return Data_Wrap_Struct(self, NULL, integer_free, data);
}

[silverhammermba]

You can handle it however you like. The "nice" way would be

rb_raise(rb_eNoMemError, "failed to allocate mpz_t");

But if you're actually out of memory I don't even know if that would succeed since Ruby might try to allocate an object for the exception. You could even try fancier stuff like invoking the GC to free up some memory and then retrying, but again the chance of success really depends on how the internals of the VM are set up.

Personally I would find it completely reasonable to do nothing and let the program crash in whatever way it happens to. Recovering from malloc failure is really only possible in a C program because you can structure your code such that the failure path is allocation-free, but when your C code is getting called from within the hugely complicated Ruby VM, all bets are off.