Using XRay Function Patching in a Shared Library

[kavon]

Beyond some linking issues with weak symbols, which I've taken care of, the main issue is that the XRay implementation mades some assumptions about its usage. Currently, to patch a nop sled at function entry points to invoke the handler, XRay will patch in the following code that calls their trampoline:

movd $FuncID, %r10d
call __xray_FunctionEntry

where the call is actually a near call, i.e., CALL rel32 (starting with 0xE8). See here: https://www.felixcloutier.com/x86/call

The problem is that __xray_FunctionEntry is currently linked in as part of the halomon shared library, which is loaded at an address that is too far away from the program's .text section (more than 32-bit relative offset), so we get errors like this:

; non PIE
==11819==XRay Entry trampoline (0x7eff75c60be0) too far from sled (0x000000400660)
; PIE
==11935==XRay Entry trampoline (0x7f18e1167be0) too far from sled (0x5633fe2d1a70)

Potential Solutions:

1. (most preferred) Make halomon a static library. The limitation is only for the call to the trampoline, not the handler's address. However, we will face a similar issue in the future when we want to fully redirect calls to JIT'd code instead of invoking a logging handler. In that case, we can write a different trampoline that simply loads the far function pointer from a jump table indexed by the XRay function ID, and readjusts the stack before doing a far-jump.

2. Change XRay codegen & runtime to use more nops so we can perform the mov immediate, register needed before the call. I'm not sure if this will work on x86-64, though, since you can't move a 64-bit immediate into a register. More thinking would be needed to figure out how to make this work.

[kavon]

This has been fixed through static linking of halomon, which includes parts of XRay.