build: Utilize Modern Compiler Flags to Address Potential Security Issues

[kenny-y]

I've done some search in issues list and I feel I should log a new one.

It's a common best practice to utilize the flags of modern compilers, e.g. the following flags are from GCC

Stack execution protection:                    LDFLAGS="-z noexecstack" 
Data relocation and protection (RELRO):        LDLFAGS="-z relro -z now" 
Stack-based Buffer Overrun Detection:          CFLAGS=”-fstack-protector-strong” if using GCC 4.9 or newer,
                                                                                 otherwise CFLAGS="-fstack-protector"
Position Independent Execution (PIE)           CFLAGS="-fPIE -fPIC" LDFLAGS="-pie" (PIE for executables only)
Fortify source:                                CFLAGS="-O2 -D_FORTIFY_SOURCE=2"
Format string vulnerabilities:                 CFLAGS="-Wformat -Wformat-security"

noexecstack was addressed in #17933

• Version: Future versions
• Platform: All supported platforms
• Subsystem:

I've done a little experiment and found that most of the flags can be directly applied on Linux platform without build errors or loss of basic functionalities. Further test on functionality and the viability on other platforms remains unknown.

[bnoordhuis]

Pull requests welcome. :-)

I've investigated most of these over the years. Performance concerns and toolchain issues make it complicated.

Data relocation and protection (RELRO)

I forgot the details but IIRC -z now didn't play well with add-ons.

Stack-based Buffer Overrun Detection

I believe this was blocked on clang 3.4 not supporting -fstack-protector. A quick git log of clang's source tree suggests that 3.4.2 (our current baseline) does so this could probably be enabled.

Position Independent Execution (PIE)

Was too costly on some architectures. Would have to be benchmarked and checked if it interacts properly with add-ons.

Fortify source

I think this was blocked on one of our bundled dependencies (openssl or v8?) not being compatible with -D_FORTIFY_SOURCE=2. Perhaps -D_FORTIFY_SOURCE=1 would work.

Format string vulnerabilities

Covered by -Wall -Wextra.

[tingshao]

I created a new issue #20122 to track the Data Relocation and Protection (RELRO) flag to make it independent and more clear to track. Further discussions could be put there, and I also presents some testing results, thanks. @kenny-y @bnoordhuis

[tingshao]

I created a new issue #20928 to track the Stack-based Buffer Overrun Detection issue, let's put further discussions there, thanks.

[kenny-y]

Close this since all things can be done are already done.