GEOSLineSubstring segfault with fraction = NaN

[gabe-sherman]

A segmentation fault occurs in the below program. This behavior occurs at line 2401 in geos_ts_c.cpp.

#include "geos_c.h"
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <math.h>
typedef uint8_t   u8;
typedef uint16_t  u16;
typedef uint32_t  u32;
typedef uint64_t  u64;
typedef unsigned int usize;
typedef int8_t  i8;
typedef int16_t i16;
typedef int32_t i32;
typedef int64_t i64;
typedef int isize;
typedef float f32;
typedef double f64;
int main() {
    struct GEOSContextHandle_HS *v0 = GEOS_init_r(); // handle
    if (v0 == NULL) return 0;
    struct GEOSContextHandle_HS *v2 = v0; // handle
    struct GEOSGeom_t *v3 = GEOSGeom_createEmptyLineString_r(v2); // g
    if (v3 == NULL) return 0;
    struct GEOSGeom_t *v5 = v3; // g
    f64 v6 = NAN; // start_fraction
    f64 v7 = 0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000022250738585072014; // end_fdraction
    struct GEOSGeom_t *v8 = GEOSLineSubstring_r(v2, v5, v6, v7); // $target
}

Test Environment

Ubuntu 22.04, 64bit

How to trigger

./filename

Version

Latest: c8b889b

Address Sanitizer Output

AddressSanitizer:DEADLYSIGNAL
=================================================================
==2233847==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x555555973e0b bp 0x7fffffffd6f0 sp 0x7fffffffd540 T0)
==2233847==The signal is caused by a READ memory access.
==2233847==Hint: address points to the zero page.
    #0 0x555555973e0b in GEOSLineSubstring_r::$_116::operator()() const /home/gabesherman/harness_test/AutoHarn-Evaluation/geos/lib_asan/capi/geos_ts_c.cpp:2401:18
    #1 0x555555973e0b in _Z7executeIZ19GEOSLineSubstring_rE5$_116LDn0EEDTclfp0_EEP20GEOSContextHandle_HSOT_ /home/gabesherman/harness_test/AutoHarn-Evaluation/geos/lib_asan/capi/geos_ts_c.cpp:430:16
    #2 0x55555597386c in GEOSLineSubstring_r /home/gabesherman/harness_test/AutoHarn-Evaluation/geos/lib_asan/capi/geos_ts_c.cpp:2388:16
    #3 0x55555592d161 in main /home/gabesherman/harness_test/AutoHarn-Results/geos/hopper-03/reproducer.c:29:29
    #4 0x7ffff7029d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #5 0x7ffff7029e3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #6 0x55555586f524 in _start (/home/gabesherman/harness_test/AutoHarn-Results/geos/hopper-03/reproducer+0x31b524) (BuildId: e9e6f2449c2164c0b3993bd653651a89d488693e)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/gabesherman/harness_test/AutoHarn-Evaluation/geos/lib_asan/capi/geos_ts_c.cpp:2401:18 in GEOSLineSubstring_r::$_116::operator()() const
==2233847==ABORTING

[dbaston]

Simplified test case:

// NaN start fraction
// https://github.com/libgeos/geos/issues/1077
template<>
template<>
void object::test<7>
()
{
    input_ = fromWKT("LINESTRING EMPTY");
    double start = std::numeric_limits<double>::quiet_NaN();
    double end = 0.1;
    result_ = GEOSLineSubstring(input_, start, end);

    ensure(result_ == nullptr);
}

[dr-jts]

Does this indicate we should sanitize all numeric inputs (e.g. by replacing NaN with 0, or else throwing)?

What do other C libs do?

[dbaston]

I think we just need more testing of NaN inputs. I don't think there's a universal right way to handle them.