Infinite loop for certain regular expressions

[Sjlver]

Calling agrep '\\)' results in an infinite loop. Agrep allocates memory inside the loop, and will quickly run out of memory.

Compiling agrep with debugging enabled gives the following output:

tre_compile: parsing '\\)'
tre_parse: parsing '\\)', len = 3
tre_parse:  bleep: '\\)'
tre_parse:     escaped: '\\)'
tre_mem_alloc: allocating new 1024 byte block
tre_parse:          empty: ')'
tre_parse:          empty: ')'
tre_parse:          empty: ')'
tre_parse:          empty: ')'
tre_parse:          empty: ')'
tre_parse:          empty: ')'
tre_parse:          empty: ')'
tre_mem_alloc: allocating new 1024 byte block
tre_parse:          empty: ')'
...

[Sjlver]

I believe that the following patch fixes the issue:

diff --git a/lib/tre-parse.c b/lib/tre-parse.c
index e113896..19bad4a 100644
--- a/lib/tre-parse.c
+++ b/lib/tre-parse.c
@@ -1346,8 +1346,9 @@ tre_parse(tre_parse_ctx_t *ctx)
              break;
 
            case CHAR_RPAREN:  /* end of current subexpression */
-             if ((ctx->cflags & REG_EXTENDED && depth > 0)
-                 || (ctx->re > ctx->re_start
+             if (((ctx->cflags & REG_EXTENDED) && depth > 0)
+                 || (!(ctx->cflags & REG_EXTENDED)
+                      && ctx->re > ctx->re_start
                      && *(ctx->re - 1) == CHAR_BACKSLASH))
                {
                  DPRINT(("tre_parse:       empty: '%.*" STRF "'\n",

[Sjlver]

This issue was found using LLVM's LibFuzzer.

[dag-erling]

Fixed in 6739b91.