/re2

Ruby bindings to RE2, a "fast, safe, thread-friendly alternative to backtracking regular expression engines like those used in PCRE, Perl, and Python".

Primary LanguageRubyBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

re2 - safer regular expressions in Ruby

Ruby bindings to RE2, a "fast, safe, thread-friendly alternative to backtracking regular expression engines like those used in PCRE, Perl, and Python".

Build Status

Current version: 2.14.0
Bundled RE2 version: libre2.11 (2024-07-02)

RE2('h.*o').full_match?("hello")    #=> true
RE2('e').full_match?("hello")       #=> false
RE2('h.*o').partial_match?("hello") #=> true
RE2('e').partial_match?("hello")    #=> true
RE2('(\w+):(\d+)').full_match("ruby:1234")
#=> #<RE2::MatchData "ruby:1234" 1:"ruby" 2:"1234">

Table of Contents

Why RE2?

While recent versions of Ruby have improved defences against regular expression denial of service (ReDoS) attacks, it is still possible for users to craft malicious patterns that take a long time to process by using syntactic features such as back-references, lookaheads and possessive quantifiers. RE2 aims to eliminate ReDoS by design:

Safety is RE2's raison d'être.

RE2 was designed and implemented with an explicit goal of being able to handle regular expressions from untrusted users without risk. One of its primary guarantees is that the match time is linear in the length of the input string. It was also written with production concerns in mind: the parser, the compiler and the execution engines limit their memory usage by working within a configurable budget – failing gracefully when exhausted – and they avoid stack overflow by eschewing recursion.

— Why RE2?

Usage

Install re2 as a dependency:

# In your Gemfile
gem "re2"

# Or without Bundler
gem install re2

Include in your code:

require "re2"

Full API documentation automatically generated from the latest version is available at https://mudge.name/re2/.

While re2 uses the same naming scheme as Ruby's built-in regular expression library (with Regexp and MatchData), its API is slightly different:

Compiling regular expressions

Warning

RE2's regular expression syntax differs from PCRE and Ruby's built-in Regexp library, see the official syntax page for more details.

The core class is RE2::Regexp which takes a regular expression as a string and compiles it internally into an RE2 object. A global function RE2 is available to concisely compile a new RE2::Regexp:

re = RE2('(\w+):(\d+)')
#=> #<RE2::Regexp /(\w+):(\d+)/>
re.ok? #=> true

re = RE2('abc)def')
re.ok?   #=> false
re.error #=> "missing ): abc(def"

Tip

Note the use of single quotes when passing the regular expression as a string to RE2 so that the backslashes aren't interpreted as escapes.

When compiling a regular expression, an optional second argument can be used to change RE2's default options, e.g. stop logging syntax and execution errors to stderr with log_errors:

RE2('abc)def', log_errors: false)

See the API documentation for RE2::Regexp#initialize for all the available options.

Matching interface

There are two main methods for matching: RE2::Regexp#full_match? requires the regular expression to match the entire input text, and RE2::Regexp#partial_match? looks for a match for a substring of the input text, returning a boolean to indicate whether a match was successful or not.

RE2('h.*o').full_match?("hello")    #=> true
RE2('e').full_match?("hello")       #=> false

RE2('h.*o').partial_match?("hello") #=> true
RE2('e').partial_match?("hello")    #=> true

Submatch extraction

Tip

Only extract the number of submatches you need as performance is improved with fewer submatches (with the best performance when avoiding submatch extraction altogether).

Both matching methods have a second form that can extract submatches as RE2::MatchData objects: RE2::Regexp#full_match and RE2::Regexp#partial_match.

m = RE2('(\w+):(\d+)').full_match("ruby:1234")
#=> #<RE2::MatchData "ruby:1234" 1:"ruby" 2:"1234">

m[0] #=> "ruby:1234"
m[1] #=> "ruby"
m[2] #=> "1234"

m = RE2('(\w+):(\d+)').full_match("r")
#=> nil

RE2::MatchData supports retrieving submatches by numeric index or by name if present in the regular expression:

m = RE2('(?P<word>\w+):(?P<number>\d+)').full_match("ruby:1234")
#=> #<RE2::MatchData "ruby:1234" 1:"ruby" 2:"1234">

m["word"]   #=> "ruby"
m["number"] #=> "1234"

They can also be used with Ruby's pattern matching:

case RE2('(\w+):(\d+)').full_match("ruby:1234")
in [word, number]
  puts "Word: #{word}, Number: #{number}"
else
  puts "No match"
end
# Word: ruby, Number: 1234

case RE2('(?P<word>\w+):(?P<number>\d+)').full_match("ruby:1234")
in word:, number:
  puts "Word: #{word}, Number: #{number}"
else
  puts "No match"
end
# Word: ruby, Number: 1234

By default, both full_match and partial_match will extract all submatches into the RE2::MatchData based on the number of capturing groups in the regular expression. This can be changed by passing an optional second argument when matching:

m = RE2('(\w+):(\d+)').full_match("ruby:1234", submatches: 1)
=> #<RE2::MatchData "ruby:1234" 1:"ruby">

Warning

If the regular expression has no capturing groups or you pass submatches: 0, the matching method will behave like its full_match? or partial_match? form and only return true or false rather than RE2::MatchData.

Scanning text incrementally

If you want to repeatedly match regular expressions from the start of some input text, you can use RE2::Regexp#scan to return an Enumerable RE2::Scanner object which will lazily consume matches as you iterate over it:

scanner = RE2('(\w+)').scan(" one two three 4")
scanner.each do |match|
  puts match.inspect
end
# ["one"]
# ["two"]
# ["three"]
# ["4"]

Searching simultaneously

RE2::Set represents a collection of regular expressions that can be searched for simultaneously. Calling RE2::Set#add with a regular expression will return the integer index at which it is stored within the set. After all patterns have been added, the set can be compiled using RE2::Set#compile, and then RE2::Set#match will return an array containing the indices of all the patterns that matched.

set = RE2::Set.new
set.add("abc")         #=> 0
set.add("def")         #=> 1
set.add("ghi")         #=> 2
set.compile            #=> true
set.match("abcdefghi") #=> [0, 1, 2]
set.match("ghidefabc") #=> [2, 1, 0]

Encoding

Warning

Note RE2 only supports UTF-8 and ISO-8859-1 encoding so strings will be returned in UTF-8 by default or ISO-8859-1 if the :utf8 option for the RE2::Regexp is set to false (any other encoding's behaviour is undefined).

For backward compatibility: re2 won't automatically convert string inputs to the right encoding so this is the responsibility of the caller, e.g.

# By default, RE2 will process patterns and text as UTF-8
RE2(non_utf8_pattern.encode("UTF-8")).match(non_utf8_text.encode("UTF-8"))

# If the :utf8 option is false, RE2 will process patterns and text as ISO-8859-1
RE2(non_latin1_pattern.encode("ISO-8859-1"), utf8: false).match(non_latin1_text.encode("ISO-8859-1"))

Requirements

This gem requires the following to run:

It supports the following RE2 ABI versions:

  • libre2.0 (prior to release 2020-03-02) to libre2.11 (2023-07-01 to 2024-07-02)

Native gems

Where possible, a pre-compiled native gem will be provided for the following platforms:

Verifying the gems

SHA256 checksums are included in the release notes for each version and can be checked with sha256sum, e.g.

$ gem fetch re2 -v 2.14.0
Fetching re2-2.14.0-arm64-darwin.gem
Downloaded re2-2.14.0-arm64-darwin
$ sha256sum re2-2.14.0-arm64-darwin.gem
3c922d54a44ac88499f6391bc2f9740559381deaf7f4e49eef5634cf32efc2ce  re2-2.14.0-arm64-darwin.gem

GPG signatures are attached to each release (the assets ending in .sig) and can be verified if you import our signing key 0x39AC3530070E0F75 (or fetch it from a public keyserver, e.g. gpg --keyserver keyserver.ubuntu.com --recv-key 0x39AC3530070E0F75):

$ gpg --verify re2-2.14.0-arm64-darwin.gem.sig re2-2.14.0-arm64-darwin.gem
gpg: Signature made Fri  2 Aug 12:39:12 2024 BST
gpg:                using RSA key 702609D9C790F45B577D7BEC39AC3530070E0F75
gpg: Good signature from "Paul Mucur <mudge@mudge.name>" [unknown]
gpg:                 aka "Paul Mucur <paul@ghostcassette.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7026 09D9 C790 F45B 577D  7BEC 39AC 3530 070E 0F75

The fingerprint should be as shown above or you can independently verify it with the ones shown in the footer of https://mudge.name.

Installing the ruby platform gem

Warning

We strongly recommend using the native gems where possible to avoid the need for compiling the C++ extension and its dependencies which will take longer and be less reliable.

If you wish to compile the gem, you will need to explicitly install the ruby platform gem:

# In your Gemfile with Bundler 2.3.18+
gem "re2", force_ruby_platform: true

# With Bundler 2.1+
bundle config set force_ruby_platform true

# With older versions of Bundler
bundle config force_ruby_platform true

# Without Bundler
gem install re2 --platform=ruby

You will need a full compiler toolchain for compiling Ruby C extensions (see Nokogiri's "The Compiler Toolchain") plus the toolchain required for compiling the vendored version of RE2 and its dependency Abseil which includes CMake, a compiler with C++14 support such as clang 3.4 or gcc 5 and a recent version of pkg-config. On Windows, you'll also need pkgconf 2.1.0+ to avoid undefined reference errors when attempting to compile Abseil.

Using system libraries

If you already have RE2 installed, you can instruct the gem not to use its own vendored version:

gem install re2 --platform=ruby -- --enable-system-libraries

# If RE2 is not installed in /usr/local, /usr, or /opt/homebrew:
gem install re2 --platform=ruby -- --enable-system-libraries --with-re2-dir=/path/to/re2/prefix

Alternatively, you can set the RE2_USE_SYSTEM_LIBRARIES environment variable instead of passing --enable-system-libraries to the gem command.

Thanks

  • Thanks to Jason Woods who contributed the original implementations of RE2::MatchData#begin and RE2::MatchData#end.
  • Thanks to Stefano Rivera who first contributed C++11 support.
  • Thanks to Stan Hu for reporting a bug with empty patterns and RE2::Regexp#scan, contributing support for libre2.11 (2023-07-01) and for vendoring RE2 and abseil and compiling native gems in 2.0.
  • Thanks to Sebastian Reitenbach for reporting the deprecation and removal of the utf8 encoding option in RE2.
  • Thanks to Sergio Medina for reporting a bug when using RE2::Scanner#scan with an invalid regular expression.
  • Thanks to Pritam Baral for contributing the initial support for RE2::Set.
  • Thanks to Mike Dalessio for reviewing the precompilation of native gems in 2.0.
  • Thanks to Peter Zhu for ruby_memcheck and helping find the memory leaks fixed in 2.1.3.
  • Thanks to Jean Boussier for contributing the switch to Ruby's TypedData API and the resulting garbage collection improvements in 2.4.0.
  • Thanks to Manuel Jacob for reporting a bug when passing strings with null bytes.

Contact

All issues and suggestions should go to GitHub Issues.

License

This library is licensed under the BSD 3-Clause License, see LICENSE.txt.

Copyright © 2010, Paul Mucur.

Dependencies

The source code of RE2 is distributed in the ruby platform gem. This code is licensed under the BSD 3-Clause License, see LICENSE-DEPENDENCIES.txt.

The source code of Abseil is distributed in the ruby platform gem. This code is licensed under the Apache License 2.0, see LICENSE-DEPENDENCIES.txt.