Trouble splitting string with Regexp

Question

Trouble splitting string with Regexp

Closed this issue 9 years ago · 50 comments

Hi,

I'm having some issues splitting strings with Regexps in mruby-onig-regexp

The following:

"<%= 1 + 1 %>".split(/(<%=)|(%>)/)

doesn't split and returns ["<%= 1 + 1 %>"]. The same expression splits fine in MRI. I checked that mruby-onig-regexp substitutes String#split with a Regexp-aware implementation, so either I'm doing something seriously wrong or maybe there's a bug in mruby-onig-regexp

thanks,
Ricardo

Answer 1 · 2013-04-02T00:20:29.000Z

Also, this returns nil, which is a bit strange:

/(%)/.match("<%= 1 + 1 %>")

Answer 2 · 2013-04-02T00:27:31.000Z

Another thing that may be relevant, I'm getting a segfault with the following code (in mirb) adapted from the docs:

Regexp.compile('abc').match('abcdef')

Answer 3 · 2013-04-02T00:32:13.000Z

I get following

Answer 4 · 2013-04-02T00:33:11.000Z

Are you using mruby-require? Then do require 'mruby-onig-regexp'. If not, you will get same result as mine.

Answer 5 · 2013-04-02T10:13:20.000Z

Hi,

Thanks for your answer.

Requiring mruby-onig-regexp does improve things, but not completely. For example, this no longer returns nil and matches as supposed:

Regexp.compile('abc').match('abcdef')

but

"<%= 1 + 1 %>".split(/(<%=)|(%>)/)

still doesn't split properly (returns an array with a single item instead of 4 items). The same expression works well in MRI.

thanks,
Ricardo

Answer 6 · 2013-04-02T10:54:32.000Z

Ah, I understand your said just now. It seems broken. I'll fix in later.
Try to use mruby-pcre-regexp.

Answer 7 · 2013-04-02T11:01:34.000Z

I tried that, but mruby-pcre-regexp isn't compiling at this time (against mruby/master). I get the following error:

build/mrbgems/mruby-pcre-regexp//src/mruby_pcre_regexp.c:168: error: ‘PCRE_CASELESS’ undeclared (first use in this function)

Answer 8 · 2013-04-02T11:06:49.000Z

PCRE_CASELESS is defined as 0x00000001 in my pcre.h

Below is a part of version information.

#define PCRE_MAJOR          8
#define PCRE_MINOR          32
#define PCRE_PRERELEASE     
#define PCRE_DATE           2012-11-30

Answer 9 · 2013-04-02T15:00:24.000Z

didn't have pcre properly installed, it's working now. it doesn't really give the same result as MRI, but does split the string (returns ["", " 1 + 1 ", ""]). I'll recheck later tonight and get back to you.

Answer 10 · 2013-04-03T01:11:06.000Z

Hi,

PCRE does work better, but since the regular expressions I'm using were written for oniguruma, it doesn't split exactly how I need it to. since I'm porting existing software, it wouldn't be ideal to change the regexps.

the ideal for me would be to use oniguruma, if the bug above is fixed, otherwise I'll change the regexps to use PCRE (or I'll try fixing the oniguruma bug).

thanks,
Ricardo

Answer 11 · 2013-04-03T01:16:10.000Z

Could you please write test case that you expected?

Answer 12 · 2013-04-03T09:17:24.000Z

ok, I'll do that. I tried running mrbtest, but it segfaults. is there any way to fix it, or should I install something like https://github.com/iij/mruby-mtest ?

thanks,
Ricardo

Answer 13 · 2013-04-03T09:22:11.000Z

Ah, I don't mind if you write simple script that reproduce it.

Answer 14 · 2013-04-03T09:58:33.000Z

Hi,

This very basic script demonstrates the problem. If you run it in MRI it will work fine, in mruby with onig-regexp it will fail.

def test_split
  raise 'Split by regexp returned wrong results' unless '<%= 1 + 1 %>'.split(/(<%=)|(%>)/) == ['', '<%=', ' 1 + 1 ', '%>']
  'Split by regexp correct'
end

test_split

After I get the test suite running I'll add it as a proper test.

thanks,
Ricardo

Answer 15 · 2013-04-03T10:00:17.000Z

Currently, you have two cases issue, isn't it? :)

Answer 16 · 2013-04-03T15:36:36.000Z

Sorry mattn, couldn't understand your question :(

Answer 17 · 2013-04-03T16:35:16.000Z

Sorry, I was thinking you has more problems, so I wonder it's enough
cases to check.

On 4/4/13, Ricardo Trindade notifications@github.com wrote:

Sorry mattn, couldn't understand your question :(

Reply to this email directly or view it on GitHub:
#4 (comment)

Yasuhiro Matsumoto

Answer 18 · 2013-04-03T17:08:07.000Z

For now this is the only problem I'm having. If I find other problems with onig-regexp, I'll create test scripts for them.

thanks,
Ricardo

Answer 19 · 2013-04-04T02:11:16.000Z

I fixed it. Please try in latest.

Answer 20 · 2013-04-04T09:39:44.000Z

It does work, but only if mruby-require is being used. If mruby-require is not being included in the build, it still fails as before. This is not a problem for me as I can include mruby-require, but it doesn't appear to be 100% ok yet.

thanks,
Ricardo

Answer 21 · 2013-04-04T10:12:29.000Z

Did you do require ?

Answer 22 · 2013-04-04T10:16:38.000Z

mruby define String#split even if Regexp is disabled.

Answer 23 · 2013-04-04T10:19:15.000Z

When not using mruby-require, I didn't require since require isn't defined. But in that case Regexp == OnigRegexp, so I'm assuming that onig-regexp gets included, no ?

Answer 24 · 2013-04-04T10:48:58.000Z

include mruby-onig-regexp', not includemruby-require`
should work correctly.
include mruby-onig-regexp', includemruby-require you need dorequire 'mruby-onig-regexp'`

Answer 25 · 2013-04-04T18:09:05.000Z

This case doesn't work correctly for me:

include mruby-onig-regexp', not includemruby-require`
should work correctly (in this case the array returned from splitting the string above has just one item, instead of 4)

Answer 26 · 2013-04-05T01:36:06.000Z

In this case, mruby defines default String#split that Regexp disabled. It works but it treat pattern as string. So return non-splited strings.

Answer 27 · 2013-04-05T08:44:27.000Z

So mruby-onig-regexp doesn't fully work unless it's used with mruby-require ?

Answer 28 · 2013-04-05T08:55:18.000Z

It should work. :-(

Answer 29 · 2013-04-05T09:34:56.000Z

I get the same results as in your screenshot when I include mruby-require in the build. That case works well for me. My problem is when I don't include mruby-require in the build.

Answer 30 · 2013-04-05T09:47:14.000Z

You mean follow?

MRuby::Build.new do |conf|
  # load specific toolchain settings
  toolchain :gcc

  conf.gem "#{root}/mrbgems/mruby-sprintf"
  conf.gem "#{root}/mrbgems/mruby-print"
  conf.gem "#{root}/mrbgems/mruby-math"
  conf.gem "#{root}/mrbgems/mruby-time"
  conf.gem "#{root}/mrbgems/mruby-struct"
  conf.gem "#{root}/mrbgems/mruby-enum-ext"
  conf.gem "#{root}/mrbgems/mruby-string-ext"
  conf.gem "#{root}/mrbgems/mruby-numeric-ext"
  conf.gem "#{root}/mrbgems/mruby-array-ext"
  conf.gem "#{root}/mrbgems/mruby-hash-ext"
  conf.gem "#{root}/mrbgems/mruby-random"
  conf.gem "#{root}/mrbgems/mruby-eval"
  conf.gem :git => 'git@github.com:mattn/mruby-onig-regexp.git'
  #conf.gem :git => 'git@mattn/mruby-require.git'
end

I built mruby with build_config.rb above. And working good for me.

Answer 31 · 2013-04-05T09:53:57.000Z

This is what happens to me - http://cl.ly/image/3g3C130e2g3B

Probably there is a difference in the mrbgems we're including, I'll compare both lists

Answer 32 · 2013-04-05T09:54:37.000Z

Did you update mruby-onig-regexp?

Answer 33 · 2013-04-05T10:05:25.000Z

I'm cleaning mruby/build between tests, so I'm getting a new checkout of mruby-onig-regexp each time. I'm also using latest mruby (just updated now).

I'm still getting the same result - http://cl.ly/image/1k14130s2U0y

Answer 34 · 2013-04-05T10:09:10.000Z

What toolchain do you use? mingw? msys? visual studio?

Answer 35 · 2013-04-05T10:10:32.000Z

Could you please compile & run follow?

#include <stdio.h>

int
main(int argc, char* argv[]) {
  puts(onig_version());
  return 0;
}

I get 5.9.3

Answer 36 · 2013-04-05T10:13:43.000Z

I'm using gcc (on macOS X).

I'm also with oniguruma 5.9.3

Answer 37 · 2013-04-05T10:41:51.000Z

Try following

onig.c

#include <stdio.h>
#include <memory.h>
#include <onigposix.h>

int
main(int argc, char* argv[]) {
  size_t nmatch = 999;
  regmatch_t match[nmatch];
  int regerr;
  int i;
  char err[256];
  char *ptr = "<%= 1 + 1 %>";
  regex_t reg;

  regerr = regcomp(&reg, "(<%=)|(%>)", REG_EXTENDED | REG_NEWLINE);
  if (regerr) {
    regerror(regerr, &reg, err, sizeof(err));
    regfree(&reg);
    puts(err);
    exit(1);
  }
  for (i = 0; i < nmatch; i++)
    match[i].rm_so = -1;
  regerr = regexec(&reg, "<%= 1 + 1 %>", nmatch, match, 0);
  if (regerr == REG_NOMATCH) {
    puts("no match");
    exit(1);
  }
  if (regerr) {
    regerror(regerr, &reg, err, sizeof(err));
    puts(err);
    exit(1);
  }

  for (i = 0; i < nmatch; i++) {
    if (match[i].rm_so != -1) {
      fwrite(ptr + match[i].rm_so, match[i].rm_eo - match[i].rm_so, 1, stdout);
      putchar('\n');
    }
  }
  regfree(&reg);
  return 0;
}

$ gcc -o onig onig.c -lonig
$ ./onig

I got

<%=
<%=

Answer 38 · 2013-04-05T13:09:32.000Z

Same here - http://cl.ly/image/2a2y2O2E0t2u

Answer 39 · 2013-04-05T13:32:45.000Z

Hmm, it seems working for me. I can't understand.

> "a1b".split /\d/
 => ["a", "b"]

Is this working?

Answer 40 · 2013-04-05T13:52:43.000Z

No, also doesn't work - http://cl.ly/image/3c3t2e0W2C3Z

At this point the only thing I can think of is the platform difference - mac vs windows. Do you have anyone near you with a Mac that can test ? I do have windows machines, but they don't have cygwin etc.. so it wouldn't be so easy (I can do it on the weekend if no other option exists)

Answer 41 · 2013-04-05T14:03:42.000Z

But you get same result in onig.c, so I don't think platform differences.

Answer 42 · 2013-04-05T14:08:40.000Z

Yes, you are right. I'll try again later tonight with a fresh checkout of mruby, to rule out any other issue.

Answer 43 · 2013-04-05T14:28:33.000Z

Thanks for your work

On 4/5/13, Ricardo Trindade notifications@github.com wrote:

Yes, you are right. I'll try again later tonight with a fresh checkout of
mruby, to rule out any other issue.

Reply to this email directly or view it on GitHub:
#4 (comment)

Yasuhiro Matsumoto

Answer 44 · 2013-04-11T23:06:51.000Z

Just tried again with a new git checkout, and getting the same behaviour.

With mruby-require, splitting works fine
When not using mruby-require, an array with a single element is returned, instead of the expected result

Answer 45 · 2013-04-11T23:17:36.000Z

Could you please try to add printf("something\n"); code to make sure
passing C's functions?

On 4/12/13, Ricardo Trindade notifications@github.com wrote:

Just tried again with a new git checkout, and getting the same behaviour.

With mruby-require, splitting works fine

When not using mruby-require, an array with a single element is returned,
instead of the expected result

Reply to this email directly or view it on GitHub:
#4 (comment)

Yasuhiro Matsumoto

Answer 46 · 2013-04-11T23:28:29.000Z

I did that and found a difference. With mruby-require, the onig_regexp_match C function is called twice. Without mruby-require, it's only called once.

http://cl.ly/image/0o3k3X3k3S3D - The difference between both mirb sessions show shows this (there was a build in between)

Answer 47 · 2013-04-12T05:34:26.000Z

Do you get follow?

> /\d/.match('a3d')[0]
 => "3"

Answer 48 · 2013-04-12T08:36:14.000Z

No - http://cl.ly/image/1i2b1h3H0J06

Answer 49 · 2013-04-12T08:51:07.000Z

Cases that match method return nil,

Invalid argument for match (ex match("a3d", -1))
https://github.com/mattn/mruby-onig-regexp/blob/master/src/mruby_onig_regexp.c#L93

Or no matches
https://github.com/mattn/mruby-onig-regexp/blob/master/src/mruby_onig_regexp.c#L106

First cast should be checked with

> /\d/.match("a3d", -1)
 => nil
> /\d/.match("a3d", 0)
 => #<OnigMatchData:0xe0e6f0>

And second case should be checked with

> //
 => #<OnigRegexp:0xe0ea98 @source="", @regexp=#<Object:0xe0ea80>>

@regexp shouldn't be nil

Answer 50 · 2014-02-17T00:42:20.000Z

Probably, this issue was fixed in cc6b0b3