IOOBE when parsing LightGBM model files

Question

IOOBE when parsing LightGBM model files

xiaoxingbai opened this issue 5 months ago · 9 comments

java.lang.IndexOutOfBoundsException: Index 3 out of bounds for length 1
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:359)
	at java.base/java.util.ArrayList.get(ArrayList.java:427)
	at org.jpmml.lightgbm.Tree.selectValues(Tree.java:319)
	at org.jpmml.lightgbm.Tree.encodeNode(Tree.java:214)
	at org.jpmml.lightgbm.Tree.encodeNode(Tree.java:267)
	at org.jpmml.lightgbm.Tree.encodeTreeModel(Tree.java:110)
	at org.jpmml.lightgbm.ObjectiveFunction.createMiningModel(ObjectiveFunction.java:74)
	at org.jpmml.lightgbm.BinomialLogisticRegression.encodeModel(BinomialLogisticRegression.java:46)
	at org.jpmml.lightgbm.GBDT.encodeModel(GBDT.java:417)
	at org.jpmml.lightgbm.GBDT.encodeModel(GBDT.java:404)
	at org.jpmml.lightgbm.GBDT.encodePMML(GBDT.java:392)
	at org.jpmml.lightgbm.example.Main.run(Main.java:175)
	at org.jpmml.lightgbm.example.Main.main(Main.java:136)

Answer 1 · 2024-07-11T03:39:34.000Z

模型训练数据量大的时候正常，数据量少的话去做txt到pmml 转换的时候就会报越界的错误

Answer 2 · 2024-07-11T06:44:10.000Z

Looks like the conversion was initiated using the command-line application (as opposed to using some ML framework wrapper such as SkLearn2PMML or PySpark2PMML).

The command-line application reads model schema information (number and type of features, the cardinality of categorical features, etc) from the model file. In comparison, when using ML frameworks wrappers, this model schema information would be provided by the wrapper component (eg. Scikit-Learn pipeline).

模型训练数据量大的时候正常，数据量少的话去做txt到pmml 转换的时候就会报越界的错误

I don't speak the language, sorry. Google translate tells me that the above means that this exception is raised with small datasets and not with big datasets.

if so, can you provide a small self-containing LightGBM model file that causes this IndexOutOfBoundsException to be thrown? Can you reproduce it using my default Audit dataset?

Also, what is your LightGBM version?

I need more evidence before I can implement any code changes.

Answer 3 · 2024-07-11T10:54:10.000Z

When the model training data is large, it is normal, and when the data is small, the error will be reported when the txt is converted to pmml.I trained the model directly with lightgbm, version 4.3.0

Answer 4 · 2024-07-11T11:12:42.000Z

Hello, nice to meet you.          I have provided three models of lightgbm training conversion errors. Can you see what causes the errors? At present, I found that there are 100,000 million training data conversion errors, and tens of millions of training data conversion into pmml format is normal, I do not know why?

…

------------------ 原始邮件 ------------------ 发件人: "jpmml/jpmml-lightgbm" ***@***.***>; 发送时间: 2024年7月11日(星期四) 下午2:44 ***@***.***>; ***@***.******@***.***>; 主题: Re: [jpmml/jpmml-lightgbm] Index 3 out of bounds for length 1 (Issue #63) Looks like the conversion was initiated using the command-line application (as opposed to using some ML framework wrapper such as SkLearn2PMML or PySpark2PMML). The command-line application reads model schema information (number and type of features, the cardinality of categorical features, etc) from the model file. In comparison, when using ML frameworks wrappers, this model schema information would be provided by the wrapper component (eg. Scikit-Learn pipeline). 模型训练数据量大的时候正常，数据量少的话去做txt到pmml 转换的时候就会报越界的错误 I don't speak the language, sorry. Google translate tells me that the above means that this exception is raised with small datasets and not with big datasets. if so, can you provide a small self-containing LightGBM model file that causes this IndexOutOfBoundsException to be thrown? Can you reproduce it using my default Audit dataset? Also, what is your LightGBM version? I need more evidence before I can implement any code changes. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 5 · 2024-07-11T11:15:06.000Z

I trained the model directly with lightgbm, version 4.8.3.

In PyPI, the latest LightGBM version is only 4.4.0.

I have provided three models of lightgbm training conversion errors. Can you see what causes the errors?

Where can I see those models? They are not attached here, and they're not in my e-mail inbox.

I can't do anything without sample model files.

Answer 6 · 2024-07-12T01:56:00.000Z

I trained the model directly with lightgbm, version 4.8.3

In PyPI, the latest LightGBM version is only 4.4.0.
sorry, in jupyter lightgbm version 4.3.0

I have provided three models of lightgbm training conversion errors. Can you see what causes the errors?

Where can I see those models? They are not attached here, and they're not in my e-mail inbox.
I put three attachment in Email.Now I'm reattached here,So you can see it directly.
lightgbm_model_shouhuan_optimize_v6_cvr_20240711010046.txt
lightgbm_model_shouhuan_optimize_v6_cvr_20240711101909.txt
lightgbm_model_shouhuan_optimize_v6_cvr_20240711120223.txt

I can't do anything without sample model files.
Try again.

Answer 7 · 2024-07-12T05:35:38.000Z

I have provided three models of lightgbm training conversion errors.

Got all three models. And they are also failing with an IOOBE when I try to convert them using the command-line application.

Will investigate soon.

Answer 8 · 2024-07-14T20:13:05.000Z

This IOOBE happened when the training dataset contained constant-value categorical columns. For example, in the provided three sample models, the value of the "os" field is always "android".

The fix is included in JPMML-LightGBM version 1.5.4 and newer.

Answer 9 · 2024-07-15T06:34:26.000Z

Thank you very much. I'll try later.