jpmml/r2pmml

random java errors when converting xgboost

evilantal opened this issue · 4 comments

evilantal commented

I'm trying to convert an xgboost model to pmml.

About 5 times in 6 I'll get java errors when converting after running an xgboost model on the same dataset with the same hyperparameter settings.

Here's the error

y: java.io.IOException: Expected 32-element array of zeroes, got [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32527]
	at org.jpmml.xgboost.XGBoostDataInput.readReserved(XGBoostDataInput.java:129)
	at org.jpmml.xgboost.GBTree.load(GBTree.java:58)
	at org.jpmml.xgboost.Learner.load(Learner.java:103)
	at org.jpmml.xgboost.XGBoostUtil.loadLearner(XGBoostUtil.java:53)
	at org.jpmml.xgboost.XGBoostUtil.loadLearner(XGBoostUtil.java:45)
	at org.jpmml.rexp.XGBoostConverter.loadLearner(XGBoostConverter.java:201)
	at org.jpmml.rexp.XGBoostConverter.loadLearner(XGBoostConverter.java:143)
	... 6 more
Error in .convert(tempfile, file, converter, converter_classpath, verbose) : 
  1
Calls: r2pmml -> tryCatch -> tryCatchList -> main -> .convert

Here's the code I'm using to run r2pmml (which is also the same every time):
r2pmml(xgbmodel,"xgb_model.pmml",fmap=fmap,response_name="target",response_levels=c("0","1"),compact=TRUE)

I'm running the process through R script on a remote machine so it's kinda difficult to troubleshoot.
Can anyone tell me what this error might point to?

vruusmann commented

About 5 times in 6 I'll get java errors.

That doesn't make sense as the layout of the XGBoost binary file is deterministic - there should always be a 32-byte empty space, or a 31-byte empty space plus one non-empty byte, but not a "mix" of the two layouts.

What's your XGBoost version, and what's the configuration of the learning task (objective function, number of trees, etc)?

evilantal commented

Thanks for the reply!

Due to the limitations of the server I'm using xgboost 0.4-4.
Objective is "binary:logistic", eval_metric="auc"
I've used r2pmml::genFMap and r2pmml::genDMatrix to create the feature map and DMatrix objects

Through hyperparameter optimisation I'm using these parameter settings:
nrounds=228, eta=0.0219740211397875, max_depth=13, gamma=0, min_child_weight=3, colsample_bytree=0.706007500481792

vruusmann commented

Due to the limitations of the server I'm using xgboost 0.4-4.

I'm not sure if this XGBoost version is "officially" supported by the underlying JPMML-XGBoost library at all. The commit log suggests that its development was launched when XGBoost was already at the 0.6 version stage.

Troubleshooting would be possible if the XGBoost binary file was always broken (6 out of 6 times). I suspect there's something wrong with the XGBoost binary file itself - is the size of this file smaller/larger in some cases than others? Perhaps some other tool is interacting with it?

If you're willing to dig in XGBoost source code and find out what's the "meaning" of this 32527 magical constant in the reserved space, then I could do something about it. Otherwise, my resolution is - please upgrade your XGBoost to 0.7 or newer, and reopen this issue if the problem persists even after that.

evilantal commented

Hmmm... thanks for that....
Trouble is, I'll first have to have the server upgraded to SLES12 from SLES11.

Maybe I can see something in the model objects when the conversion does and doesn't success