Importing Keras Tokenizer from Json Exception Java
zax-ma opened this issue · 0 comments
Hello,
Can't import tokenizer created in python and saved to JSON as recomended in method KerasTokenizer.fromJson
Please help me to resolve the issue. What could be done wrong?
Thank you,
Issue Description
Please describe our issue, along with:
- expected behavior:
Tokenizer, created by
tokenizer_json = tokenizer.to_json()
with io.open('tokenizer.json', 'w', encoding='utf-8') as f:
f.write(json.dumps(tokenizer_json, ensure_ascii=False))
importing to Java project by the command:
KerasTokenizer.fromJson(TOKEN_PATH);
as a result an object of tokenizer should be created.
- encountered behavior:
Exception appeared
Exception in thread "main" org.nd4j.shade.jackson.databind.exc.MismatchedInputException: Cannot construct instance of java.util.HashMap
(although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{"class_name": "Tokenizer", "config": {"num_words": 10, "filters": "!"#$%&()*+,-./:;<=>?@[]^_{|}~tn", "lower": true, "split": " ", "char_level": false, "oov_token": null, "document_count": 23, "word_counts": "{"url1": 12, "url2": 12, "url3": 10, "url4": 16, "url5": 15, "url6": 16}", "word_docs": "{"url1": 12, "url2": 10, "url3": 10, "url4": 16, "url5": 14, "url6": 13}", "index_docs": "{"4": 12, "5": 10, "6": 10, "1": 16, "3": 14, "2": 13}", "index_word": "{"1": "url4", "2": "url6", "3": "url5", "4": "url1", "5": "url2", "6": "url3"}", "word_index": "{"url4": 1, "url6": 2, "url5": 3, "url1": 4, "url2": 5, "url3": 6}"}}') at [Source: (String)""{\"class_name\": \"Tokenizer\", \"config\": {\"num_words\": 10, \"filters\": \"!\"#$%&()*+,-.\/:;<=>?@[]^_
{|}~tn", "lower": true, "split": " ", "char_level": false, "oov_token": null, "document_count": 23, "word_counts": "{"url1": 12, "url2": 12, "url3": 10, "url4": 16, "url5": 15, "url6": 16}", "word_docs": "{"url1": 12, "url2": 10, "url3": 10, "url4": 16, "url5": 14, "url6": 13}", "index_docs": "{"4": 12, "5": 10, "6": 10, "1": 16, "3"[truncated 246 chars]; line: 1, column: 1]
at org.nd4j.shade.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
at org.nd4j.shade.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1728)
at org.nd4j.shade.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1353)
at org.nd4j.shade.jackson.databind.deser.std.StdDeserializer._deserializeFromString(StdDeserializer.java:311)
at org.nd4j.shade.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:444)
at org.nd4j.shade.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32)
at org.nd4j.shade.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at org.nd4j.shade.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
at org.nd4j.shade.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
at org.nd4j.shade.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3612)
at org.deeplearning4j.nn.modelimport.keras.utils.KerasModelUtils.parseJsonString(KerasModelUtils.java:400)
at org.deeplearning4j.nn.modelimport.keras.preprocessing.text.KerasTokenizer.fromJson(KerasTokenizer.java:107)
at com.example.importh5.ImportKerasModel.main(ImportKerasModel.java:40)
Version Information
Please indicate relevant versions, including, if relevant:
- Deeplearning4j version
- Platform information (OS, etc)
- CUDA version, if used
- NVIDIA driver version, if in use
Additional Information
Where applicable, please also provide:
version 1.0.0-M2.1
- Full log or exception stack trace (ideally in a Gist: gist.github.com)
- pom.xml file or similar (also in a Gist)
Contributing
If you'd like to help us fix the issue by contributing some code, but would
like guidance or help in doing so, please mention it!