Error when running notebooks: Answer from Java side is empty

[Jaganatha]

Hello Team,

bert_model = BertEmbeddings.pretrained('bert_base_cased', 'en').setInputCols(["sentence",'token']).setOutputCol("bert").setCaseSensitive(False).setPoolingLayer(0)

df_bert_train = bert_model.transform(sparkNLP_transformed_full_train)

nerTagger = NerDLApproach().setInputCols(["sentence", "token", "bert"]).setLabelColumn("label").setOutputCol("ner")
.setMaxEpochs(1).setRandomSeed(0).setVerbose(1).setValidationSplit(0.2).setEvaluationLogExtended(True).setEnableOutputLogs(True).setIncludeConfidence(True)

#above code were successful and I can see bert embeddings added to the traindata.However the below code gives error
ner_tag_model_final = nerTagger.fit(df_bert_train)

I am trying to create a NER DL model and I am successful in creating the pipeline.

However, when I feed the train data to fit() the model, I am receiving following error

Exception happened during processing of request from ('127.0.0.1', 47346)
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 1159, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 985, in send_command
response = connection.send_command(command)
File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 1164, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
Traceback (most recent call last):
File "/usr/lib/python3.6/socketserver.py", line 320, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.6/socketserver.py", line 724, in init
self.handle()
File "/usr/local/lib/python3.6/dist-packages/pyspark/accumulators.py", line 269, in handle
poll(accum_updates)
File "/usr/local/lib/python3.6/dist-packages/pyspark/accumulators.py", line 241, in poll
if func():
File "/usr/local/lib/python3.6/dist-packages/pyspark/accumulators.py", line 245, in accum_updates
num_updates = read_int(self.rfile)
File "/usr/local/lib/python3.6/dist-packages/pyspark/serializers.py", line 717, in read_int
raise EOFError
EOFError

---

Py4JError Traceback (most recent call last)
in ()
----> 1 ner_tag_model_final = nerTagger.fit(df_bert_train)

5 frames
/usr/local/lib/python3.6/dist-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
334 raise Py4JError(
335 "An error occurred while calling {0}{1}{2}".
--> 336 format(target_id, ".", name))
337 else:
338 type = answer[1]

Py4JError: An error occurred while calling o860.fit

[maziyarpanahi]

Could you please complete this template (also please be detail in how and where you are running this code):

Description

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce

Context

Your Environment

• Spark NLP version:
• Apache NLP version:
• Java version (java -version):
• Setup and installation (Pypi, Conda, Maven, etc.):
• Operating System and version:
• Link to your project (if any):

[Jaganatha]

Hello Maziyar, *MANY THANKS FOR YOUR EMAIL * Description: I am trying to build a BERT embedding model for Named entity recognition task (MIT movie corpus data). I have done all the preprocessing stepsand converted the raw data into documents, sentences and tokens and also generated Bert Embeddings from the BertEmbeddings pretrained model annotator in the SparkNLP.Following these steps, my final step is to build the NER model using NER DL Approach using the following code: nerTagger = NerDLApproach().setInputCols(["sentence", "token", "bert"]) .setLabelColumn("label").setOutputCol("ner")\ .setMaxEpochs(1).setRandomSeed(0).setVerbose(1).setValidationSplit(0.2) .setEvaluationLogExtended(True).setEnableOutputLogs(True) .setIncludeConfidence(True) nerTagger variable contains the estimator built using the above code. (so far no error). Finally when I try to use fit() function to train the model on my training set data, it throws me the error. ner_tag_model_final = nerTagger.fit(df_bert_train) * error * ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:44675) Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 929, in _get_connection connection = self.deque.pop() IndexError: pop from an empty deque During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 1067, in start self.socket.connect((self.address, self.port)) ConnectionRefusedError: [Errno 111] Connection refused

…

--------------------------------------------------------------------------- IndexError Traceback (most recent call last) /usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py in _get_connection(self) 928 try: --> 929 connection = self.deque.pop() 930 except IndexError: IndexError: pop from an empty deque During handling of the above exception, another exception occurred: ConnectionRefusedError Traceback (most recent call last)

------------------------------ 11 frames ------------------------------ ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Py4JNetworkError Traceback (most recent call last) /usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py in start(self) 1077 "server ({0}:{1})".format(self.address, self.port) 1078 logger.exception(msg) -> 1079 raise Py4JNetworkError(msg, e) 1080 1081 def _authenticate_connection(self): Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:44675) Expected Behavior: successful train without errorCurrent Behavior: THROWS ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:44675) Possible SolutionSteps to Reproduce 1. 2. 3. 4. ContextYour Environment - Spark NLP version: Collecting spark-nlp==2.5.1 Downloading https://files.pythonhosted.org/packages/df/b4/db653f8080a446de8ce981b262d85c85c61de7e920930726da0d1c6b4c65/spark_nlp-2.5.1-py2.py3-none-any.whl (121kB) |████████████████████████████████| 122kB 5.2MB/s Installing collected packages: spark-nlp Successfully installed spark-nlp-2.5.1 - - Apache NLP version: *spark* version 2.4.4 - Java version (java -version): openjdk version "1.8.0_252" OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~18.04-b09) OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode) - Setup and installation (Pypi, Conda, Maven, etc.): *GOOGLE COLAB NOTEBOOK PYTHON* - Operating System and version: Windows 10 - Link to your project (if any) *Regards,* *Jaganath* <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Sat, Jul 11, 2020 at 11:02 AM Maziyar Panahi ***@***.***> wrote: Could you please complete this template (also please be detail in how and where you are running this code): Description Expected Behavior Current Behavior Possible Solution Steps to Reproduce 1. 2. 3. 4. Context Your Environment - Spark NLP version: - Apache NLP version: - Java version (java -version): - Setup and installation (Pypi, Conda, Maven, etc.): - Operating System and version: - Link to your project (if any): — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#86 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYZ7BVYLMTZRMDXROBZQZTR3A2B7ANCNFSM4OWO4XGQ> .

[maziyarpanahi]

The error doesn't suggest Spark NLP, it's more about Apache Spark and Java. Without having the template I am afraid I can't guess what could be wrong, you're code, you're dataset, Apache Spark or Spark NLP.
In addition to all the info required in the template, please also provide the full snippet code from the line you read your dataset to CoNLL() class and all the processing. You are getting an error in Apache Spark which means it could be in one of those lines.