COR is down
carueda opened this issue · 4 comments
carueda commented
COR is down. The AWS instance is up - could others scope.
carueda commented
Problem was that the ORR ran out of memory:
Last few lines of the ORR container:
[cor-admin1@ip-172-30-0-37 ~]$ docker logs -f --tail=40 orr
...
2019-09-28 18:59:08.720 [http-nio-8080-exec-14] DEBUG org.mmisw.orr.ont.app.OntController - raw Accept: Some(*/*)
2019-09-28 19:45:38.846 [http-nio-8080-exec-14] DEBUG org.mmisw.orr.ont.app.OntController - acceptHeader: List(*/*)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/tomcat/hs_err_pid1.log
# [ timer expired, abort... ]
[thread 25 also had an error]
Then:
[cor-admin1@ip-172-30-0-37 COR]$ docker-compose stop orr
[cor-admin1@ip-172-30-0-37 COR]$ docker-compose rm orr
[cor-admin1@ip-172-30-0-37 COR]$ docker-compose up -d
The ORR is up now, but then saw exceptions in the log related with the triple store so decided to restart the whole set of containers:
[cor-admin1@ip-172-30-0-37 COR]$ docker-compose down
[cor-admin1@ip-172-30-0-37 COR]$ docker-compose up -d
Now, all looks good in the log.
Finally, reloaded the triple store by using the Admin option for this in the GUI.
carueda commented
Checking memory settings for the ORR container, it does seem like the max heap size is too small:
[cor-admin1@ip-172-30-0-37 ~]$ docker exec -it orr java -XshowSettings:vm -version
VM settings:
Max. Heap Size (Estimated): 988.00M
Using VM: OpenJDK 64-Bit Server VM
openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment 18.9 (build 11.0.4+11)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11, mixed mode)
Some references:
carueda commented
[cor-admin1@ip-172-30-0-37 COR]$ docker exec -it orr java -XX:+PrintFlagsFinal -version | grep MaxHeap
uintx MaxHeapFreeRatio = 70 {manageable} {default}
size_t MaxHeapSize = 1035993088 {product} {ergonomic}