Impossible to use v2.2.1 with OF 4.6.3

[danieltet]

I have tried for about a week to get monitoring v2.2.1 to work with OF 4.6.3 and the whole setup becomes unstable and refuses connections, complains about write locks and database connections being lost. Here are some of the errors I see. But first for the record we have about 50K+ MUC rooms and 70+ million rows in ofMessageArchive and 7+ million in ofMucConversationLog. We are running on Amazon Linux with Java 1.8.0_241-b07. The net result is that we would like to upgrde to the latest but are stuck using monitoring 2.0.0. No cluster (that's a separate problem).

Error 1 - Inconsistent connections:

• Even using Adium with two users (from same IP) one connects consistently the second one does not. The second waits at 25% connection but does not complete. I do a netstat -an | grep IP and see the tcp connection made but nothing in logs even with debug turned on. Once I remove montoring plugin and restart I can connect consistently with both users.

Error 2 - Write Lock
I get this error consistently at startup. I've tried removing the write.lock file and restarting and same issue. I've read about issues with two writers to lucene, not sure if that's what's going on here:

05:25:21.517 [pool-monitoring7] ERROR org.jivesoftware.openfire.archive.ArchiveIndexer[CONVERSATION] - An exception occurred while updating the Lucene index.
org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /opt/openfire-4.6.3/monitoring/search/write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139) ~[lucene-core-8.4.0.jar!/:?]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.4.0.jar!/:?]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.4.0.jar!/:?]
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:729) ~[lucene-core-8.4.0.jar!/:?]
at org.jivesoftware.openfire.index.LuceneIndexer.updateIndex(LuceneIndexer.java:243) [monitoring-2.2.1.jar!/:?]
at org.jivesoftware.openfire.index.LuceneIndexer$1.run(LuceneIndexer.java:134) [monitoring-2.2.1.jar!/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]

Error 3 - DB time-outs
This error appears after the setup runs for just a few minutes. It's as if it uses a stale or released connection from the connection pool:

01:01:11.067 [TaskEngine-pool-770] ERROR org.jivesoftware.database.DbConnectionManager - No operations allowed after connection closed.
java.sql.SQLNonTransientConnectionException: No operations allowed after connection closed.
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) ~[mysql-connector-java-8.0.21.jar:8.0.21]
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.21.jar:8.0.21]
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89) ~[mysql-connector-java-8.0.21.jar:8.0.21]
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63) ~[mysql-connector-java-8.0.21.jar:8.0.21]

...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: com.mysql.cj.exceptions.ConnectionIsClosedException: No operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_241]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_241]

Warning 4 - Keeps wanting to connect to proxy.eu
I found this in the log as well, very strange:

00:54:29.282 [pool-26-thread-3] WARN org.jivesoftware.openfire.net.SocketUtil - Unable to create a socket connection to XMPP domain 'proxy.eu.jabber.org' using remote host: proxy.eu.jabber.org:5269. Cause: proxy.eu.jabber.org (a full stacktrace is logged on debug level)
00:54:29.282 [pool-26-thread-3] WARN org.jivesoftware.openfire.net.SocketUtil - Unable to create a socket connection to XMPP domain 'proxy.eu.jabber.org': Unable to connect to any of its remote hosts.

[Fishbowler]

I've looked into the 4th one, as the simplest one to tackle. I'm fairly certain this isn't coming from Openfire or Monitoring, but from a client/user trying to connect to something there via Openfire. A quick Google for that address suggests that it used to be a default to connect to in Pidgin, and possibly others.