ERROR Uncaught error from Logz.io sender java.lang.IndexOutOfBoundsException: Index 214 is out of bounds [217 - 937]

Question

ERROR Uncaught error from Logz.io sender java.lang.IndexOutOfBoundsException: Index 214 is out of bounds [217 - 937]

Opened this issue a year ago · 36 comments

Hi,
We're using logz.io appender for log4j2 to ship logs. While on x64 servers it's running fine, in the ARM instances we see next exception

2023-07-25 13:20:55,788 Log4j2-TF-1-LogzioAppender-1 ERROR Uncaught error from [Logz.io](http://logz.io/) sender java.lang.IndexOutOfBoundsException: Index 214 is out of bounds [217 - 937]
	at [io.logz.sender.org](http://io.logz.sender.org/).kairosdb.bigqueue.BigArrayImpl.validateIndex(BigArrayImpl.java:471)
	at [io.logz.sender.org](http://io.logz.sender.org/).kairosdb.bigqueue.BigArrayImpl.get(BigArrayImpl.java:411)
	at [io.logz.sender.org](http://io.logz.sender.org/).kairosdb.bigqueue.BigQueueImpl.dequeue(BigQueueImpl.java:124)
	at io.logz.sender.DiskQueue.dequeue(DiskQueue.java:65)
	at io.logz.sender.LogzioSender.dequeueUpToMaxBatchSize(LogzioSender.java:218)
	at io.logz.sender.LogzioSender.drainQueue(LogzioSender.java:234)
	at io.logz.sender.LogzioSender.drainQueueAndSend(LogzioSender.java:133)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)

Please advise , what can be an issue here ?
Thanks

Answer 1 · 2023-07-26T08:55:57.000Z

Hi @slavag, can you provide the appender version you are using, and JDK version?

Answer 2 · 2023-07-26T09:00:22.000Z

Appender is 1.0.19 and JDK is AWS-Corretto 15.And also with OpenJDK 8
Thanks

Answer 3 · 2023-07-26T12:00:57.000Z

@tamir-michaeli As temp solution we put <inMemoryQueue>true</inMemoryQueue> and it solved the issue.

Answer 4 · 2023-07-26T14:04:19.000Z

@slavag Good to hear. We will release a new version next week that will have a better maintained bigqueue module for the underlying queue implementation, hopefully it will fix the issue.

Answer 5 · 2023-07-30T08:18:14.000Z

@slavag new version is out. Please note that it is supported only with JDK 11 and above.

Answer 6 · 2023-08-06T13:58:46.000Z

@slavag Did you had a chance to check the new version?

Answer 7 · 2023-08-06T19:01:49.000Z

@tamir-michaeli sorry, missed that, will check in the coming days and let you know.
Thanks

Answer 8 · 2023-08-07T11:02:18.000Z

@tamir-michaeli Hi, unfortunately not fixed :

2023-08-07 10:32:37,886 Log4j2-TF-1-LogzioAppender-2 ERROR Uncaught error from Logz.io sender java.lang.IndexOutOfBoundsException: Index 214 is out of bounds [217 - 1430]
        at io.logz.sender.org.ikasan.bigqueue.BigArrayImpl.validateIndex(BigArrayImpl.java:444)
        at io.logz.sender.org.ikasan.bigqueue.BigArrayImpl.get(BigArrayImpl.java:385)
        at io.logz.sender.orgikasan.bigqueue.BigQueueImpl.dequeue(BigQueueImpl.java:112)
        at io.logz.sender.DiskQueue.dequeue(DiskQueue.java:65)
        at io.logz.sender.LogzioSender.dequeueUpToMaxBatchSize(LogzioSender.java:222)
        at io.logz.sender.LogzioSender.drainQueue(LogzioSender.java:238)
        at io.logz.sender.LogzioSender.drainQueueAndSend(LogzioSender.java:133)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:832)

Answer 9 · 2023-08-07T12:31:10.000Z

Hey @tamir-michaeli ,
Our java application is using /mnt as tmp folder (-Djava.io.tmpdir=/mnt).
Premissions for that folder is 777.

If we are not using that folder the appender is working without true flag.

Any suggestions?

Thanks Nicolas

Answer 10 · 2023-08-07T13:24:02.000Z

Hey @tamir-michaeli , Our java application is using /mnt as tmp folder (-Djava.io.tmpdir=/mnt). Premissions for that folder is 777.

If we are not using that folder the appender is working without true flag.

Any suggestions?

Thanks Nicolas

Need to add that /mnt is different disk. But as Nico said it's writable to every user

Answer 11 · 2023-08-15T09:38:16.000Z

@slavag Did the appender worked for some time when setting the tmpdir, or not at all?

Answer 12 · 2023-08-16T13:26:38.000Z

@tamir-michaeli not at all, failing during initialization.

Answer 13 · 2023-08-22T12:00:50.000Z

@tamir-michaeli any update ?

Answer 14 · 2023-08-23T08:12:21.000Z

@slavag We will probably start working on it next week.

Answer 15 · 2023-08-24T13:26:40.000Z

@slavag Can you please share your config?

Answer 16 · 2023-08-24T16:03:31.000Z

@tamir-michaeli

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<Configuration status="WARN">
    <Properties>
        <Property name="LOG_PATTERN">%d{HH:mm:ss,SSS} [%p] [%t] %notEmpty{[SID:%X{SID}]} - %m [%c]%n</Property>
    </Properties>

    <Appenders>
        <Console name="Console" target="SYSTEM_OUT" follow="true">
            <PatternLayout pattern="${LOG_PATTERN}"/>
        </Console>
        <LogzioAppender name="Logzio">
            <logzioToken>....</logzioToken>
            <logzioType>Tika</logzioType>
            <logzioUrl>https://listener-eu.logz.io:8071</logzioUrl>
            <additionalFields>hostname=${hostName}</additionalFields>
            <inMemoryQueue>true</inMemoryQueue>
        </LogzioAppender>
        <RollingFile name="File" fileName="/mnt/logs/tikalogs.log"
                     filePattern="mnt/logs/tikalogs-%d{MM-dd-yyyy}-%i.log.gz">
            <PatternLayout>
                <Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
            </PatternLayout>
            <Policies>
                <TimeBasedTriggeringPolicy />
                <SizeBasedTriggeringPolicy size="1 G"/>
            </Policies>
            <DefaultRolloverStrategy>
                <Delete basePath="/mnt/logs/">
                    <IfFileName glob="*/tikalogs-*.log.gz"/>
                    <IfAccumulatedFileCount exceeds="10"/>
                </Delete>
            </DefaultRolloverStrategy>
        </RollingFile>
    </Appenders>

    <Loggers>
        <Root level="debug">
            <AppenderRef ref="Console" level="warn"/>
            <AppenderRef ref="Logzio" level="warn"/>
            <AppenderRef ref="File" level="info"/>
        </Root>
        <Logger name="com.yyy.tika.parser" level="error" additivity="false">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="Logzio"/>
            <AppenderRef ref="File"/>
        </Logger>

        <!-- disable the below, as sometimes they fill our logs with grabage and increase log size in logz.io -->
        <Logger name="org.apache.pdfbox" level="error" additivity="false">
        </Logger>
        <Logger name="org.apache.fontbox" level="error" additivity="false">
        </Logger>
        <Logger name="org.mp4parser" level="error" additivity="false">
        </Logger>
    </Loggers>
</Configuration>

Please note that with <inMemoryQueue>true</inMemoryQueue> it's working ok, without it - failing.
Thanks

Answer 17 · 2023-08-30T10:00:04.000Z

Hi @slavag ,
unfortunately i could not reproduce the issue, these are the steps i took:

Build jar with latest log4j2 appender, setting inMemoryQueue to false.
Create ec2 instance (arm64) with 2 volumes.
Run jar from the first drive while setting the tempdir to a folder on the second drive.

The jar ran with no issues, for multiple runs as well (to test the loading of data saved in disk).

Maybe setting the appender debug flag to true will provide us more information.

Answer 18 · 2023-08-30T20:08:39.000Z

@tamir-michaeli Hi, is there any update ? Thanks

Answer 19 · 2023-08-30T21:29:12.000Z

@slavag Please see the message above. I could not reproduce the issue. Can you try running the appender with debug mode?

Answer 20 · 2023-08-31T06:23:45.000Z

@tamir-michaeli sorry , somehow didn't see your message. Sure we'll run it in debug, can you please tell me where to put that debug property ?
Thanks

Answer 21 · 2023-09-03T12:47:53.000Z

@slavag , add <debug>true</debug>:

        <LogzioAppender name="Logzio">
            <logzioToken>....</logzioToken>
            <logzioType>Tika</logzioType>
            <debug>true</debug>
            <logzioUrl>https://listener-eu.logz.io:8071</logzioUrl>
            <additionalFields>hostname=${hostName}</additionalFields>
            <inMemoryQueue>true</inMemoryQueue>
        </LogzioAppender>

Answer 22 · 2023-09-04T06:11:43.000Z

Hey @tamir-michaeli,
I've tested with true.
The error is the same.

INFO [main] 05:59:38,972 org.apache.tika.server.core.TikaServerConfig As of Tika 2.5.0, you can set the digester via the AutoDetectParserConfig in tika-config.xml. We plan to remove this commandline option in 2.8.0
I> No access restrictor found, access to any MBean is allowed
Jolokia: Agent started with URL http://127.0.0.1:8778/jolokia/
2023-09-04 05:59:39,564 Log4j2-TF-2-LogzioAppender-2 ERROR Uncaught error from Logz.io sender java.lang.IndexOutOfBoundsException: Index 214 is out of bounds [217 - 1744]
at io.logz.sender.org.ikasan.bigqueue.BigArrayImpl.validateIndex(BigArrayImpl.java:444)
at io.logz.sender.org.ikasan.bigqueue.BigArrayImpl.get(BigArrayImpl.java:385)
at io.logz.sender.org.ikasan.bigqueue.BigQueueImpl.dequeue(BigQueueImpl.java:112)
at io.logz.sender.DiskQueue.dequeue(DiskQueue.java:65)
at io.logz.sender.LogzioSender.dequeueUpToMaxBatchSize(LogzioSender.java:222)
at io.logz.sender.LogzioSender.drainQueue(LogzioSender.java:238)
at io.logz.sender.LogzioSender.drainQueueAndSend(LogzioSender.java:133)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Sep 04, 2023 5:59:40 AM org.apache.cxf.endpoint.ServerImpl initDestination

When using true
it works with no issues.

Answer 23 · 2023-09-10T07:05:34.000Z

@tamir-michaeli Hi, is there any update ? Thanks

Answer 24 · 2023-09-11T07:54:08.000Z

@slavag Unfortunately no, i could not reproduce the issue, tried multiple combinations of disks & jar locations.
Will keep this issue open.

Answer 25 · 2023-09-11T09:03:06.000Z

@tamir-michaeli we can debug with you in our servers where the issue is constantly reproduced, would you like to do conf call (screen sharing) ?

Answer 26 · 2023-09-11T12:18:44.000Z

@slavag What kind of disk you are using (efs, ebs, fsx)?

Answer 27 · 2023-09-11T17:53:51.000Z

@tamir-michaeli we're using EBS gp3, maybe I forgot to mention it's LVM managed disk on top of EBS gp3

Answer 28 · 2023-09-13T07:22:38.000Z

@slavag I will try to reproduce the issue with LVM.

Answer 29 · 2023-09-13T18:36:30.000Z

@tamir-michaeli thanks, please use XFS file system on it.

Answer 30 · 2023-09-20T07:43:43.000Z

@slavag Im keeping the issue open until we can solve it, no ETA.
As of today our recommendation to not store integration queue's on LVM.

Answer 31 · 2023-09-20T17:30:24.000Z

@tamir-michaeli the problem is that java temp folder in the app that uses logger is pointing LVM disk, and this affects of course integration queue location. So, if we can somehow define different location for integration queue , we definitely can put into /tmp standard folder. Is there any way to define different from default Java temp folder to integration queue storage ?
Thanks

Answer 32 · 2023-09-26T08:59:17.000Z

@slavag Not sure that i understand, -Djava.io.tmpdir does not work?

Answer 33 · 2023-09-26T20:17:38.000Z

@tamir-michaeli we're using -Djava.io.tmpdir for our needs to point on LVM volume, and same for logs.io appender.
I'm asking if there's another option in appender to point to different location, other than -Djava.io.tmpdir ?

Answer 34 · 2023-09-27T08:11:32.000Z

@slavag You can use queueDir parameter for the disk queue - https://github.com/logzio/logzio-log4j2-appender#parameters-for-disk-queue

Answer 35 · 2023-09-27T10:02:57.000Z

@tamir-michaeli thanks, will check.

Answer 36 · 2023-10-02T13:25:26.000Z

@tamir-michaeli works, thanks