[Fatal Error] :-1:-1: Premature end of file when recreating geoserver container (version 2.23.0)

[Ja-Woj]

What is the bug or the crash?

Geoserver logs :
[Fatal Error] 👎-1: Premature end of file
On re-create attempt of geoserver container
Geoserver has mounted geoserver data dir and geowebcache data dir in form of volumes.
Geoserver has configuration applied via api. During investigation we narrowed down cause of failure to loading user configuration from files (that is located on volumes/ pvc on kubernetes)

Steps to reproduce the issue

1. run docker compose up using beneath file :

version: '3.9'

volumes:
  geoserver-data:

services:
  geoserver:
    image: kartoza/geoserver:2.23.0
    volumes:
      - geoserver-data:/opt/geoserver/data_dir
    ports:
      - "8600:8080"
    restart: on-failure
    environment:
        - GEOSERVER_DATA_DIR=/opt/geoserver/data_dir
        - GEOSERVER_ADMIN_USER=user
        - GEOSERVER_ADMIN_PASSWORD=password
        - STABLE_EXTENSIONS=charts-plugin,sqlserver-plugin,params-extractor-plugin
        - TOMCAT_EXTRAS=false
        - DISABLE_WEB_INTERFACE=false
    healthcheck:
      test: "curl --fail --silent --write-out 'HTTP CODE : %{http_code}\n' --output /dev/null -u user:'password' http://localhost:8600/geoserver/rest/about/version.xml"
      interval: 1m30s
      timeout: 10s
      retries: 3

2. add user vi api
   POST http://localhost:8600/geoserver/rest/security/usergroup/users/
   request body :

<?xml version="1.0" encoding="UTF-8"?>
<user>
    <userName>userxml</userName>
    <password>passwordxml</password>
    <enabled>true</enabled>
</user>

3. delete container and run docker compose up again (it will create a new container) Make sure you run a new container , not starting exising one

4. Find in logs : [Fatal Error] 👎-1: Premature end of file
   Geoserver won't sucessfully start

This is local reproduction of the same issue that happens on kubernetes cluster

Versions

Geoserver - 2.23.0

Additional context

No response

[NyakudyaA]

Have you tried without specifying any plugins at all? Note that the majority if not all test in https://github.com/kartoza/docker-geoserver/tree/develop/scenario_tests run without plugins except the clustering

[JakubBakowski]

@NyakudyaA I work with OP on problem he described, and since he's offline now I took liberty of testing this scenario without specifying any plugins, result it the same "Fatal error premature end of file"

EDIT Also if I am correct in tests you've send link to, you are stopping container and starting it again. In scenario we provided, we need to delete container and create it from scratch, only thing left should be volume ( or if working on k8s PVC)

EDIT2 Maybe its worth mentioning, same scenario worked fine on version 2.21.1

[Ja-Woj]

@NyakudyaA I confirm that the same result occurs without any plugins.

It is crucial to recreate container in this scenario. This is reproducion of scenario that happens on kuberenetes cluster. You cannot stop and then start the same container . Kubernetes scheduler will always spin up a new pod with a new instance of container but it will use the same volume. Typical scenario when it can happen are - kuberenetes cluster upgrade, rescheduling geoserver pod on a new k8s worker node, geoserver deployment upgrade.

[NyakudyaA]

@Ja-Woj cc @JakubBakowski This works locally on my docker-compose setup, we could also add this to the tests #586

for now, this works on a clean environment using docker-compose.yml

Maybe this concerns how you are mounting your volume in Kubernetes and what storage class you are using. Does it fully support the chown and are all the files owned by the correct users?

[NyakudyaA]

[Ja-Woj]

Hi @NyakudyaA

Thanks for reply but I am not sure wheter you reproduced full scenerio, precisely point 3.

delete container and run docker compose up again (it will create a new container) Make sure you run a new container , not starting exising one

So please execute :

1. docker stop 6b85n24de81
2. docker rm 6b8c65461ff7
3. docker compose up

I believe this should reproduce this failure

Maybe this concerns how you are mounting your volume in Kubernetes and what storage class you are using. Does it fully support the chown and are all the files owned by the correct users?

We verified that all files are owned by the correct user group in k8s

[jeremypioche]

Hi,

I experience the same behavior as @Ja-Woj, but on v2.23.1.

To add more information to the issue, the root cause of Premature end of file error is that the users.xml located in data_dir/security/usergroup/default/ is empty. If i restore the content of this file to default, i can start successfully the container.

Our scenario to reproduce the bug:

1. start the container
2. add a user with the admin panel

at this point, users.xml looks like this :

<?xml version="1.0" encoding="UTF-8"?><userRegistry xmlns="http://www.geoserver.org/security/users" version="1.0">
<users>
<user enabled="true" name="admin" password="digest1:password_placeholder"/>
<user enabled="true" name="test" password="digest1:password_placeholder"/>
</users>
<groups/>
</userRegistry>

3. restart the container
   at this point, admin panel is not accessible anymore, and the users.xml file is empty

[NyakudyaA]

I think what could be causing this is that the users are most likely being created at each restart,
Can you try to set the env
EXISTING_DATA_DIR=TRUE
Before you restart your container. This should allow reset logic to be skipped1

[NyakudyaA]

@Ja-Woj cc @jeremypioche I managed to replicate this. The issue is caused by the location of the lockfile which the password script looks for. It currently defaults to https://github.com/kartoza/docker-geoserver/blob/develop/scripts/update_passwords.sh#L18

which in Kubernetes is not persistent by it's nature because each time a container restart it will delete settings. There are two solutions here

1. Set an extra env variable and a volume to manage lockflie persistence:

-e EXTRA_CONFIG_DIR=/settings

then mount a volume to this path

-v settings:/settings

2. Run your container and when it is up and running, update your env variables to add

 EXISTING_DATA_DIR=TRUE 

Maybe we just need to be more verbose about this in the README