odd behavior when replicating
3commascapital opened this issue · 1 comments
3commascapital commented
hi there, it seems that both the latest (2.4.2) as well as previous versions have some odd behaviors when streaming is setup. i am not sure if this is the right repo to be reporting this to, but here is an example of my logs after i run docker compose up CONTAINER_NAMES to start up my containers. This is after I have run docker compose down as well as docker volume rm $(docker volume ls -q)
infura-postgres-rw:
container_name: infura-postgres-rw
image: timescale/timescaledb:2.4.2-pg13-bitnami
ports:
- 4020:5432
# shm_size: 512m
volumes:
- 'postgresql_master_data:/bitnami/postgresql'
environment:
- "POSTGRESQL_REPLICATION_MODE=master"
- "POSTGRESQL_REPLICATION_USER=my_repl_user"
- "POSTGRESQL_REPLICATION_PASSWORD=my_repl_password"
- "ALLOW_EMPTY_PASSWORD=yes"
- "POSTGRESQL_USERNAME=infura"
- "POSTGRESQL_PASSWORD=password"
- "POSTGRESQL_DATABASE=infura"
- "POSTGRESQL_POSTGRES_PASSWORD=password"
# - "POSTGRESQL_SHARED_PRELOAD_LIBRARIES=timescaledb, pgaudit"
- "POSTGRES_NUM_SYNCHRONOUS_REPLICAS=1"
infura-postgres-ro:
container_name: infura-postgres-ro
image: timescale/timescaledb:2.4.2-pg13-bitnami
ports:
- '4021:5432'
depends_on:
- infura-postgres-rw
volumes:
- 'postgresql_slave_data:/bitnami/postgresql'
environment:
- "POSTGRESQL_REPLICATION_MODE=slave"
- "POSTGRESQL_REPLICATION_USER=my_repl_user"
- "POSTGRESQL_REPLICATION_PASSWORD=my_repl_password"
- "POSTGRESQL_MASTER_HOST=infura-postgres-rw"
- "POSTGRESQL_USERNAME=infura"
- "POSTGRESQL_PASSWORD=password"
- "POSTGRESQL_MASTER_PORT_NUMBER=5432"
# - "POSTGRESQL_SHARED_PRELOAD_LIBRARIES=timescaledb, pgaudit"➜ infura git:(master) ✗ docker compose up infura-postgres-rw infura-postgres-ro
[+] Running 6/6
⠿ Network infura_infura Created 0.0s
⠿ Network infura_default Created 0.0s
⠿ Volume "infura_postgresql_master_data" Created 0.0s
⠿ Volume "infura_postgresql_slave_data" Created 0.0s
⠿ Container infura-postgres-rw Created 0.1s
⠿ Container infura-postgres-ro Created 0.0s
Attaching to infura-postgres-ro, infura-postgres-rw
infura-postgres-rw | postgresql 03:39:41.80
infura-postgres-rw | postgresql 03:39:41.82 Welcome to the Bitnami postgresql container
infura-postgres-rw | postgresql 03:39:41.84 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
infura-postgres-rw | postgresql 03:39:41.86 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
infura-postgres-rw | postgresql 03:39:41.87
infura-postgres-rw | postgresql 03:39:42.01 INFO ==> ** Starting PostgreSQL setup **
infura-postgres-rw | postgresql 03:39:42.14 INFO ==> Validating settings in POSTGRESQL_* env vars..
infura-postgres-rw | postgresql 03:39:42.17 WARN ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
infura-postgres-rw | postgresql 03:39:42.24 INFO ==> Loading custom pre-init scripts...
infura-postgres-rw | postgresql 03:39:42.30 INFO ==> Initializing PostgreSQL database...
infura-postgres-rw | postgresql 03:39:42.52 INFO ==> pg_hba.conf file not detected. Generating it...
infura-postgres-rw | postgresql 03:39:42.55 INFO ==> Generating local authentication configuration
infura-postgres-ro | postgresql 03:39:42.55
infura-postgres-ro | postgresql 03:39:42.58 Welcome to the Bitnami postgresql container
infura-postgres-ro | postgresql 03:39:42.60 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
infura-postgres-ro | postgresql 03:39:42.62 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
infura-postgres-ro | postgresql 03:39:42.64
infura-postgres-ro | postgresql 03:39:42.79 INFO ==> ** Starting PostgreSQL setup **
infura-postgres-ro | postgresql 03:39:42.92 INFO ==> Validating settings in POSTGRESQL_* env vars..
infura-postgres-ro | postgresql 03:39:42.99 INFO ==> Loading custom pre-init scripts...
infura-postgres-ro | postgresql 03:39:43.04 INFO ==> Initializing PostgreSQL database...
infura-postgres-ro | postgresql 03:39:43.26 INFO ==> pg_hba.conf file not detected. Generating it...
infura-postgres-ro | postgresql 03:39:43.28 INFO ==> Generating local authentication configuration
infura-postgres-ro | postgresql 03:39:43.35 INFO ==> Waiting for replication master to accept connections (60 timeout)...
infura-postgres-ro | infura-postgres-rw:5432 - no response
infura-postgres-ro | infura-postgres-rw:5432 - no response
infura-postgres-ro | infura-postgres-rw:5432 - no response
infura-postgres-rw | postgresql 03:39:45.85 INFO ==> Starting PostgreSQL in background...
infura-postgres-rw | postgresql 03:39:46.36 INFO ==> Changing password of postgres
infura-postgres-rw | postgresql 03:39:46.46 INFO ==> Creating user infura
infura-postgres-rw | postgresql 03:39:46.56 INFO ==> Granting access to "infura" to the database "infura"
infura-postgres-ro | infura-postgres-rw:5432 - no response
infura-postgres-rw | postgresql 03:39:46.76 INFO ==> Setting ownership for the 'public' schema database "infura" to "infura"
infura-postgres-rw | postgresql 03:39:46.90 INFO ==> Creating replication user my_repl_user
infura-postgres-rw | postgresql 03:39:47.05 INFO ==> Configuring replication parameters
infura-postgres-rw | postgresql 03:39:47.44 INFO ==> Configuring fsync
infura-postgres-ro | infura-postgres-rw:5432 - no response
infura-postgres-rw | postgresql 03:39:47.67 INFO ==> Loading custom scripts...
infura-postgres-rw | postgresql 03:39:47.72 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
infura-postgres-rw | postgresql 03:39:47.74 INFO ==> Starting PostgreSQL in background...
infura-postgres-rw | psql:/tmp/tmp.egBpPIKpFC:1: ERROR: permission denied to create extension "timescaledb"
infura-postgres-rw | HINT: Must have CREATE privilege on current database to create this extension.
infura-postgres-rw | psql:/tmp/tmp.egBpPIKpFC:1: ERROR: permission denied to create extension "timescaledb"
infura-postgres-rw | HINT: Must have CREATE privilege on current database to create this extension.
infura-postgres-rw | psql:/tmp/tmp.egBpPIKpFC:1: WARNING:
infura-postgres-rw | WELCOME TO
infura-postgres-rw | _____ _ _ ____________
infura-postgres-rw | |_ _(_) | | | _ \ ___ \
infura-postgres-rw | | | _ _ __ ___ ___ ___ ___ __ _| | ___| | | | |_/ /
infura-postgres-rw | | | | | _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
infura-postgres-rw | | | | | | | | | | __/\__ \ (_| (_| | | __/ |/ /| |_/ /
infura-postgres-rw | |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
infura-postgres-rw | Running version 2.4.2
infura-postgres-rw | For more information on TimescaleDB, please visit the following links:
infura-postgres-rw |
infura-postgres-rw | 1. Getting started: https://docs.timescale.com/timescaledb/latest/getting-started
infura-postgres-rw | 2. API reference documentation: https://docs.timescale.com/api/latest
infura-postgres-rw | 3. How TimescaleDB is designed: https://docs.timescale.com/timescaledb/latest/overview/core-concepts
infura-postgres-rw |
infura-postgres-rw | Note: TimescaleDB collects anonymous reports to better understand and assist our users.
infura-postgres-rw | For more information and how to disable, please see our docs https://docs.timescale.com/timescaledb/latest/how-to-guides/configuration/telemetry.
infura-postgres-rw |
infura-postgres-rw | CREATE EXTENSION
infura-postgres-ro | infura-postgres-rw:5432 - no response
infura-postgres-rw | Using postgresql.conf at this path:
infura-postgres-rw | /opt/bitnami/postgresql/conf/postgresql.conf
infura-postgres-rw |
infura-postgres-rw | Writing backup to:
infura-postgres-rw | /tmp/timescaledb_tune.backup202109230339
infura-postgres-rw |
infura-postgres-rw | Recommendations based on 3.84 GB of available memory and 2 CPUs for PostgreSQL 13
infura-postgres-rw | shared_preload_libraries = 'pgaudit,timescaledb'
infura-postgres-rw | shared_buffers = 1007021kB
infura-postgres-rw | effective_cache_size = 2950MB
infura-postgres-rw | maintenance_work_mem = 503510kB
infura-postgres-rw | work_mem = 10070kB
infura-postgres-rw | timescaledb.max_background_workers = 8
infura-postgres-rw | max_worker_processes = 13
infura-postgres-rw | max_parallel_workers_per_gather = 1
infura-postgres-rw | max_parallel_workers = 2
infura-postgres-rw | wal_buffers = 16MB
infura-postgres-rw | min_wal_size = 512MB
infura-postgres-rw | max_wal_size = 1GB
infura-postgres-rw | default_statistics_target = 500
infura-postgres-rw | random_page_cost = 1.1
infura-postgres-rw | checkpoint_completion_target = 0.9
infura-postgres-rw | max_connections = 50
infura-postgres-rw | max_locks_per_transaction = 64
infura-postgres-rw | autovacuum_max_workers = 10
infura-postgres-rw | autovacuum_naptime = 10
infura-postgres-rw | effective_io_concurrency = 256
infura-postgres-rw | timescaledb.last_tuned = '2021-09-23T03:39:49Z'
infura-postgres-rw | timescaledb.last_tuned_version = '0.11.0'
infura-postgres-rw | Saving changes to: /opt/bitnami/postgresql/conf/postgresql.conf
infura-postgres-rw | postgresql 03:39:49.06 INFO ==> Enabling remote connections
infura-postgres-rw | postgresql 03:39:49.19 INFO ==> Stopping PostgreSQL...
infura-postgres-rw | waiting for server to shut down.... done
infura-postgres-rw | server stopped
infura-postgres-rw | postgresql 03:39:49.37 INFO ==> ** PostgreSQL setup finished! **
infura-postgres-rw |
infura-postgres-rw | postgresql 03:39:49.50 INFO ==> ** Starting PostgreSQL **
infura-postgres-rw | 2021-09-23 03:39:49.594 GMT [1] LOG: pgaudit extension initialized
infura-postgres-rw | 2021-09-23 03:39:49.668 GMT [1] LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
infura-postgres-rw | 2021-09-23 03:39:49.672 GMT [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
infura-postgres-rw | 2021-09-23 03:39:49.673 GMT [1] LOG: listening on IPv6 address "::", port 5432
infura-postgres-rw | 2021-09-23 03:39:49.676 GMT [1] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
infura-postgres-rw | 2021-09-23 03:39:49.691 GMT [533] LOG: database system was shut down at 2021-09-23 03:39:49 GMT
infura-postgres-rw | 2021-09-23 03:39:49.708 GMT [1] LOG: database system is ready to accept connections
infura-postgres-rw | 2021-09-23 03:39:49.733 GMT [542] LOG: TimescaleDB background worker launcher connected to shared catalogs
infura-postgres-ro | infura-postgres-rw:5432 - accepting connections
infura-postgres-ro | postgresql 03:39:49.86 INFO ==> Replicating the initial database
infura-postgres-ro | pg_basebackup: initiating base backup, waiting for checkpoint to complete
infura-postgres-ro | pg_basebackup: checkpoint completed
infura-postgres-ro | pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
infura-postgres-ro | pg_basebackup: starting background WAL receiver
infura-postgres-ro | pg_basebackup: created temporary replication slot "pg_basebackup_557"
infura-postgres-ro | 0/33384 kB (0%), 0/1 tablespace (...ami/postgresql/data/backup_label)
infura-postgres-ro | 27137/33384 kB (81%), 0/1 tablespace (.../postgresql/data/base/13397/2838)
infura-postgres-ro | 33393/33393 kB (100%), 0/1 tablespace (...ostgresql/data/global/pg_control)
infura-postgres-ro | 33393/33393 kB (100%), 1/1 tablespace
infura-postgres-ro | pg_basebackup: write-ahead log end point: 0/2000100
infura-postgres-ro | pg_basebackup: waiting for background process to finish streaming ...
infura-postgres-ro | pg_basebackup: syncing data to disk ...
infura-postgres-ro | pg_basebackup: renaming backup_manifest.tmp to backup_manifest
infura-postgres-ro | pg_basebackup: base backup completed
infura-postgres-ro | postgresql 03:39:51.41 INFO ==> Configuring replication parameters
infura-postgres-ro | postgresql 03:39:51.72 INFO ==> Configuring fsync
infura-postgres-ro | postgresql 03:39:51.79 INFO ==> Setting up streaming replication slave...
infura-postgres-ro | postgresql 03:39:52.11 INFO ==> Loading custom scripts...
infura-postgres-ro | postgresql 03:39:52.16 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
infura-postgres-ro | postgresql 03:39:52.18 INFO ==> Starting PostgreSQL in background...
infura-postgres-ro exited with code 1
i have been seeing a variety of behaviors when trying to get these to play nice with each other, however, it seems that regardless of whether or not i add the POSTGRESQL_SHARED_PRELOAD_LIBRARIES env i get the section about not having permission to create the extension timescaledb, but then the timescaledb always ends up running later anyway, which is kind of confusing.
also, if i don't start them together, on the second run, i am able to get the master to eventually be at a stable state, but no matter how many times i restart the slave, it always ends up exiting with code 1
erimatnor commented
This repository is for the Docker image, so if this is a general replication-related issue, you should either file a bug with PostgreSQL or if it is related to TimescaleDB, file a bug report in the main TimescaleDB repository.