set-up guidance

Question

set-up guidance

Closed this issue 4 years ago · 46 comments

@pini-gh , thanks for making this container and configuration available. I am in the process to set things up - this is what I did so far:

Assuming my server domain is example.com and I have the docker-mailman setup running in the mailman-docker stack - with the postorius web interface available at https://lists.example.com)

To start, the first thing I did was to generate locally on the host machine a pair of TLS perm and key with the following instruction:

/usr/bin/openssl req -x509 -sha256 -days 9000 -nodes -newkey rsa:4096 -keyout example.com.key -out example.com.cert

I left those files in the root directory of the docker-compose file. So the .env file did not require any change (besides using my real domain instead of 'example') and this is how it looks like:

COMPOSE_PROJECT_NAME=exim4

SERVER_HOSTNAME=exim4
SERVER_DOMAINNAME=example.com
SERVER_MAILNAME=exim4.example.com

# Semicolon separated list of local domains for incoming mail
LOCAL_DOMAINS=lists.example.com

# TLS enabled if and only if set to "TRUE" (case sensitive)
EXIM4_TLS_ENABLE=TRUE
EXIM4_TLS_CERTIFICATE=example.com.crt
EXIM4_TLS_PRIVATEKEY=example.com.key

Q: I wonder how to make sure those files (.crt and .key) are available to the exim container - should I mount them as volumes in a special location (e.g.: /var/spool/exim4/cert) or is the path to those files the one I set in the .env file?

As mentioned in maxking/docker-mailman#414 I am trying to make this environment working within the mailman setup. So I guess I need to enable the mailman plug-in in the env_plugin file uncommenting the relative line which now looks like:

# Json table for mailman servers if any
PLUGIN_SPEC_MAILMAN={"plugin": "mailman", "instances": [{"domainlist": "lists.example.com", "host": "mailman-core", "home": "/opt/lists.example.com/var"}]}

Do I need to make sure the "/opt/lists.example.com/var" directory is available (e.g.: run on the host machine: mkdir -p /opt/lists.example.com/var)

I then uncommented the line related to mailman in the volumes

    volumes:
      - cert:/var/spool/exim4/cert
      - mailman_core-vol:/opt/lists.example.com:ro

Finally, I run the docker-compose and when attempting to use the postorius web-interface ... I got it to work!
I no longer have the error:

smtplib.SMTPRecipientsRefused: {'my_personal_email@me.com': (451, b'4.3.0 <my_personal_email@me.com>: Temporary lookup failure')}

And I was actually able to use the interface - I was able to reset the password by receiving the :

[lists.example.com] Password Reset E-mail

I logged in and created a domain and I received another e-mail:

[lists.example.com] Please Confirm Your E-mail Address

I then created a new mailing list for testing purpose, and I received a welcoming e-maiol:

Welcome to the "example" mailing list

To post to this list, send your email to:

  example@example.com

You can unsubscribe or make adjustments to your options via email by
sending a message to:

  example-request@example.com

Yeah! ... I then tried to send a message, but It failed - this the message I got:

Undelivered Mail Returned to Sender

This is a system-generated message to inform you that your email could not
be delivered to one or more recipients. Details of the email and the error are as follows:


<example@example.com>: host mail.example.com[178.63.52.22] said: 550 relay not
    permitted (in reply to RCPT TO command)
Reporting-MTA: dns; ms11p00im-qufo17282101.me.com
X-Postfix-Queue-ID: 9354A7801F1
X-Postfix-Sender: rfc822; my_personal_email@icloud.com
Arrival-Date: Sun,  8 Nov 2020 19:40:12 +0000 (UTC)

Final-Recipient: rfc822; example@example.com
Original-Recipient: rfc822;example@example.com
Action: failed
Status: 5.0.0
Remote-MTA: dns; mail.example.com
Diagnostic-Code: smtp; 550 relay not permitted

A first search on the web pointed me to:

"""You need to add the IP address of your host(s) to the list of the "permitted relayers"."""

I guess I am still missing few bits ...

Thanks a lot for your help!

Answer 1 · 2020-11-08T20:46:43.000Z

little update, as the error was related to a missing host, I guessed it was looking for the mailman-core hostname, from the mailman docker stack - so I moved this docker instruction inside my mailman docker stack.

Within these settings, I tried to send a test email to the mailing list address - this time no delivery failure message ... but nothing seems to be received on the server - by looking at the logs for the exim4 service I notice some issues: log attached as gist
https://gist.github.com/98b927cf8f2db4fdaab2b9531de2b367

Answer 2 · 2020-11-08T20:48:04.000Z

Q: I wonder how to make sure those files (.crt and .key) are available to the exim container - should I mount them as volumes in a special location (e.g.: /var/spool/exim4/cert) or is the path to those files the one I set in the .env file?

The cert volume is there for hosting the key and the certificate. The environment variables EXIM4_TLS_CERTIFICATE and EXIM4_TLS_PRIVATEKEY hold their relative path into the volume.

Do I need to make sure the "/opt/lists.example.com/var" directory is available (e.g.: run on the host machine: mkdir -p /opt/lists.example.com/var)

No, you don't need to create a directory. The path /opt/lists.example.com/ is the mount point of tthe mailman-core volume into the exim4 container.

I then uncommented the line related to mailman in the volumes

Fine. The mailman-core volume should be declared as external. This is what I have into my mailman's compose file:

services:
  mailman-core:
  ...
    volumes:
    - core-vol:/opt/mailman
...
volumes:
  core-vol:
  web-vol:
  database-vol:

And into my exim4's compose file:

services:
  exim4:
  ...
    volumes:
      - cert:/var/spool/exim4/cert
      - mailman_core-vol:/opt/lists.example.com:ro
...
volumes:
  cert:
    external: true
    name: reverse-proxy_certs
  mailman_core-vol:
    external: true

Note: the cert volume is external as well because on my server the key / certificate pair is managed via docker-letsencrypt-nginx-proxy-companion.

"""You need to add the IP address of your host(s) to the list of the "permitted relayers"."""

I guess I am still missing few bits ...

Did you set the LOCAL_DOMAINS variable into the .env file?

# Semicolon separated list of localdomains for incoming mail
LOCAL_DOMAINS=lists.example.com

Answer 3 · 2020-11-08T21:32:50.000Z

by looking at the logs for the exim4 service I notice some issues

The logs read as if you didn't define any EXIM4_* variable.

Answer 4 · 2020-11-09T12:47:32.000Z

I did some fixes, thanks for the pointers :) - still something missing/misconfigured from my side.

To avoid typos and misconfiguration I am no longer hiding the domain I use for the server - which is: epinux.com -- while the postorius interface is exposed via traefik-https at : https://lists.epinux.com

A check on the mailman network setup which is based on this docker-swarm file, returns the following configuration

For the TLS issue:

The logs read as if you didn't define any EXIM4_* variable.

Maybe the problem was the owner of those files - I regenerated the epinux.key and epinux.crt and then changed the ownership with:

chown Debian-exim:Debian-exim /var/spool/exim4/cert/epinux.*

For the LOCAL_DOMAINS - I am struggling to find a solution - I 'maybe added too many entries after several attempts - my .env file looks like:

COMPOSE_PROJECT_NAME=exim4

SERVER_HOSTNAME=exim4
SERVER_DOMAINNAME=epinux.com
SERVER_MAILNAME=exim4.epinux.com

# Semicolon separated list of local domains for incoming mail
LOCAL_DOMAINS=lists.epinux.com;mailman-core;mailman-web;10.0.0.2;172.18.0.1;epinux.com;xxx.xx.xx.1,xxx.xx.xx.xx

# TLS enabled if and only if set to "TRUE" (case sensitive)
EXIM4_TLS_ENABLE=TRUE
EXIM4_TLS_CERTIFICATE=epinux.crt
EXIM4_TLS_PRIVATEKEY=epinux.key

where:

- epinux.com -> my domain
- lists.epinux.com -> my mailing list subdomain+domain
- mailman-core  & mailman-web -> I guess unused as it assumes knowledge of the mailman docker environment
- 10.0.0.2 ->  ... from the exim logs
- 172.18.0.1 -> the  results from `ip route` from inside the mailman-core container
- xxx.xx.xx.1  & xxx.xx.xx.xx -> my serve IP

Which is loaded by my Exim docker-compose file

I repeated all the process from the start, using new volumes for the mailman docker environment.

I am attaching the whole log from the Exim container - which includes the steps of reset password, user registration/confirmation, domain and email-list creation, etc .. (I used )
exim log

which shows the following error, when attempting to send an email to the list form a registered address:

exim4    | Nov  9 12:19:16 exim4 exim[419]: 2020-11-09 12:19:16 H=ms11p00im-qufo17291501.me.com [17.58.38.44] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<epiesasha@icloud.com> rejected RCPT <epinux@lists.epinux.com>: Unrouteable address

I am probably doing something wrong when registering a domain and a new mailing list on postorius ..

Mail	Host	Description	Alias Domain	Owners	Action
lists.epinux.com	epinux		lists.epinux.com (lists.epinux.com) (Edit) SITE_ID = 1	epiesasha@me.com	Templates Edit Delete

Answer 5 · 2020-11-09T13:35:48.000Z

The exim{ log looks good. These lines:

exim4    | Nov  9 12:12:03 exim4 exim[404]: 2020-11-09 12:12:03 1kc61j-00006W-4i <= postorius@lists.epinux.com H=(mailman-web) [172.30.0.1] P=esmtp S=857 id=160492392314.1568.13205874217900846180@mailman-web
exim4    | Nov  9 12:12:07 exim4 exim[406]: 2020-11-09 12:12:07 1kc61j-00006W-4i => epiesasha@me.com R=dnslookup T=remote_smtp H=mx02.mail.icloud.com [17.57.152.14] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes DN="CN=mx01.mail.icloud.com,OU=management:idms.group.859635,O=Apple Inc.,ST=California,C=US" K C="250 2.0.0 Ok: 879 bytes queued as B36FFD06B51"
exim4    | Nov  9 12:12:07 exim4 exim[406]: 2020-11-09 12:12:07 1kc61j-00006W-4i Completed

say that an email was sent to you by Postorious. Have you received it? If not, have you checked your spam folder?

The Unrouteable address could be caused by Exim4 not finding the list mount point. Try using /opt/lists.epinux.com instead of /opt/lists.epinux.com/var in PLUGIN_SPEC_MAILMAN.

Update: I was wrong about this last part. It should be "home": "/opt/lists.epinux.com/var".

Answer 6 · 2020-11-09T14:00:53.000Z

Hi! @pini-gh - Yes I can receive the e-mail sent from postorius, the ones regarding all the configuration processes.
I then tried to create a mailing list and I subscribed to my email as a user of such a list - the Unrouteable address arises when attempting to send an email from my registered personal email to the newly created list.

looking at the PLUGIN_SPEC_MAILMAN what should be the "host"? for now, I have: "host": "mailman-core" which maybe is not reachable? I tried replacing it with the mailman-core IP .. but that didn't work.

Answer 7 · 2020-11-09T14:35:27.000Z

PLease log into the exim4 instance and check the existence of the path /opt/lists.epinux.com/var/lists/epinux.lists.epinux.com.

Answer 8 · 2020-11-09T14:49:39.000Z

Another easy test to run :

log into the exim4 container
use this command to check whether your list address is routable:

# exim4 -bt mylist@lists.example.com
R: system_aliases for mylist@lists.example.com
mylist@lists.example.com
  router = mailman3_router_0, transport = mailman3_transport_0

If the result is not good, check the exim4 configuration file /etc/exim4/conf.d/main/25_mm3_macros which should resemble:

# Place this file at
# /etc/exim4/conf.d/main/25_mm3_macros

domainlist mm3_domains_0=lists.example.com
MM3_LMTP_HOST_0=mailman-core
MM3_LMTP_PORT_0=8024
MM3_HOME_0=/opt/lists.example.com/var

################################################################
# The configuration below is boilerplate:
# you should not need to change it.

# The path to the list receipt (used as the required file when
# matching list addresses)
MM3_LISTCHK_0=MM3_HOME_0/lists/${local_part}.${domain}

You can also try to ping mailman-core from inside your exim4 container.

Answer 9 · 2020-11-09T17:12:35.000Z

PLease log into the exim4 instance and check the existence of the path /opt/lists.epinux.com/var/lists/epinux.lists.epinux.com.

root@exim4:/# ls opt/
lists.epinux.com   #  <-- this exist
root@exim4:/# ls opt/lists.epinux.com/ # <--- but it is empty
root@exim4:/#

log into the exim4 container

use this command to check whether your list address is routable:

root@exim4:/# exim4 -bt epinux@lists.epinux.com
R: system_aliases for epinux@lists.epinux.com
epinux@lists.epinux.com is undeliverable: Unrouteable address

check the exim4 configuration file /etc/exim4/conf.d/main/25_mm3_macros

root@exim4:/# cat /etc/exim4/conf.d/main/25_mm3_macros
# Place this file at
# /etc/exim4/conf.d/main/25_mm3_macros

domainlist mm3_domains_0=lists.epinux.com
MM3_LMTP_HOST_0=mailman-core
MM3_LMTP_PORT_0=8024
MM3_HOME_0=/opt/lists.epinux.com/var

################################################################
# The configuration below is boilerplate:
# you should not need to change it.

# The path to the list receipt (used as the required file when
# matching list addresses)
MM3_LISTCHK_0=MM3_HOME_0/lists/${local_part}.${domain}

ping mailman-core from inside your exim4 container.

root@exim4:/# ping mailman-core
ping: mailman-core: Name or service not known

Answer 10 · 2020-11-09T17:36:17.000Z

root@exim4:/# ls opt/lists.epinux.com/ # <--- but it is empty
root@exim4:/#

This problem should be fixed first. What if you log into mailman-core and inspect this path /opt/mailman/var/lists/?

Answer 11 · 2020-11-09T17:38:16.000Z

This problem should be fixed first. What if you log into mailman-core and inspect this path /opt/mailman/var/lists/?

bash-5.0# ls /opt/mailman/var/lists/
epinux.lists.epinux.com
bash-5.0# ls /opt/mailman/var/lists/epinux.lists.epinux.com/
bash-5.0#

Answer 12 · 2020-11-10T09:14:08.000Z

@pini-gh, your help is being invaluable, thanks!

Following your directions, I discovered I had a mismatching between the volume names used in Exim which has to re-use the same volumes from the mailman-swarm settings. I didn't know the swarm settings was attaching a prefix swarm-name_ to volume-name- in my case, the swarm was named lists and the volumes declaration were:

volumes:
  core-vol:
  web-vol:
  database-vol:

So the matching exim4 configuration was:

volumes:
  cert:
    external: true
  mailman_core-vol:
    external: true
    name: lists_core-vol

The other issue was about the network settings. Exim4needs to be attached to the same mailman network, - this was possible by specifying the attachable: true option in the swarm settings - so the final networks configuration for the two environments looked like:

swarm network:

networks:
  net:
    driver: overlay
    attachable: true
  traefik-public:
    external: true
  mailman:
    driver: overlay
    attachable: true

exim4 network:

networks:
  bridge:
    external: true
    name: lists_mailman

I can now have access to the list, register new users and create a domain and a mailing list - I then try to send a message to the list using the registered account and the log says it has been accepted:

exim4    | Nov 10 09:06:13 exim4 exim[696]: 2020-11-10 09:06:13 1kcPbQ-0000BE-Tb <= epiesasha@me.com H=ms11p00im-qufo17281401.me.com [17.58.38.51] P=esmtps X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no S=2756 DKIM=me.com id=1dcd1ac5-1db2-4352-8f92-9a570646b31e@me.com
exim4    | Nov 10 09:06:13 exim4 exim[698]: 2020-11-10 09:06:13 1kcPbQ-0000BE-Tb => epinux@lists.epinux.com R=mailman3_router_0 T=mailman3_transport_0 H=mailman-core [10.0.126.2] C="250 Ok"
exim4    | Nov 10 09:06:13 exim4 exim[698]: 2020-11-10 09:06:13 1kcPbQ-0000BE-Tb Completed
exim4    | Nov 10 09:06:15 exim4 exim[701]: 2020-11-10 09:06:15 no host name found for IP address 172.18.0.1
exim4    | Nov 10 09:06:15 exim4 exim[701]: 2020-11-10 09:06:15 1kcPbT-0000BJ-1Z <= epinux-bounces@lists.epinux.com H=([10.0.126.3]) [172.18.0.1] P=esmtp S=4168 id=1dcd1ac5-1db2-4352-8f92-9a570646b31e@me.com
exim4    | Nov 10 09:06:16 exim4 exim[705]: 2020-11-10 09:06:16 no host name found for IP address 172.18.0.1
exim4    | Nov 10 09:06:16 exim4 exim[705]: 2020-11-10 09:06:16 1kcPbU-0000BN-5j <= epinux-bounces@lists.epinux.com H=([10.0.126.3]) [172.18.0.1] P=esmtp S=649 id=160499917460.36.3988928030605058723@mailman-core
exim4    | Nov 10 09:06:18 exim4 exim[706]: 2020-11-10 09:06:18 1kcPbU-0000BN-5j => epiesasha@me.com R=dnslookup T=remote_smtp H=mx01.mail.icloud.com [17.57.152.9] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes DN="CN=mx01.mail.icloud.com,OU=management:idms.group.859635,O=Apple Inc.,ST=California,C=US" K C="250 2.0.0 Ok: 670 bytes queued as 8A8436401C6"
exim4    | Nov 10 09:06:18 exim4 exim[706]: 2020-11-10 09:06:18 1kcPbU-0000BN-5j Completed
exim4    | Nov 10 09:06:20 exim4 exim[702]: 2020-11-10 09:06:20 1kcPbT-0000BJ-1Z => epiesasha@me.com R=dnslookup T=remote_smtp H=mx02.mail.icloud.com [17.42.251.12] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes DN="CN=mx01.mail.icloud.com,OU=management:idms.group.859635,O=Apple Inc.,ST=California,C=US" K C="250 2.0.0 Ok: 4262 bytes queued as A48E0200276"
exim4    | Nov 10 09:06:20 exim4 exim[702]: 2020-11-10 09:06:20 1kcPbT-0000BJ-1Z Completed

Also, the notification of a successful post (one activated in the settings) works as expected - I got the message:

Your message entitled

    [Epinux] test

was successfully received by the Epinux mailing list.

But, doesn't seem the messages get collected in the archive - which looks empty, -- I guess I am still missing something in the mailman/hyperkitty settings?
the only ** warning ** I see from the log is the following line:

exim4    | Nov 10 09:06:16 exim4 exim[705]: 2020-11-10 09:06:16 no hostname found for IP address 172.18.0.1

Answer 13 · 2020-11-10T18:17:08.000Z

exim4 | Nov 10 09:06:16 exim4 exim[705]: 2020-11-10 09:06:16 no hostname found for IP address 172.18.0.1

This is not related to hyperkitty AFAICT.

Log into the mailman-web container and check this log for clues: /opt/mailman-web-data/logs/mailmanweb.log. Look for this string: hyperkitty.views.mailman.

Answer 14 · 2020-11-10T18:43:26.000Z

I can see this:

ERROR 2020-11-10 18:39:32,073 1696 hyperkitty.views.mailman Access to the archiving API endpoint was forbidden from IP 10.0.1.13, your MAILMAN_ARCHIVER_FROM setting may be misconfigured
WARNING 2020-11-10 18:39:32,073 1696 django.request Forbidden: /hyperkitty/api/mailman/archive
WARNING 2020-11-10 18:39:32,073 1696 django.request Forbidden: /hyperkitty/api/mailman/archive

My ENV settings for the mailman-web container has:

MAILMAN_ARCHIVER_FROM=lists.epinux.com

while for the mailman-core I just set:

HYPERKITTY_URL=https://lists.epinux.com/hyperkitty

update:

googling I found someone with similar issue: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/RIXJRJTMICPWZGWIRVEY7UEECJ33QGJ3/
I tried to log-in into the web container and I run the following in a python shell:

>>> import os
>>> (gethostbyname(os.environ.get('MAILMAN_HOSTNAME', os.environ.get('MAILMAN_HOST_IP', 'mailman-core'))),)
('10.0.129.5',)
>>>

I thought it could be helpful .. as in settings.py I found the line below used to set the MAILMAN_ARCHIVER_FROM

MAILMAN_ARCHIVER_FROM = (gethostbyname(os.environ.get('MAILMAN_HOSTNAME', os.environ.get('MAILMAN_HOST_IP', 'mailman-core'))),)

Answer 15 · 2020-11-10T19:41:31.000Z

Are MAILMAN_HOSTNAME or MAILMAN_HOST_IP set in mailman-web environment?

What is returned by ping mailman-core when logged into mailman-web?

Answer 16 · 2020-11-10T20:53:07.000Z

Are MAILMAN_HOSTNAME or MAILMAN_HOST_IP set in mailman-web environment?

No, they are not set

What is returned by ping mailman-core when logged into mailman-web?

This works fine. I guess I should set it to be MAILMAN_HOSTNAME - what about the MAILMAN_HOST_IP?

Answer 17 · 2020-11-10T21:11:00.000Z

What is returned by ping mailman-core when logged into mailman-web?

This works fine. I guess I should set it to be MAILMAN_HOSTNAME - what about the MAILMAN_HOST_IP?

What is the returned IP for mailman-core then?

Answer 18 · 2020-11-10T21:16:31.000Z

It is 10.0.131.7

ping mailman-core
PING mailman-core (10.0.131.7): 56 data bytes
64 bytes from 10.0.131.7: seq=0 ttl=64 time=0.138 ms
64 bytes from 10.0.131.7: seq=1 ttl=64 time=0.137 ms
64 bytes from 10.0.131.7: seq=2 ttl=64 time=0.139 ms
64 bytes from 10.0.131.7: seq=3 ttl=64 time=0.148 ms
64 bytes from 10.0.131.7: seq=4 ttl=64 time=0.195 ms
64 bytes from 10.0.131.7: seq=5 ttl=64 time=0.147 ms

Answer 19 · 2020-11-10T21:27:59.000Z

It is 10.0.131.7

Then it's strange that gethostbyname(os.environ.get('MAILMAN_HOSTNAME', os.environ.get('MAILMAN_HOST_IP', 'mailman-core'))) returns 10.0.129.5

Answer 20 · 2020-11-10T21:32:58.000Z

my fault ... seems it changes everytime I start and stop the service.

this time I did it in a single session, and they match:

import os
from socket import gethostbyname
>>> gethostbyname(os.environ.get('MAILMAN_HOSTNAME', os.environ.get('MAILMAN_HOST_IP', 'mailman-core')))
'10.0.132.2'
>>> exit()
bash-5.0# ping mailman-core
PING mailman-core (10.0.132.2): 56 data bytes
64 bytes from 10.0.132.2: seq=0 ttl=64 time=0.169 ms
64 bytes from 10.0.132.2: seq=1 ttl=64 time=0.148 ms
64 bytes from 10.0.132.2: seq=2 ttl=64 time=0.137 ms

Answer 21 · 2020-11-10T21:48:53.000Z

OK. Then, in the same session, please check those two + the hyperkitty.views.mailman line of the log file. Logically it should report the very same IP.

Answer 22 · 2020-11-10T22:19:00.000Z

gethostbyname and ping return the same IP (10.0.137.5) - while: cat /opt/mailman-web-data/logs/mailmanweb.log | grep hyperkitty.views.mailman returns 10.0.1.13 when 'HYPERKITTY_URL' is not set or set to 'https://lists.epinux.com/hyperkitty'

AILMAN_ARCHIVER_FROM setting may be misconfigured
WARNING 2020-11-10 21:35:50,417 1701 django.request Forbidden: /hyperkitty/api/mailman/archive
WARNING 2020-11-10 21:35:50,417 1701 django.request Forbidden: /hyperkitty/api/mailman/archive
ERROR 2020-11-10 21:35:50,503 1701 hyperkitty.views.mailman Access to the archiving API endpoint was forbidden from IP 10.0.1.13, your MAILMAN_ARCHIVER_FROM setting may be misconfigured
WARNING 2020-11-10 21:35:50,503 1701 django.request Forbidden: /hyperkitty/api/mailman/archive
WARNING 2020-11-10 21:35:50,503 1701 django.request Forbidden: /hyperkitty/api/mailman/archive

Q: what is the value for HYPERKITTY_URL ?

Answer 23 · 2020-11-10T22:38:54.000Z

Looks like several networks are attached to mailman-core. Is that right?

Answer 24 · 2020-11-10T22:58:01.000Z

mailman-core has only:

    networks:
      mailman:

while mailman-web has:

    networks:
      mailman:
      traefik-public:

and at the end of my docker-swarm file I have:

networks:
  net:
    driver: overlay
    attachable: true
  traefik-public:
    external: true
  mailman:
    driver: overlay
    attachable: true

where net is not used by any service, I can try to remove it.

epinux@epinux:~/dev$ sudo docker network inspect lists_mailman
[
    {
        "Name": "lists_mailman",
        "Id": "sal2ynzpsl3fjgw4chz36qga8",
        "Created": "2020-11-10T23:16:48.879502704+01:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.137.0/24",
                    "Gateway": "10.0.137.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "05f8dd9a6a8e48ca57c01135a9562ce4cc263d59acc3eab833e4b3416b31b6d4": {
                "Name": "lists_database.1.f9zpdsxv5yu7j1ewb5fwp0bbp",
                "EndpointID": "f3aa577a81bfa143337f685abbfa26c0fdd08b83d5d6c701b6f855d3c5737cbf",
                "MacAddress": "02:42:0a:00:89:03",
                "IPv4Address": "10.0.137.3/24",
                "IPv6Address": ""
            },
            "19596bb595b53afd3d978506fd700d9fee757d40697e92f323ae97245ef3e3c2": {
                "Name": "exim4",
                "EndpointID": "fec82e58a63a8d5fb56e531e3eb238a73f8f3dd1e241388835add4a5e4d15b53",
                "MacAddress": "02:42:0a:00:89:09",
                "IPv4Address": "10.0.137.9/24",
                "IPv6Address": ""
            },
            "4fa7fc96917029fe81d68a4a9149f8f423476412797395beb592fae0f621d95b": {
                "Name": "lists_mailman-core.1.tmuwlwzaqfalk1v8fz9wt5w5t",
                "EndpointID": "c5a18dc6a0b74311680d34634521080410f9b47c8935c196a6181310319245c0",
                "MacAddress": "02:42:0a:00:89:06",
                "IPv4Address": "10.0.137.6/24",
                "IPv6Address": ""
            },
            "99147df5f78a22810201151a2e2828c3495c312330dbbf1f97b1fc67cbed3fae": {
                "Name": "lists_mailman-web.1.r1nhrmuraeb3oop5b3cbaak0x",
                "EndpointID": "cfa08eb80ab739aace5989d8b6b64602eaf743248f62e1b0af604ecca31779ea",
                "MacAddress": "02:42:0a:00:89:08",
                "IPv4Address": "10.0.137.8/24",
                "IPv6Address": ""
            },
            "lb-lists_mailman": {
                "Name": "lists_mailman-endpoint",
                "EndpointID": "06d71e1c6ca48d92e705720d79adc28bf674489ef6356e4de5a44e270b50d3fe",
                "MacAddress": "02:42:0a:00:89:04",
                "IPv4Address": "10.0.137.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4241"
        },
        "Labels": {
            "com.docker.stack.namespace": "lists"
        },
        "Peers": [
            {
                "Name": "81d98c4d2c5c",
                "IP": "xxx.xx.xx.xx.xx"
            }
        ]
    }
]

Answer 25 · 2020-11-10T23:03:59.000Z

the class of IP on the range 10.0.1.x seems the one from the traefik-public network, which is my proxy - where the mailman-web container is attached:

            "99147df5f78a22810201151a2e2828c3495c312330dbbf1f97b1fc67cbed3fae": {
                "Name": "lists_mailman-web.1.r1nhrmuraeb3oop5b3cbaak0x",
                "EndpointID": "1a7252935f080e5a4602428212d7684c26ecaa50fbe0ac6c0ff2f35ae37fc3c8",
                "MacAddress": "02:42:0a:00:01:b3",
                "IPv4Address": "10.0.1.179/24",
                "IPv6Address": ""
            },

Answer 26 · 2020-11-10T23:08:03.000Z

Which container does 10.0.1.13 belong to?

Answer 27 · 2020-11-10T23:22:02.000Z

none of my containers has this IP. or at least none of the one from the list below:

docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)

/lists_mailman-web.1.r1nhrmuraeb3oop5b3cbaak0x - 10.0.0.18410.0.137.810.0.1.179
/exim4 - 10.0.137.9
/lists_mailman-core.1.tmuwlwzaqfalk1v8fz9wt5w5t - 10.0.137.6
/lists_database.1.f9zpdsxv5yu7j1ewb5fwp0bbp - 10.0.137.3
/csw_postgis.1.z81wxvqyzh7mayw8nwg5k1q16 - 10.0.133.6
/csw_pycsw.1.hmghp4ou0mxkx2dpqtzuiii5e - 10.0.133.310.0.0.17310.0.1.170
/swarmpit_agent.j7p1g8b9b7p828w3gsptw4dn2.ukfaexs4w3t1z5rdy3oa45d47 - 10.0.2.16
/swarmpit_agent.j7p1g8b9b7p828w3gsptw4dn2.i9p2rcyu3n64u76emma43hp5o - 
/gallant_williams - 
/gracious_feistel - 
/ncplot_ncplot.1.zpnibz1yxs41klblao46duaiw - 10.0.0.19710.0.72.310.0.1.137
/pybasket_rabbit.1.ioufdnnfjj0e2etxslmz6be90 - 10.0.0.18910.0.69.3
/pybasket_worker.1.cnd0gm9bur1eo0c4l9ivt5sr8 - 10.0.68.6
/pybasket_fastapi.1.akpxwao2j7mgnr1v3ucnmxxgh - 10.0.0.19110.0.69.610.0.1.131
/pybasket_flower.1.xalsyy994mn7yiz4km5akaa3x - 10.0.0.18710.0.68.3
/phpmyadmin - 172.29.0.4
/docker_swarm_wordpress_1 - 172.29.0.2
/docker_swarm_wpdb_1 - 172.29.0.3
/swarmpit_agent.j7p1g8b9b7p828w3gsptw4dn2.njp3qu3rplliusbfxfj0bkk5u - 
/56725672_rabbit_1 - 192.168.96.2
/portainer_portainer.1.or8yiq2j4pdcsq69dvlt66ebh - 10.0.3.610.0.1.18
/thelounge_app.1.h0ihuigbkrrhrauo54w1ckw37 - 10.0.1.20
/swarmpit_influxdb.1.gixjc2kquu0w5qxjfjq7mgzhv - 10.0.2.11
/swarmpit_app.1.dmw192qb90nloq7311qw0ktfy - 10.0.0.1210.0.2.1010.0.1.17
/jupyter_jupyter.1.z1izsr5jitj9fdxphveh4y3kd - 10.0.0.1110.0.50.310.0.1.16
/traefik_traefik.1.zwo396stx4o3fyk456t78h87u - 10.0.0.810.0.1.13
/swarmpit_db.1.a8t4nd30qbiyhpf4bah007noq - 10.0.2.12
/hyrax_hyrax.1.qrixiuzga4zko73g7mpfv4pz2 - 10.0.36.310.0.0.710.0.1.11
/cloud_db.1.tmste97hmp5quz3ydninu7k23 - 10.0.27.610.0.1.12
/portainer_agent.j7p1g8b9b7p828w3gsptw4dn2.9sttg86wwb4kkxq1fxdfunu9c - 10.0.3.3
/cloud_nextcloud.1.ovw8s064k9nz0crgcwu06uwxi - 10.0.27.810.0.0.910.0.1.14

Answer 28 · 2020-11-10T23:24:55.000Z

Well, if the error message is consistent with this 10.0.1.13 address, then you could try setting MAILMAN_HOST_IP with this address and see if it makes it work.

Answer 29 · 2020-11-10T23:30:28.000Z

what should I use for:

MAILMAN_ARCHIVER_FROM and HYPERKITTY_URL

in the web container at the moment, this is the list of envs I have:

      - DATABASE_TYPE=postgres
      - DATABASE_URL=postgres://mailman:mailmanpass@database/mailmandb
      - HYPERKITTY_API_KEY=----------------------------------------------------------------
      - SECRET_KEY=ksjdbaksdba
      - UWSGI_STATIC_MAP=/static=/opt/mailman-web-data/static 
      - SERVE_FROM_DOMAIN=lists.epinux.com 
      - DJANGO_ALLOWED_HOSTS=lists.epinux.com
      - MAILMAN_ADMIN_USER=epinux
      - MAILMAN_ADMIN_EMAIL=epiesasha@me.com
      - MAILMAN_ARCHIVER_FROM=10.0.131.7 # lists.epinux.com # 
      - MAILMAN_HOSTNAME=mailman-core
      - MAILMAN_HOST_IP=10.0.1.13

Answer 30 · 2020-11-10T23:34:58.000Z

with the settings above, I got the same log:

ERROR 2020-11-10 23:31:57,293 1715 hyperkitty.views.mailman Access to the archiving API endpoint was forbidden from IP 10.0.1.13, your MAILMAN_ARCHIVER_FROM setting may be misconfigured
WARNING 2020-11-10 23:31:57,294 1715 django.request Forbidden: /hyperkitty/api/mailman/archive
WARNING 2020-11-10 23:31:57,294 1715 django.request Forbidden: /hyperkitty/api/mailman/archive

Answer 31 · 2020-11-10T23:35:39.000Z

You shouldn't have to set HYPERKITTY_URL. What you have to try is setting MAILMAN_HOST_IP=10.0.1.13 in mailman-web.

Answer 32 · 2020-11-10T23:39:35.000Z

Scrolling the log, instead of using grep .. I found this:

ERROR 2020-11-10 21:35:50,503 1701 hyperkitty.views.mailman Access to the archiving API endpoint was forbidden from IP 10.0.1.13, your MAILMAN_ARCHIVER_FROM setting may be misconfigured
WARNING 2020-11-10 21:35:50,503 1701 django.request Forbidden: /hyperkitty/api/mailman/archive
WARNING 2020-11-10 21:35:50,503 1701 django.request Forbidden: /hyperkitty/api/mailman/archive
ERROR 2020-11-10 23:30:45,432 1715 django.request Internal Server Error: /hyperkitty/list/epinux@lists.epinux.com/message/new
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.8/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/hyperkitty/lib/view_helpers.py", line 134, in inner
    return func(request, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/hyperkitty/views/message.py", line 231, in new_message
    form = get_posting_form(PostForm, request, mlist)
  File "/usr/lib/python3.8/site-packages/hyperkitty/lib/view_helpers.py", line 157, in get_posting_form
    (a, a) for a in request.user.hyperkitty_profile.addresses]
  File "/usr/lib/python3.8/site-packages/django/utils/functional.py", line 257, in inner
    return func(self._wrapped, *args)
  File "/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 412, in __get__
    raise self.RelatedObjectDoesNotExist(
django.contrib.auth.models.User.hyperkitty_profile.RelatedObjectDoesNotExist: User has no hyperkitty_profile.
ERROR 2020-11-10 23:30:45,432 1715 django.request Internal Server Error: /hyperkitty/list/epinux@lists.epinux.com/message/new
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.8/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/hyperkitty/lib/view_helpers.py", line 134, in inner
    return func(request, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/hyperkitty/views/message.py", line 231, in new_message
    form = get_posting_form(PostForm, request, mlist)
  File "/usr/lib/python3.8/site-packages/hyperkitty/lib/view_helpers.py", line 157, in get_posting_form
    (a, a) for a in request.user.hyperkitty_profile.addresses]
  File "/usr/lib/python3.8/site-packages/django/utils/functional.py", line 257, in inner
    return func(self._wrapped, *args)
  File "/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 412, in __get__
    raise self.RelatedObjectDoesNotExist(
django.contrib.auth.models.User.hyperkitty_profile.RelatedObjectDoesNotExist: User has no hyperkitty_profile.

Answer 33 · 2020-11-10T23:42:09.000Z

Try setting MAILMAN_HOST_IP=10.0.1.13 in mailman-web.

Answer 34 · 2020-11-10T23:44:09.000Z

it is already there:

    environment:
      - DATABASE_TYPE=postgres
      - DATABASE_URL=postgres://mailman:mailmanpass@database/mailmandb
      - HYPERKITTY_API_KEY=--------------------------------------
      - SECRET_KEY=ksjdbaksdba
      - UWSGI_STATIC_MAP=/static=/opt/mailman-web-data/static 
      - SERVE_FROM_DOMAIN=lists.epinux.com 
      - DJANGO_ALLOWED_HOSTS=lists.epinux.com
      - MAILMAN_ADMIN_USER=epinux
      - MAILMAN_ADMIN_EMAIL=epiesasha@me.com
      - MAILMAN_ARCHIVER_FROM=10.0.1.13 # 10.0.131.7 # lists.epinux.com # 10.0.1.13 # 
      - MAILMAN_HOSTNAME=mailman-core
      - MAILMAN_HOST_IP=10.0.1.13

Answer 35 · 2020-11-10T23:47:16.000Z

I'd rather not use these end line comments. Check the MAILMAN_ARCHIVER_FROM value from inside the mailman-web container.

Answer 36 · 2020-11-10T23:52:42.000Z

Oops no, sorry. Actually, drop this MAILMAN_ARCHIVER_FROM setting from the compose file. It's useless, because this is a python variable.

You could try pinging 10.0.1.13 from inside mailman-web. If you can't find out where this IP comes from, I won't know either...

Answer 37 · 2020-11-11T00:06:35.000Z

none of my containers has this IP. or at least none of the one from the list below:

docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)
...
/traefik_traefik.1.zwo396stx4o3fyk456t78h87u - 10.0.0.810.0.1.13

Hrm... Isn't it 10.0.1.13 just above?

Answer 38 · 2020-11-11T00:16:55.000Z

I can ping 10.0.1.13 from inside mailman-web - while I can't from mailman-core .. so I guess is on the traefik-public network.

I looped over all the services, I found the IP in :

0tbf2o2hg1im portainer_agent-network overlay swarm

root@epinux /home/epinux/dev/docker_swarm # docker inspect 0tbf2o2hg1im 
[
    {
        "Name": "portainer_agent-network",
        "Id": "0tbf2o2hg1im1lcnwbuskuty2",
        "Created": "2020-10-23T09:26:47.111496189+02:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.3.0/24",
                    "Gateway": "10.0.3.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "438b6d2a9ad0f017ad57ac490e3502111e0f739421a240ede15d1594d5d70f7f": {
                "Name": "portainer_agent.j7p1g8b9b7p828w3gsptw4dn2.9sttg86wwb4kkxq1fxdfunu9c",
                "EndpointID": "0a1991b3a82e1b10ec33fabd64f7e3b99a983b20ff1cc9c32e617fe5cf690587",
                "MacAddress": "02:42:0a:00:03:03",
                "IPv4Address": "10.0.3.3/24",
                "IPv6Address": ""
            },
            "8fa10742ac0bd8518fbd5ea883d357d7dcd519dbf0f928666ffaa9f1a47712f9": {
                "Name": "portainer_portainer.1.or8yiq2j4pdcsq69dvlt66ebh",
                "EndpointID": "889f7f63b2cce5bcca385ead88cbfc3536e9c34daec2fd26cc6ce81c853c8f85",
                "MacAddress": "02:42:0a:00:03:06",
                "IPv4Address": "10.0.3.6/24",
                "IPv6Address": ""
            },
            "lb-portainer_agent-network": {
                "Name": "portainer_agent-network-endpoint",
                "EndpointID": "eab6b7b78ed233f10a6fb592fac33429f48286347ca9e9c8ddf336f785b5e0fe",
                "MacAddress": "02:42:0a:00:03:04",
                "IPv4Address": "10.0.3.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4099"
        },
        "Labels": {
            "com.docker.stack.namespace": "portainer"
        },
        "Peers": [
            {
                "Name": "81d98c4d2c5c",
                "IP": "xxx.xx.xx.xx"
            }
        ]
    }
]

Answer 39 · 2020-11-11T00:18:05.000Z

Hrm... Isn't it 10.0.1.13 just above?

I missed it :( ... lack of sleep :)

Answer 40 · 2020-11-11T00:31:51.000Z

this the right inspection, ignore the one above ... it is quite complex to read ;/ it is the traefik proxy which I use to proxy mailman_web

docker inspect traefik_traefik.1.zwo396stx4o3fyk456t78h87u

[
    {
        "Id": "6f622d00e4b96421bc152bac1ef6f2651ebaf8ce41d0830ebee4f5b417bbc83f",
        "Created": "2020-10-23T07:26:53.494884129Z",
        "Path": "/entrypoint.sh",
        "Args": [
            "--providers.docker",
            "--providers.docker.constraints=Label(`traefik.constraint-label`, `traefik-public`)",
            "--providers.docker.exposedbydefault=false",
            "--providers.docker.swarmmode",
            "--entrypoints.http.address=:80",
            "--entrypoints.https.address=:443",
            "--certificatesresolvers.le.acme.email=epiesasha@me.com",
            "--certificatesresolvers.le.acme.storage=/certificates/acme.json",
            "--certificatesresolvers.le.acme.tlschallenge=true",
            "--accesslog",
            "--log",
            "--api"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 3602554,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-10-23T07:27:27.172163715Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:f965f5a1fff83fd0c440bb12a20f14e40f0ad33f1dc6d4f57736751ded8c5f87",
        "ResolvConfPath": "/var/lib/docker/containers/6f622d00e4b96421bc152bac1ef6f2651ebaf8ce41d0830ebee4f5b417bbc83f/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/6f622d00e4b96421bc152bac1ef6f2651ebaf8ce41d0830ebee4f5b417bbc83f/hostname",
        "HostsPath": "/var/lib/docker/containers/6f622d00e4b96421bc152bac1ef6f2651ebaf8ce41d0830ebee4f5b417bbc83f/hosts",
        "LogPath": "/var/lib/docker/containers/6f622d00e4b96421bc152bac1ef6f2651ebaf8ce41d0830ebee4f5b417bbc83f/6f622d00e4b96421bc152bac1ef6f2651ebaf8ce41d0830ebee4f5b417bbc83f-json.log",
        "Name": "/traefik_traefik.1.zwo396stx4o3fyk456t78h87u",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "default",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/var/run/docker.sock",
                    "Target": "/var/run/docker.sock",
                    "ReadOnly": true
                },
                {
                    "Type": "volume",
                    "Source": "traefik_traefik-public-certificates",
                    "Target": "/certificates",
                    "VolumeOptions": {
                        "Labels": {
                            "com.docker.stack.namespace": "traefik"
                        }
                    }
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/fd26fc0a9e6f67e7233ae432c026b147a8f33172614975ab258446c64f3abe5d-init/diff:/var/lib/docker/overlay2/ab282f26432cb91fb666d457e68ce5651a93caff614de65086c709727c6bc4df/diff:/var/lib/docker/overlay2/11a72bcf437d5cb0b6708ed45b729265360aede1a8683ecfec72b5a326e5fda2/diff:/var/lib/docker/overlay2/bb077f0720614aff486da6b129e3782b699261b361a415380cdb914d80741d53/diff:/var/lib/docker/overlay2/a06ab3512e0e2dfeafa7e884611d0a85f32a94fbf350d746566212d59b525070/diff",
                "MergedDir": "/var/lib/docker/overlay2/fd26fc0a9e6f67e7233ae432c026b147a8f33172614975ab258446c64f3abe5d/merged",
                "UpperDir": "/var/lib/docker/overlay2/fd26fc0a9e6f67e7233ae432c026b147a8f33172614975ab258446c64f3abe5d/diff",
                "WorkDir": "/var/lib/docker/overlay2/fd26fc0a9e6f67e7233ae432c026b147a8f33172614975ab258446c64f3abe5d/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "traefik_traefik-public-certificates",
                "Source": "/var/lib/docker/volumes/traefik_traefik-public-certificates/_data",
                "Destination": "/certificates",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "6f622d00e4b9",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "--providers.docker",
                "--providers.docker.constraints=Label(`traefik.constraint-label`, `traefik-public`)",
                "--providers.docker.exposedbydefault=false",
                "--providers.docker.swarmmode",
                "--entrypoints.http.address=:80",
                "--entrypoints.https.address=:443",
                "--certificatesresolvers.le.acme.email=epiesasha@me.com",
                "--certificatesresolvers.le.acme.storage=/certificates/acme.json",
                "--certificatesresolvers.le.acme.tlschallenge=true",
                "--accesslog",
                "--log",
                "--api"
            ],
            "Image": "traefik:v2.2@sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.stack.namespace": "traefik",
                "com.docker.swarm.node.id": "j7p1g8b9b7p828w3gsptw4dn2",
                "com.docker.swarm.service.id": "ikkvwme4ijlg96wv8mcj1y4kg",
                "com.docker.swarm.service.name": "traefik_traefik",
                "com.docker.swarm.task": "",
                "com.docker.swarm.task.id": "zwo396stx4o3fyk456t78h87u",
                "com.docker.swarm.task.name": "traefik_traefik.1.zwo396stx4o3fyk456t78h87u",
                "org.opencontainers.image.description": "A modern reverse-proxy",
                "org.opencontainers.image.documentation": "https://docs.traefik.io",
                "org.opencontainers.image.title": "Traefik",
                "org.opencontainers.image.url": "https://traefik.io",
                "org.opencontainers.image.vendor": "Containous",
                "org.opencontainers.image.version": "v2.2.11"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "93c4b053d78b6ff785b746bab7a50ed3d9a291e9e6bedc70ff0eb9391836874d",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "80/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/93c4b053d78b",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "ingress": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.0.8"
                    },
                    "Links": null,
                    "Aliases": [
                        "6f622d00e4b9"
                    ],
                    "NetworkID": "snsff6gmmu3bsz9yy72o2gtet",
                    "EndpointID": "098686405ef7f9b1ff32f8b853e0f66740d2f0d5824e2ff61f6f234d2d2da3da",
                    "Gateway": "",
                    "IPAddress": "10.0.0.8",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:00:08",
                    "DriverOpts": null
                },
                "traefik-public": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.1.13"
                    },
                    "Links": null,
                    "Aliases": [
                        "6f622d00e4b9"
                    ],
                    "NetworkID": "2co7wmfuo3ce2k233sd0t5mza",
                    "EndpointID": "20494cd93d713bfa1ef722be65a7777c619ede0644c867d3a64325c38c5db483",
                    "Gateway": "",
                    "IPAddress": "10.0.1.13",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:01:0d",
                    "DriverOpts": null
                }
            }
        }
    }
]

Answer 41 · 2020-11-11T00:42:39.000Z

Well, I don't use traefik, and I can't see why it would try to connect to the archiving API endpoint. I'm afraid you'll need to figure out this one yourself.

Everywhere I look for such an issue, I read that MAILMAN_ARCHIVER_FROM should resolve to the mailman-core's IP address.

Answer 42 · 2020-11-11T00:50:47.000Z

Did you try to drop the HYPERKITTY_URL variable? I don't have this variable defined in my settings. And that may explain why the hyperkitty API is accessed from the traefik public network.

Answer 43 · 2020-11-11T15:17:19.000Z

I am trying to srtart from scratch - I temporary shutted down the services running on my host so that docker ps shows:

docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                NAMES
94ef3bc9322f        epinux/exim4                 "/sbin/my_init"          12 minutes ago      Up 12 minutes       0.0.0.0:25->25/tcp   exim4
751b9e412a9c        postgres:9.6-alpine          "docker-entrypoint.s…"   16 minutes ago      Up 16 minutes       5432/tcp             lists_database.1.rb7acdg9lxy512dlbghe48g3i
9c40ac0990c2        epinux/mailman-web:latest    "docker-entrypoint.s…"   16 minutes ago      Up 16 minutes       8000/tcp, 8080/tcp   lists_mailman-web.1.qc1zanyueogk2g85f8gm1riba
a9aa8ff26bc7        epinux/mailman-core:latest   "docker-entrypoint.s…"   16 minutes ago      Up 16 minutes       8001/tcp, 8024/tcp   lists_mailman-core.1.vam0e2meun9kbf2pxzm9rkqbf
bcaeefef6788        traefik:v2.2                 "/entrypoint.sh --pr…"   7 hours ago         Up 7 hours          80/tcp               traefik_traefik.1.49nwdfxpfg4zqnqnqcvgqd964

Now I am creating a new domain with the postorius web-app - first Q: .. what should I put as mail host in the "Add a new domain" page?

I was going to use the following:

Mail Host    lists.epinux.com
Description  epinux test mailing list
Alias Domain  empty
Web Host lists.epinux.com (lists.epinux.com)

Is that correct?

Answer 44 · 2020-11-11T15:53:09.000Z

Is that correct?

Yes.

Have you dropped the HYPERKITTY_URL variable?

Answer 45 · 2020-11-11T18:31:28.000Z

No luck :(

I removed the HYPERKITTY_URL variable - My ENV

Mailman logs says:

ERROR 2020-11-11 17:25:46,898 1728 hyperkitty.views.mailman Access to the archiving API endpoint was forbidden from IP 10.0.8.4, your MAILMAN_ARCHIVER_FROM setting may be misconfigured
WARNING 2020-11-11 17:25:46,899 1728 django.request Forbidden: /hyperkitty/api/mailman/archive
WARNING 2020-11-11 17:25:46,899 1728 django.request Forbidden: /hyperkitty/api/mailman/archive

and I can't find the forbidden IP: 10.0.8.4

A quick check on my docker IP's:

# docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)
/exim4 - 10.0.8.9
/lists_database.1.rb7acdg9lxy512dlbghe48g3i - 10.0.8.8
/lists_mailman-web.1.qc1zanyueogk2g85f8gm1riba - 10.0.0.3510.0.8.610.0.1.38
/lists_mailman-core.1.vam0e2meun9kbf2pxzm9rkqbf - 10.0.8.3
/traefik_traefik.1.49nwdfxpfg4zqnqnqcvgqd964 - 10.0.0.1910.0.1.22

For what I can see (compared to my previous debug) this time around the refused IP changes at any restart and its value seems somehow correlated to the mailman-core IP .. the refused IP looks like one unit greater than the mailman-core IP (on the last digit):

e.g.:

mailmman log: forbidden from IP 10.0.8.4
mailman-core IP: /lists_mailman-core.1.vam0e2meun9kbf2pxzm9rkqbf - 10.0.8.3

To test: I stop the service, restart, and try again (I did it several times) ... and that is what I get:

#docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)
/exim4 - 10.0.9.9
/lists_database.1.tn6qsbr0asoj0byrns7rm5yx2 - 10.0.9.8
/lists_mailman-web.1.wjpxci1l0zlh4udxtu3wj8dvf - 10.0.0.3710.0.9.610.0.1.40
/lists_mailman-core.1.khihgn9fyq4u2a6nu8vr8h1ko - 10.0.9.3
/traefik_traefik.1.49nwdfxpfg4zqnqnqcvgqd964 - 10.0.0.1910.0.1.22

And, sending an email to the list ad checking the logs I get:

ERROR 2020-11-11 18:19:24,670 1714 hyperkitty.views.mailman Access to the archiving API endpoint was forbidden from IP 10.0.9.4, your MAILMAN_ARCHIVER_FROM setting may be misconfigured
WARNING 2020-11-11 18:19:24,671 1714 django.request Forbidden: /hyperkitty/api/mailman/urls
WARNING 2020-11-11 18:19:24,671 1714 django.request Forbidden: /hyperkitty/api/mailman/urls

Which results in:

mailmman log: forbidden from IP 10.0.9.4
mailman-core IP: /lists_mailman-core.1.vam0e2meun9kbf2pxzm9rkqbf - 10.0.9.3

... Is there any way to get into the python code that does this? I'm searching in the mailman src code .. but not sue it can be in the hyperkitty srtc code instead.

attaching more complete logs as gist: https://gist.github.com/epifanio/1768a91be1fd3753cef76eff26d19768

Answer 46 · 2020-11-13T15:22:43.000Z

Issue solved after private discussion with @epifanio.