MARTI dice server not configuration managed
Closed this issue · 27 comments
The MARTI dice server does not appear to be configuration managed. Presumably this is because sensitive information (e.g. credentials) is embedded within the source code.
The dice server code should be changed so that sensitive information is pulled from a separate configuration (e.g. environment variables). In addition, any other non-sensitive configuration data (e.g. database schemas) should be captured and stored in version control.
Picking up from our previous discussion...
I can of course also give you the source code with removed credentials...
It seems that extracting that kind of configuration information from the source code and into the environment or something would be a prerequisite to creating a Docker image.
I can try to do this. I don't think I'll need access to the server at this time. As you suggested, making the secret-free code accessible someplace should be sufficient (e.g. publish to a branch on your fork or just email it to me).
It would also help if you could provide me with the schema for each database table used by the code. You can capture the schema by starting the MySQL shell on the machine hosting the database and running the describe (or equivalent) command. See here for examples. (Actually, running show create table may be good, too, because it will ensure I don't screw up reverse engineering the schema in the first place. 😃)
Can you also provide versions for each of the following components on the production machine?
- nginx
- PHP
- MySQL
Thanks!
I'll do that real quick... Hang on!
PHP: 7.0.15
MySQL: 5.7.18
nginx: 1.10.0
There are 3 tables;
CREATE TABLE `dice_emails` (
`registered_email` varchar(255) NOT NULL,
UNIQUE KEY `registered_email` (`registered_email`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
CREATE TABLE `dice_table` (
`request_ID` int(11) NOT NULL AUTO_INCREMENT,
`UUID` varchar(36) NOT NULL,
`dice` varchar(256) NOT NULL,
`subject` varchar(70) NOT NULL,
`time_stamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`request_ID`),
KEY `UUID` (`UUID`)
) ENGINE=MyISAM AUTO_INCREMENT=57 DEFAULT CHARSET=latin1;
CREATE TABLE `pending_validations` (
`email` varchar(255) NOT NULL,
`validation_key` char(32) NOT NULL,
`time_stamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`IP` int(10) unsigned NOT NULL,
UNIQUE KEY `email` (`email`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
I'd recommed using InnoDB as engine for MySQL and unicode as charset would probably be the better option
I'd recommed using InnoDB as engine for MySQL and unicode as charset would probably be the better option
Agreed.
BTW, could you please recheck the MySQL version? You have 14.14 above, but latest stable is 5.7 (and latest preview is only 8.0). Did you mean 4.14? Or could that be the console version?
Sorry, meant 5.7.18...
mysql Ver 14.14 Distrib 5.7.18, for Linux (x86_64) using EditLine wrapper
I updated my previous comment
Who coded this ****?
Why does PHP even allow such flaws?!
@RoiEXLab The scrubbed code you sent does not include the database name (in dice.class.php). Is the database name really considered sensitive information that should not be published to a public repo? It seems no different than a table name to me. Am I misunderstanding something regarding the security of the production system?
I don't actually need it for the Docker container I'm creating. I can use any name and just require that it be specified by the user as an environment variable. But, if possible, I'd like to minimize the number of external configuration items that are required to spin up the server.
@RoiEXLab I'm making progress... I have everything running, but MARTI is unable to send emails because I'm using the default PHP mail configuration.
I would like to capture the production mail configuration to reproduce it in the Docker image. It should be in the [mail function] section of php.ini and possibly in a sendmail.ini file. I'm not sure where these files are located on the production machine because each install is different. The hack I use to locate them is to embed a <?php phpinfo(); ?> directive on a page served only to localhost and look at the Loaded Configuration File and Additional .ini files parsed rows.
If the production machine authenticates to the SMTP server, you'll probably have to scrub the credentials from the .ini file(s).
Nice to hear @ssoloff !
php.ini:
; For Win32 only.
; http://php.net/smtp
SMTP = localhost
; http://php.net/smtp-port
smtp_port = 25
; For Win32 only.
; http://php.net/sendmail-from
;sendmail_from = me@example.com
; For Unix only. You may supply arguments as well (default: "sendmail -t -i").
; http://php.net/sendmail-path
;sendmail_path =
; Force the addition of the specified parameters to be passed as extra parameters
; to the sendmail binary. These parameters will always replace the value of
; the 5th parameter to mail().
;mail.force_extra_parameters =
; Add X-PHP-Originating-Script: that will include uid of the script followed by the filename
mail.add_x_header = On
; The path to a log file that will log all mail() calls. Log entries include
; the full path of the script, line number, To address and headers.
;mail.log =
; Log mail to syslog (Event Log on Windows).
;mail.log = syslog
There's no sendmail.ini file...
I hope this helps!
Thanks, @RoiEXLab. Does there happen to be any files under /etc/mail/? For example, sendmail.conf and/or sendmail.mc? If the PHP mail function is using sendmail, then one of those files should contain the relay configuration (again, authentication credentials may be embedded).
Both of them exist:
sendmail.conf (I removed the comments, because they were 90% of the file):
DAEMON_NETMODE="Static";
DAEMON_NETIF="eth0";
DAEMON_MODE="Daemon";
DAEMON_PARMS="";
DAEMON_HOSTSTATS="No";
DAEMON_MAILSTATS="No";
QUEUE_MODE="${DAEMON_MODE}";
QUEUE_INTERVAL="10m";
QUEUE_PARMS="";
MSP_MODE="Cron";
MSP_INTERVAL="20m";
MSP_PARMS="";
MSP_MAILSTATS="${DAEMON_MAILSTATS}";
MISC_PARMS="";
CRON_MAILTO="root";
CRON_PARMS="";
LOG_CMDS="No";
HANDS_OFF="No";
AGE_DATA="";
DAEMON_RUNASUSER="No";
DAEMON_STATS="${DAEMON_MAILSTATS}";
MSP_STATS="${MSP_MAILSTATS}";
sendmail.mc:
divert(-1)dnl
divert(0)dnl
define(`_USE_ETC_MAIL_')dnl
include(`/usr/share/sendmail/cf/m4/cf.m4')dnl
VERSIONID(`$Id: sendmail.mc, v 8.14.9-4 2015-07-26 18:37:21 cowboy Exp $')
OSTYPE(`debian')dnl
DOMAIN(`debian-mta')dnl
dnl # Items controlled by /etc/mail/sendmail.conf - DO NOT TOUCH HERE
undefine(`confHOST_STATUS_DIRECTORY')dnl #DAEMON_HOSTSTATS=
dnl # Items controlled by /etc/mail/sendmail.conf - DO NOT TOUCH HERE
dnl #
dnl # General defines
dnl #
dnl # SAFE_FILE_ENV: [undefined] If set, sendmail will do a chroot()
dnl # into this directory before writing files.
dnl # If *all* your user accounts are under /home then use that
dnl # instead - it will prevent any writes outside of /home !
dnl # define(`confSAFE_FILE_ENV', `')dnl
dnl #
dnl # Daemon options - restrict to servicing LOCALHOST ONLY !!!
dnl # Remove `, Addr=' clauses to receive from any interface
dnl # If you want to support IPv6, switch the commented/uncommentd lines
dnl #
FEATURE(`no_default_msa')dnl
dnl DAEMON_OPTIONS(`Family=inet6, Name=MTA-v6, Port=smtp, Addr=::1')dnl
DAEMON_OPTIONS(`Family=inet, Name=MTA-v4, Port=smtp, Addr=127.0.0.1')dnl
dnl DAEMON_OPTIONS(`Family=inet6, Name=MSP-v6, Port=submission, M=Ea, Addr=::1')dnl
DAEMON_OPTIONS(`Family=inet, Name=MSP-v4, Port=submission, M=Ea, Addr=127.0.0.1')dnl
dnl #
dnl # Be somewhat anal in what we allow
define(`confPRIVACY_FLAGS',dnl
`needmailhelo,needexpnhelo,needvrfyhelo,restrictqrun,restrictexpand,nobodyreturn,authwarnings')dnl
dnl #
dnl # Define connection throttling and window length
define(`confCONNECTION_RATE_THROTTLE', `15')dnl
define(`confCONNECTION_RATE_WINDOW_SIZE',`10m')dnl
dnl #
dnl # Features
dnl #
dnl # use /etc/mail/local-host-names
FEATURE(`use_cw_file')dnl
dnl #
dnl # The access db is the basis for most of sendmail's checking
FEATURE(`access_db', , `skip')dnl
dnl #
dnl # The greet_pause feature stops some automail bots - but check the
dnl # provided access db for details on excluding localhosts...
FEATURE(`greet_pause', `1000')dnl 1 seconds
dnl #
dnl # Delay_checks allows sender<->recipient checking
FEATURE(`delay_checks', `friend', `n')dnl
dnl #
dnl # If we get too many bad recipients, slow things down...
define(`confBAD_RCPT_THROTTLE',`3')dnl
dnl #
dnl # Stop connections that overflow our concurrent and time connection rates
FEATURE(`conncontrol', `nodelay', `terminate')dnl
FEATURE(`ratecontrol', `nodelay', `terminate')dnl
dnl #
dnl # If you're on a dialup link, you should enable this - so sendmail
dnl # will not bring up the link (it will queue mail for later)
dnl define(`confCON_EXPENSIVE',`True')dnl
dnl #
dnl # Dialup/LAN connection overrides
dnl #
include(`/etc/mail/m4/dialup.m4')dnl
include(`/etc/mail/m4/provider.m4')dnl
dnl #
dnl # Masquerading options
FEATURE(`always_add_domain')dnl
MASQUERADE_AS(`sanjuro.ca')dnl #This is an unused domain which still has an empty config on the server. I don't know what it's for
FEATURE(`allmasquerade')dnl
FEATURE(`masquerade_envelope')dnl
dnl #
dnl # Default Mailer setup
MAILER_DEFINITIONS
MAILER(`local')dnl
MAILER(`smtp')dnl
@RoiEXLab Ugh, that looks like a default config.
I went back and examined the headers of one of the emails I received from MARTI. From this line:
Received: by tripleawarclub.org (Postfix, from userid 33) id E431A10A496; Sat, 29 Apr 2017 15:45:14 -0400 (EDT)
I'm guessing the production server is actually using Postfix and not Sendmail. (But since I didn't see that specified in php.ini, I'm wondering if sendmail is just a link to postfix on the production machine.) Does there happen to be an /etc/postfix/main.cf or similar file? If so, can you post that please? From what I've read, that file should not contain any credentials, but you'll want to double check just to be sure.
If you find the Postfix config file, there should be a file in the same directory named saslpasswd. That file contains hashed credentials, but I would appreciate if you could verify there is a line for the same SMTP host listed in main.cf under the relayhost key.
There is no file calles saslpasswd, but an empty directory called sasl...
Nonetheless here's the main.cf file... It looks promising:
# See /usr/share/postfix/main.cf.dist for a commented, more complete version
# Debian specific: Specifying a file name will cause the first
# line of that file to be used as the name. The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
readme_directory = no
# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = tripleawarclub.org
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = tripleawarclub.org, localhost.ca, , localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all
Thanks, @RoiEXLab. I was hoping relayhost was set here. This again looks like a default config generated when Postfix is installed. I'll need to do some further research to figure out what other configuration settings control how Postfix knows where to relay mail.
@RoiEXLab Thanks for all your help digging through config files. I think I've resolved the issue.
My theory is that the production server is using the Google G Suite SMTP relay service to send emails. I base that on the following headers from one of the MARTI emails I received:
Received-SPF: pass (google.com: domain of www-data@tripleawarclub.org designates 45.79.144.53 as permitted sender) client-ip=45.79.144.53;
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of www-data@tripleawarclub.org designates 45.79.144.53 as permitted sender) smtp.mailfrom=www-data@tripleawarclub.org
That looks like Google is authenticating the sender based on IP address, which is what the G Suite SMTP relay service does. Thus, there doesn't need to be any credentials stored on the machine for sending emails.
However, that's obviously not going to work for a Docker dev/test image. So my plan is to allow the user to specify an SMTP host, port, username, and password when they build the Docker image so it's customizable for each dev (and their responsibility to keep the image secure so those credentials aren't leaked). I did this using Google's standard credential-based-authentication SMTP service (smtp.gmail.com), and my MARTI mails are being delivered. (What threw me for a loop was that, because I have two-factor authentication enabled on my account, I had to generate a custom app password to avoid Postfix constantly getting authentication failures when it tried to connect to smtp.gmail.com.)
Anyway, using that approach, I was able to register with my local MARTI server, received a confirmation email, and validated the registration. Joy! 🎉 Now I just need to clean everything up and get it pushed to my fork. Hopefully, I'll be able to open a PR for you to review within a few days on the changes I had to make to the MARTI code to get it to run without hard-coded credentials.
Excellent! @ssoloff
Btw, forgot to mention, the reason I removed tge database name was not, because I considered it sensitive, but because I thought it was a hardcoded value which should be moved outside the code...
Off-Topic question:
One thing I realized: MARTI mails are sent without encryption (at least this is what Gmail tells me).
Any ideas on how to fix that considering this complex configuration?
the reason I removed tge database name was not, because I considered it sensitive, but because I thought it was a hardcoded value which should be moved outside the code...
Agreed, and I've already done this, so I'll leave it that way in my PR. However, so I have a meaningful value to work with in my Docker image (and not just something I made up 😃), could you please share what the production database name is?
MARTI mails are sent without encryption (at least this is what Gmail tells me).
Any ideas on how to fix that considering this complex configuration?
Yeah, this is pretty simple. If you look at the link I posted above, you have the option of connecting to the Google SMTP relay on ports 25, 465 (SSL), or 587 (TLS). My guess is MARTI is connecting on port 25. The emails sent from my local MARTI server are not showing the "XXX did not encrypt this message" warning because I am connecting to the SMTP relay on port 587 (TLS). (And, to be honest, I only did it this way because my ISP blocks all outbound traffic on port 25 in an effort to combat spam, so I really had no choice. 😃)
So, if we can find the SMTP configuration in production, it should only require changing the port and possibly a flag that says to use TLS when connecting. However, our search for this configuration has been unfruitful so far. You may just have to hunker down for an hour and put on your deerstalker cap looking for it. I'd recommend starting again in the /etc/postfix folder. You'll know you've found it when you see a reference to smtp-relay.gmail.com.
You can also look at /var/log/mail.log or /var/log/maillog to see if there's any clue about where the configuration is being read from (I didn't see anything on my system, but the base Docker image I'm using could be configured in such a way as to make the mail logs very terse). /var/log/syslog might contain something useful, as well.
BTW, not sure why I didn't ask before, but who originally set up the tripleawarclub.org server? If they're still around, they might remember how they configured SMTP...
The db name is simply marti
I don't know who setup this server originally.
I believe @veqryn or @DanVanAtta should know.
I recently upgraded it to Ubuntu 16.04 which introduced some errors into the tripleawarclub.org forum since php5 is no longer supported in this Ubuntu version.
Xoops has a php7 version, but a lot of plugins are no longer maintained and call methods which no longer exist, or call methods which in the new version require a specific object type.
docker containers anyone?
@veqryn That was actually the motivation for this issue. 😃 We're working on building a Docker image for testing MARTI. If that works out, maybe using Docker in production is the next step...?
Good to hear!
Having used docker in production at my company for the last 2 years, I definitely sing its praises.
It looks like there are official PHP 5 images of docker that you can use as the base layer here: https://hub.docker.com/_/php/
I know absolutely nothing about the MARTI dice server (Bung or SGB might, but good luck getting in touch with them), but I can definitely be of use if you have questions about docker.
According to the text file I made to log steps when I migrated us a couple years back, I used the following guide to send mail (had to find an archive version):
There was some setup around using gmail for mail on a much earlier server, so that I could have accounts @tripleawarclub.org, but I don't think I did anything specific for google on the last linode setup I did because it's not in my document at all.
Thanks, @bung!
@RoiEXLab, when you have some time, maybe you can try to reverse engineer exactly how the mail() call in PHP code effectively works its way down the technology stack of the production server to the point where the email is sent to a relay? Once we understand how it's working and how to configure it, we can document what we've learned in this repo's wiki.
Closing as resolved by triplea-game/dice-server#1.