/srv01

[mirror] Run your own network services - sharing/communication/collaboration systems, file storage, websites, multimedia, office/organization, development, automation, IT infrastructure, and many more, hosted on a server you control.

Primary LanguagePython

░░░░░█▀█ ┏━┓┏━┓╻ ╻┏━┓╺┓ 
░░░░░█░█ ┗━┓┣┳┛┃┏┛┃┃┃ ┃ 
░▀▀▀░▀▀▀ ┗━┛╹┗╸┗┛ ┗━┛╺┻╸

Easily run your own network services - sharing/communication/collaboration systems, file storage, websites, multimedia, office/organization, development, automation, IT infrastructure, and many more, hosted on a server you control.

Provided services

Each component can be installed independently.

  • Simple command-line tool to manage and configure server functionality, single-command installation.
  • Base system: Automated server deployment/configuration/upgrades/maintenance, backups, firewall/IPS, SSH remote access, SFTP file storage.
  • Apache web server.
  • Nextcloud private cloud solution with file storage/sharing/synchronization, desktop/mobile clients, calendars/tasks/address books management and CalDAV/CardDAV synchronization, groupware/collaboration features, and more.
  • Shaarli bookmarks manager and link sharing application.
  • Tiny Tiny RSS news feed reader/aggregator.
  • Transmission web client for the BitTorrent file sharing network.
  • Gitea Git service with issues/project management and wiki.
  • Mumble low-latency, high quality voice chat server.
  • Icecast multimedia streaming server.
  • PulseAudio streaming audio server.
  • Developer/administrator tools: Gitlab Runner, Docker

Components (roles) can be installed independently.

Screenshots

Preparing the server

Hardware requirements: srv01 can be installed on the following hardware:

  • Physical machine (server-grade or repurposed desktop/laptop computer)
  • Virtual Private Server (VPS)
  • Virtual Machine on your home computer or a dedicated hypervisor.
    Computer with x86/64 compatible CPU
    512MB+ RAM
    8GB-∞ drive space
    Same capacity external backup drive
    Internet connection during installation
    Remote control/administration computer
    1GB+ USB drive or DVD for the installation media

CPU/Memory: RAM/CPU usage will vary according to installed/running services and applications and the number simultaneous of users. 1GB ram and 1-2 CPU cores should ensure good performance for 1-5 users. Installation in a Virtual Machine is recommended. Open-source virtualization software includes virt-manager/libvirt on Linux, Proxmox, and Virtualbox on Windows. See Setting up KVM/libvirt.

Storage: A basic installation without user data requires about ~??GB of disk space. 40GB+ is recommended to store documents, shared files and other data.

Power: Prefer newer hardware as it will decrease your server's power consumption. Use low power consumption devices. If your budget allows it and reliability/performance is important, prefer server-grade hardware and ECC RAM. If availability is important, protect your server from power loss with an UPS. Some BIOS configuration tools allow turning the power back on after a power loss.

Network: The server must have Internet connectivity to download software and updates. In the default firewall configuration, connections are accepted both from your LAN and the Internet. Adjust the configuration if you want to allow connections from your LAN only. The default configuration assumes the server is behind a NAT. Setup port forwarding on your router if your server is behind a NAT device/Internet Service Provider box, and you want to be able to access your services from the Internet. How to do so depends on your router model, refer to your hardware vendor or ISP documentation. Ports that need to be forwarded to your server are:

SSH server (initial deployment): TCP 22
SSH server:                      TCP 823
Apache web server:               TCP 443
Transmission torrent client:     TCP/UDP 55702
Mumble VoIP server:              TCP/UDP 64738

Prefer a stable, wired network with decent upload/download bandwidth. A wireless connection is possible with good enough signal quality if latency is not a concern - always use WPA2, use strong passphrases. Use VLANs or other methods to isolate your server from unsafe machines on your network.

Register a DNS domain name (A record) and point it to your server's public IP address. You can get a free public DNS subdomain at freedns.afraid.org. If the server is not reachable from the Internet (LAN only), you should add a DNS record in your local DNS server (your home router/Internet Service Provider box may support this), or update the hosts file on every client that will need to connect. In case the server is used both locally and from the Internet, consider configuring Split DNS/NAT Loopback/Hairpinning/Reflection on your router.

Install Debian:

  • Setup your hardware or virtual machine.
  • Download Debian, and write the downloaded ISO image to USB using dd or GNOME disks on Linux, or win32diskimager on Windows, or burn it on optical media. This is not required for installation in a VM.
  • Boot your server/VM from the Debian installer USB/DVD/ISO image.
  • Select Advanced > Graphical advanced install.
  • Follow the installation procedure (example), using the following options:
    • Setup a static (manual) IP address (eg 192.168.1.90)
    • Enter the IP address of your local DNS server, or use a trusted public DNS server.
    • Allow root (administrator) logins (set a strong password).
    • Do not create an additional normal user account.
    • Any disk partitioning scheme is OK, but prefer separate /var and /tmp partitions. Most data storage locations are in /var so make sure to give it as much space as possible. 10-15GB should be enough for the root filesystem partition. Setup LVM if possible. Setup RAID if required.
    • Only install the following tasks:
      • Standard system utilities
      • SSH server
  • Finish installation, reboot, login as root.
  • To prepare the server to receive commands from a remote machine, run:
# Install the requirements for remote admin/ansible access
apt install python aptitude sudo

# Create an administrator user account
adduser deploy

# Give the account admin (sudo) privileges
adduser deploy sudo

# Allow this user to connect over SSH
adduser deploy ssh

# test internet connectivity
ping -c1 debian.org

Remember the server's domain name and sudo username/password and logout.

Preparing the administration machine

These steps must be run on the machine that will be used for remote administration.

A remote administration computer is required to perform administrative tasks (deployment, reconfiguration...) and store configuration and secrets. This could be a desktop/laptop, a separate server/VM, or the hypervisor itself. This computer (the controller) will be granted remote administrative access via SSH to the server. Only Debian-based systems are tested, but it should work from any Linux distribution where Ansible 2.7+ and Bash are available.

If you do not have a computer for remote administration, you may also use this tool from the server itself. This is not recommended.

# Install ansible
sudo apt install git python3-pip
sudo pip3 install ansible==2.8.0 netaddr

# clone the repository
git clone https://github.com/nodiscc/srv01

Configuration

Initial server deployment

# All commands must be run from the `srv01` directory
cd srv01/

# Create default configuration files, test connectivity, authorize your SSH key on the server
./srv01

The script will propose deploying the server with the default configuration. If you want to edit the config before initial deployment, do it like described below, then run ./srv01 --initial to continue.

TODO asciinema of install process

Editing configuration

To enable/disable components (roles), edit data/config/srv01-playbook.yml (just comment out the relevant role). By default the following roles are enabled (the installer will allow you to edit this file before initial deployment):

- hosts: example.net
  gather_facts: True
  become: True
  become_user: root
  vars_files:
    - data/config/example.net.yml
  roles:
    - common
    - apache2
    - nextcloud
    - shaarli
    - tt-rss
    - transmission
    - mumble
    - icecast
    - gitea
#    - pulseaudio
#    - gitlab-runner
#    - docker

To change any of the default settings, add them to data/config/srv01-config.yml with the appropriate value. By default, only variables that need to be set are present in this file:

#### HOST CONFIGURATION FILE ###

# DNS name of the host (Fully Qualified Domain Name)
host_fqdn: "example.net"

# username/password for the administrator (sudo) account
admin_username: "deploy"
admin_password: "hunter2"

# server administrator e-mail address
admin_email: "admin@localhost"

# username/password for default admin account created on your services/web applications...
services_username: "nodiscc"
services_password: "{{ lookup('password', playbook_dir + '/data/secrets/' + host_fqdn + '.services_password length=15') }}"

# Port for initial SSH connection (before first deployment)
init_ssh_port: "22"

# SSH private key
ansible_ssh_private_key_file: "{{ playbook_dir }}/data/secrets/{{ host_fqdn }}.id_rsa"

Passwords and keys are stored in data/secrets/.

Keep backups of these files.

More settings can be changed through the web administration interfaces of your services.

Applying configuration changes

To configure all components exactly as defined in the enabled roles/config file, run:

./srv01

Partial deployment/configuration

If you only want to reconfigure a single component/role to save some time, and not the whole playbook, you may specify one or more tags:

./srv01 --tags=tag1,tag2,tag3...

List of tags:

apache          # configure apache/php/mysql (webserver) services
backups         # configure rsnapshot backup utility
common          # configure base system components (system/clock, hostname, ssh, users, firewall...)
docker          # configure docker container management service
fail2ban        # configure fail2ban IPS/bruteforce prevention system
firewall        # configure firehol firewall service
gitea           # configure gitea git/project management service
gitlab-runner   # configure gitlab-runner continuous integration service
icecast         # configure icecast media streaming server
monitoring      # configure monitoring/reports/IDS/security scans
mumble          # configure mumble voice communication server
nextcloud       # configure nextcloud groupware/collaboration application suite
pulseaudio      # configure pulseaudio network sound server
services        # update services enabled/disabled status according to configuration
shaarli         # configure shaarli link/bookmark sharing application
ssh             # configure SSH remote access/SFTP file storage
transmission    # configure transmission bittorrent client/web interface
tt-rss          # configure tt-rss feed reader application

Some tasks will not run as part of the default playbook, but are useful to automate occasional intervention on the server.

utils-apt-upgrade   # force immediate upgrade of Debian packages
utils-backup        # force backups to run immediately
utils-backup-sync   # transfer a copy of latest daily backups to the controller
utils-cleanup       # run various cleanup tasks (delete temporary files, old logs, clear caches...)
utils-icecast-skip  # skip the track icecast is currently playing
utils-poweroff      # force immediate reboot of the server
utils-reboot        # force immediate power off/shutdown of the server (use with caution)

Usage

Access your services using a Web browser such as Firefox, or other software from the Clients section of each service.

A HTML page listing useful shortcuts and configuration tips will be generated at data/srv01.html.

Upgrades: Security upgrades for Debian packages are handled automatically. To upgrade all components, default settings, configuration tasks and applications to the latest versions, run git pull && ./srv01. Doing so will preserve your data and settings. You can also schedule this using a cron job or similar for a fully unattended system.

Notes

Self-hosting places your services and data under your own responsibility (uptime, backups, security...). Always have a plan in place if your server crashes, gets compromised or damaged.

Backups: Important data is backed up periodically to a local directory on the server. Run ./srv01 utils-backup-sync to synchronise latest backups to the controller, or setup a separate backup machine. Keep a third copy of valuable data off site. The default is to keep 6 daily backups, 4 weekly backups, 6 monthly backups. Attempt restoring and/or accessing your backups on a regular basis (eg. just after running the backup procedure) to make sure they are valid. Operating system files are not backed up by default (only application/long term storage data. The system can be redeployed identically by running the playbook, then restoring the backups). There is no High Availability mechanism configured by default. You can setup RAID manually but keep in mind that RAID is not a backup.

Passwords: Use long and unique passwords. Do not disclose or lose the data/ directory which contains server configuration and secrets. A .gitignore file is used to prevent accidentally pushing these files to git. Consider protecting the directory using disk encryption or other secure storage/access control mechanisms.

Using roles in plain ansible: The srv01 script uses some hardcoded paths for organization an general ease of use. The limitation is that the script only works to manage a single server. If you want to build your own playbook/inventory around the provided roles, and use them in a multi-server setup: edit srv01-playbook.yml and data/config/srv01-inventory.yml to include your other hosts, and run ansible-playbook --inventory data/config/srv01-inventory.yml srv01-playbook.yml. You can also move your vars files (data/config/your.domain.yml) to a directory host_vars/ at the root of the repository and remove the vars_files directives from the playbook.

Contributing

See TODO for a list of work-in-progress items and prospective ideas.

Please report issues or send requests on Gitlab or GitHub issue trackers.

See awesome-selfhosted for a list of similar projects, and other network services and applications.

License

GNU GPLv3

This project is based on Debian GNU/Linux and uses ansible roles for configuration. You can use the srv01 command line tool for easy management of a single server, or you can use these roles to build your own ansible playbooks.

All components are Free and open-source software. See each component's README/COPYING/AUTHORS/LICENSE file for more information. License information for software in Debian packages can be found at /usr/share/doc/*/copyright.