/linuxgateway

VPN concentrator for direct AWS integration

Primary LanguageShellApache License 2.0Apache-2.0

Linux VPN Gateway

General system design

  • traffic forwarding between the public Internet and SV2 is governed by /etc/init.d/firewall:

    • it's a very simple bash script running a sequence of iptable commands
    • it includes both firewall and NAT rules
      • classic port forwarding is knows as DNAT
      • classic outbound connection natting is known as SNAT
    • NOTE: rules for the VPCs in AWS are dynamically generated:
      • /gen_capo_firewall.rb: runs once an hour

  • VPN is taken care of by strongswan, default IKE service provided by ubuntu.

    • strongswan / ipsec encrypts/decrypts traffic based on connection IDs
      • the actual traffic needs to be somehow marked with the right IDs to make it into the matching tunnel
      • usually this is accomplished by iptables policies, in our case (due to a route-based VPN using dynamic routing) this is not possible
      • ip tunnel kernel interfaces are used instead
      • read on for more details on this

  • failover between the 2 devices is handled in 2 different ways:

    1. OSPF/BGP dynamic routing for VPN traffic (/etc/bird/bird.conf)
    2. VRRP/Keepalive for default gateway/outbound internet traffic (/etc/keepalived/keepalived.conf)

  • the routing is taken care of by bird. it runs OSPF on the SV2 side and exports any routes learnd over the VPNs by BGP into SV2 through OSPF

  • all global config is found in the correspondig service folder in /etc -> i.e. /etc/bird/ or /etc/ipsec.conf -> these are GLOBAL settings shared by all connections

  • individual VPN tunnel configs are generated dynamically by /gen_configs.sh

    • the script will take 3x arguments
      • the location of XML config file from AWS portal
        • you can get the XML config by downloading the Sophos ASG version from the AWS portal
        • this can also be a URL
      • assigned VPC CIDR subnet (i.e. 172.16.12.0/24)
      • teamname or very very short description of the tunnel (capo-usw1, agentprofile, ...)
      • example how I built the logstash VPN: /gen_configs.sh http://myserver.intranet.com/temp/ops-lnxgwy2.xml 172.16.10.0/24 logstash .
    • it generates 4x brand new config files that all go in /vpns/ (the global configs include these by wildcard-globbing)
      • *.bird.conf -> routing config bird/BGP
      • *.ipsec.conf -> remote ipsec tunnel IP and to be selected connection ID
      • *.ipsec.secrets -> tunnel secrets
      • *.ipsec.sh -> configuring virtual tunnel interfaces matching the connection ID selected above by ipsec
        • this is a regular shell script that sets up required kernel interfaces.
        • NOTE: it needs to be run once after a reboot.
        • it is NOT started automatically at bootup by design to prevent preemting the VPN tunnels back to the freshly rebooted box.
    • cleanup a old or incorrect tunnel simply delete these 4x files in /vpns and reload the various congigs (see below for more details there)

useful commands

  • ipsec status: list current SAs
  • ipsec reload: reload ipsec config after you added or removed a tunnel
  • ipsec rereadsecrets: if you change the secrets in a file a simle ipsec reload is not enough, you need to reload the actual secrets
  • ipsec update: reload ipsec.conf file
  • ipsec status: check VPN:
  • birdc: routing shell:
    • you can VERIFY any routing related info in here.
    • configure will reload the config files
      • syntax check configure check
      • and soft reload configure soft
    • support cisco-stype "?" and "tab" for autocompletion and quick help
    • for more details see here
  • ip tunnel show: shows currently configured kernel tunnel interfaces (the VPN does NOT have to be up to show here in this list)
    • ip tunnel add/remove ... for adding or removing a tunnel
  • iptables -L -n -v --line-num: shows the whole firewall ruleset
  • iptables -L -n -v --line-num -t nat: shows the firewall nat table (Port Forwarding and Source NAT rules etc)
  • iptables -L -n -v --line-num -t mangle: shows special mangle table (i.e MSS adjust, other advanced packet manipulation actions)

example

Setting up a new VPN tunnel to a VPC

run ./gen_configs.sh [file/url to xml-file] [VPC-cidr] [consumer/team/project requesting the service]
i.e. ./gen_configs.sh https://ops-jenkins.sv2.trulia.com/vpns/zgtdb.xml 172.16.10.0/24 zgtdb
the XML file is completely generated by Terraform and is ingested by this script completely untouched. usually the URL for this XML file should point directly to the jenkins XML store location the part above will only take care of the routing/VPN part. the firewall part needs to be updated/adjusted in /etc/init.d/firewall

Firewall part

Dynamic files design and internals

  • by default each VPN tunnel gets it's own iptables chain. an iptables chain is similar to a legacy firewall zone.
  • all packets floing through the firewall are handled by the FORWARD chain.
    • this chain delegates individual zones to individual chains
    • the FORWARD chain is build statically on bootup or on initial VPN spinup. it does NOT change dynamically
    • dynamic updates happen ONLY to the individual chains to limit the blast radius
  • the rule-set for each chain is found as a simple CSV file in /capo-firewalls/**.list on each node
  • some are statically defined chains -> these rulesets are checked into this repo.
  • some firewall chains are generated dynamically by pulling up2date information from consul and converting it to a CSV file. (through /gen_capo_firewall.rb)
    • this script does some initial error checking
    • it will only update the CSV if it connected successfully
  • both dynamic and static CSV files are pushed live by default once an hour (check /refresh-capo-firewall.sh)
  • the cron job calls the script gen_iptables_from_csv.sh that pushes a CSV table into the actual running ACL chain.
    • this script does some extra error checking and can actually handle FQDN in the CSV and convert these to IPs
    • it only runs if there are actual changes in the CSV file since the last run (it does this by creating a copy of the original CSV by appending .last to the filename
    • if there are changes to the last run it will email the changes (as well as any potential errors occured) to the root mail alias
    • it will also log all output to syslog