-
traffic forwarding between the public Internet and SV2 is governed by
/etc/init.d/firewall
:- it's a very simple bash script running a sequence of iptable commands
- it includes both firewall and NAT rules
- classic port forwarding is knows as DNAT
- classic outbound connection natting is known as SNAT
- NOTE: rules for the VPCs in AWS are dynamically generated:
/gen_capo_firewall.rb
: runs once an hour
-
VPN is taken care of by strongswan, default IKE service provided by ubuntu.
- strongswan / ipsec encrypts/decrypts traffic based on connection IDs
- the actual traffic needs to be somehow marked with the right IDs to make it into the matching tunnel
- usually this is accomplished by iptables policies, in our case (due to a route-based VPN using dynamic routing) this is not possible
- ip tunnel kernel interfaces are used instead
- read on for more details on this
- strongswan / ipsec encrypts/decrypts traffic based on connection IDs
-
failover between the 2 devices is handled in 2 different ways:
- OSPF/BGP dynamic routing for VPN traffic (
/etc/bird/bird.conf
) - VRRP/Keepalive for default gateway/outbound internet traffic (
/etc/keepalived/keepalived.conf
)
- OSPF/BGP dynamic routing for VPN traffic (
-
the routing is taken care of by bird. it runs OSPF on the SV2 side and exports any routes learnd over the VPNs by BGP into SV2 through OSPF
-
all global config is found in the correspondig service folder in /etc -> i.e. /etc/bird/ or /etc/ipsec.conf -> these are GLOBAL settings shared by all connections
-
individual VPN tunnel configs are generated dynamically by
/gen_configs.sh
- the script will take 3x arguments
- the location of XML config file from AWS portal
- you can get the XML config by downloading the Sophos ASG version from the AWS portal
- this can also be a URL
- assigned VPC CIDR subnet (i.e. 172.16.12.0/24)
- teamname or very very short description of the tunnel (capo-usw1, agentprofile, ...)
- example how I built the logstash VPN:
/gen_configs.sh http://myserver.intranet.com/temp/ops-lnxgwy2.xml 172.16.10.0/24 logstash
.
- the location of XML config file from AWS portal
- it generates 4x brand new config files that all go in
/vpns/
(the global configs include these by wildcard-globbing)- *.bird.conf -> routing config bird/BGP
- *.ipsec.conf -> remote ipsec tunnel IP and to be selected connection ID
- *.ipsec.secrets -> tunnel secrets
- *.ipsec.sh -> configuring virtual tunnel interfaces matching the connection ID selected above by ipsec
- this is a regular shell script that sets up required kernel interfaces.
- NOTE: it needs to be run once after a reboot.
- it is NOT started automatically at bootup by design to prevent preemting the VPN tunnels back to the freshly rebooted box.
- cleanup a old or incorrect tunnel simply delete these 4x files in
/vpns
and reload the various congigs (see below for more details there)
- the script will take 3x arguments
ipsec status
: list current SAsipsec reload
: reload ipsec config after you added or removed a tunnelipsec rereadsecrets
: if you change the secrets in a file a simleipsec reload
is not enough, you need to reload the actual secretsipsec update
: reload ipsec.conf fileipsec status
: check VPN:birdc
: routing shell:- you can VERIFY any routing related info in here.
configure
will reload the config files- syntax check
configure check
- and soft reload
configure soft
- syntax check
- support cisco-stype "?" and "tab" for autocompletion and quick help
- for more details see here
ip tunnel show
: shows currently configured kernel tunnel interfaces (the VPN does NOT have to be up to show here in this list)ip tunnel add/remove ...
for adding or removing a tunnel
iptables -L -n -v --line-num
: shows the whole firewall rulesetiptables -L -n -v --line-num -t nat
: shows the firewall nat table (Port Forwarding and Source NAT rules etc)iptables -L -n -v --line-num -t mangle
: shows special mangle table (i.e MSS adjust, other advanced packet manipulation actions)
Setting up a new VPN tunnel to a VPC
run ./gen_configs.sh [file/url to xml-file] [VPC-cidr] [consumer/team/project requesting the service]
i.e. ./gen_configs.sh https://ops-jenkins.sv2.trulia.com/vpns/zgtdb.xml 172.16.10.0/24 zgtdb
the XML file is completely generated by Terraform and is ingested by this script completely untouched. usually the URL for this XML file should point directly to the jenkins XML store location
the part above will only take care of the routing/VPN part. the firewall part needs to be updated/adjusted in /etc/init.d/firewall
- by default each VPN tunnel gets it's own iptables chain. an iptables chain is similar to a legacy firewall zone.
- all packets floing through the firewall are handled by the FORWARD chain.
- this chain delegates individual zones to individual chains
- the FORWARD chain is build statically on bootup or on initial VPN spinup. it does NOT change dynamically
- dynamic updates happen ONLY to the individual chains to limit the blast radius
- the rule-set for each chain is found as a simple CSV file in
/capo-firewalls/**.list
on each node - some are statically defined chains -> these rulesets are checked into this repo.
- some firewall chains are generated dynamically by pulling up2date information from consul and converting it to a CSV file. (through
/gen_capo_firewall.rb
)- this script does some initial error checking
- it will only update the CSV if it connected successfully
- both dynamic and static CSV files are pushed live by default once an hour (check
/refresh-capo-firewall.sh
) - the cron job calls the script
gen_iptables_from_csv.sh
that pushes a CSV table into the actual running ACL chain.- this script does some extra error checking and can actually handle FQDN in the CSV and convert these to IPs
- it only runs if there are actual changes in the CSV file since the last run (it does this by creating a copy of the original CSV by appending
.last
to the filename - if there are changes to the last run it will email the changes (as well as any potential errors occured) to the root mail alias
- it will also log all output to syslog