Virtual Machine / Deployment Setup
qjhart opened this issue · 2 comments
The plan is to replicate the setup for aggie experts, and the dams in order
to run a zero down-time system. This requires the library website going to
a router that switches between two potential backend systems.
Because the website is editable, the setup in this case will be
Required Virtual Machines
We require 3 virtual machines in order to run the setup.
machine | specs |
---|---|
blue.library.ucdavis.edu | 8Gb, 250Gb, 8cpu |
gold.library.ucdavis.edu | 8Gb, 250Gb, 8cpu |
router.library.ucdavis.edu | 4Gb, 25Gb, 8cpu |
Apache Router
We are currently using apache as our router. Although there are some more modern alternatives to this setup; nginx and traffic server, my inclination is to continue using apache. The reasons being 1) we are most used to this setup and 2) It appears that the nginx load balance requires a commercial setup for the type of load balancing we use, and 2a) I found trafficserver very confusing.
<VirtualHost *:80>
ServerName library.ucdavis.edu
ServerAlias lib.ucdavis.edu
ServerAlias www.library.ucdavis.edu
ServerAlias www.lib.ucdavis.edu
ServerAdmin library@ucdavis.edu
RewriteEngine on
RewriteRule ^ https://library.ucdavis.edu%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>
<VirtualHost *:443>
ServerName lib.ucdavis.edu
ServerAlias www.library.ucdavis.edu
ServerAlias www.lib.ucdavis.edu
ServerAdmin library@ucdavis.edu
RewriteEngine on
RewriteRule ^ https://library.ucdavis.edu%{REQUEST_URI} [END,QSA,R=permanent]
SSLCertificateFile /etc/ssl/certs/__library_ucdavis_edu_cert.cer
SSLCertificateKeyFile /etc/ssl/private/__library.ucdavis.edu.key
SSLCertificateChainFile /etc/ssl/certs/incommon_interm.cer
</VirtualHost>
<VirtualHost *:443>
ServerName library.ucdavis.edu
Header add Set-Cookie "ROUTEID=d.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
SSLProxyEngine on
<Proxy "balancer://prod">
BalancerMember "http://blue.web.library.ucdavis.edu:80" route=blue
BalancerMember "http://gold.web.library.ucdavis.edu:80" route=gold
ProxySet stickysession=ROUTEID
ProxySet failonstatus=503
ProxySet lbmethod=byrequests
</Proxy>
AllowEncodedSlashes On
ProxyPreserveHost On
# ForensicLog ${APACHE_LOG_DIR}/forensic.log
ProxyRequests off
ProxyPass "/" "balancer://dev/" nocanon
ProxyPassReverse "/" "balancer://dev/" nocanon
# IMPORTANT!
RequestHeader unset X-Forwarded-Host
RequestHeader unset X-Forwarded-For
RequestHeader unset X-Forwarded-Server
SSLCertificateFile /etc/ssl/certs/__library_ucdavis_edu_cert.cer
SSLCertificateKeyFile /etc/ssl/private/__library.ucdavis.edu.key
SSLCertificateChainFile /etc/ssl/certs/incommon_interm.cer
</VirtualHost>
Routing setup
The router will continuously monitor a standard port (80?) on both blue and gold, and route to whichever is currently working. In our setup, the standard practice will be that only one computer will be running at any time. After an update, one computer or the other will be turned off.
The same blue and gold machines can also run stage and dev machines. In fact, this is considered best form, as we can more easily upgrade from stage to production in that setup. blud and gold use different ports to identify the version they are running.
Update Path
All stage and production deployment configurations will be located at /etc/library-website/{version}
where version is a number that corresponds to the current data setup. This is setup, so that the volumes are identified with data updates, and not with something else (dns eg.). This allows us to move an implementation from stage to production with a simple docker compose up/down without needing a seperate copy of the data. dev and other setups should be in /etc/library-website/dev
The typical update path then is for the new setup to be tested in stage. For switching, stage is brought down, the production setup is set to read-only; a final database / fs update is performed, the .env file is switched to production, and the machine is brought back up. At that point two servers respond to library requests, the old server in read-only, the new server ready for updates. Once the new server is verified to be working, the old server is brought down.
Hot Spare
The previous server configuration can act as a ready to go spare, Without modification, this spare would bring up a slightly outdated website (both functionality and data). It is possible to keep a more up-to-date spare, aggie experts uses rsync techniques between docker volumnes for this task; the library website may have a different setup.
@spelkey-ucd / @jrmerz Let me know what you think about these notes. Questions that I have:
- I choose above to 301 everything to library.ucdavis.edu
- we will use *.library.ucdavis.edu, a subset in the staff VPN
- I would suggest we use dev.library.ucdavis.edu for testing
- Let me know about the hot spare
@jniffenegger and others suggested using *.library as the subdomain. I've edited the above to show that.
I've added a diagram that combines this with the production site, and the special collection redirect issues.