The Software Defined Software Development Process aka sd2 (pronounced sd-square) aims to assist developers who work concurrently with multiple source bases, each with multiple branches and each potentially with its own technology stack.
sd2 relies heavily on a single text configuration file that describes a developer's development environments. It also relies heavily on containers that isolate the different environments. sd2 seperates the editing environment from the environment where compilation and testing takes place. In some sense, sd2 brings the advantages of a microservices environment to the development phase, even when you do not use microservices in your architecture.
This repository provides a tool called sd2
that assists developers embracing the
sd2 process.
In sd2 you split your editing, browsing and source control activities from all the other development activities. Editing is done on your developer workstation that we will call editing host(EH). All the other development activities are performed in containers (DC) that can run anywhere you want. The containers run on hosts that we call development hosts (DH).
- First of all, sd2 builds on top of all the lightweight virtualization advantages:
- Start a development stack in milliseconds, exactly the same stack you started last time.
- Keep multiple development stacks isolated from each other. So now you can work on multiple projects (or releases of the same project) with incompatible stacks at the same time.
- Version independently your EH from your DH & DC. Gone are the days where upgrading to the next version of MacOS broke a number of your development projects.
- Ensure that every developer in the team has exactly the same development stack independent of the editing environment they use.
- sd2 uses a combination of technologies to make sure that when you change a file on the EH, the file is copied to the destination(s) almost instanteneoulsy.
- All generated files/artifacts are generated in the DHs. Since you only commit on EHs, there is no chance for accidentally commit generated artifacts.
- Modern IDEs are heavyweight and have perofrmance requirements that increase with bigger projects. sd2 uses multiple replicas of the source code so both the IDE and the compilers/transpilers/toolchain can work independently.
- If your source code targets multiple output platforms you can very efficiently simultaneoulsy make a change in one place and have it compile and run in all the target platforms.
sd2 relies on one or more configuration files that define your DHs, your conainers/images, and your repositories/workspaces. The configuration file needs to be in your home directory under .sd2 (ie ~/.sd2/config.yaml).
sd2 reads the configuration file and takes all the actions needed to maintain the connections and replicate the repositories in real time as they change in the EH computer. If you change the configuration file, sd2 will automatically change what it is doing to match the new configuration.
You can download the appropriate sd2 binary for your platform from the releases section. Put it somewhere so that it is in your PATH, create your configuration file that describes your environment, open a terminal and run sd2 from the command line.
Make sure you read the detailed directions
Location: When sd2 starts it reads its configuration from ~/.sd2/config.yaml
(or in $SD2_CONFIG_DIR/config.yaml
if this variable is set).
Syntax: This file
has four main sections. One that describes the DHs, one that describes the
containers images, one that describes containers
and one that describes the workspaces that you want to be
synced into the DHs. There is a json schema for the file that you can find
here.
There can be multiple configuration files, multiple sections with the same
name can be there. For instance if you work on project foo and bar with different
stacks you can have three files in the directory: config.yaml
for the common
things (e.g. shared hosts), foo-config.yaml
for the foo project and
bar-config.yaml
for the bar project. sd2
will parse all of them.
Examples A very simple configuration file is shown here It will start two containers on a host called paros that run an image that has ubuntu 16.04, ssh and nginx. The first container will be called paros-0 and the second will be called paros-1. From the EH, you should be able to ssh to paros-0 and paros-1. You will find yourself in your own home directory in paros, using your username. On your EH you should be able to open a browser and point it to http://paros-0 or http://paros-1 and see the nginx page.
A second example is shown here. It builds on the first one by adding two workspaces 0.sd2 and 1.sd2 that are replicated into the containers under /tmp. Note how the workspaces configuration sections are built using inheritance to avoid replicating the "exclude" paths. 0.sd2 and 1.sd2 inherit from sd2 which inherits from any.
Another example shown here shows two images ubuntu1404 and ubuntu1604. It also adds four containers paros-0, paros-1, paros-2 and paros-3. paros-0 and paros-1 use the ubuntu1404 image. The other two containers use the ubuntu1604 image. Finally, there are two workspaces 0.sd2 and 1.sd2. 0.sd2 is replicated to paros-0 and paros-2. 1.sd2 is replicated to paros-1 and paros-3.
Expansion:
String values are treated as python string templates with the already parsed
config file as the context. So for instance if you put $name
or ${name}
somewhere it will
be substituted by the earlier value of name. (If you need to
use the dollar sign character $$
to escape $
). Environment variables are also
available this way.
Jinja Template Support
If the config.yaml file starts with the line #!jinja2
sd2 treats the configuration file as a jinja2 template.
The tool first processes the jinja2 file
and then parses it as yaml.
By default, a dictionary with the environment variables
is passed as the context for the jinja2 template rendering. So for instance if
you have {{USER}}
in the file it will be substituted with the USER environment
variable.
Furthermore, if there is an executable called ~/.sd2/config, the executable is first executed, and its output is treated as json. Then this json is parsed and provided as context to the the jinja2 parser. This allows to do very cool initializations based on the source code. For instance, you can have each branch use a different image and have the version of the docker image in the source code. The executable will make this information to the configuration file and the daemon.
Inheritance In the workspaces and hosts sections you can create abstract sections that do not describe a workspace or host but provide some key/values pairs that will be inherited by other sections.
Multiple configuration files
If the tool finds more that one files in ~/.sd2 that end with config.yaml
it will read and parse all the files and merge them together. This is a great
way to separate the configuration of unrelated projects.
- Bring your own editor/IDE. sd2 is agnostic on how you edit your source code.
- Bring your own container images. sd2 does not provide tools to generate docker images, we expect that are already available and published in a local or remote repository.
- Although this might change in the future, you are currently responsible to set up your DH (a DH really only needs an account with your user name, ssh access and docker + rsync installed).
- Start the command as
sd2 --showconfig
. It will show the config file as it is after substitutions and will exit. - Start the daemon by running
sd2
. You can see logs runningsd2 logs
- The logs rotate every time you restart sd2. You can find the current log
file in
/var/logs/sd2/sd2.log
, the previous one insd2.log.1
etc - Look for the events starting with
HH:
to see whether the hosts are reported healthy as expected.
- The logs rotate every time you restart sd2. You can find the current log
file in
- Start the command as
sd2 --showschema
. It will show the json schema file. - Try to ssh to the DHs. You should be able to ssh to any of them from a terminal in your EH.
- You can access a container in a DH with the following command:
ssh -t <<DH>> sudo docker exec -it <<CONTAINER NAME>> su - $USER
Of course in the local case you do not need the ssh part. This is the preferred way to get shell access to the container. - You can see the containers running on a DH by runnign
sudo ssh <<DH>> sudo docker ps
- You can delete all the containers runnin on a DH by running
sudo ssh <<DH>> 'sudo docker rm -f $(docker ps -qa)'
-
When I run docker directly on the MacOS can I just mount the MacOS file system on the container?
This will work but in most cases, but it will have severe performance implications for many common use cases. You can read more about the issue here. -
Is sd2 secure when the development host is across the internet?
sd2 always uses ssh to communicate from the editing machine to the development machine so it is as secure as ssh itself. -
Is sd2 slow when my EH is my mac and my DH is in AWS or some other cloud? Try it, you will be amazed how fast it is. This is a very common use case of sd2.
-
Why would I ever want to copy the files to the DH instead of the container itself?
The lifetime of a container is expected to be much shorter than the lifetime of a DH. By replicating the repositories to the DH and mount them in the container you save the time of continously replicating to new containers and save space when you want multiple containers to access the same repository. -
What are the system files that sd2 alters:
/etc/hosts
~/.ssh/config
All the changes are done in a very safe way that does not alter what is already in the files.
-
What IP addresses do the containers use?
The system automatically assigns addresses in the 172.30.X.X private address range. It uses a mini json cache in ~/.config/.sd2-cidr-db to give the same IP every time. You can override these defaults with environment variables. -
Can I have more than one configuration file? sd2 reads all the files in
~/.sd2
ending in -config.yaml so it is possible and highly encouraged if you work in multiple projects
- Many times I have thought that this separation should take one more step where the development environment can be different from the execution environment. In this directions, you have EH, DH, DC and one more set of environments called ECs where the code runs. Currently the generated artifacts are deployed in the DC itself that plays a dual role.
- Many frameworks, e.g. kubernetes and swarm slightly overlap on the container lifetime management. For now, I have decided to keep things simple and only rely on ssh & friends but I can see how in the future this could be expanded to embrace and leverage such frameworks.
- The implemenation is in python but relies a lot on unix tools. I am sure people would want a Windows port.
- Fork the repository
- Build the source code by running
cd build; make
- Add the full path to the src/lib directory in the PYTHONPATH env variable by running
export PYTHONPATH=/path/to/sd2/src/lib
(also consider adding it to your profile or bashrc) - The main program is in bin/sd2 start from there for testing personal user
- When ready to push:
- change the version number at the top of build/Makefile
- commit all changes
- run
make push