Develop, Operate and Collaborate on Data and Analytics
Automated - Flexible - Secure - Scalable
Data scientists use a variety of Python and R modules to create relevant insights based on multiple sets of data from many different sources. core4os enables data scientists and other users to integrate an existing insight-generation processing chain into a fault-tolerant, distributed system, thereby automating the whole data analytics process from data transformation to insight generation without the usual need to worry about the underlying software or hardware.
core4os takes care of everything that is essential to using and operating such a distributed system, from central logging and configuration to deployment, all while scaling to hundreds of servers, allowing for rapid progress from development to production deployment and even enabling the developer to deploy a HTTP API quickly based on the output of the data-processing, which provides a shortcut for creating beautiful, frontend applications.
core4os has on the following prerequisites:
- Linux flavor operating system, preferably Debian 9 or Ubuntu 18.04
- Python 3.5 or higher
- MongoDB database instance version 3.6 or higher up-and-running, see https://www.mongodb.com/download-center#community
- pip 18.1 or higher
- git 2 or higher
Install pip for Python 3, python-venv and git with:
# install prerequisites
sudo apt-get install python3-pip python3-venv python3-dev git dirmngr --yes
Install MongoDB and enable the service to start at boot time with:
# install MongoDB
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb http://repo.mongodb.org/apt/debian "$(lsb_release -sc)"/mongodb-org/4.0 main" | sudo tee /etc/apt/sources.list.d/mongodb.list
sudo apt-get update
sudo apt-get install mongodb-org --yes
sudo systemctl start mongod.service
sudo systemctl enable mongod.service
Please note that MongoDB requires further configuration. See below.
Install nodejs, yarn and npm to build and setup web tools:
# install nodejs and npm
wget -qO- https://deb.nodesource.com/setup_11.x | sudo bash -
sudo apt-get update
sudo apt-get install -y nodejs
# install yarn
wget -qO- https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update
sudo apt-get install yarn --yes
sudo npm -g install vue-cli
Clone and install core4os framework in a Python virtual environment:
# clone core4
git clone https://github.com/plan-net/core4.git
# setup and enter Python virtual environment
cd core4
python3 -m venv .venv
source enter_env
# install core4
pip install --upgrade pip
pip install .
MongoDB requires further configuration to setup a replica set. core4os uses some special features of MongoDB which are only available with replica set.
The interactive script local_setup.py
simplifies this configuration. Start
the script with python local_setup.py
in the Python virtual environment
created above.
Please note that running the core4os regression tests requires two additional
steps. First, the user core
must exists as a MongoDB user with password
654321
. Second, the MongoDB hostname must have an additional domain name
testmongo
. For production the password and hostname must be changed.
Ensure the setup is reflected with your core4os local.yaml
configuration
file.
Pass the following JavaScript to the mongo shell to create user core
with p
assword 654321
:
mongo <<- EOF
use admin
db.createUser(
{
user: "core",
pwd: "654321",
roles: [ { role: "root", db: "admin" } ]
}
);
EOF
Edit the MongoDB configuration file /etc/mongod.conf
and add the following
lines:
security:
authorization: enabled
Add hostname testmongo
to the /etc/hosts
file:
127.0.0.1 localhost
127.0.0.1 devops
127.0.0.1 testmongo
Finally, restart mongod with:
service mongod restart
Test the settings and MongoDB connection with:
mongo --username=core --password=654321 --authenticationDatabase admin
Use core4os tool cadmin
to build all web tools inside the Python virtual
environment created above:
# build core4 web apps
cadmin build
Find the latest core4os documentation at https://core4os.readthedocs.io/en/latest/ or build the sphinx documentation with
cd core4
pip install -e ".[tests]"
cd core4/docs
make html
Both methods of accessing the documentation provide further installation instructions and an Ubuntu 18.04 step-by-step installation guide.
All external packages used within core4os have the associated license placed
within the LICENSES
directory.