*event here means a happening where people get together and do something.
Linked Events is event information:
- Aggregator => using Python importers which have the logic to import events information from different data sources
- Creator => by offering PUT/POST
/event
API endpoint with granular user permissions and a hierarchical organization structure supporting different publishing rights for different organizations - Publisher => by offering API endpoints from which interested parties can retrieve information about events
Linked Events was originally developed for City of Helsinki organization and you can see the Linked Events API in action for Helsinki capital region here. It contains data from all Helsinki City Departments as well as data from Helsinki Marketing and the Helmet metropolitan area public libraries. Viewing the API should give a reasonable view for the kind of information Linked Events is targeted for.
Target audience of this README.md are developers who may or may not have a lot of Python experience and would like to get things running as quickly as possible. Therefore, instructions written in this README.md should be written accordingly.
The best way to contribute is to open a new PR for discussion. We strive to be able to support various cities with various use cases, so suggestions and new features (as long as they fit in with existing functionality) are welcome.
If all you want is a barebone application to work with for your own city:
-
Copy
./docker/django/.env.example
to./docker/django/.env
and change the variable values to your liking. -
Start django application and database server:
docker-compose up
-
Access application on localhost:8000
-
You are done 🔥
If you wish to use locations, addresses and events data from the Helsinki capital region:
-
Read linked-events-importers.md and decide the importers or commands you would like to use.
-
You can then serve the UI for Linked Events API for example by setting authentication keys to
local_settings.py
-
UI app is specific to Helsinki at the moment and requires general Finnish ontology as well as additional Helsinki specific audiences and keywords to be present. However, UI code should be easily adaptable to your own city if you have an OAuth2 authentication server present
These instructions assume an $INSTALL_BASE, like so:
INSTALL_BASE=$HOME/linkedevents
If you've already cloned this repository, just move repository root into $INSTALL_BASE/linkedevents. Otherwise just clone the repository, like so:
git clone https://github.com/City-of-Helsinki/linkedevents.git $INSTALL_BASE/linkedevents
Prepare Python 3.x virtualenv using your favorite tools and activate it. Plain virtualenv is like so:
virtualenv -p python3 $INSTALL_BASE/venv
source $INSTALL_BASE/venv/bin/activate
Install required Python packages into the virtualenv
cd $INSTALL_BASE/linkedevents
pip install -r requirements.txt
Create the database, like so: (we have only tested on PostgreSQL)
cd $INSTALL_BASE/linkedevents
sudo -u postgres createuser -R -S linkedevents
# Following is for US locale, we are not certain whether Linkedevents
# behaves differently depending on DB collation & ctype
#sudo -u postgres createdb -Olinkedevents linkedevents
# This is is for Finnish locale
sudo -u postgres createdb -Olinkedevents -Ttemplate0 -lfi_FI.UTF-8 linkedevents
# Create extensions in the database
sudo -u postgres psql linkedevents -c "CREATE EXTENSION postgis;"
sudo -u postgres psql linkedevents -c "CREATE EXTENSION hstore;"
# This fills the database with a basic skeleton
python manage.py migrate
# This adds language fields based on settings.LANGUAGES (which may be missing in external dependencies)
python manage.py sync_translation_fields
# This creates language objects with correct translations
python manage.py create_languages
If you wish to install Linkedevents without any Helsinki specific data (an empty database), and instead customize everything for your own city, you have a working install right now.
The last steps are needed if you wish to use location, address or event data from the Helsinki metropolitan region, or if you wish to run the Helsinki UI (https://linkedevents.hel.fi) from https://github.com/City-of-Helsinki/linkedevents-ui. Currently, the UI is specific to Helsinki and requires the general Finnish ontology as well as additional Helsinki specific audiences and keywords to be present, though its code should be easily adaptable to your own city if you have an OAuth2 authentication server present.
The commands below are documented in more detail in linked-events-importers.md.
cd $INSTALL_BASE/linkedevents
# Import general Finnish ontology (used by Helsinki UI and Helsinki events)
python manage.py event_import yso --all
# Add keyword set to display in the UI event audience selection
python manage.py add_helsinki_audience
# Add keyword set to display in the UI main category selection
python manage.py add_helsinki_topics
# Import places from Helsinki metropolitan region service registry (used by events from following sources)
python manage.py event_import tprek --places
# Import places from Helsinki metropolitan region address registry (used as fallback locations)
python manage.py event_import osoite --places
# Import events from Helsinki metropolitan region libraries
python manage.py event_import helmet --events
# Import events from Espoo
python manage.py event_import espoo --events
# Import City of Helsinki hierarchical organization for UI user rights management
python manage.py import_organizations https://dev.hel.fi/paatokset/v1/organization/?limit=1000 -c openahjo -s OpenAhjoAPI:ahjo
# Import municipalities in Finland
python manage.py geo_import finland --municipalities
# Import districts in Helsinki
python manage.py geo_import helsinki --divisions
# install API frontend templates:
python manage.py install_templates helevents
The last command installs the helevents/templates/rest_framework/api.html
template,
which contains Helsinki event data summary and license. You may customize the template
for your favorite city by creating your_favorite_city/templates/rest_framework/api.html
.
For further erudition, take a look at the DRF documentation on customizing the browsable API
After this, everything but search endpoint (/search) is working. See search
- Note: Processor architecture on Apple-silicon machines is ARM64, not AMD64. If you have one of these extra tinkering is needed. On Intel-silicon machines, this is not necessary.
- Note 2: As AMD64 is emulated on top of ARM64 Linux, this isn't fast.
- Pre-build linux/AMD64-versions
- PostgreSQL-container:
podman build --platform=linux/amd64 \ -f docker/postgres/Dockerfile \ .
- Django-container:
podman build --platform=linux/amd64 \ -f docker/postgres/Dockerfile \ .
- PostgreSQL-container:
- Now
podman-compose
will find pre-built alternate architecture container images and will run ok:podman-compose up
- Done, test @ http://127.0.0.1:8000/
- Problem on macOS:
django.contrib.gis.geos.error.GEOSException: Error encountered checking Geometry returned from GEOS C function "GEOSGeom_createCollection_r".
- See: django/django#15214
- Fix:
- Edit
linkedevents/settings.py
, add to end:# macOS: GDAL_LIBRARY_PATH="/opt/homebrew/opt/gdal/lib/libgdal.dylib" GEOS_LIBRARY_PATH="/opt/homebrew/opt/geos/lib/libgeos_c.dylib"
- Edit
-
Problem: Error: no container with name or ID "linkedevents-backend" found: no such container
- Solution:
You missed the part with: Copy
./docker/django/.env.example
to./docker/django/.env
- Solution:
You missed the part with: Copy
-
Problem: TCP-ports not available in host
- Solution: If port forwarding from VM is flaky, native ports are visible. Django should be visible in http://127.0.0.1:8080/
Development installation above will give you quite a serviceable production installation for lightish usage. You can serve out the application using your favorite WSGI-capable application server. The WSGI-entrypoint for Linked Events is linkedevents.wsgi
or in file linkedevents/wsgi.py
. Former is used by gunicorn, latter by uwsgi. The callable is application
.
You will also need to serve out static
and media
folders at /static
and /media
in your URL space.
Tests must be run using an user who can create (and drop) databases and write the directories your linkedevents installation resides in. Also the template database must include Postgis and HSTORE-extensions. If you are developing, you probably want to give those permissions to the database user configured in your development instance. Like so:
# Change this if you have different DB user
DATABASE_USER=linkedevents
# Most likely you have a postgres system user that can log into postgres as DB postgres user
sudo -u postgres psql << EOF
ALTER USER "$DATABASE_USER" CREATEDB;
\c template1
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS hstore;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
EOF
Afterwards you can run the tests:
cd $INSTALL_BASE/linkedevents
py.test events
Note that search tests will fail unless you configure search
The project uses django-mailer for queuing and sending email. When a normal Django email function like send_mail
is called, the email will be stored into a queue that exists in the database for later sending.
django-mailer
comes with a number of management commands for interacting with the email queue. Out of those, the send_mail
command is for us perhaps the most notable one as it is responsible for the actual sending of the queued messages.
To send messages locally, you can run the send_mail
command:
./manage.py send_mail
For more information about django-mailer
and the management commands, you can refer to the usage documentation.
Linked Events uses two files for requirements. The workflow is as follows.
requirements.txt
is not edited manually, but is generated
with pip-compile
.
requirements.txt
always contains fully tested, pinned versions
of the requirements. requirements.in
contains the primary, unpinned
requirements of the project without their dependencies.
In production, deployments should always use requirements.txt
and the versions pinned therein. In development, new virtualenvs
and development environments should also be initialised using
requirements.txt
. pip-sync
will synchronize the active
virtualenv to match exactly the packages in requirements.txt
.
In development and testing, to update to the latest versions
of requirements, use the command pip-compile
. You can
use requires.io to monitor the
pinned versions for updates.
To remove a dependency, remove it from requirements.in
,
run pip-compile
and then pip-sync
. If everything works
as expected, commit the changes.
This project uses
black
,
flake8
and
isort
for code formatting and quality checking. Project follows the basic
black config, without any modifications.
Basic black
commands:
- To let
black
do its magic:black .
- To see which files
black
would change:black --check .
pre-commit
can be used to install and
run all the formatting tools as git hooks automatically before a
commit.
Project includes a .git-blame-ignore-revs
file for ignoring certain commits from git blame
.
This can be useful for ignoring e.g. formatting commits, so that it is more clear from git blame
where the actual code change came from. Configure your git to use it for this project with the
following command:
git config blame.ignoreRevsFile .git-blame-ignore-revs
New commit messages must adhere to the Conventional Commits specification, and line length is limited to 72 characters.
When pre-commit
is in use, commitlint
checks new commit messages for the correct format.
Linkedevents uses Elasticsearch for generating results on the /search-endpoint. If you wish to use that functionality, proceed like so:
-
Install elasticsearch
We've only tested using the rather ancient 1.7 version. If you are using Ubuntu 16.04, 1.7 will be available in the official repository. This limitation was originally due to django-haystack not supporting versions above 1. As of writing this the django-haystack version in use does support versions 1, 2, 5 and 7.
-
(For Finnish support) Install elasticsearch-analyzer-voikko, libvoikko and needed dictionaries
/usr/share/elasticsearch/bin/plugin -i fi.evident.elasticsearch/elasticsearch-analysis-voikko/0.4.0
This specific command is for Debian derivatives. The path toplugin
command might be different on yours. Note that version 0.4.0 is the one compatible with Elasticsearch 1.7Installing libvoikko:
apt-get install libvoikko1
Installing the dictionaries (v5 dictionaries are needed for libvoikko version included in Ubuntu 16.04):
wget -P $INSTALL_BASE http://www.puimula.org/htp/testing/voikko-snapshot-v5/dict-morpho.zip unzip $INSTALL_BASE/dict-morpho.zip -d /etc/voikko ```
-
Configure the thing
Set the
ELASTICSEARCH_URL
environment variable (or variable inconfig_dev.env
, if you are running in development mode) to your elasticsearch instance. The default value ishttp://localhost:9200/
.Haystack configuration for all Linkedevents languages happens automatically if
ELASTICSEARCH_URL
is set, but you may customize it manually usinglocal_settings.py
if you know Haystack and wish to do so. -
Rebuild the search indexes
python manage.py rebuild_index
You should now have a working /search endpoint, give or take a few.
It is possible to extend event data and API without touching events
application by implementing separate extension applications. These extensions will be wired under field extension_<extension idenfier>
in event API. If not auto enabled (see 6. below), extensions can be enabled per request using query param extensions
with comma separated identifiers as values, or all
for enabling all the extensions.
To implement an extension:
-
Create a new Django application, preferably named
extension_<unique identifier for the extension>
. -
If you need to add new data for events, implement that using model(s) in the extension application.
-
Inherit
events.extensions.EventExtension
and implement needed attributes and methods. See extensions.py for details. -
Add
event_extension: <your EventExtension subclass>
attribute to the extension applications'sAppConfig
. -
Make the extension available by adding the extension application to
INSTALLED_APPS
. -
If you want to force the extension to be enabled on every request, add the extension's identifier to
AUTO_ENABLED_EXTENSIONS
in Django settings.
For an example extension implementation, see course extension.