DEPRECATED: Although this is the repository where NIS packages started, latest commits (features, fixes) are available in the fork at https://github.com/ENVIRO-Module/nis-backend)
A MuSIASEM instance
NIS (Nexus Information System) is a software system being developed inside MAGIC project (H2020 grant #689669).
NIS enables accounting the biophysical flows in complex bioeconomic systems according to MuSIASEM concepts and methodology, as a way to assess the sustainability of current -or future- scenarios in socio-ecologic systems.
NIS is made of four components: nis-backend, nis-frontend, nis-client and nis-eda. nis-backend is the backend component of NIS, which is deployed embedding another NIS component, nis-frontend (an Angular web application) and it is (programmatically) actionable through a third, nis-client, using Python or R. nis-eda is an exploratory data analysis tool written in R and Shiny which connects to a nis-backend.
nis-backend "runs" MuSIASEM models by interpreting (lexically, syntactically and semantically) and resolving (quantitatively) a representation of a MuSIASEM grammar (an open model), using a tabular syntax ("dataframe" compatible), and producing a set of outputs exportable for further use by external, complementary tools.
It is an open system as it can be used to integrate MuSIASEM as a formalism in analytic/scientific workflows thanks to the RESTful API (and the easier to use nis-client, for Python and R scripts).
Disclaimer: this README is still under elaboration, details may be missing or innacurate.
- Documentation
- Getting started
- Accessing nis-backend in Python and R scripts with nis-client
- Configuration
- Contributing
- People
- Contact
- License
- Acknowledgements
- Keywords
Full documentation:
- How-to (Annex 4 - "NIS frontend how-to manual").
- Technical reference (Annex 2 - "Report on NIsys toolset development").
- Format of input files (Annex 3 - "Format for specification of NIS case studies").
is available in the annexes of the deliverable of the project regarding "nis-backend", in the official web page, at http://www.magic-nexus.eu/documents/deliverable-33-report-datasets-and-software-tools.
NOTE: the "Format of input files" document is updated outside of the main document.
- MuSIASEM adaptation. MuSIASEM concepts have been adapted to an object model using a DSL (Domain Specific Language) with tabular syntax (spreadsheet, dataframe).
- Formalization, standardization. The former point has as consequence a formal and standard definition of MuSIASEM grammars/models, for repeatability, reproducibility, replicability, findability of case studies and of MuSIASEM itself, and also for automatic processing of specifications.
- Uses. Apt for both learning and teaching; for research, economic analysis, decision making, policy making, diagnosis and prognosis, etc.
- Integrability. Easy to integrate into analytical workflows (GIS, statistical, engineering, ...).
- Inputs - External. Integrated use of statistical data sources: Eurostat, OECD, FAO, FADN.
- Inputs - Internal. Integration of data from custom external spreadsheets.
- Reusability of model parts. Similar to libraries/packages in programming languages.
- Integrated solver. Capable of exploiting available quantifications and relationships, in multiple:
- Systems, e.g. interconnected countries.
- Scales, e.g. food and macronutrients.
- Scenarios, which are named sets of values for parameters, e.g. "optimistic", "expected", "pessimistic".
- Sources, so several observations can be made about the same fact.
- Times, in years or in months.
- Outputs. Exportable as datasets (statistical cubes), graphs (networks), matrices, models (transformations of the input model), geolayers, scripts (Python or R).
- Deployment. It can be deployed in multiple ways: personal computer (pip, source, Docker), or server (source or Docker).
- Configurable. Deployment for server can be configured to use different database servers and REDIS for session management.
- Open. All functionality can be accessed through a RESTful service. Behind the scenes, because HTTP protocol is stateless, a complex serialization/deserialization function combined with a key/value store and a browser session enable saving the state in memory after a service call ends, just to recover it when another invocation is done.
- Two expertise levels. Two components wrap the RESTful interface, nis-frontend and nis-client. These components match two expertise levels: nis-frontend does not require programming knowledge, while nis-client can be used in Python/R scripts.
nis-backend is a Python package. There are several options to have it installed.
The pip version is obviously for execution as package, and does not include the frontend, the other two deployment options below include the frontend.
- Set a Python environment
- Install the package with
pip install nexinfosys
- Use class NIS, with methods similar to NISClient class in nexinfosys-client package.
For a lab setup, this option allows to configure and run nis-backend inside a Docker environment (Docker must be installed in the target computer).
The instructions are at "Docker Hub" site, please follow the link:
https://hub.docker.com/r/magicnexush2020/magic-nis-backend
NOTE: Python3 and git required.
Clone this repository and execute "service_main.py":
git clone https://github.com/MAGIC-nexus/nis-backend
cd nis-backend
git checkout develop
pip install -r requirements.txt
PYTHONPATH=. python3 nexinfosys/restful_service/service_main.py
Alternately to the last line, for a more robust server (gunicorn assumed to be installed):
gunicorn --workers=3 --log-level=2000 --bind 0.0.0.0:5000 nexinfosys.restful_service.service_main:appChange the port to the one desired
To specify MuSIASEM grammars (models), NIS proposes a syntactically simple "block oriented DSL" (DSL - Domain Specific Language), structured in a spreadsheet, where each worksheet is considered a batch of Commands of the same kind, each row is an instance of the command and each column is a field for the command.
In order to take advantage of nis-backend, analysts write a workbook considering that commands (worksheets) are read and executed from left to right, and inside a worksheet from the first row to the last.
The reference of a command appears when a command from the "Commands" tab is selected in nis-frontend. A document with a complete reference is available at ¿ZENODO?. An HTML called "Commands reference" can be obtained from "Exportable datasets" tab, containing an up-to-date reference.
The next section introduces nis-frontend which is a tool embedded in nis-backend whose goal is to help in writing MuSIASEM grammars.
nis-backend embeds the compiled version of a full Angular application called nis-frontend which is the indicated interface to nis-backend for most users.
Once nis-backend is executing in your computer, open a browser a go to the address:
http://localhost:5000/nis_client
A screen similar to the following should open:
The screen has a top bar (blue colored) where a menu, the Play button and a field allowing the specification of a URL are present.
The rest of the screen is divided in two zones, at left and right.
The right zone is where the process of representation of a MuSIASEM grammar (model) can be edited (textually or graphically), issues are showed for correction, and outputs can be explored or exported for further elaboration.
The left zone contains helpers allowing the insertion of empty Commands (worksheets) and the selection of external Datasets for the elaboration of internal ones.
The URL field allows editing a workbook using Google Sheets (the application must be authorized first). nis-backend reads directly from the specified worksheet, so it needs access enabled to the file.
For extensive documentation see the deliverable. Online help for commands appears below the list of commands:
and help for each field in a command appears when clicking on the name of a field in a worksheet:
When information is typed for a field in the embedded worksheet, syntactic validity is checked.
IMPORTANT: nis-frontend is closed source due to a restriction in the license of a commercial component.
nis-backend is, from a software engineering point-of-view, a RESTful service. All its functionality is accessed through this interface.
nis-frontend makes use of it behind the scenes, it is not necessary for users to know the technical details in this regards. The other alternative designed to access this service is a Python component called "nexinfosys-client". It can be installed using "pip":
pip install nexinfosys-client
With this package it possible to integrate the MuSIASEM related capabilities of nis-backend in a processing workflow. See nis-client README for basic usage example.
Thanks to R capabilities to use Python packages, the component can be used from R, see also the former README.
If no configuration file is provided (environment variable MAGIC_NIS_SERVICE_CONFIG_FILE not specified), a default configuration file is generated which can be later modified with a text editor.
To specify a custom configuration, a text file with the typical syntax of a Variable and its Value per line must be created:
VAR1="VALUE"
VAR2="VALUE"
| Variable name | Value | Example |
|---|---|---|
| DB_CONNECTION_STRING | Metadata database, SQLAlchemy compliant connection string | "sqlite:///nis_metadata.db" |
| DATA_CONNECTION_STRING | Dataset cache database, SQLAlchemy compliant connection string | "sqlite:///nis_cached_data.db" |
| CASE_STUDIES_DIR | Directory where case studies would be stored | "/srv/nis_data/cs/" |
| FAO_DATASETS_DIR | Directory where FAO datasets are downloaded and cached | "/srv/faostat/" |
| FADN_FILES_LOCATION | Directory where FADN datasets are downloaded and cached | "/srv/fadn" |
| CACHE_FILE_LOCATION | Directory where SDMX datasets are downloaded and cached | "/srv/sdmx_datasets_cache" |
| REDIS_HOST_FILESYSTEM_DIR | If REDIS_HOST='filesystem:local_session', directory where sessions are stored | "/srv/sessions" |
| SSP_FILES_DIR | Not used | "" |
| NIS_FILES_LIST | A comma-separated list of URLs to CSV files where NIS case studies or parts of them are enumerated. Each CSV file starts with a header, with four columns: name, url, description and example (True if it is an example) | "" |
| REDIS_HOST | "localhost" expects a REDIS server available at localhost:6379; "redis-local" creates a local REDIS instance; "filesystem:local_session" uses filesystem to store sessions (a good option for execution in PC/laptop) | "" |
| TESTING | "True" | "True" |
| SELF_SCHEMA | Name of the host where Backend RESTful service responds, preceded by the protocol (http or https) | "https://one.nis.magic-nexus.eu/" |
| FS_TYPE | "Webdav" | "Webdav" |
| FS_SERVER | Host name of the WebDAV server | "nextcloud.data.magic-nexus.eu" |
| FS_USER | User name used. Files and folders must be readable and writable by this user | "" |
| FS_PASSWORD | Password for the previous user | "" |
| GAPI_CREDENTIALS_FILE | Path to a file obtained from Google API management web, to directly access a NIS workbook file in Google Sheets | "/srv/credentials.json" |
| GAPI_TOKEN_FILE | Path to a file used to stored authorization token | "/srv/tocken.pickle" |
Contributions are welcome. Follows a list of possible ways in which you could do it:
- Testing as end user. Install "nis-backend" (this repository) and execute it, launching the user interface to specify and submit case studies, maybe with the help of an external spreadsheet software. Use Github issues when you find incorrect behavior. Please follow Mastering Issues guidelines.
- Testing as scripts developer. Install "nis-backend" and use it in your scripts using either NIS class or the subproject nis-client connecting it to a running "nis-backend".
- Prepare reusable MuSIASEM library files, containing model parts, like structures of high level functions in different domains (water, energy, food), interface type hierarchies, technical data about structurals, mappings from specific official statistical datasets to MuSIASEM interface types, calculation of scalar indicators, benchmarks, etc. All of them may be compiled in separate files, as long as the composition of them is consistent.
- If you are a developer willing to contribute to "nis-backend", please mail one of the contact persons (see Contact below) to discuss some of the features and improvements which could be interesting to have implemented. Once you are ready to contribute, please do it through Github Pull Requests, following the standard PR workflow.
The standard contributor covenant code of conduct is earnestly applied in interactions with the community.
An enumeration of people who have contributed in different manners to the elaboration of NIS during MAGIC project lifetime:
- Rafael Nebot. ITC-DCCT (Instituto Tecnológico de Canarias - Departamento de Computación).
- Marco Galluzzi.
- Michele Staiano. UniNa (Università degli Studi di Napoli Federico II). NIS supervisor, mentor, catalyzer.
- Paula Moreno. ITC-DCCT.
- Mario Giampietro, Ansel Renner, Violeta Cabello, Cristina Madrid, Maddalena Rippa, Juan Cadillo, Raúl Velasco, Louisa di Felice, Sandra Bukkens. ICTA-UAB (Institut de Ciència i Tecnologia Ambientals - Universitat Autònoma de Barcelona). MuSIASEM creators and mentors, analysts.
- Ignacio López.
- Alberto Sosa, Francisco Socorro, María Artiles, Carlos Caraballo, Ivet Cabrera. Internship students at ITC-DCCT.
Please send any question regarding this repository to rnebot@itccanarias.org.
This project is licensed under the BSD-3 License - see the LICENSE file for details
The development of this software was supported by the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 689669 (MAGIC). This work reflects the authors' view only; the funding agencies are not responsible for any use that may be made of the information it contains.
Sustainability - Bioeconomy - Socio-Ecological Systems - Complex Adaptive Systems - Water-Energy-Food Nexus