Hao Xu, Ben Keller, Antoine de Torcy, Jason Coposky (2016) QueryArrow: Bidirectional Integration of Multiple Metadata Sources. 8th iRODS User Group Meeting, University of North Carolina at Chapel Hill. June 2016.
Slides: https://irods.org/uploads/2016/06/queryarrow2_Hao-Xu_DICE_iRODS-UGM-2016.pdf https://irods.org/uploads/2017/Xu-RENCI-QueryArrow-slides.pdf
Technical Report: https://irods.org/uploads/2015/01/xu-queryarrow-2016.pdf
Paper: https://irods.org/uploads/2017/Xu-RENCI-QueryArrow-paper.pdf
Specification: https://github.com/xu-hao/CertifiedQueryArrow
QueryArrow is motivated by the following applications: bidirectional metadata integration from different metadata sources, metadata policies, metadata migration from different databases, metadata indexing.
QueryArrow provides a systematic solution to shared namespace and unshared namespace federation of metadata. In particular, QueryArrow allows querying multiple multiple data sources including NoSQL databases, and updating data sources. For data sources that support two-phase commit, QueryArrow also supports distributed transactions. QueryArrow also enables poly-fill for features that the underlying database does not support.
A QueryArrow instance includes a QueryArrow Service and QueryArrow plugins (QAPs). Each plugin provides interface with one data store. The queries are issued from the client in the QueryArrow Language (QAL). QAL is a unified query and update language for SQL and NoSQL databases.
cd docker/build
docker build -t queryarrow-stack-build:0.1.0 .
cd docker/run
docker build -t queryarrow-stack-run:0.1.0 .
If you use customized bases from Internet, then
stack docker pull
stack build
stack image container
From source:
https://www.haskell.org/ghc/download_ghc_8_0_2#sources
Find out where ghc is installed.
If built from source, it is <prefix>/lib/ghc-8.0.2/. Default <prefix> is /usr/local.
Follow the instructions on this page:
If you install from system repo, make sure you run
stack upgrade
On Ubuntu
apt-get install postgresql-server-dev-all libsqlite3-dev -y
On CentOS
yum install postgresql-devel sqlite3-devel -y
git clone http://github.com/xu-hao/QueryArrow
cd QueryArrow
stack build
cd ..
Make a new directory
mkdir build
cd build
../QueryArrow/find_dependencies.sh ../QueryArrow
cpack --config CPackConfig.cmake
On Ubuntu
dpkg -i queryarrow-0.2-Linux-amd64.deb
On CentOS
yum install queryarrow-0.2-Linux-amd64.rpm
By default QueryArrow Configuration files are stored in the /etc/QueryArrow/tdb-plugin-gen-abs.yaml file.
An example is
db_plugin:
qap_name: cache
catalog_database_type: Cache
db_config:
max_cc: 1024
cache_db_plugin:
qap_name: trans
catalog_database_type: Translation
db_config:
rewriting_file_path: "../QueryArrow-gen/rewriting-plugin-gen.rules"
include_file_path:
- "../QueryArrow-gen"
trans_db_plugin:
qap_name: sum
catalog_database_type: Sum
db_config:
summands:
- qap_name: ICAT
catalog_database_type: SQL/HDBC/PostgreSQL
db_config:
db_port: 5432
db_name: ICAT
db_password: testpassword
db_host: localhost
db_username: irods
db_predicates: "../QueryArrow-gen/gen/ICATGen"
db_namespace: ICAT
db_sql_mapping: "../QueryArrow-gen/gen/SQL/ICATGen"
- qap_name: ''
catalog_database_type: FileSystem
db_config:
fs_port: 0
db_namespace: FileSystem
fs_host: ''
fs_root: "/tmp"
fs_hostmap:
- - ''
- 0
- "/tmp"
- qap_name: ''
catalog_database_type: InMemory/BuiltIn
db_config:
db_namespace: BuiltIn
servers:
- server_protocol: service/tcp
server_config:
tcp_server_addr: "*"
tcp_server_port: 12345Currently, the implemented QAPs include
| Name | Description | db_config |
|---|---|---|
| Sum | aggregation | summands, db_namespace |
| Translation | policy support | rewriting_file_path, include_file_path,trans_db_plugin |
| Cache | caching | max_cc, cache_db_plugin |
| Remote/TCP | remoting | db_host, db_port |
| FileSystem | interfacing with file system | fs_host, fs_port, fs_root, fs_hostmap, db_namespace |
| ElasticSearch/ElasticSearch | interfacing with ElasticSearch | db_name, db_namespace, db_predicates, db_sql_mapping, db_host, db_port, db_username, db_password |
| Cypher/Neo4j | interfacing with Neo4j | db_namespace, db_predicates, db_sql_mapping, db_host, db_port, db_username, db_password |
| SQL/HDBC/PostgreSQL | interfacing with Postgres | db_name, db_namespace, db_predicates, db_sql_mapping, db_host, db_port, db_username, db_password |
| SQL/HDBC/SQLite3 | interfacing with SQLite3 | db_file_path, db_namespace, db_predicates, db_sql_mapping |
| SQL/HDBC/CockroachDB | interfacing with CockroachDB | db_name, db_namespace, db_predicates, db_sql_mapping, db_host, db_port, db_username, db_password |
| Include | include other config files | include |
| InMemory/StateMap | in-memory mutable map | db_namespace,predicate_name,db_map |
| InMemory/Map | in-memory immutable map | db_namespace,predicate_name,db_map |
| InMemory/BuiltIn | built-in predicates: like_regex, not_like_regex, eq, ne, le, ge, lt, gt, concat, substr, strlen, add, sub, mul, div, mod, exp, like, not_like, in, replace, regex_replace, sleep, encode |
db_namespace |
QueryArrow provides a CLI command QueryArrow.
QueryArrow provides server protocols. The service protocol is used for clients to communicate with QueryArrow. The remote protocol is used by the Remote QAP. The file system protocol is used by the FileSystem QAP.
| unix domain socket | tcp socket | http | |
|---|---|---|---|
| service | implemented | implemented | implemented |
| remote | implemented | implemented | |
| file system | implemented |
| Service | server_config |
|---|---|
| service/tcp, remote/tcp | tcp_server_addr, tcp_server_port |
| service/http | http_server_port |
| service/unix domain socket, remote/unix domain socket | uds_server_addr |
| file system/tcp | fs_server_port, fs_server_root, fs_server_addr |
The command for starting the server is QueryArrowServer.
In addition to running QueryArrow as a standalone service, QueryArrow provides an iRODS database plugin. To build it, follow instructions for building iRODS database plugins.