in order for the coverage model to be loaded and used the pyon.yml name needs to be set to ion
other wise the data will not be loaded.
cd to the ooi-extern directory and do a python bootstrap.py
, once complete do a bin/buildout
.
This service handles the communication between the DMS and Geoserver. The importer service allows the modification (add,remove) of geoserver data layers from ooi coverages
run bin/ipython init.py
the service will tell you which port it is on. you can simply then pass a query to the service eg.
is the service alive
http://localhost:8844/service=alive&name=ooi&id=ooi
add a layer
http://localhost:8844/service=addlayer&name="DATASET_ID"&id="DATASET_ID"
reset the datastore
http://localhost:8844/service=resetstore&name=ooi&id=ooi
NOTE: postgres needs to be buildout against pyon 2.7.X other wise things will not run correctly you can modify your postgres setup to bypass some of the issues if you are using supd
but this requires adding paths to eggs for coverage and everything else.
i suggest reading information on FDW from the multicorn page before proceding
- Create the extension
CREATE EXTENSION multicorn
- create the server using the FDW
CREATE SERVER cov_srv foreign data wrapper multicorn options (
wrapper 'covfdw.CovFdw'
);
- create the foreign data table with the path to the dataset
drop foreign table covtest;
create foreign table covtest (
dataset_id character varying,
time timestamp,
cond real,
temp real,
lat real,
lon real,
"geom" geometry(Point,4326)
) server cov_srv options (cov_path '/path/to/dataset/44afbd5858c44a8494f171d15e76d0ab',cov_id '44afbd5858c44a8494f171d15e76d0ab');
- you can add this to a postgres function as follows
CREATE OR REPLACE FUNCTION runCovTest() returns text as $$
drop foreign table covtest;
create foreign table covtest (
dataset_id character varying,
time timestamp,
cond real,
temp real,
lat real,
lon real,
"geom" geometry(Point,4326)
) server cov_srv options (cov_path '/path/to/datasets/44afbd5858c44a8494f171d15e76d0ab',cov_id '44afbd5858c44a8494f171d15e76d0ab');
$$ LANGUAGE SQL ;
- Because the srid is fixed geom can be overridden using postgres mk_point, to caluclate the SRID from the lat,lon of a coverage by generating a postgres view as follows.
CREATE or replace VIEW covproj as
SELECT ST_SetSRID(ST_MakePoint(lon, lat),4326) as proj, dataset_id, time, cond, temp from covtest;
- notice that the server is called
cov_src
, and the data table is calledcov_test
and the view containing the srid information is calledcovproj
. To view the information stored you could simply do:
SELECT * [time,cond,temp] from covproj [limit #] [where 'field' = 'condition'];
###Install FDW alongside coverage model using ooivm.
before proceding review the following site, particually the "how do we do that?" section:
http://multicorn.org/implementing-an-fdw/
Create a new user
sudo adduser eoitest
sudo passwd eoitest
Add eoitest to /etc/sudoers
sudo vi /etc/sudoers
move to postgres user to create some python functions
sudo su - postgres
create language
CREATE LANGUAGE plpythonu;
create path verification function
CREATE or replace FUNCTION checkpath ()
RETURNS text
AS $$
import sys
import os
print sys.version
print sys.path
return '\n'.join(sys.path)
$$ LANGUAGE plpythonu;
test it
select checkPath();
quit
\q
go to the eoi user
sudo su - eoitest
modify default python version
vi ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=/opt/python2.7/bin:$PATH
PATH=/opt/python2.7/bin:$PATH
PATH=/home/eoitest/ooi-extern/fdw:$PATH
export PATH
as the postgres user do the same as above
switch to the postgres user
sudo su - postgres
drop existing multicorn items
drop extension multicorn cascade;
as sudo in eoitest
easy_install pip
then clone the repo git clone [url for repo] ooi-extern
. install the coverage fdw by going to the directory cd /home/eoitest/ooi-extern/fdw
then sudo /opt/python2.7/bin/python setup.py install
you should see the following at the end of the build, this shows the egg was build and moved to the correct location. take note that it says py2.7.egg!
Installed /opt/python2.7/lib/python2.7/site-packages/covfdw-0.0.1-py2.7.egg
Processing dependencies for covfdw==0.0.1
Finished processing dependencies for covfdw==0.0.1
change to postgres user then do psql
verify that the egg is on the package list i.e (/opt/python2.7/lib/python2.7/site-packages/covfdw-0.0.1-py2.7.egg)
select checkPath();
quit and login to eoitest user, and go to ooi-extern directory
\q
sudo su - eoitest
cd ~/ooi-extern
create virtual env to hold all the python stuff
mkvirtualenv --no-site-packages eoipg
then do the following
pip install numpy==1.7.1
pip install -U setuptools==0.8
pip install --upgrade setuptools
edit the profile
vi ~/.bash_profile
add workon eoipg
get the submodules and initalize them using the virtual machine python
git submodule update --init
cd extern/gsconfig
use virtual machine python
sudo ~/.virtualenvs/eoipg/bin/python setup.py develop
build out the coverage model.
cd ../coverage-model
git submodule update --init
sudo ~/.virtualenvs/eoipg/bin/python bootstrap.py -v 2.2.0
build out the coverage model bin/buildout
, incase of permissions issues do this: sudo chown eoitest:eoitest coverage-model
- if you get an error for psycopg2 do the following
pip install psycopg2==2.5.1
thenbin/buildout
build out the interfaces bin/generate_interfaces
you should get the following generate_interfaces: Completed with exit code: 0
cd ../pyon
git submodule update --init
sudo ~/.virtualenvs/eoipg/bin/python bootstrap.py -v 2.2.0
incase of permissions issues sudo chown eoitest:eoitest pyon
bin/buildout
bin/generate_interfaces
coverage model dependancy on pyon is here
https://github.com/ooici/coverage-model/blob/master/coverage_model/utils.py
the site packages for postgres should be located in /opt/python2.7/lib/python2.7/site-packages
sort out python pathing issues, i.e the python path in psql
for notes see http://www.postgresql.org/docs/9.2/static/plpython-envar.html
the PYTHONPATH needs to be set in the following file /etc/profile.d/postgresql.sh
with all the eggs in the coverage model.
cd to dir and make backup
cd /etc/profile.d/
sudo cp postgresql.sh postgresql.sh.bk
in ~/ooi-extern of eoitest do
find /home/eoitest/ooi-extern/extern/coverage-model/eggs -name *.egg > eoieggs.txt
then create the new file to contain the python path information
python path_tool.py
as main user not POSTGRES or EOITEST
sudo bash -c 'cat /home/eoitest/ooi-extern/python_path.txt >> /etc/profile.d/postgresql.sh'
set python in postgressql.sh
export PYTHON=/opt/python2.7/bin/python:$PYTHON
install the following (maybe dont do this) instead change permissions (chmod 755 user dir eoitest)
sudo /opt/python2.7/bin/easy_install pip
sudo /opt/python2.7/bin/pip install numexpr
sudo /opt/python2.7/bin/pip install simplejson
make sure numpy is installed
sudo /opt/python2.7/bin/pip install numpy
sudo /opt/python2.7/bin/pip install psycopg2
if requirement is satisfied Requirement already satisfied
install psycopg2 and regular user not eoitest or postgres
pip install psycopg2
make sure numpy and psycopg2 are on the select checkPath();
list, if it is not add it
i.e
/opt/python2.7/lib/python2.7/site-packages/numpy-1.8.1-py2.7.egg-info
/opt/python2.7/lib/python2.7/site-packages/psycopg2-2.5.2-py2.7.egg-info
restart the services by doing a sudo /etc/init.d/postgresql-9.3 stop
then ps -ef | grep psql or postgres to see if it stopped running then sudo /etc/init.d/postgresql-9.3 start
. check that the path has been updated using...
sudo su - postgres
psql
select checkPath();
create and run the imports that are used by the covfdw to verify they will work
CREATE or replace FUNCTION test ()
RETURNS text
AS $$
import sys
import math
import numpy as np
import numpy
import string
import time
from multicorn import ColumnDefinition
from multicorn import ForeignDataWrapper
from multicorn import Qual
from multicorn.compat import unicode_
from multicorn.utils import log_to_postgres,WARNING,ERROR
from numpy.random import random
import numexpr as ne
import simplejson
from gevent import server
from gevent.monkey import patch_all; patch_all()
import gevent
import random
from random import randrange
import os
import datetime
from coverage_model import SimplexCoverage, AbstractCoverage,QuantityType, ArrayType, ConstantType, CategoryType
return '\n'.join(sys.path)
$$ LANGUAGE plpythonu;
verify it works by doing a select test();
in postgres
if not errors occur do the following
CREATE SERVER cov_srv foreign data wrapper multicorn options (
wrapper 'covfdw.CovFdw'
);