@author tjolibois@cls.fr: Project manager & Product owner
@author smarty@cls.fr: Scrum master, Software architect, Quality assurance, Continuous Integration manager
How to read this file? Use a markdown reader: plugins chrome exists (Once installed in Chrome, open URL chrome://extensions/, and check "Markdown Preview"/Authorise access to file URL.), or for firefox (anchor tags do not work) and also plugin for notepadd++.
Be careful: Markdown format has issue while rendering underscore "_" character which can lead to bad variable name or path.
- Overview
- Architecture
- Development
- Installation
- Configuration
- Operation
- Motu clients & REST API
- Docker image
Motu is a robust web server allowing the distribution of met/ocean gridded data files through the web.
Subsetter allows user to extract the data of a dataset, with geospatial, temporal and variable criterias.
Thus, user download only the data of interest.
A graphic web interface and machine to machine interfaces allow to access data and information on data (metadata).
The machine-to-machine interface can be used through a client written in python, freely available here https://github.com/clstoulouse/motu-client-python.
Output data files format can be netCDF3 or netCDF4.
An important characteristic of Motu is its robustness: in order to be able to answer many users without crashing, Motu manages its incoming requests in a queue server.
The aim is to obtain complete control over the requests processing by balancing the processing load according to criteria (volume of data to extract, number of requests to fulfill
for a user at a given time, number of requests to process simultaneously).
Moreover, Motu implements a request size threshold. Motu masters the amount of data to extract per request by computing, without any data processing, the result data size of the request.
Beyond the allowed threshold, every request is rejected. The threshold is set in the configuration file.
Motu can be secured behind an authentication server and thus implements authorization. A CAS server can implement the authentication.
Motu receives with authentication process user information, including a user profile associated with the account.
Motu is configured to authorize or not the user to access the dataset or group of datasets which user is trying to access.
For administrators, Motu allows to monitor the usage of the server: the logs produced by Motu allow to know who (login) requests what (dataset) and when, with extraction criterias.
Motu is a Java Web Application running inside the Apache Tomcat application server. Motu can be run as a single application or can be scaled over several instances.
The clients "motu-client-python" or an HTTP client like a web browser are used to connect to Motu services.
A frontal web, Apache HTTPd for example, is used as a reverse proxy to redirect request to Motu server and also to serve the downloaded data from Motu download folder.
Motu server, runs on a Apache Tomcat server and can serve files either directly "DGF" or by delegating extraction to Thredds server with NCSS or OpenDap protocols.
A NFS server is used to share the netcdf files between Thredds and Motu DGF when they are not deployed on the same host.
An (SSO CAS server)[#ConfigurationSystemCASSSO] is used for the authentication of users but Motu can also be deployed without any authentication system.
The Apache HTTPd, on the top right corner is used to serve the graphic chart when several Motu Web server are deployed.
The schema below shows an example of Motu scalability architecture. The "i1, i2" are the Motu server deployed. They have to share the same business configuration file and the download folder.
To run Motu over several instances, a Redis server has to be deployed in order to share to request id and status. The download folder of Motu has also to be shared between the different Motu instances.
It can be on a NFS server or a GLusterFS server.
The frontal web server "Apache HTTPd" must serve the downloaded files and implement the load balencer between all Motu instances.
All other servers, CAS, NFS remains as on the single instance architecture.
The same source code is used to run Motu with a single architecture or with several instances. It is just done by configuration.
When Motu is scalable, one Motu server instance can run a download request, another distinct Motu server instance can respond to a "get status" request and a last one can respond the URL of the result file.
All ports are defined in motu.properties configuration file.
- HTTP: Apache tomcat manages incoming requests with HTTP protocol.
- HTTPs: Used to manage HTTPs incoming requests. This is delegated to Apache HTTPd frontal web server. Apache Tomcat is not intended to be used with HTTPs.
- AJP: Used to communicate with an Apache HTTPd frontal server
- Socket for Shutdown: Port opened by Tomcat to shutdown the server
- JMX: Used to monitor the application
- Debug: In development mode, used to remotely debug the application
Motu has interfaces with other systems:
- DGF: Direct Get File: Read dataset from the file system. (See how to configure it.)
- Unidata Thredds Data Server: It connects with the NCSS or OpenDap HTTP REST API to run download request for example. (See how to configure it.)
- HTTP CAS Server: Use for Single Sign On (SSO) in order to manager user authentication. (See how to configure it "CAS SSO server" and check profiles attribute set on the dataset.)
- CDO command line tool: CDO is used to deal with 2 types of download requests, which are not covered by NCSS service of Thredds Data Server:
- a download request on a range of depths,
- a download request that come across the boundary of the datasets (for global datasets)
- Redis: Stores the request id and status into the Redis server when Motu is scaled over several instances.
The Motu application has been designed by implementing the Three-Layered Services Application design. It takes many advantages in maintenance cost efficiency and in the ease of its future evolutivity.
Three layers are set in the core "motu-web" project:
- USL: User Service Layer: This layer manages all incoming actions through HTTP request
- BLL: Business Logic Layer: This layer manages all Motu business
- DAL: Data Access Layer: This layer manages all access to Motu external interfaces: DGF, Unidata server, CDO, ...
Each layer is an entry point of the application designed with a singleton. These three singletons gives access to high level managers which provides services by implementing a Java interface. High level managers handle for example the configuration, the request, the catalog, the users.
A common package is also defined to provide utilities: log4j custom format, XML, String ...
The main project is "motu-web". This project is divided in three main layers detailed below:
- usl/servlet/context/MotuWebEngineContextListener.java: Apache tomcat ServletContextListener used to init and stop the application.
- usl/request: All requests are managed with the "motu/web/servlet/MotuServlet.java" by following a command pattern. "action" HTTP parameter matches one of the "usl/request/actions" classes.
- usl/response: XML and Apache velocity data model.
- usl/wcs: All WCS requests and responses are managed in this package. The servlet entry point is defined with the class "motu/web/servlet/MotuWCSServlet.java".
- bll/catalog: Catalog and Product managers. Package "bll/catalog/product/cache" contains the product cache.
- bll/config: Configuration and version manager.
- bll/exception: Business exceptions. MotuException is a generic one. MotuExceptionBase defines all other business exceptions.
- bll/messageserror: Message error manager
- bll/request: The queue server used to download requests
- bll/users: User manager
- dal/catalog: OpenDap, TDS or FILE catalog access.
- dal/config: Configuration (/motu-web/src/main/resources/schema/MotuConfig.xsd) and version manager (configuration, distribution, products).
- dal/messageserror: Manage messages for a specific error (/motu-web/src/main/resources/MessagesError.properties).
- dal/request: NCSS, OpenDAP and CDO
- dal/tds: NCSS and OpenDap datamodel
- dal/users: User manager
"motu-web" project starts daemon threads for:
- bll/catalog/product/cache: Keep a product cache and refresh it asynchronously to improve response time
- bll/request/cleaner: Clean files (extracted files, java temp files) and request status (stored into java map or list objects)
- bll/request/queueserver: Contains a thread pool executor to treat download requests
- dal/request/cdo: A queue server used to manage requests using CDO tool to avoid using too much RAM. As CDO uses a lot of RAM memory, requests that require CDO to be processed are in sequence and only one is processed at a time
- motu-api-message: JAXB API: All errors types, status mode, ... /motu-api-message/src/main/schema/XmlMessageModel.xsd
- motu-api-rest: @Deprecated. Not used since Motu v3.
- motu-distribution: Deployment tools (ant script, install folder structure, ...)
- motu-library-cas: Used to manage JAVA HTTP client with CAS server.
- motu-library-converter: JodaTime, ...
- motu-library-inventory: Used for DGF access. JAXBmodels are /motu-library-inventory/src/main/resources/fr/cls/atoll/motu/library/inventory/Inventory.xsd and CatalogOLA.xsd
- motu-parent: Maven parent, eclipse launchers, documentation
- motu-poducts: Source code and scripts used to build archive motu-products.tar.gz (JDK, Apache tomcat, CDO tools)
- motu-scripts: ./motu bash script
- motu-web: Main Motu project. See Motu-web project for details.
Schema below displays a subset of a dataset variable as an array of 2 longitudes and 2 latitudes. At each intersection, we have got 1 real value (10, 11, 12, 13) as defined in the gridded data.
But which result value is returned by Motu when the request target a location between those longitudes and latitudes ?
As we can see below, 4 areas are displayed and the nearest value from the requested location is returned.
Source code can be downloaded directly from Github.
mkdir motugithub
cd motugithub
git clone https://github.com/clstoulouse/motu.git
#In order to work on a specific version
git tag -l
git checkout motu-X.Y.Z
git status
cd motu
cd motu-parent
-
Add variable in order to run/debug Motu on your localhost:
From Eclipse menu bar: Run/Debug > String substitution
MOTU_HOME=J:\dev\cmems-cis-motu\motu-install-dir
This variable represent the folder where Motu is installed. -
From a file explorer, create folders:
$MOTU_HOME/log
$MOTU_HOME/config
$MOTU_HOME/data/public/download -
Copy configuration files from Eclipse to configuration folder:
Note: If you do not have any motu-config folder available, default configuration files are folders are available in the "/motu-web/src/main/resources" folder
If "motu-config" exists, copy:
cp $eclipse/motu-config/src/config/common/config $MOTU_HOME/config
cp $eclipse/motu-config/src/config/cls/dev-win7 $MOTU_HOME/config -
Add an application server in Eclipse: Window>Preferences>Server>Runtime environment
Name=Apache Tomcat 7.0
Tomcat installation directory=C:\dvlt\java\servers\tomcat\apache-tomcat-7.0.65 -
J2EE perspective > Under the Servers view > Right click > New > Server
Server Name: Tomcat v7.0 Server at localhost -
Edit /Servers/Tomcat v7.0 Server at localhost/server.xml and add
<Context docBase="J:/dev/cmems-cis-motu/motu-install-dir/data-deliveries" path="/mis-gateway/deliveries" />
just under the line:
<Host appBase="webapps" autoDeploy="true" name="localhost" unpackWARs="true">
Now Tomcat can serve downloaded files directly
Click Debug configurations...> Under Apache Tomcat, debug "Motu Tomcat v7.0 Server at localhost"
Open a web browser and test:
http://localhost:8080/motu-web/Motu?action=ping
it displays "OK - response action=ping"
For more details about Eclipse launchers, refers to /motu-parent/README-eclipseLaunchers.md.
Maven is used in order to compile Motu.
You have to set maven settings in order to compile.
Copy/paste content below in a new file settings.xml and adapt it to your information system by reading comments inside.
<settings>
<!-- localRepository: Path to the maven local repository used to store artifacts. (Default: ~/.m2/repository) -->
<localRepository>J:/dev/cmems-cis-motu/testGitHub/m2/repository</localRepository>
<!-- proxies: Optional. Set it if you need to connect to a proxy to access to Internet -->
<!--
<proxies>
<proxy>
<id>cls-proxy</id>
<active>true</active>
<protocol>http</protocol>
<host></host>
<port></port>
<username></username>
<password></password>
<nonProxyHosts></nonProxyHosts>
</proxy>
</proxies>
-->
<!-- Repositories used to download Maven artifacts in addition to https://repo.maven.apache.org
cls-to-ext-thirdparty : contains patched libraries and non public maven packaged libraries
geotoolkit: contains geographical tools libraries
-->
<profiles>
<profile>
<id>profile-cls-cmems-motu</id>
<repositories>
<repository>
<id>cls-to-ext-thirdparty</id>
<name>CLS maven central repository, used for CMEMS Motu project</name>
<url>http://mvnrepo.cls.fr:8081/nexus/content/repositories/cls-to-ext-thirdparty</url>
</repository>
<repository>
<id>geotoolkit</id>
<name>geotoolkit</name>
<url>http://maven.geotoolkit.org/</url>
</repository>
</repositories>
</profile>
</profiles>
</settings>This step is used to generate JAR (Java ARchives) and WAR (Web application ARchive).
In the GitLab Continuous Integration context, the access to the Maven repository are configured with maven password encryption using the ci/settings-security.xml master key, according to https://maven.apache.org/guides/mini/guide-encryption.html. It is recommended to use maven encryption, and for changing a password, ensure that the encryption is done on a maven repository using the same master key than the one used on the continuous integration server, or update with your local master key of your own settings-security.xml.
mkdir motu
cd motu
#Copy paste the content above inside settings.xml
vi settings.xml
#Get source code of the last Motu version
git clone https://github.com/clstoulouse/motu.git
cd motu/motu-parent
# This remove the maven parent artifact from pom.xml, or remove lines below manually:
# <parent>
# <artifactId>cls-project-config</artifactId>
# <groupId>cls.commons</groupId>
# <version>1.2.00</version>
# </parent>
sed -i '6,10d' pom.xml
#Compile the source code
mvn -s ../../settings.xml -gs ../../settings.xml -Pprofile-cls-cmems-motu -Dmaven.test.skip=true clean install
...
[INFO] BUILD SUCCESS
...
All projects are built under target folder.
The Motu war is built under "/motu-web/target/motu-web-X.Y.Z-classifier.war".
It embeds all necessary jar libraries.
This packaging process can be run only on CLS development environment.
This is an helpful script used to packaged as tar.gz the different projects (products, distribution (server and client), configuration, themes).
So if you try to run it outside of CLS development environment, you will have to tune and remove many things to run it successfully (in particular all which is related to motu-config and motu-products).
This step includes the compilation step. Once all projects are compiled, it groups all archives in a same folder in order to easy the final delivery.
You have to set ANT script inputs parameter before running it.
See /motu-distribution/build.xml header to get more details about inputs.
cd /motu-distribution
ant
cd target-ant/delivery/YYYYMMDDhhmmssSSS
4 folders are built containing archives:
- src: contains sources of motu application and the configuration files
- motu-$version-src.tar.gz
- motu-client-python-$version-src.tar.gz
- motu-config-$version-src.tar.gz
- motu-web-static-files-$version-src.tar.gz
- motu: contains the built application archive and the products (java, tomcat, cdo) archive
- motu-distribution-$version.tar.gz
- motu-products-$version.tar.gz
- config: contains two kind of archives:
- motu-config-X.Y.Z-classifier-$timestamp-$target.tar.gz: the built configurations for each target platform
- motu-web-static-files-X.Y.Z-classifier-$timestamp-$target.tar.gz: The public static files (css, js) for each target platform
- motu-client
- motu-client-python-$version-bin.tar.gz
In this chapter some paths are set. For example "/opt/cmems-cis" is often written to talk about the installation path.
You are free to install Motu in any other folder, so in the case, replace "/opt/cmems-cis" by your installation folder.
This installation is used to install Motu on a single instance. To scale Motu on several instances, refers to Install a scalable Motu over several instances.
OS target: Linux 64bits (Tested on centos-7.2.1511)
Minimal configuration for an operational usage:
- CPU: 4 CPU, 2,4GHz
- RAM: 32 Gb RAM
- Storage:
- Motu installation folder 15Gb: can be install on the OS partition (default folder /opt/cmems-cis)
- Motu download folder 200Gb: by default /opt/cmems-cis/motu/data/public/download
Has to be installed in a dedicated partition to avoid freezing Motu if disk is full. Note that the available space of the download folder has to be tuned, depending on:- The number of users which run requests at the same time on the server
- The size of the data distributed
Once started, you can check performance.
For test usage we recommend:
- CPU: 2 CPU, 2,4GHz
- RAM: 10 Gb RAM
- Storage:
- Motu installation folder 15Gb
- Motu download folder 50Gb: by default motu/data/public/download
Motu embeds all its dependencies (Java , Tomcat, CDO). All versions of these dependencies will be visible in the folder name once the Motu product archive is extracted.
For example:
ls -1 /opt/cmems-cis/motu/products
apache-tomcat-7.0.69
cdo-group
jdk1.7.0_79
README
version-products.txt
So bash shell is only required on the Linux host machine.
Motu is able to communicate with different external servers:
- Unidata | THREDDS Data Server (TDS): Motu has been only tested with TDS v4.6.14 2019-07-29. The links to this server are set in the Business settings and are used to run OpenDap or subsetter interfaces. If Motu runs only with DGF, this server is not required.
Note that some specific characters have to be relaxed, e.g. when TDS is installed on Apache Tomcat, add attribute relaxedQueryChars="<>[]{|}" in the connector node by editing conf/server.xml from your TDS tomcat installation folder:
<Connector relaxedQueryChars="<>[\]{|}" port="8080" ...
as reported in this forum topic.
Without this configuration Motu server can raised exeptions visible in the Motu "errors.log", e.g.:
ERROR fr.cls.atoll.motu.web.bll.catalog.product.cache.CacheUpdateService.updateConfigService Error during refresh of the describe product cache, config service=..., productId=...
fr.cls.atoll.motu.web.bll.exception.MotuException: Error in NetCdfReader open - Unable to aquire dataset - location data:
Caused by: java.io.IOException: http://.../thredds/dodsC/$dataset is not a valid URL, return status=400
- Single Sign-On - CAS: The link to this server is set in the System settings. If Motu does not use SSO, this server is not required.
The installation of these two servers is not detailed in this document. Refer to their official web site to know how to install them.
If you need to instance several instances of Motu server on a same host, you have to:
- RAM: set 32Go of RAM for each instance. For example, two Motu instances on a same host requires 64Go
- Storage: allocate disk space, in relationship with the Motu usage. Download dedicated partition can be shared or dedicated.
- Folders: Install each Motu instance in a dedicated folder:
- /opt/motu1/motu,
- /opt/motu2/motu,
- ...,
- /opt/motuN/motu
Check this section only if you have installed Motu v2.x and you want to install Motu 3.x. In this section we consider that your Motu installation folder of version 2.x is "/opt/atoll/misgw/".
First stop Motu v2.x: /opt/atoll/misgw/stop-motu.
Then install the version 3.x of Motu from scratch. Before starting the new Motu version, upgrade its configuration by ready section below.
Once the version 3.x of Motu runs well, you can fully remove the folder of version 2.x is "/opt/atoll/misgw/".
To avoid any issue, perhaps backup the folder of Motu v2.x before removing it definitively.
motu2xInstallFolder=/opt/atoll/misgw
rm -rf $motu2xInstallFolder/deliveries
rm -rf $motu2xInstallFolder/motu-configuration-common-2.1.16
rm -rf $motu2xInstallFolder/motu-configuration-sample-misgw-1.0.5
rm -rf $motu2xInstallFolder/motu-web
rm -rf $motu2xInstallFolder/start-motu
rm -rf $motu2xInstallFolder/stop-motu
rm -rf $motu2xInstallFolder/tomcat7-motu
rm -rf $motu2xInstallFolder/tomcat-motu-cas
The new version of Motu is compatible with the motuConfiguration.xml file configured in Motu v3.x.
So you can use exactly the same file, but it is important to update some fields to improve performance and future compatibility.
Copy your old motuConfiguration.xml file to the folder /opt/cmems-cis/motu/config, for example:
cp /opt/atoll/misgw/motu-configuration-sample-misgw/resources/motuConfiguration.xml /opt/cmems-cis/motu/config
Then update attributes below:
- use the TDS subsetter protocol to improve performance of product download
- motuConfig/configService/catalog
- ncss: See ncss protocol
- urlSite: Update URL to use the TDS subsetter URL. See Add a dataset, TDS NCSS protocol
- motuConfig/configService/catalog
- check the attributes used to serve downloaded datasets
- motuConfig
- extractionPath See extractionPath
- downloadHttpUrl Set the URL used to serve files from extractionPath
- motuConfig
- remove all @deprecated attributes listed below to ease future migrations. You can read the attribute description in Business configuration.
- XML header Remove header below
< !DOCTYPE rdf:RDF [
<!ENTITY myoceanUrn "http://purl.org/myocean/ontology/service/database#">
]>
- motuConfig
- maxSizePerFile This attribute definition has been updated. See maxSizePerFile parameter configuration
- maxSizePerFileTDS Rename this attribute to maxSizePerFileSub. See maxSizePerFileSub parameter configuration
- runGCInterval Remove the attribute
- httpDocumentRoot Remove the attribute
- useAuthentication Remove the attribute
- defaultActionIsListServices Remove the attribute
- useProxy, proxyHost, proxyPort, proxyLogin, proxyPwd Remove the attributes
- defaultService Remove the attribute, See defaultService . It was previously used to declare a default config Service. Now this attribute sets a default action. If the action set is unknown, an error log will be written and user is redirected to the listServices action which is the default one.
- motuConfig/configService
- defaultLanguage Remove the attribute
- defaultPriority Remove the attribute
- httpBaseRef Remove the attribute
- veloTemplatePrefix Remove the attribute
- motuConfig/queueServerConfig
- defaultPriority Remove the attribute
- motuConfig/queues
- batch Remove the attribute
- lowPriorityWaiting Remove the attribute
- motuConfig/configFileSystem Remove the node
- after starting Motu, if there is issues with the graphic chart, check the attributes below.
- motuConfig
- httpBaseRef See httpBaseRef attribute
- motuConfig/configService
- httpBaseRef Remove the attribute
- motuConfig
Resource URN attributes are not handled in the same way in Motu v3.
In Motu v2.x, URN attributes had values set with an ontology URL:
<resource urn="http://purl.org/myocean/ontology/product/database#dataset-bal-analysis-forecast-phys-V2-dailymeans">
In Motu v3.x, you have to upgrade URN attributes with only the value found after the # character:
<resource urn="dataset-bal-analysis-forecast-phys-V2-dailymeans">
In CMEMS-CIS context the log file motuQSlog.xml is stored in a specific folder in order to be shared.
You have so to check that with the new version this log file is well written in the shared folder.
Here is where this log files were written in Motu v2.x:
grep -i "motuQSlog.xml" /opt/atoll/misgw/motu-configuration-sample-misgw-1.0.5/resources/log4j.xml
<param name="file" value="/opt/atoll/misgw/tomcat-motu-cas/logs/motuQSlog.xml" />
The folder set in the value attribute shall be the same as the one defined in new the Motu configuration file. Replace $path below by the folder path:
grep -i "motuQSlog.xml" /opt/cmems-cis/motu/config/log4j.xml
fileName="$path/motuQSlog.xml"
filePattern="$path/motuQSlog.xml.%d{MM-yyyy}"
Motu installation needs two main step: the software installation and optionally the theme installation.
The software installation brings by default the CLS theme. The theme installation is there to customize or change this default theme.
Install Motu software, for example on a Dissemination Unit
Copy the installation archives and extract them.
cd /opt/cmems-cis
cp motu-products-A.B.tar.gz .
cp motu-distribution-X.Y.Z.tar.gz .
cp motu-config-X.Y.Z-$BUILDID-$TARGET-$PLATFORM.tar.gz .
tar xzf motu-products-A.B.tar.gz
tar xzf motu-distribution-X.Y.Z.tar.gz
tar xzf motu-config-X.Y.Z-$BUILDID-$TARGET-$PLATFORM.tar.gz
cd motu
At this step, Motu is able to start. But static files used for customizing the web theme can be installed.
In the CMEMS context, the installation on a dissemination unit is ended, static files are installed on a central server.
Now you can configure the server:
- Set the system properties: http port, ...
- Configure dataset
- Configure the logs
Refer to configuration in order to check your configuration settings.
Motu is installed and configured. You can start Motu server.
Then you can check installation.
As a dissemination unit administrator, in CMEMS context, this section is not applicable.
Public static files are used to customized Motu theme. When several Motu are installed, a central server eases the installation and the update by
referencing static files only once on a unique machine. This is the case in the CMEMS context, where each dissemination unit host a Motu server, and
a central server hosts static files.
If you runs only one install of Motu, you can install static files directly on Motu Apache tomcat server.
Extract this archive on a server.
tar xvzf motu-web-static-files-X.Y.Z-classifier-$timestamp-$target.tar.gz
Then use a server to make these extracted folders and files accessible from an HTTP address.
Example: The archive contains a motu folder at its root. Then a particular file is "motu/css/motu/motu.css" and this file is served by the URL http://resources.myocean.eu/motu/css/motu/motu.css in the CMEMS CIS context.
If you do not use a central entity to serve public static files, you can optionally extract the archive
and serve files directly by configuring Motu.
First extract the archive:
tar xzf motu-web-static-files-X.Y.Z-classifier-$timestamp-$target.tar.gz -C /opt/cmems-cis/motu/data/public/static-files
Then edit "motu/tomcat-motu/conf/server.xml" in order to serve files from Motu.
Add then "Context" node as shown below. Note that severals "Context" nodes can be declared under the Host node.
[...]
<Host appBase="webapps" [...]
<!-- Used to serve public static files -->
<Context docBase="/opt/cmems-cis/motu/data/public/static-files/motu" path="/motu"/>
[...]
Finally in motuConfiguration.xml, remove all occurrences of the attribute named: httpBaseRef in motuConfig and configService nodes. (Do not set it empty, remove it).
If you want to set another path instead of "/motu", you have to set also the business configuration parameter named httpBaseRef.
./motu start
When you start Motu, the only message shall be:
tomcat-motu - start
Optionaly, when this is your first installation or when a software update is done, an INFO message is displayed:
INFO: War updated: tomcat-motu/webapps/motu-web.war [$version]
If any other messages appear, you have to treat them.
As Motu relies on binary software like CDO, error could be raised meaning that CDO does not runs well.
ERROR: cdo tool does not run well: $cdo --version
[...]
In this case, you have to install CDO manually.
Open a Web browser, and enter:
http://$motuUrl/motu-web/Motu?action=ping
Where $motuUrl is: ip adress of the server:tomcat port
Refer to configuration regarding the tomcat port
Response has to be:
OK - response action=ping
Open a Web browser, and enter:
http://$motuUrl/motu-web/Motu
If nothing appears, it is because you have to add dataset.
Check that no error appears in Motu errors log files.
This section has to be read only if Motu does not start successfully.
Select one option below to install "cdo". If you have no idea about cdo installation, choose the Default option.
- Default option: Install cdo
- cdo is already installed on this machine
- Try MOTU without cdo installation
- How cdo is built?
"cdo" (Climate Data operators) are commands which has to be available in the PATH when Motu starts.
By default, Motu provides a built of CDO and add the "cdo" command to the PATH, but with some Linux distribution it is necessary to install it.
Motu provides some help in order to install CDO.
First check your GLibC version:
ldd --version
ldd (GNU libc) 2.12
[...]
If your GlibC lower than 2.14, you have to install GLIBC 2.14, but to highly recommend to upgrade your Linux operating system to get an up to date GLIBC version:
INSTALL GLIBC 2.14
export MotuInstallDir=/opt/cmems-cis
cd $MotuInstallDir/motu/products/cdo-group
wget http://ftp.gnu.org/gnu/glibc/glibc-2.14.tar.gz
tar zxvf glibc-2.14.tar.gz
cd glibc-2.14
mkdir build
cd build
mkdir $MotuInstallDir/motu/products/cdo-group/glibc-2.14-home
../configure --prefix=$MotuInstallDir/motu/products/cdo-group/glibc-2.14-home
make -j4
make install
cd $MotuInstallDir/motu
Now check if "cdo" runs well:
export MotuInstallDir=/opt/cmems-cis
$MotuInstallDir/motu/products/cdo-group/cdo.sh --version
Climate Data Operators version x.y.z (http://mpimet.mpg.de/cdo)
[...]
If error appear like ones below, it certainly means that GLIC is not in the LD_LIBRARY_PATH.
$MotuInstallDir/motu/products/cdo-group/cdo-x.z.z-home/bin/cdo: error while loading shared libraries: libhdf5.so.10: cannot open shared object file: No such file or directory
or
cdo: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by cdo)
cdo: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/cmems-cis-validation/motu/products/cdo-group/hdf5-1.8.17-home/lib/libhdf5.so.10)
In this case, edit $MotuInstallDir/products/cdo-group/cdo.sh and add "$GLIBC-home/lib" to LD_LIBRARY_PATH.
Now check again if "cdo" runs well.
If it runs well, you can now start Motu.
If "cdo" is installed in another folder on the machine, you can add its path in "$MotuInstallDir/motu/motu" script:
__setPathWithCdoTools() {
PATH=$MOTU_PRODUCTS_CDO_HOME_DIR/bin:$PATH
}
Optionnaly set LD_LIBRAY_PATH in $MotuInstallDir/products/cdo-group/setLDLIBRARYPATH.sh
Note that without CDO, some functionalities on depth requests or on download product won't work successfully. If any case, you can disable the CDO check by commented the check call:
- Disable check:
cd /opt/cmems-cis/motu/
sed -i 's/ __checkCDOToolAvailable/# __checkCDOToolAvailable/g' motu
- Enable check:
cd /opt/cmems-cis/motu/
sed -i 's/# __checkCDOToolAvailable/ __checkCDOToolAvailable/g' motu
CDO is automaticcly build from the script $MotuInstallDir/motu/products/cdo-group/install-cdo.sh Also in order to get full details about CDO installation, you can get details in /opt/cmems-cis/motu/products/README and search for 'Download CDO tools'.
Once archives have been extracted, a "motu" folder is created and contains several sub-folders and files:
motu/
- config: Folder which contains the motu configuration files. Refers to Configuration for more details.
- data: Folder used to managed Motu data.
- public: Folders which contain files exposed to public. It can be published through a frontal Apache HTTPd Web server, through Motu Apache Tomcat or any other access.
- download: Folder used to save the products downloaded. This folder is sometimes elsewhere, for example in Motu v2: /datalocal/atoll/mis-gateway/deliveries/. A best practice is to create a symbolic link to a dedicated partition to avoid to freeze Motu when there is no space left.
- inventories: This folder can be used to store the DGF files.
- transaction: This folder is used to serve the transaction accounting logs
- static-files: Used to store public static files. This folder can be served by a frontal Apache HTTPd Web server or Motu Apache Tomcat. In the CMEMS-CIS context, it is not used as static files are deployed on a central web server.
- public: Folders which contain files exposed to public. It can be published through a frontal Apache HTTPd Web server, through Motu Apache Tomcat or any other access.
- log: Folder which contains all log files. Daily logging are suffixed by yyyy-MM-dd.
- logbook.log: Motu application logs (errors and warning are included)
- warnings.log: Motu application warnings
- errors.log: Motu application errors
- motuQSlog.xml,motuQSlog.csv: Motu queue server logs messages (transaction accounting logs), format is either xml or csv
- motu file: Script used to start, stop Motu application. Refers to Start & Stop Motu for more details.
- pid: Folder which contains pid files of the running Motu application.
- tomcat-motu.pid: Contains the UNIX PID of the Motu process.
- products: Folder which contains Java, Tomcat ($CATALINE_BASE folder) and CDO products.
- apache-tomcat-X.Y.Z: Apache tomcat default installation folder from http://tomcat.apache.org/
- cdo-group: CDO tools from https://code.zmaw.de/projects/cdo
- jdkX.Y.Z: Oracle JDK from http://www.oracle.com/technetwork/java
- version-products.txt: Contains the version of the current Motu products.
- tomcat-motu: Tomcat is deployed inside this folder. This folder is built by setting the $CATALINE_HOME environment variable.
- version-distribution.txt file: Contains the version of the current Motu distribution.
Apache HTTPd is used as a frontal HTTP server in front of Motu Http server. It has several aims:
- Delegate HTTP requests to Motu HTTP server
- Serve directly extracted dataset files written after a download action. The folder in which requests are written is configurable. URL used to download those files is "http://$ipMotuServer/mis-gateway/deliveries". This URL is configurable.
- Serve the download transaction logbook files. The folder in which log files are written is configurable.
- Manage errors like 403, 404. Motu server manages errors web page by displaying a custom message. Redirect to the URL "http://$ipMotuServer/motu-web/Motu?action=httperror&code=$errorCode" by replacing $errorCode with the HTTP error code.
- Optionally acts as a load balancer when several instances of Motu.
See sample of the Apache HTTPd configuration in the Motu installation folder: config/apache-httpd-conf-sample
The configuration is described for Apache2 contains files:
- 001_httpd-vhosts-motu.conf: Main apache configuration file used for Motu, replace $serverHostName by the server host, and $webmasterEmail by the webmaster or administrator email.
- apache2.conf: Use to show how timeout parameter shall be set
- enableApacheModules.sh Describe the Apache modules to enable
When an SSO cas server is used, you have to st the property cas-auth-serverName to http://$serverHostName
Apache HTTPd can be used at different levels. The Apache HTTPd above is the one installed on the same machine as Motu. An Apache HTTPd can be used as a frontal to manage Apache HTTPd load balancing. In the case, you can set up with the following example:
# Use to authenticate users which want to download transaction files
< Location /datastore-gateway/transactions/* >
|--AuthType Basic
|--AuthName "XXX"
|--AuthUserFile /XXX/password.conf
|--Require valid-user
< / Location>
# Used to serve URL requested after a CAS authentication
# Because Motu SSO client set a redirection URL directly to its webapp name
# so we have to take into account the webapp name in Apache HTTPd
ProxyPass /motu-web http://$motuTomcatIp:$motuTomcatPort/motu-web
ProxyPassReverse /motu-web http://$motuTomcatIp:$motuTomcatPort/motu-web
# Used to serve Motu requests
# /mis-gateway-servlet These rules are used for retro compatibility between Motu v2.x and Motu v3.x
ProxyPass /mis-gateway-servlet http://$motuTomcatIp:$motuTomcatPort/motu-web
ProxyPassReverse /mis-gateway-servlet http://$motuTomcatIp:$motuTomcatPort/motu-web
ProxyPreserveHost On
# /motu-web-servlet This URL is sometimes used.
# It can be customized depending of your current installation. If you have any doubt, keep this rule.
ProxyPass /motu-web-servlet http://$motuTomcatIp:$motuTomcatPort/motu-web
ProxyPassReverse /motu-web-servlet http://$motuTomcatIp:$motuTomcatPort/motu-web
ProxyPass /datastore-gateway/transactions http://$apacheHTTPdOnMotuHost/datastore-gateway/transactions
ProxyPassReverse /datastore-gateway/transactions http://$apacheHTTPdOnMotuHost/datastore-gateway/transactions
ProxyPass /datastore-gateway/deliveries http://$apacheHTTPdOnMotuHost/datastore-gateway/deliveries
ProxyPassReverse /datastore-gateway/deliveries http://$apacheHTTPdOnMotuHost/datastore-gateway/deliveries
< Location /motu-web-servlet/supervision>
|--Order allow,deny
|--Allow from All
< /Location>
Motu is a web server based on Apache Tomcat.
In order to secure HTTP connections with a client, HTTPs protocol can be used.
You have two choices:
- Motu as a standalone web server
In this case only Motu is installed.
Refers to the Apache Tomcat official documentation to know how to set SSL certificates: SSL/TLS Configuration HOW-TO - Motu with an Apache HTTPd frontal web server
In this case Motu is installed and also a frontal Apache HTTPd server.
Refers to the Apache HTTPd official documentation: SSL/TLS Strong Encryption: How-To
Apache HTTPd communicates with Apache Tomcat with the AJP protocol.
In order to manage SSO (Single Sign On) connections to Motu web server, Motu uses an HTTPs client.
All documentation about how to setup is written in chapter CAS SSO server.
Scalability is provided with two main components, the Redis database, for sharing the status of the requests between Motu instances, and Tomcat with the session replication mechanism to share login and authentication data.
- install a Redis server. (Motu has been tested with Redis version 4.0.8, 64 bit). Redis shares the request ids and status between all Motu instances
- share the download folder between all instances with a NFS mount, GlusterFS or any other file sharing system.
- set a frontal web server to serve the downloaded files from the Motu server and to load balance the requests between all Motu servers.
- $installDir/motu/config/motuConfiguration.xml: set the Redis settings in the business configuration file. For example:
<motuConfig dataBlockSize="1024000" ...>
....
<redisConfig host="localhost" port="16379"/>
....
</motuConfig>- $installDir/motu/tomcat-motu/conf/web.xml file: add the distributable element inside of the web-app element
<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee
http://xmlns.jcp.org/xml/ns/javaee/web-app_4_0.xsd"
version="4.0">
....
<distributable />
....
</web-app>- $installDir/motu/tomcat-motu/conf/server.xml file
- add a Cluster element in the Engine element
<Server port="16005" shutdown="SHUTDOWN">
<Service name="Catalina">
<Engine name="Catalina" defaultHost="localhost">
....
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="6">
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="500"
dropTime="3000"/>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
....
</Engine>
</Service>
</Server>Ensure that the configured multicast ip is authorized for multicast (here 228.0.0.4). The configured broadcast port (here 45564) has to be available.
Each Motu instance will get allocated a port in the configurable range starting at the port 4000 as configured in the Receiver element. Ensure those ports are available.
channelSendOptions at
6ensures that a request creating a session on a server of the cluster is shared among all other servers and acknowledged, before of being responded.
- Ensure no
jvmRoutefield is configured in the Engine element:
<Server port="16005" shutdown="SHUTDOWN">
<Service name="Catalina">
....
<Engine name="Catalina" defaultHost="localhost">
<!-- instead of <Engine name="Catalina" defaultHost="localhost" jvmRoute="server1"> -->
....
</Engine>
....
</Service>
</Server>- $installDir/motu/tomcat-motu/conf/context.xml: add a Manager in the Context element.
<Context>
....
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"
persistAuthentication="true"/>
....
</Context>Deactivate sticky session mechanisms:
- for httpd VirtualHost blocks
- no route attribute in
BalancerMemberdirectives:
- no route attribute in
<Proxy balancer://cluster/>
BalancerMember http://my-host:6080/motu-web
BalancerMember http://my-host:7080/motu-web
</Proxy>- no stickysession attribute in
ProxyPassandProxyPassReversedirectives:
ProxyPass /motu-web/ balancer://cluster/
ProxyPassReverse /motu-web/ balancer://cluster/- for HAProxy configurations, do not use
stick ondirectives on IP or session
This chapter describes the Motu configuration settings.
All the configuration files are set in the $installDir/motu/config folder.
cd $installDir/motu/config
- config: Folder which contains the motu configuration files.
- motu.properties: JVM memory, network ports of JVM (JMX, Debug) and Tomcat (HTTP, HTTPS, AJP, SHUTDOWN). CAS SSO server settings.
- motuConfiguration.xml: Motu settings (Service, Catalog via Thredds, Proxy, Queues, ....)
- log4j.xml: Log4j v2 configuration file
- standardNames.xml: Contains the standard names
- version-configuration.txt: Contains the version of the current Motu configuration.
This file is watched and updated automatically. This means that when Motu is running, this file has to be written in a atomic way.
You can configure 3 main categories:
- MotuConfig node : general settings
- ConfigService node : catalog settings
- QueueServerConfig node : request queue settings
- RedisConfig node : Redis server config
If you have not this file, you can extract it (set the good version motu-web-X.Y.Z.jar):
/opt/cmems-cis/motu/products/jdk1.7.0_79/bin/jar xf /opt/cmems-cis/motu/tomcat-motu/webapps/motu-web/WEB-INF/lib/motu-web-X.Y.Z.jar motuConfiguration.xml
If you have this file from a version anterior to Motu v3.x, you can reuse it. In order to improve global performance, you have to upgrade some fields:
- ncss Set it to "enabled" to use a faster protocol named subsetter rather than OpenDap to communicate with TDS server. ncss must be enabled only with regular grid. The datasets using curvilinear coordinates (like ORCA grid) can not be published with ncss. Thus, ncss option must be set to disable or empty.
- httpBaseRef shall be set to the ULR of the central repository to display the new theme
- ExtractionFilePatterns to give a custom name to the downloaded dataset file
A string representing the default action in the URL /Motu?action=$defaultService
The default one is "listservices".
All values can be found in the method USLRequestManager#onNewRequest with the different ACTION_NAME.
Amount of data in Ko that can be requested in a single query from Motu to TDS. Default is 2048Kb.
If this amount is lower than the maxSizePerFile (in MegaBytes), Motu will launch several sub-requests to TDS to gather all the data.
Higher value (up to the maxSizePerFile) leads to less requests, but with higher data volume to transfer by request from TDS to Motu. And the tds.http.sotimeout has to be long enough for letting TDS the time to read and transfer the whole data to Motu.
Lower values will imply more requests, but shorter. It consumes more CPU on both Motu and TDS with the advantages to allow TDS to answer to other parallel request it would receive, and to reduce communication times for each request.
Concerning performance, we tried 200Mb and 1024Mb for a 1024Mb request, and durations were similar, but note that the shapes of the sub-requests may not match the shapes of the netcdf files, and that could imply a supplementary delay for hard drive data storage.
If the tds.http.sotimeout can be set to a high value (such as 900s for a 1Gb max size request), the safest is to use the maxSizePerFile value for the dataBlockSize parameter (mind the units Kb/Mb).
Else from the max timeout the environment can support (external constraints or "004-27" error, see tds.http.sotimeout), extrapolate a volume of data that can be transfered during this delay, lower it to keep a margin, and use it as the dataBlockSize.
This parameter is only used with a catalog type set to "FILE" meaning a DGF access.
It allows download requests to be executed only if data extraction is lower than this parameter value.
Unit of this value is MegaBytes.
Default is 1024 MegaBytes.
Example: maxSizePerFile="2048" to limit a request result file size to 2GB.
This parameter is only used with a catalog type used with Opendap or Ncss.
It allows download requests to be executed only if data extraction is lower that this parameter value.
Unit of this value is MegaBytes.
Default is 1024 MegaBytes.
Example: maxSizePerFileSub="2048" to limit request result file size to 2GB.
@Deprecated from v3 This parameter is not used and has been replaced by maxSizePerFile and maxSizePerFileSub.
Number of data in Megabytes that can be written and download for a Netcdf file. Default is 1024Mb.
The absolute path where files downloaded from TDS are stored.
For example: /opt/cmems-cis/motu/data/public/download
It is recommended to set this folder on an hard drive with very good performances in write mode.
It is recommended to have a dedicated partition disk to avoid freezing Motu if the hard drive is full.
By default value is $MOTU_HOME/data/public/download, this folder can be a symbolic link to another folder.
String with format ${var} will be substituted with Java property variables. @See System.getProperty(var)
Http URL used to download files stored in the "extractionPath" described above. It is used to allow users to download the result data files.
This URL is concatenated to the result data file name found in the folder "extractionPath".
When a frontal HTTPd server is used, it is this URL that shall be configured to access to the folder "extractionPath".
String with format ${var} will be substituted with Java property variables. @See System.getProperty(var)
Http URL used to serve files from to the path where archive motu-web-static-files-X.Y.Z-classifier-buildId.tar.gz has been extracted.
For example:
- When httpBaseRef is set to an URL, for example "http://resources.myocean.eu/motu", this URL serves a folder which contains ./css/motu/motu.css.
For example, it enables to serve the file http://resources.myocean.eu/motu/css/motu/motu.css - When httpBaseRef is set to ".", it serves static files which are included by default in Motu application
- When httpBaseRef is removed (not just empty but attribute is removed), it serves a path accessible from URL
$motuIP/$ {motuContext}/motu
IMPORTANT: When Motu URL starts with "HTTPS", if you set an URL in httpBaseRef, this URL has also to start with "HTTPS". On the contrary, when Motu URL starts with "HTTP", if you set an URL in httpBaseRef, this URL can start with "HTTP" or "HTTPS".
In minutes, oldest result files from extraction request which are stored in the folder set by extractionpath are deleted. This check is done each "runCleanInterval" minutes.
Default = 60min
In minutes, oldest status (visible in debug view) than this time are removed from Motu. This check is done each "runCleanInterval" minutes.
Default = 60min
In minutes, the waiting time between each clean process. The first clean work is triggered when Motu starts.
A clean process does:
- delete files inside java.io.tmpdir
- delete all files found in extractionFolder bigger than extractionFileCacheSize is Mb
- delete all files found in extractionFolder oldest than cleanExtractionFileInterval minutes
- remove all status oldest than cleanRequestInterval minutes
Default = 1min
Patterns (as regular expression) that match extraction file name to delete in folders:
- java.io.tmpdir
- extractionPath
Default is "..nc$|..zip$|..tar$|..gz$|.*.extract$"
Size in Mbytes.
A clean job runs each "runCleanInterval". All files with a size higher than this value are deleted by this job.
If value is zero, files are not deleted.
Default value = 0.
Provide the delay to wait to refresh the meta-data of products cache after the last refresh.
Motu has a cache which is refreshed asynchronously. Cache is first refreshed as soon as Motu starts.
Then Motu waits for this delay before refreshing again the cache.
This delay is provided in millisecond.
The default value is 60000 meaning 1 minute.
Logbook file (motu/log/logbook.log) gives details about time taken to refresh cache, for example:
INFO CatalogAndProductCacheRefreshThread.runProcess Product and catalog caches refreshed in 2min 19sec 75msec
Logbook file gives details per config service ($configServiceId) about dedicated time taken to refresh cache, for example:
INFO CatalogAndProductCacheRefreshThread.runProcess Refreshed statistics: $configServiceId@Index=0min 34sec 180msec, $configServiceId@Index=0min 31sec 46msec, ...
They are sorted by config service which has taken the most time first.
@Index All config services are refreshed sequentially. This index is the sequence number for which this cached has been refreshed.
Example of archived data with several TB of data. Cache is refreshed daily: describeProductCacheRefreshInMilliSec=86400000
Example of real time data with several GB of data. Cache is refreshed each minute: describeProductCacheRefreshInMilliSec=60000
@Deprecated from v3 This parameter is not used.
@Deprecated from v3 This parameter is not used. Document root of the servlet server.
Optional attribute. Used to set the tag value "DCP" in the response of the WCS GetCapabilities request with a full URL. The WCS DCP URL value is define using the following priority order: - The value of this parameter defines on the motuConfiguration.xml file. The value can be directly the URL to use or the name of a java property define between {} which contains the value of the URL. - The java property "wcs-dcp-url" value - The URL of the web server on which Motu webapps is deployed This attribute can be set when you use a frontal web server to serve the WCS requests, e.g. http://myFrontalWebServer/motu/wcs and your frontal is an HTTP proxy to http://motuWebServer/motu-web/wcs.
@Deprecated from v3 This parameter is not used. It is redundant with parameter config/motu.properties#cas-activated.
@Deprecated from v3 This parameter is not used.
@Deprecated from v3 This parameter is not used. To use a proxy in order to access to a Threads, use the JVM properties, for example:
tomcat-motu-jvm-javaOpts=-server -Xmx4096M ... -Dhttp.proxyHost=monProxy.host.fr -Dhttp.proxyPort=XXXX -Dhttp.nonProxyHosts='localhost|127.0.0.1'
- useProxy
- proxyHost
- proxyPort
- proxyLogin
- proxyPwd
This token is a key value which is checked to authorize the execution of the cache refresh when it is request by the administrator . If the token value provided by the administrator doesn't match the configured token value, the refresh is not executed and an error is returned. A default value "a7de6d69afa2111e7fa7038a0e89f7e2" is configured but it's hardly recommended to change this value. If this token is not changed, it is a security breach and a log ERROR will be written while the configuration will be loaded. The value can contains the characters [A-Za-z] and specials listed here ( -_@$*!:;.,?()[] ) It's recommended to configure a token with a length of 29 characters minimum.
Format of the file name result of a download request.
2 dynamic parameters can be used to configure this attribute:
- @@requestId@@: this pattern will be replaced in the final file name by the id of the request.
- @@productId@@: this pattern will be replaced in the final file name by the id of the requested product.
If this attribute is not present, default value is: "@@productId@@_@@requestId@@.nc"
Configure how motu configuration is reloaded.
Arguments are only 'inotify' or an 'integer in seconds'. 'inotify' is the default value.
- 'inotify': reload as soon as the file is updated (works only on local file system, not for NFS file system).
- 'integer in seconds': reload each X second the configuration in 'polling' mode. If this integer is equals or lower than 0, it disables the refresh of the configuration.
String to set the config service name If the value of this attribute contains some special caracters, those caracters have not to be encoded. For example, if the value is an URL, the caracters ":" and "/" have not to be encoded like "%2E" or "%3A".
String which describes the group
String which describes the service
Optional string containing one value, several values separated by a comma or empty (meaning everybody can access).
Used to manage access right from a SSO cas server.
In the frame of CMEMS, three profiles exist:
- internal: internal users of the CMEMS project
- major: major accounts
- external: external users
Otherwise, it's possible to configure as many profiles as needed.
Profiles are configured in LDAP within the attribute "memberUid" of each user. This attribute is read by CAS and is sent to Motu
once a user is logged in, in order to check if it matches profiles configured in Motu to allow a user accessing the data.
In LDAP, "memberUid" attribute can be empty, contains one value or several values separated by a comma.
Optional, string used to target the default velocity template. It is used to set a specific theme.
Value is the velocity template file name without the extension.
Default value is "index".
Optional, boolean used to determine if the current config service have its cache updated automatically by Motu or not. Default value is "true". "true" means that the config service cache update is executed automatically by Motu.
Optional, used to override motuConfig httpBaseRef attribute for this specific service.
@Deprecated from v3 This parameter is not used.
This catalog name refers a TDS catalog name available from the URL: http://$ip:$port/thredds/m_HR_MOD.xml Example: m_HR_OBS.xml
- tds: Dataset is downloaded from TDS server. In this case, you can use Opendap or NCSS protocol.
- file: Dataset is downloaded from DGF
Example: tds
Optional parameter used to enable or disable the use of NetCDF Subset Service (NCSS) in order to request the TDS server.
ncss must be enabled only with regular grid. The datasets using curvilinear coordinates (like ORCA grid) can not be published with ncss. Thus, ncss option must be set to disable or empty.
Without this attribute or when empty, Motu connects to TDS with Opendap protocol. If this attribute is set to "enabled" Motu connects to TDS with NCSS protocol in order to improve performance.
We recommend to use "enabled" for regular grid datasets.
Values are: "enabled", "disabled" or empty.
-
TDS URL
For example: http://$ip:$port/thredds/ -
DGF URL
For example: file:///opt/publication/inventories
Maximum number of request that an anonymous user can send to Motu before throwing an error message.
Value of -1 means no check is done so an unlimited number of user can request the server.
Default value is 10
In case where an SSO server is used for authentication, this parameter is not used. In this you you will be able to fix a limit by setting "maxPoolAuth" parameter value.
Maximum number of request that an authenticated user can send to Motu before throwing an error message.
Value of -1 means no check is done so an unlimited number of user can request the server.
Default value is 1
In case where no SSO server is used for authentication, this parameter is not used. In this you you will be able to fix a limit by setting "maxPoolAnonymous" parameter value.
@Deprecated from v3 This parameter is not used.
An id to identify the queue.
Description of the queue.
@Deprecated from v3 This parameter is not used.
Use to build a java.util.concurrent.ThreadPoolExecutor an to set "corePoolSize" and "maximumPoolSize" values.
Default value is 1
The total number of threads should not be up to the total number of core of the processor on which Motu is running.
Request are put in a queue before being executed by the ThreadPoolExecutor. Before being put in the queue, the queue size
is checked. If it is upper than this value maxPoolSize, an error message is returned.
Value of -1 means no check is done.
Default value is -1
Size in Megabyte. A request has a size. The queue in which this request will be processed is defined by the request size.
All queues are sorted by size ascending. A request is put in the last queue which has a size lower than the request size.
If the request size if higher than the bigger queue dataThreshold, request is not treated and an error message is returned.
This parameter is really useful when a Motu is used to server several kind of file size and you want to be sure that file with a specific size does no slow down request of small data size.
In this case you can configure two queues and set a number of threads for each in order to match the number of processors. The JVM, even if requests for high volume are running, will be able to
process smallest requests by running the thread on the other processor core. Sp processing high volume requests will not block the smallest requests.
@Deprecated from v3 This parameter is not used.
This optional node is used to run Motu in a scalable architecture. Do not add this node when you just run one single Motu instance.
Once this node is added, Motu stores all its request ids and status in Redis.
Define the host (ip or server name) where is deployed the Redis server od Redis cluster used by Motu to share the RequestId and RequestStatus data. Default value is localhost
Define the port used by the Redis server or Redis cluster used by Motu to share the requestId and RequestStatus data. Default value is 6379
Define the prefix used to build the RequestId value of the shared RequestStatus data. Default value is requestStatus
Define if the redis server in in cluster mode. This is a boolean value. By default is set to false and the cluster mode is not activate. To activate the cluster, the value have to be set on true.
System settings are configured in file config/motu.properties
All parameters can be updated in the file.
The three parameters below are used to tune the Java Virtual Machine, and the tomcat-motu-jvm-javaOpts parameter can include any Java property in the form "-D<java property name>=<value>":
# -server: tells the Hostspot compiler to run the JVM in "server" mode (for performance)
tomcat-motu-jvm-javaOpts=-server -Xmx4096M -Xms512M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=512M
tomcat-motu-jvm-port-jmx=9010
tomcat-motu-jvm-address-debug=9090
tomcat-motu-jvm-umask=tomcat|umask|0000 (More details...)
By default, if tomcat-motu-jvm-umask is not set, motu sets the umask with result of the command umask
tomcat-motu-jvm-umask=umask|tomcat|0000
- umask: By default, if tomcat-motu-jvm-umask is not set, motu sets the umask with result of the command
umask - tomcat: Apache Tomcat process forces umask to 0027 (https://tomcat.apache.org/tomcat-8.5-doc/security-howto.html)
- 0000: Custom umask value
Values 0002 or umask are recommended if Motu download results are served by a frontal web server
By default this parameter is at "300". It represents the maximum delay in seconds for TDS to answer a MOTU request (reading timeout of the socket).
For queries involving lots of files, TDS might need more than the default 5 minutes to answer, and to avoid the error "004-27 : Error in NetcdfWriter finish", this parameter can be set to a higher value in:
tomcat-motu-jvm-javaOpts=-server [...] -XX:MaxPermSize=512M -Dtds.http.sotimeout=4000
By default this parameter is at "60". It represents the maximum delay in seconds for TDS to accept a MOTU request (connection timeout on the socket).
The paramater can be customized and added in:
tomcat-motu-jvm-javaOpts=-server [...] -XX:MaxPermSize=512M -Dtds.http.conntimeout=100
The parameters below are used to set the different network ports used by Apache Tomcat.
At startup, these ports are set in the file "$installdir/motu/tomcat-motu/conf/server.xml".
But if this file already exist, it won't be replaced. So in order to apply these parameters, remove the file "$installdir/motu/tomcat-motu/conf/server.xml".
tomcat-motu-port-http=9080
# HTTPs is in a common way managed from a frontal Apache HTTPd server. If you really need to use it from Tomcat, you have to tune the SSL certificates and the protocols directly in the file "$installdir/motu/tomcat-motu/conf/server.xml".
tomcat-motu-port-https=9443
tomcat-motu-port-ajp=9009
tomcat-motu-port-shutdown=9005
# true or false to enable the SSO connection to a CAS server
cas-activated=false
# Cas server configuration to allow Motu to access it
# @see https://wiki.jasig.org/display/casc/configuring+the+jasig+cas+client+for+java+in+the+web.xml
# The start of the CAS server URL, i.e. https://cas-cis.cls.fr/cas
cas-server-url=https://cas-cis.cls.fr/cas
# The Motu HTTP server URL, for example: http://misgw-ddo-qt.cls.fr:9080 or http://motu.cls.fr
# If you use a frontal HTTPd server, you have to known if its URL will be called once the user will be login on CAS server.
# In this case, set the Apache HTTPd server. The value will be http://$apacheHTTPdServer/motu-web/Motu So, in Apache HTTPd, you have to redirect this URL to the Motu Web server
cas-auth-serverName=http://$motuServerIp:$motuServerPort
# The proxy callback HTTPs URL of the Motu server ($motuServerIp is either the Motu host or the frontal Apache HTTPs host ip or name. $motuServerHttpsPort is optional if default HTTPs port 443 is used, otherwise it is the same value as defined above with the key "tomcat-motu-port-https", or it is the port defined for the HTTPs server on the frontal Apache HTTPd)
cas-validationFilter-proxyCallbackUrl=https://$motuServerIp:$motuServerHttpsPort/motu-web/proxyCallback
IMPORTANT: Motu uses a Java HTTPs client to communicate with the CAS server. When the CAS server has an untrusted SSL certificate, you have to add it to Java default certificates or to add the Java property named "javax.net.ssl.trustStore" to target a CA keystore which contains the CAS Server SSL CA public key. For example, add this property by setting Java option tomcat-motu-jvm-javaOpts:
tomcat-motu-jvm-javaOpts=-server -Xmx4096M -Xms512M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=512M -Djavax.net.ssl.trustStore=/opt/cmems-cis/motu/config/security/cacerts-with-cas-qt-ca.jks
The following part is not relevant in the CMEMS context as the SSO CAS server has been signed by a known certification authority.
If you need to run tests with your own SSO CAS server without any certificate signed by a known certification authority, you have to follow the following steps.
How to build the file cacerts-with-cas-qt-ca.jks on Motu server?
- Download the certificate file (for example "ca.crt") of the authority which has signed the CAS SSO certificate on the CAS server machine (/opt/atoll/ssl/ca.crt) and copy it to "${MOTU_HOME}/config/security/", then rename it "cas-qt-ca.crt"
- Copy the default Java cacerts "/opt/cmems-cis-validation/motu/products/jdk1.7.0_79/jre/lib/security/cacerts" file into ${MOTU_HOME}/config/security/
and rename this file to "cacerts-with-cas-qt-ca.jks"
cp /opt/cmems-cis-validation/motu/products/jdk1.7.0_79/jre/lib/security/cacerts /opt/cmems-cis/motu/config/security/ mv /opt/cmems-cis/motu/config/security/cacerts /opt/cmems-cis/motu/config/security/cacerts-with-cas-qt-ca.jks - Then import "cas-qt-ca.crt" inside "cacerts-with-cas-qt-ca.jks", Trust the certificate=yes
/opt/cmems-cis-validation/motu/products/jdk1.7.0_79/bin/keytool -import -v -trustcacerts -alias $CAS_HOST_NAME -file cas-qt-ca.crt -keystore cacerts-with-cas-qt-ca.jks -keypass XXX
When NetCdf variables are read in data files, either by Threads or directly by Motu, Motu wait for a standard name metadata sttribute to be found for each variable as requiered by the CF convention.
Due to any production constraints, some netcdf files does not have any standard_name attribute.
In the case, you can add directly in the configuration folder, a file named standardNames.xml in order to map a standard_name to a netcdf variable name.
You can find an example in Motu source: /motu-web/src/main/resources/standardNames.xml
To enable the status supervision, set the parameter below:
tomcat-motu-urlrewrite-statusEnabledOnHosts=localhost,*.cls.fr
This parameter is used to set the property below in the WEB.XML file:
<!-- Documentation from http://tuckey.org/urlrewrite/manual/3.0/
you may want to allow more hosts to look at the status page
statusEnabledOnHosts is a comma delimited list of hosts, * can
be used as a wildcard (defaults to "localhost, local, 127.0.0.1") -->
<init-param>
<param-name>statusEnabledOnHosts</param-name>
<param-value>${tomcat-motu-urlrewrite-statusEnabledOnHosts}</param-value>
</init-param>
For more detail read:
org.tuckey UrlRewriteFilter FILTERS : see http://tuckey.org/urlrewrite/manual/3.0/
Log are configured by using log4j2 in file config/log4j.xml
This log files are used to compute statistics about Motu server usage.
Two format are managed by this log, either XML or CSV.
To configure it, edit config/log4j.xml
Update the fileFormat attribute of the node "MotuCustomLayout":
A string either "xml" or "csv" to select the format in which log message are written.
Also update the log file name extension of the attributes "fileName" and "filePattern" in order to get a coherent content in relationship with value set for MotuCustomLayout file format.
If this attribute is not set, the default format is "xml".
<RollingFile name="log-file-infos.queue"
fileName="${sys:motu-log-dir}/motuQSlog.xml"
filePattern="${sys:motu-log-dir}/motuQSlog.xml.%d{MM-yyyy}"
append="true">
<!-- fileFormat=xml or csv -->
<MotuCustomLayout fileFormat="xml" />
<Policies>
<TimeBasedTriggeringPolicy interval="1" modulate="true"/>
</Policies>
</RollingFile>
In the dissemination unit, Motu shares its log files with a central server.
Log files have to be save on a public access folder.
Set absolute path in "fileName" and "filePattern" attributes. This path shall be serve by the frontal Apache HTTPd or Apache Tomcat.
For example, if you want to share account transaction log files, you edit config/log4j.xml. Update content below:
<RollingFile name="log-file-infos.queue" fileName="${sys:motu-log-dir}/motuQSlog.xml"
filePattern="${sys:motu-log-dir}/motuQSlog.xml.%d{MM-yyyy}"
with:
<RollingFile name="log-file-infos.queue" fileName="/opt/cmems-cis/motu/data/public/transaction/motuQSlog.xml"
filePattern="/opt/cmems-cis/motu/data/public/transaction/motuQSlog.xml.%d{MM-yyyy}"
Note that both attributes fileName and filePattern have been updated.
Then the frontal Apache HTTPd server has to serve this folder.
In Motu you can update the theme of the website. There is 2 mains things in order to understand how it works?
- [Template] velocity: The velocity templates are used to generated HTML pages from Java objects.
- [Style] CSS, Images and JS: These files are used to control style and behaviour of the web UI.
By default, the template and style are integrated in the "war". But the Motu design enable to customize it easily.
- [Template] velocity: You can change all templates defined in: motu/tomcat-motu/webapps/motu-web/WEB-INF/lib/motu-web-2.6.00-SNAPSHOT.jar/velocityTemplates/*.vm by defining them in motu/config/velocityTemplates.
The main HTML web page structure is defined by the index.vm velocity template. For example, in you create a file motu/config/velocityTemplates/index.vm containing an empty html page, website will render empty web pages.
"index.vm" is the default theme. The name can be updated for each motuConfig#configService by setting veloTemplatePrefix="".
By default veloTemplatePrefix="index".
- [Style] CSS, Images and JS: Those files are integrated with the default theme motu-web-2.6.00-SNAPSHOT.war/css/, motu-web-2.6.00-SNAPSHOT.war/js/. These files can be downloaded from an external server which enable to benefit to several mMotu server at he same time. The external server name can be updated for each motuConfig#configService by setting httpBaseRef="".
By default httpBaseRef search static files from the Motu web server, for example:
service.getHttpBaseRef()/css/motu/screen/images/favicon.ico"
All operations are done from the Motu installation folder.
For example:
cd /opt/cmems-cis/motu
Start the Motu process.
./motu start
At the shutdown of Motu, the server waits for none of the pending or in progress request to be in execution.
If it's the case, the server waits the end of the request before shutdown.
Note that after waiting 10 minutes, server will automatically shutdown without waiting any running requests.
So command below can respond quickly if no requests are in the queue server or takes time to process them.
./motu stop
If you needs to understand what Motu is waiting for, you can check the logbook:
tail -f log/logbook.log
Stop in progress...
Stop: Pending=0; InProgress=2
Stop: Pending=0; InProgress=2
...
Stop: Pending=0; InProgress=1
Stop: Pending=0; InProgress=0
...
Stop done
During the stop step, from a web browser, the user will be able to ends its download request if a front web server (Apache HTTPd) serves the statics files and the downloaded product. In case where Motu is installed as a standalone web server, user will get a 500 HTTP error. For example in development or qualification environment, this could lead to block the download of the files if Motu is used to serve both static and requested product files.
./motu restart
./motu status
Status are the following:
- tomcat-motu started A pid file exists
- tomcat-motu stopped No pid file exists
./motu ?
Once started, you can use the Linux command "top" to check performance:
- load average the three numbers shall be low and under the number of CPU (lscpu | grep Proc). For example if you have 4 processors this indicator can rise up to 4 but not above. If it is above, you have to add more CPU power.
- %CpuX, parameter wa This indicator shall be near 0 to indicate that processes does not wait to access to the disks. When this number is above 0.5 you have to improve access disk performance.
- KiB Mem Be sure that free memory is available. If it is less than 5000000, meaning less than 5GB, you have to add RAM memory in order to manage pic load.
Example of top command:
top - 11:07:01 up 19:46, 3 users, ***load average: 0,05, 0,09, 0,25***
Tasks: 395 total, 2 running, 393 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1,0 us, 1,0 sy, 0,0 ni, 98,1 id, ***0,0 wa***, 0,0 hi, 0,0 si, 0,0 st
%Cpu1 : 1,0 us, 0,0 sy, 0,0 ni, 99,0 id, ***0,0 wa***, 0,0 hi, 0,0 si, 0,0 st
KiB Mem : 10224968 total, ***4034876 free***, 3334576 used, 2855516 buff/cache
...
Log messages are generated by Apache Log4j 2. The configuration file is "config/log4j.xml".
By default, log files are created in the folder $MOTU_HOME/log. This folder contains Motu log messages.
Tomcat log messages are generated in the tomcat-motu/logs folder.
-
Motu log messages
- logbook.log: All Motu log messages including WARN and ERROR(without stacktrace) messages.
- warnings.log: Only Motu log messages with a WARN level
- errors.log: Only Motu log messages with an ERROR level. When this file is not empty, it means that at least an error has been generated by the Motu application.
- motuQSlog.xml, motuQSlog.csv: Either a "CSV" or "XML" format which logs all queue events.
- CSV: On one unique line, writes:
[OK | ERR;ErrCode;ErrMsg;ErrDate];
queueId;queueDesc; requestId;
elapsedWaitQueueTime;elapsedRunTime;elapsedTotalTime;totalIOTime;preparingTime;readingTime;
inQueueTime;startTime;endTime;
amountDataSize;
downloadUrlPath;extractLocationData;
serviceName;TemporalCoverageInDays;ProductId;UserId;UserHost;isAnonymousUser;
variable1;variable2;...;variableN;
temporalMin,temporalMax;
LatitudeMin;LongitudeMin;LatitudeMax;LongitudeMax: DepthMin;DepthMax; - XML: XStream is used to serialized a Java Object to XML from fr.cls.atoll.motu.web.bll.request.queueserver.queue.log.QueueLogInfo
Same data are represented. - Field details
- queueId, queueDesc: Queue used to process the request. Id and description found in config/motuConfiguration.xml
- requestId: A timestamp representing the request id.
- inQueueTime: Timestamp with format "yyyy-MM-dd' 'HH:mm:ss.SSS" when the request has been put in the queue
- startTime: Timestamp with format "yyyy-MM-dd' 'HH:mm:ss.SSS" when the request has been started to be processed
- endTime: Timestamp with format "yyyy-MM-dd' 'HH:mm:ss.SSS" when the request has been ended to be processed
- elapsedWaitQueueTime: Duration in milliseconds, [startTime - inQueueTime]
- elapsedRunTime: Duration in milliseconds, [endTime - startTime]
- elapsedTotalTime: Duration in milliseconds, [endTime - inQueueTime]
- totalIOTime: Duration in nanoseconds: reading + writing + copying + compressing times.
- readingTime: Duration in nanoseconds.
- writingTime: Duration in nanoseconds.
- preparingTime: Duration in nanoseconds, same value as reading time.
- copyingTime: Duration in nanoseconds, only set in DGF mode.
- compressingTime: Duration in nanoseconds, only set in DGF mode.
- amountDataSize: Size in MegaBytes
- downloadUrlPath: URL to download the product
- extractLocationData: Absolute path on the server
- serviceName: The service name found in the configuration file motuConfiguration.xml
- TemporalCoverageInDays: duration in days
- ProductId: Product id
- UserId: User login if user is not anonymous, otherwise its host or IP address from which he is connected
- UserHost: Host or ip address from which user is connected
- isAnonymousUser: true or false
- variable1;variable2;...;variableN; Extracted variable names
- temporalMin,temporalMax: Temporal coverage
- LatitudeMin;LongitudeMin;LatitudeMax;LongitudeMax: Geographical coverage (latitude:-90;+90; longitude:180;+180)
- DepthMin;DepthMax;: Depth coverage
- CSV: On one unique line, writes:
- velocity.log: Logs generated by the http://velocity.apache.org/ technology to render HTML web pages.
-
Tomcat log messages: This folder contains all Apache Tomcat log files. The file below is important to check startup logs:
- catalina.out: Catalina output matching the environment variable CATALINA_OUT.
In order to add a new Dataset, you have to add a new configService node in the Motu business configuration.
When Motu read data through TDS (Opendap or NCSS service) url, the data shall be configured in TDS before this configuration is saved in Motu. The TDS configuration is not explained here.
Within CMEMS, the datasets are organized in a tree structure, where the product granularity appears above the dataset granularity.
To be noticed:
- All gridded dataset shall be configured in TDS, to be served through the subsetter of Motu
- A product is a coherent group of datasets. The product is the granularity used in the catalogue of CMEMS
- In the XML tree structure of the TDS configuration, each product shall be configured through a unique node
- This node shall correspond to one XML file in the TDS configuration (for example GLOBAL_ANALYSIS_PHYS_001_016.xml) and shall be further referenced in the motuConfiguration.xml file as one (for example <catalog name=" GLOBAL_ANALYSIS_PHYS_001_016.xml>)
- The value of the "name" attribute of the element shall be identical to the Product Name (from CMEMS Product Information Table). In the example below named “CMEMS DU xxx Thredds Catalog” there are three datasets. The following catalog tree presents a hierarchical organization for this catalog.
< CMEMS DU xxx Thredds Catalog >
| ------ < GLOBAL_ANALYSIS_PHYS_001_016 >
|------- < dataset-armor-3d-v5-myocean >
|----------------- < GLOBAL_REP_PHYS_001_013 >
|------- < dataset-armor-3d-rep-monthly-v3-1-myocean >
|------- < dataset-armor-3d-rep-weekly-v3-1-myocean>
The Motu configuration (motuConfiguration.xml) should reference the node corresponding to one XML file in the TDS configuration.
Examples:
- TDS NCSS protocol:
This is the fastest protocol implemented by Motu. Motu select this protocol because type is set to "tds" and ncss is set to "enabled".
<configService description="Free text to describe your dataSet" group="HR-Sample" httpBaseRef="" name="HR_MOD-TDS" veloTemplatePrefix="" profiles="external">
<catalog name="m_HR_MOD.xml" type="tds" ncss="enabled" urlSite="http://$tdsUrl/thredds/"/>
</configService>
- TDS Opendap protocol:
Here OpenDap is used because it is the default protocol when tds type is set and ncss is not set or is disable.
<configService description="Free text to describe your dataSet" group="HR-Sample" httpBaseRef="" name="HR_MOD-TDS" veloTemplatePrefix="" profiles="external">
<catalog name="m_HR_MOD.xml" type="tds" ncss="" urlSite="http://$tdsUrl/thredds/"/>
</configService>
- DGF protocol:
This protocol is used to access to local files. With this protocol user download the full data source file and can run only temporal extractions on the dataset (As a reminder, a dataset is temporal aggregation of several datasource files.
<configService description="Free text to describe your dataSet" group="HR-Sample" profiles="internal, external, major" httpBaseRef="" name="HR_MOD-TDS" veloTemplatePrefix="">
<catalog name="catalogFILE_GLOBAL_ANALYSIS_PHYS_001_016.xml" type="file" urlSite="file:///opt/cmems-cis-data/data/public/inventories"/>
</configService>
An an example, the file catalogFILE_GLOBAL_ANALYSIS_PHYS_001_016.xml contains:
< ?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rdf:RDF [
<!ENTITY atoll "http://purl.org/cls/atoll/ontology/individual/atoll#">
]>
<catalogOLA xmlns="http://purl.org/cls/atoll" name="catalog GLOBAL-ANALYSIS-PHYS-001-016">
<resourcesOLA>
<resourceOLA urn="dataset-armor-3d-v5-myocean" inventoryUrl="file:///opt/cmems-cis-data/data/public/inventories/dataset-armor-3d-v5-myocean-cls-toulouse-fr-armor-motu-rest-file.xml"/>
</resourcesOLA>
</catalogOLA>
File dataset-armor-3d-v5-myocean-cls-toulouse-fr-armor-motu-rest-file.xml:
< ?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rdf:RDF [
<!ENTITY atoll "http://purl.org/cls/atoll/ontology/individual/atoll#">
<!ENTITY cf "http://purl.org/myocean/ontology/vocabulary/cf-standard-name#">
<!ENTITY cu "http://purl.org/myocean/ontology/vocabulary/cf-unofficial-standard-name#">
<!ENTITY ct "http://purl.org/myocean/ontology/vocabulary/forecasting#">
<!ENTITY cp "http://purl.org/myocean/ontology/vocabulary/grid-projection#">
]>
<inventory lastModificationDate="2016-01-27T00:10:10+00:00" xmlns="http://purl.org/cls/atoll" updateFrequency="P1D">
<service urn="cls-toulouse-fr-armor-motu-rest-file"/>
<resource urn="dataset-armor-3d-v5-myocean">
<access urlPath="file:///data/atoll/armor/armor-3d-v3/"/>
<geospatialCoverage south="-82" north="90" west="0" east="359.75"/>
<depthCoverage min="0" max="5500" units="m"/>
<timePeriod start="2014-10-01T00:00:00+00:00" end="2016-01-26T23:59:59+00:00"/>
<theoricalTimePeriod start="2014-10-01T00:00:00+00:00" end="2016-01-26T23:59:59+00:00"/>
<variables>
<variable name="zvelocity" vocabularyName="http://mmisw.org/ont/cf/parameter/eastward_sea_water_velocity" units="m/s"/>
<variable name="height" vocabularyName="http://purl.org/myocean/ontology/vocabulary/cf-standard-name#height_above_geoid" units="m"/>
<variable name="mvelocity" vocabularyName="http://mmisw.org/ont/cf/parameter/northward_sea_water_velocity" units="m/s"/>
<variable name="salinity" vocabularyName="http://mmisw.org/ont/cf/parameter/sea_water_salinity" units="1e-3"/>
<variable name="temperature" vocabularyName="http://mmisw.org/ont/cf/parameter/sea_water_temperature" units="degC"/>
</variables>
</resource>
<files>
<file name="ARMOR3D_TSHUV_20141001.nc" weight="327424008" modelPrediction="http://www.myocean.eu.org/2009/resource/vocabulary/forecasting#" startCoverageDate="2014-10-01T00:00:00+00:00" endCoverageDate="2014-10-07T23:59:59+00:00" creationDate="2015-03-17T00:00:00+00:00" availabilitySIDate="2016-01-27T00:10:10+00:00" availabilityServiceDate="2016-01-27T00:10:10+00:00" theoreticalAvailabilityDate="2015-03-17T00:00:00+00:00"/>
<file name="ARMOR3D_TSHUV_20141008.nc" weight="327424008" modelPrediction="http://www.myocean.eu.org/2009/resource/vocabulary/forecasting#" startCoverageDate="2014-10-08T00:00:00+00:00" endCoverageDate="2014-10-14T23:59:59+00:00" creationDate="2015-03-17T00:00:00+00:00" availabilitySIDate="2016-01-27T00:10:10+00:00" availabilityServiceDate="2016-01-27T00:10:10+00:00" theoreticalAvailabilityDate="2015-03-17T00:00:00+00:00"/>
...
<file name="ARMOR3D_TSHUV_20160120.nc" weight="327424008" modelPrediction="http://www.myocean.eu.org/2009/resource/vocabulary/forecasting#" startCoverageDate="2016-01-20T00:00:00+00:00" endCoverageDate="2016-01-26T23:59:59+00:00" creationDate="2016-01-26T11:11:00+00:00" availabilitySIDate="2016-01-27T00:10:10+00:00" availabilityServiceDate="2016-01-27T00:10:10+00:00" theoreticalAvailabilityDate="2016-01-26T11:11:00+00:00"/>
</files>
</inventory>
In order to improve response time, Motu uses an in-memory cache which stores datasets metadata. This cache is indexed by config service.
You can tune the cache behaviour in order to manage both real time and archived datasets effectively.
At startup, Motu loads datasets metadata of each configService by turn. Once done, cache is refreshed either periodically in an automatic manner or either when asked by triggering a specific action.
The cache is kept in memory and all Motu requests are based on it. When a cache refresh is asked, a second cache loads new metadata and when fully loaded, Motu main cache is replaced. So until the full loading, old cache is used in Motu responses.
For config services which manages real time datasets, meaning datasets which are daily updated, you can set the following configuration in motuConfiguration.xml:
<?xml version="1.0"?>
<motuConfig ...
<configService ... refreshCacheAutomaticallyEnabled="true"
...
For config services which manages archived datasets, meaning dataset which not updated frequently for example only once a week, you can set the following configuration in motuConfiguration.xml:
<?xml version="1.0"?>
<motuConfig ...
<configService ... refreshCacheAutomaticallyEnabled="false"
...
In this case, when you want to refresh metadata cache of these datasets, you can use this dedicated action.
From a web browser access to the Motu web site with the URL:
/Motu?action=debug
You can see the different requests and their status.
You change the status order by entering 4 parameters in the URL:
/Motu?action=debug&order=DONE,ERROR,PENDING,INPROGRESS
Logbook files are written by Apache Tomcat server and Motu application.
Tomcat writes log files in folder tomcat-motu/logs.
You can customize this default configuration by editing tomcat-motu/conf/logging.properties
This file is the default file provided by Apache Tomcat.
There is a daily rotation so you can clean those files to fullfill the harddrive.
crontab -e
0 * * * * find /opt/cmems-cis/motu/tomcat-motu/logs/.log -type f -mmin +14400 -delete >/dev/null 2>&1
0 * * * * find /opt/cmems-cis/motu/tomcat-motu/logs/.txt -type f -mmin +14400 -delete >/dev/null 2>&1
Logbook files are written in the folder(s) configured in the log4j.xml configuration file.
All logs are generated daily except for motuQSLog (xml or csv) which are generated monthly.
You can clean those files to avoid to fullfill the harddrive.
crontab -e
0 * * * * find /opt/cmems-cis/motu/log/.log -type f -mmin +14400 -delete >/dev/null 2>&1
0 * * * * find /opt/cmems-cis/motu/log/.out -type f -mmin +14400 -delete >/dev/null 2>&1
0 * * * * find /opt/cmems-cis/motu/log/.xml -type f -mmin +144000 -delete >/dev/null 2>&1
0 * * * * find /opt/cmems-cis/motu/log/.csv -type f -mmin +144000 -delete >/dev/null 2>&1
Note that Motu is often tuned to write the motuQSLog in a dedicated folder. So you have to clean log files in this folder too. For example:
0 * * * * find /opt/cmems-cis/motu/data/public/transaction/.xml -type f -mmin +144000 -delete >/dev/null 2>&1
0 * * * * find /opt/cmems-cis/motu/data/public/transaction/.csv -type f -mmin +144000 -delete >/dev/null 2>&1
The error codes of Motu as the following format "XXXX-Y":
- XXXX code matching the action which is executed when the error is raised. This part is the "ActionCode". The action is in general a HTTP request and matches the following HTTP parameter http://$server/motu-web/Motu?action=.
- Y code which identifies the part of the program from which the error was raised. This part is the "ErrorType".
For example, the web browser can display:
011-1 : A system error happened. Please contact the administrator of the site.
Here, we have the error code in order to understand better what happens. But the end user has a generic message and no detail is given to him. These end user messages are described in the file "/motu-web/src/main/resources/MessagesError.properties". The file provided with the project is a default one and can be customized for specific purposes. Just put this file in the "config" folder, edit it and restart Motu to take it into account. So when a user has an error, it just have to tell you the error code and you can check the two numbers with the descriptions below.
The Action Code => A number matching the HTTP request with the action parameter.
001 => UNDETERMINED_ACTION
002 => PING_ACTION
003 => DEBUG_ACTION
004 => GET_REQUEST_STATUS_ACTION
005 => GET_SIZE_ACTION
006 => DESCRIBE_PRODUCT_ACTION
007 => TIME_COVERAGE_ACTION
008 => LOGOUT_ACTION
010 => DOWNLOAD_PRODUCT_ACTION
011 => LIST_CATALOG_ACTION
012 => PRODUCT_METADATA_ACTION
013 => PRODUCT_DOWNLOAD_HOME_ACTION
014 => LIST_SERVICES_ACTION
015 => DESCRIBE_COVERAGE_ACTION
016 => ABOUT_ACTION
018 => WELCOME_ACTION
019 => REFRESH_CACHE_ACTION
020 => HEALTHZ_ACTION
021 => CACHE_STATUS_ACTION
The Error Type Code => A number defining a specific error on the server.
0 => No error.
1 => There is a system error. Please contact the Administrator.
2 => There is an error with the parameters. There are inconsistent.
3 => The date provided into the parameters is invalid.
4 => The latitude provided into the parameters is invalid.
5 => The longitude provided into the parameters is invalid.
6 => The range defined by the provided dates is invalid.
7 => The memory capacity of the motu server is exceeded.
8 => The range defined by the provided latitude/longitude parameters is invalid.
9 => The range defined by the provided depth parameters is invalid.
10 => The functionality is not yet implemented.
11 => There is an error with the provided NetCDF variables.
12 => There is not variables into the variable parameter.
13 => NetCDF parameter error. Example: Invalid date range, invalid depth range, ...
14 => There is an error with the provided NetCDF variable. Have a look at the log file to have more information.
15 => The number of maximum request in the queue server pool is reached. it's necessary to wait that some requests are finished.
16 => The number of maximum request for the user is reached. It's necessary to wait that some requests are finished for the user.
18 => The priority of the request is invalid in the queue server manager. Have a look at the log file to have more information.
19 => The id of the request is not know by the server. Have a look at the log file to have more information.
20 => The size of the request is greater than the maximum data managed by the available queue. It's impossible to select a queue for this request. It's necessary to narrow the request.
21 => The application is shutting down. it's necessary to wait a while before the application is again available.
22 => There is a problem with the loading of the motu configuration file. Have a look at the log file to have more information.
23 => There is a problem with the loading of the catalog configuration file. Have a look at the log file to have more information.
24 => There is a problem with the loading of the error message configuration file. Have a look at the log file to have more information.
25 => There is a problem with the loading of the netcdf file. Have a look at the log file to have more information.
26 => There is a problem with the provided parameters. Have a look at the log file to have more information.
27 => There is a problem with the NetCDF generation engine. Have a look at the log file to have more information.
28 => The required action is unknown. Have a look at the log file to have more information.
29 => The product is unknown.
30 => The service is unknown.
31 => The request cut the ante meridian. In this case, it's not possible to request more than one depth. It's necessary to change the depth selection and to select in the "from" and the "to" the values that have the same index into the depth list.
32 => Due to a known bug in Thredds Data Server, a request cannot be satisfied wit netCDF4. User has to request a netCDF3 output file.
101 => WCS specific error code : A WCS mandatory parameter is missing
102 => WCS specific error code : A WCS parameter doesn't match the mandatory format
103 => WCS specific error code : The WCS version parameter is not compatible with the Motu WCS server
104 => WCS specific error code : A system error append.
105 => WCS specific error code : The coverage ident doesn't exist
106 => WCS specific error code : The list of coverage id is empty
107 => WCS specific error code : The provided parameter used to define a subset is invalid
108 => WCS specific error code : The provided axis label doesn't match any available label
You can connect to Motu by using a web browser or a client.
Motu offers an easy to use Python client. Very useful in machine to machine context, it enables to download data by running a python script.
Project and all its documentation is available at https://github.com/clstoulouse/motu-client-python.
Motu offers a Web Service interface which implements the OGC WCS standard described, in particular, by the two following documents on the OGC web site:
Available Web Services are:
Parameters can be added to each request and they are described with their cardinality [x,y].
- [0,1] is an optional parameter.
- [1] is a mandatory parameter.
- [0,n] is an optional parameter which can be set several times.
- [1,n] is a mandatory parameter which can be set several times.
The GetCapabilities request retrieves all available products defined on Motu server.
URL: http://localhost:8080/motu-web/wcs?service=WCS&version=2.0.1&request=GetCapabilities
Parameters:
- service [1]: Value is fixed to "WCS"
- version [1]: Value is fixed to "2.0.1"
- request [1]: Value is fixed to "GetCapabilties"
Return: A XML document as shown below:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns3:Capabilities version="2.0.1" xmlns:ns6="http://www.opengis.net/swe/2.0" xmlns:ns5="http://www.opengis.net/gmlcov/1.0" xmlns:ns2="http://www.w3.org/1999/xlink" xmlns:ns1="http://www.opengis.net/ows/2.0" xmlns:ns4="http://www.opengis.net/gml/3.2" xmlns:ns3="http://www.opengis.net/wcs/2.0">
<ns1:ServiceIdentification>
<ns1:Title>Motu</ns1:Title>
<ns1:Abstract>Motu WCS service</ns1:Abstract>
<ns1:ServiceType>OGC WCS</ns1:ServiceType>
<ns1:ServiceTypeVersion>2.0.1</ns1:ServiceTypeVersion>
<ns1:Profile>http://www.opengis.net/spec/WCS/2.0/conf/core</ns1:Profile>
<ns1:Profile>http://www.opengis.net/spec/WCS_protocol-binding_get-kvp/1.0/conf/get-kvp</ns1:Profile>
</ns1:ServiceIdentification>
<ns1:OperationsMetadata>
<ns1:Operation name="GetCapabilities">
<ns1:DCP>
<ns1:HTTP>
<ns1:Get ns2:href="http://localhost:8080/motu-web/wcs"/>
</ns1:HTTP>
</ns1:DCP>
</ns1:Operation>
<ns1:Operation name="DescribeCoverage">
<ns1:DCP>
<ns1:HTTP>
<ns1:Get ns2:href="http://localhost:8080/motu-web/wcs"/>
</ns1:HTTP>
</ns1:DCP>
</ns1:Operation>
<ns1:Operation name="GetCoverage">
<ns1:DCP>
<ns1:HTTP>
<ns1:Get ns2:href="http://localhost:8080/motu-web/wcs"/>
</ns1:HTTP>
</ns1:DCP>
</ns1:Operation>
</ns1:OperationsMetadata>
<ns3:ServiceMetadata>
<ns3:formatSupported>application/netcdf</ns3:formatSupported>
</ns3:ServiceMetadata>
<ns3:Contents>
<ns3:CoverageSummary>
<ns3:CoverageId>HR_MOD_NCSS-TDS@HR_MOD</ns3:CoverageId>
<ns3:CoverageSubtype>ns3:GridCoverage</ns3:CoverageSubtype>
</ns3:CoverageSummary>
...
</ns3:Contents>
</ns3:Capabilities>
The DescribeCoverage request retrieves the parameters description and the list of available variables. For the parameters description, low and high values are provided.
Parameters:
- service [1]: Value is fixed to "WCS"
- version [1]: Value is fixed to "2.0.1"
- request [1]: Value is fixed to "DescribeCoverage"
- coverageId [1]: list of identifiers of the required coverages. Each coverage identifiers are separated by a comma (,). CoverageId are returned by the GetCapabilities service.
Return: A XML document as shown below:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns4:CoverageDescriptions xmlns:ns6="http://www.opengis.net/ows/2.0"
xmlns:ns5="http://www.opengis.net/swe/2.0" xmlns:ns2="http://www.w3.org/1999/xlink"
xmlns:ns1="http://www.opengis.net/gml/3.2" xmlns:ns4="http://www.opengis.net/wcs/2.0"
xmlns:ns3="http://www.opengis.net/gmlcov/1.0">
<ns4:CoverageDescription ns1:id="$covergaeId">
<ns1:boundedBy>
<ns1:Envelope
uomLabels="latitude longitude depth date in seconds since 1970, 1 jan"
axisLabels="Lat Lon Height Time">
<ns1:lowerCorner>-80.0 -180.0 0.0 1.3565232E9</ns1:lowerCorner>
<ns1:upperCorner>90.0 180.0 5728.0 1.4657328E9</ns1:upperCorner>
</ns1:Envelope>
</ns1:boundedBy>
<ns4:CoverageId>$covergaeId</ns4:CoverageId>
<ns1:domainSet>
<ns1:Grid dimension="4"
uomLabels="latitude longitude depth date in seconds since 1970, 1 jan"
axisLabels="Lat Lon Height Time" ns1:id="Grid000">
<ns1:limits>
<ns1:GridEnvelope>
<ns1:low>-80 -180 0 1356523200</ns1:low>
<ns1:high>90 180 5728 1465732800</ns1:high>
</ns1:GridEnvelope>
</ns1:limits>
</ns1:Grid>
</ns1:domainSet>
<ns3:rangeType>
<ns5:DataRecord>
<ns5:field name="uice">
<ns5:Quantity>
<ns5:uom code="m s-1" />
</ns5:Quantity>
</ns5:field>
<ns5:field name="salinity">
<ns5:Quantity>
<ns5:uom code="1e-3" />
</ns5:Quantity>
</ns5:field>
<ns5:field name="vice">
<ns5:Quantity>
<ns5:uom code="m s-1" />
</ns5:Quantity>
</ns5:field>
<ns5:field name="hice">
<ns5:Quantity>
<ns5:uom code="m" />
</ns5:Quantity>
</ns5:field>
<ns5:field name="u">
<ns5:Quantity>
<ns5:uom code="m s-1" />
</ns5:Quantity>
</ns5:field>
<ns5:field name="v">
<ns5:Quantity>
<ns5:uom code="m s-1" />
</ns5:Quantity>
</ns5:field>
<ns5:field name="temperature">
<ns5:Quantity>
<ns5:uom code="K" />
</ns5:Quantity>
</ns5:field>
<ns5:field name="ssh">
<ns5:Quantity>
<ns5:uom code="m" />
</ns5:Quantity>
</ns5:field>
<ns5:field name="fice">
<ns5:Quantity>
<ns5:uom code="1" />
</ns5:Quantity>
</ns5:field>
</ns5:DataRecord>
</ns3:rangeType>
</ns4:CoverageDescription>
</ns4:CoverageDescriptions>
The GetCoverage request is used to run an extraction on a dataset using some filtering parameters and a list of required variables.
URL:
- DGF: http://localhost:8080/motu-web/wcs?service=WCS&version=2.0.1&request=GetCoverage&coverageId=$coverageId&subset=Time(1412157600,1412244000)
- Subetter: http://localhost:8080/motu-web/wcs?service=WCS&version=2.0.1&request=GetCoverage&coverageId=$coverageId&subset=Time(1412157600,1412244000)&subset=Lat(50,70)&subset=Lon(0,10)&subset=Height(0,5728)&rangeSubset=temperature,salinity
Parameters:
- service [1]: Value is fixed to "WCS"
- version [1]: Value is fixed to "2.0.1"
- request [1]: Value is fixed to "GetCoverage"
- coverageId [1]: the identifier of the required coverage. CoverageId are returned by the GetCapabilities service.
- subset [1,n]: the list of filtering parameters.
- To define one filtering parameter, the following format have to be respected:
For the Time parameter:
Unit is epoch since 1st January 1970, in UTC. E.g. Thu Dec 01 2016 00:00:00 is set to 1480550400000.SUBSET=Time(lowTimeValue,highTimeValue)- To define multiple filtering parameters, the following format have to be respected:
For the Latitude and the Longitue:
In order to know which subset filters can be applied, you have to run a DescribeCoverage request.SUBSET=Lat(lowLatValue,highLatValue)&SUBSET=Lon(lowLonValue,highLonValue) - To define one filtering parameter, the following format have to be respected:
- rangesubset: the list of required variables for the coverage. Each variable have to be separated by a comma (,)
Return: A Netcdf file. When you request for one point, a specific algorithm is used, see Downloading 1 point.
MOTU REST API defines a set of services accessible from an HTTP URLs.
All URLs have always the same pattern: http://motuServer/${context}/Motu?action=$actionName
Other parameters can be added and they are described with their cardinality [x,y].
- [0,1] is an optional parameter.
- [1] is a mandatory parameter.
- [0,n] is an optional parameter which can be set several times.
- [1,n] is a mandatory parameter which can be set several times.
$actionName is an action, they are all listed below:
- XML API
- HTML Web pages
- Plain Text
- JSON
Display version of the archives installed on Motu server
URL: http://localhost:8080/motu-web/Motu?action=about
Parameters: No parameter.
Return: An HTML page. Motu-static-files (Graphic chart) is refreshed thanks to Ajax because its version file can be installed on a distinct server.
Example:
Motu-products: 3.0
Motu-distribution: 2.6.00-SNAPSHOT
Motu-configuration: 2.6.00-SNAPSHOT-20160623173246403
Motu-static-files (Graphic chart): 3.0.00-RC1-20160914162955422
Display all requests status managed by Motu server in the last cleanRequestInterval minutes.
Tables are sorted by time ascending.
4 status are defined:
- DONE: Request has been processed successfully. Result file can be downloaded.
- ERROR: Request has not been processed successfully. No result file is available.
- PENDING: Request has been received by the server. Server computes its size and runs some checks, it can take a while.
- INPROGRESS: Request has been delegated to the queue server which is currently processing it.
URL: http://localhost:8080/motu-web/Motu?action=debug
Parameters:
- order [0,1]: Change the order of items INPROGRESS,PENDING,ERROR,DONE. All items shall be set.
example: http://localhost:8080/motu-web/Motu?action=Debug&order=DONE,ERROR,PENDING,INPROGRESS
Without this parameter, default order is: INPROGRESS,PENDING,ERROR,DONE
Return: An HTML page
Get coverage data in relationship with a dataset.
URL: http://localhost:8080/motu-web/Motu?action=describecoverage&service=HR_MOD-TDS&datasetID=HR_MOD
Parameters:
- service [1]: The service name
- datesetID [1]: The dataset ID
Return: A XML document
<dataset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="describeDataset.xsd" name="HR_MOD" id="HR_MOD">
<boundingBox>
<lon min="-180.0" max="179.91668701171875" units="degrees_east"/>
<lat min="-80.0" max="-80.0" units="degrees_north"/>
</boundingBox>
<dimension name="time" start="2012-12-26T12:00:00.000+00:00" end="2016-06-12T12:00:00.000+00:00" units="ISO8601"/>
<dimension name="z" start="" end="" units="m"/>
<variables>
<variable id="northward_sea_water_velocity" name="v" description="Northward velocity" standardName="northward_sea_water_velocity" units="m s-1">
<dimensions></dimensions>
</variable>
...
</variables>
</dataset>
Display the product meaning dataset description. Result contains notably: the datasetid, the time coverage, the geospatial coverage, the variable(s) (with the standard_name and unit), eventually the vertical coverage.
There is 2 ways to call describe product, both returning a same response.
URL: http://localhost:8080/motu-web/Motu?action=describeproduct&service=HR_MOD-TDS&product=HR_MOD
Parameters:
- service [1]: The service name
- product [1]: The product id
Parameters:
- xmlfile [1]: The Thredds dataset, example: http://$tdsServer/thredds/m_HR_MOD.xml
- data [1]: The Thredds data, example http://$tdsServer/thredds/dodsC/path_HR_MOD
Return: An XML document
<productMetadataInfo code="OK" msg="OK" lastUpdate="Not Available" title="HR_MOD" id="HR_MOD">
<timeCoverage code="OK" msg="OK"/>
<availableTimes code="OK" msg="OK">
1993-01-15T12:00:00Z/2001-01-01T00:00:00Z/P2D,2001-01-01T00:00:00Z/2012-03-01T00:00:00Z/PT6H
</availableTimes>
<availableDepths code="OK" msg="OK">
0.49402;1.54138;2.64567;...
</availableDepths>
<geospatialCoverage code="OK" msg="OK"/>
<variablesVocabulary code="OK" msg="OK"/>
<variables code="OK" msg="OK">
<variable description="Northward velocity" units="m s-1" longName="Northward velocity" standardName="northward_sea_water_velocity" name="v" code="OK" msg="OK"/>
<variable description="Eastward velocity" units="m s-1" longName="Eastward velocity" standardName="eastward_sea_water_velocity" name="u" code="OK" msg="OK"/>
...
</variables>
<dataGeospatialCoverage code="OK" msg="OK">
<axis code="OK" msg="OK" description="Time (hours since 1950-01-01)" units="hours since 1950-01-01 00:00:00" name="time_counter" upper="582468" lower="552132" axisType="Time"/>
<axis code="OK" msg="OK" description="Longitude" units="degrees_east" name="longitude" upper="179.91668701171875" lower="-180" axisType="Lon"/>
<axis code="OK" msg="OK" description="Latitude" units="degrees_north" name="latitude" upper="90" lower="-80" axisType="Lat"/>
<axis code="OK" msg="OK" description="Depth" units="m" name="depth" upper="5727.9169921875" lower="0.4940249919891357421875" axisType="Height"/>
</dataGeospatialCoverage>
</productMetadataInfo>
In the XML result file the tag "availableTimes" provides the list of date where data are available for the requested product.
The format of the date follows the convention ISO_8601 used to represent the dates and times. (https://en.wikipedia.org/wiki/ISO_8601)
Foreach available time period, the period definition format is "StartDatePeriod/EndDatePeriod/DurationBetweenEachAvailableData".
The "availableTimes" contains a list of time period separated by a ",".
- StartDate : this the first date of the period where data are available.
- EndDate : this the last date of the period where data are available.
- DurationBetweenEachAvailableData : This the period duration between each available data in the interval defined by the the "StartDate" and "EndDate" date.
For DGF datasets, the list of available times is built from the start and end date of each file of the dataset, ignoring the other time values if any. As a consequence, there might be more available times than those listed in this attribute for DGF datasets with there are more than 2 time values per file.
The format of the StartDate and EndDate is YYYY-MM-DDThh:mm:ssZ where:
- YYYY : is the year defined on 4 digits
- MM : is the number of the month defined on 2 digits
- DD: is the number of the day in the month on 2 digits
- hh: is the hour of the day on 2 digits
- mm: is the minutes of the hour on 2 digits
- ss: is the seconds of the minutes on 2 digits
Examples:
- 1993-01-15T12:00:00Z
- 2016-07-25T06:35:45Z
- 2017-08-31T15:05:08Z
The formation of the duration is PnbyersYnbmonthsMnbdaysDTnbhoursHnbminutesMnbsecondsS.nbmillisec where:
- nbyears : is the number of years. The ISO_8601 is ambiguous on the number of days in the year. For the Motu project, the number of days is fixed to 365 as in the most of projects.
- nbmonths : is the number of month. The ISO_8601 is ambiguous on the number of days in the month. For the Motu project, the number of days is fixed to 30 as in the most of projects.
- nbdays : is the number of day. One day is 24 hours.
- nbhours : is the number of hours. One hour is 60 minutes.
- nbseconds: is the number of seconds. One seconds is 1000 milliseconds.
- nbmillisec : is the number of milliseconds.
By convention, P1M defines a duration of 1 month and PT1M defines a duration of 1 minutes.
Examples:
- each minute => PT1M
- each hour => PT1H
- each 12 hours => PT12H
- each day => P1D
- each 15 days => P15D
- each month => P1M
Note about "lastUpdate" attribute of the "productMetadataInfo" field: it has the same value than "Last update" field of the list catalog page.
Request used to download a product
URL: http://localhost:8080/motu-web/Motu?action=productdownload
example:
http://localhost:8080/motu-web/Motu?action=productdownload&service=HR_MOD-TDS&product=HR_MOD&x_lo=-2&x_hi=2&y_lo=-2&y_hi=2&output=netcdf&t_lo=2016-06-12+12%3A00%3A00&t_hi=2016-06-12+12%3A00%3A00&z_lo=0.49&z_hi=5727.92
Parameters:
-
service [1]: The service name
-
product [1]: The product id
-
variable [0,n]: physical variables to be extracted from the product. When no variable is set, all the variables of the dataset are extracted.
-
y_lo [0,1]: low latitude of a geographic extraction. Default value is -90.
-
y_hi [0,1]: high latitude of a geographic extraction. Default value is 90.
-
x_lo [0,1]: low longitude of a geographic extraction. Default value is -180.
-
x_hi [0,1]: high longitude of a geographic extraction. Default value is 180.
-
z_lo [0,1]: low vertical depth . Default value is the min available depth. If the lo value is greater than the hi value, the 2 values are switched. If the depth range is out of the available range, Motu computes the best range into the available range. Value of this parameter is a double or "Surface" string which has a value of 0.0.
-
z_hi [0,1]: high vertical depth. Default value is the max available depth. If the hi value is lower than the lo value, the 2 values are switched. If the depth range is out of the available range, Motu computes the best range into the available range. Value of this parameter is a double or "Surface" string which has a value of 0.0.
-
t_lo [0,1]: Start date of a temporal extraction. If not set, the default value is the first date/time available for the dataset. Format is "yyy-MM-dd" or "yyyy-MM-dd HH:mm:ss" or "yyyy-MM-ddTHH:mm:ss" and depends on the requested dataset.
-
t_hi [0,1]: End date of a temporal extraction. If not set, the default value is the last date/time available for the dataset. Format is "yyy-MM-dd" or "yyyy-MM-dd HH:mm:ss" or "yyyy-MM-ddTHH:mm:ss" and depends on the requested dataset.
-
output [0,1]: netcdf. Due to a TDS issue, only netcdf is available. netcdf4 will be available as soon as TDS will have resolved its issue.
-
mode [0,1]: Specify the desired result mode. Enumeration value from [url, console, status] represented as a string. If no mode, "url" value is the default mode.
- mode=url: URL of the delivery file is directly returned in the HTTP response as an HTML web page. Then Javascript read this URL to download file. The request is processed in a synchronous mode.
- mode=console: the response is a 302 HTTP redirection to the delivery file to be returned as a binary stream. The request is processed in a synchronous mode.
- mode=status: request is submitted and the status of the request processing is immediately returned as an XML. The request is processed in an asynchronous mode.
Web Portal submits the request to the Dissemination Unit Subsetter and gets an immediate response of the Subsetter. This response contains the identifier and the status of the order (pending, in progress, done, error). So long as the order is not completed (done or error), Web Portal requests the status of the order at regular and fair intervals (> 5 seconds) and gets an immediate response. When the status is “done”, Web Portal retrieves the url of the file to download, from the status response. Then Web Portal redirects response to this url. The Web Browser opens a binary stream of the file to download and shows a dialog box to allow the user saving it as a local file.
Return: Several ways depending of the selected http parameter mode. When you request for one point, a specific algorithm is used, see Downloading 1 point.
Get a request status to get more details about a download state.
URL: http://localhost:8080/motu-web/Motu?action=getreqstatus&requestid=123456789
Parameters:
- requestid [1]: A request id.
Return: An XML document or an HTML page if requestId does not exists.
Validated by the schema /motu-api-message/src/main/schema/XmlMessageModel.xsd#StatusModeResponse
Example:
<statusModeResponse code="004-0" msg="" scriptVersion="" userHost="" userId="" dateSubmit="2016-09-19T16:56:22.184Z" localUri="/$pathTo/HR_MOD_1474304182183.nc" remoteUri="http://localhost:8080/motu/deliveries/HR_MOD_1474304182183.nc" size="1152.0"dateProc="2016-09-19T16:56:22.566Z" requestId="1474304182183" status="1"/>
Size is in MegaBytes or at NaN while still not estimated.
Get the size of a download request. Result contains the size of the potential result file, with a unit, and the maximum allowed size for this service.
URL: http://localhost:8080/motu-web/Motu?action=getsize
Parameters:
Parameters below are exactly the same as for Download product
- service [1]: The service name
- product [1]: The product id
- variable [0,n]: physical variables to be extracted from the product. When no variable is set, all the variables of the dataset are extracted.
- y_lo [0,1]: low latitude of a geographic extraction. Default value is -90.
- y_hi [0,1]: high latitude of a geographic extraction. Default value is 90.
- x_lo [0,1]: low longitude of a geographic extraction. Default value is -180.
- x_hi [0,1]: high longitude of a geographic extraction. Default value is 180.
- z_lo [0,1]: low vertical depth . Default value is the min available depth. If the lo value is greater than the hi value, the 2 values are switch. If the depth range is out of the available range, Motu compute the best range into the available range.
- z_hi [0,1]: high vertical depth. Default value is the max available depth. If the hi value is less than the lo value, the 2 values are switch. If the depth range is out of the available range, Motu compute the best range into the available range.
- t_lo [0,1]: Start date of a temporal extraction. If not set, the default value is the first date/time available for the dataset. Format is yyy-mm-dd or yyyy-dd h:m:s or yyyy-ddTh:m:s.
- t_hi [0,1]: End date of a temporal extraction. If not set, the default value is the last date/time available for the dataset. Format is yyy-mm-dd or yyyy-dd h:m:s or yyyy-ddTh:m:s.
Return: An XML document.
The unit is "KB" means Kilobyte.
Validated by the schema /motu-api-message/src/main/schema/XmlMessageModel.xsd#RequestSize
Example:
<requestSize code="005-0" msg="OK" unit="kb" size="1.5104933E8" maxAllowedSize="9.961472E8"/>
Display information about a catalog (last update timestamp) and display link to access to download page and dataset metadata.
Note about field "Last update" : On MOTU start-up, this date is set to the most recent date of the dataset if it is before of the current day, or to the "last update date" returned by the TDS server also if it is before current day, or else to "Not available".
On MOTU cache update, if the most recent available date of the dataset changes, the "last update date" is set with the current date.
URL: http://localhost:8080/motu-web/Motu?action=listcatalog&service=HR_MOD-TDS
Parameters:
- service [1]: The service name
Return: An HTML page
Display the service web page URL: http://localhost:8080/motu-web/Motu?action=listcatalog&service=HR_MOD-TDS
Parameters:
- catalogtype [0,1]: The catalog type used to filter by type.
Return: An HTML page
Used to be sure that server is up. You can also use the supervision URL.
URL: http://localhost:8080/motu-web/Motu?action=ping
Parameters: No parameter.
Return: An plain text
OK - response action=ping
Force the refresh of the cache of config service metadata instead of waiting the automatic refresh.
This action is secured and is only triggered when a valid token is given.
Moreover a list of config services needed to be refreshed is shared with the automatic update process.
This add robustness because a job refreshes only cache of the config services which are is the list. So when this action is called several times, if a config service in already in this waiting list, it is not added a second time.
A soon as a cache for a config service is refreshed, config service is removed from this waiting list.
URL: http://localhost:8080/motu-web/Motu?action=refreshcache&token=tokenValid&configServiceNames=all
Parameters:
- token [1] : Used to secure this action. The token configured in the motuConfiguration.xml file which allowed the execution of the refresh. See this section for the token configured
-
configServiceNames [1] : [all,onlyauto,$configServiceNames] 3 options to tune how the cache will be resfreshed.
- all : Refresh immediately all the config service.
- onlyauto : Refresh immediately only the config services which enable the automatic refresh.
- $configServiceNames : Refresh immediately all the config services listed. Value of this parameter is a list of all config service name is separated by a comma character, e.g. configServiceNames=AAA,BBB,CCC
Return: A plain text which specify if the refresh is launched or if an error occurred, e.g. "OK: config service AAA cache refresh in progress" or "ERROR: Unknwon config service UnknownConfigService"
OK cache refresh in progress
Gives healthz information about Motu server health.
URL: http://localhost:8080/motu-web/Motu?action=healthz
Parameters: No parameter
Return: An http status and a short message:
- http status 202 (accepted) when started and cache still not refreshed, with the message "Server started and refresh in progress (remaining X / Y)." Where X is the number of Catalog to put in the cache, over the total number Y.
This message is also displayed when the Web context gets destroyed and cache gets build again, or also when the configuration file is modified and reloaded. - http status 200 when running and ready, with the message "Server is ready."
HTML page which gives access to several web pages, in particular the Motu listservices web page.
URL:
Parameters: No parameter
Return: An HTML web page
Display an HTML page in order to set the download parameters.
URL: http://localhost:8080/motu-web/Motu?action=productdownloadhome&service=HR_OBS-TDS&product=HR_OBS
Parameters:
- service [1]: The service name
- product [1]: The product id
Return: An HTML page
Display an HTML page with the geographical and temporal coverage, the last dataset update and the variables metadata.
Note about field "Date" of "Last dataset update" section: this date has the same value than the "Last update" field.
URL: http://localhost:8080/motu-web/Motu?action=listproductmetadata&service=HR_OBS-TDS&product=HR_OBS
Parameters:
- service [1]: The service name
- product [1]: The product id
Return: An HTML page
Display an HTML page with the geographical and temporal coverage, the last dataset update and the variables metadata.
URL: http://localhost:8080/motu-web/Motu?action=gettimecov&service=HR_MOD-TDS&product=HR_MOD
Parameters:
- service [1]: The service name
- product [1]: The product id
Return: A XML document
<timeCoverage code="007-0" msg="OK" end="2016-09-17T00:00:00.000Z" start="2007-05-13T00:00:00.000Z"/>Gives information about Motu server.
For more details, see [https://jolokia.org/reference/html/agents.html].
URL: http://localhost:8080/motu-web/supervision
Parameters: No parameter
Return: A JSON document
{"timestamp":1474638852,"status":200,"request":{"type":"version"},"value":{"protocol":"7.2","config":{"agentId":"10.1.20.198-18043-2df3a4-servlet","agentType":"servlet"},"agent":"1.3.3","info":{"product":"tomcat","vendor":"Apache","version":"7.0.69"}}}Gives the status of the dataset cache of Motu.
On start-up, Motu reads its configuration file, and gets a list of "configService" nodes referencing a dataset catalog with an URL, and starts caching them.
URL: http://localhost:8080/motu-web/Motu?action=cachestatus
Parameters: No parameter
Return: A JSON document
{"cachestatus":
{"state":
{"nbTotal":8,"nbSuccess":6,"nbFailure":2,"lastUpdate":"2019-11-19T10:56:37.357Z","lastUpdateDuration":"PT1M17.561S"},
"configServices": [
{"Sea_Surface_Temperature_Global-TDS":
{"state":
{"status":"FAILURE","lastUpdate":"2019-11-19T10:57:56.398Z","lastUpdateDuration":""},
"conf":
{"refreshCacheAutomaticallyEnabled":true,"type":"tds","ncss":"enabled"}
}
},
{"HR_MOD_NCSS-TDS":
{"state":
{"status":"SUCCESS","lastUpdate":"2019-11-19T10:57:01.436Z","lastUpdateDuration":"PT1.688S"},
"conf":
{"refreshCacheAutomaticallyEnabled":true,"type":"tds","ncss":"enabled"}
}
},
.....]
},
"version":
{"motu-products":"Unknow version","motu-distribution":"Unknow version","motu-configuration":"3.11.04-20190716151835979"}
}The nbTotal is the total number of ConfigServices.
The nbSuccess and the nbFailure are the number of currently loaded ConfigServices in success/failure.
Note that lastUpdate and lastUpdateDuration fields can be empty if the system hasn't still refreshed the cache or if the access failed.
HTML page which gives access to several web pages, in particular the Motu listservices web page.
URL:
Parameters: No parameter
Return: An HTML web page
Motu comes in a Docker release based on CentOS.
The Motu specific elements are located under /opt/motu.
The required libraries are installed on the Docker image.
The Tomcat is located under /opt/motu/tomcat-motu.
The CDO scripts (merge.sh and cdo.sh) can be found in the folder /opt/motu/products/cdo-group.
To configure the Motu Docker image, some of the folders have to be mounted from the execution environment:
- the motu configuration folder to mount to /opt/motu/config
- log4j.xml
- motuConfiguration.xml
- motu.properties
- standarNames.xml
- version-configuration.txt
- velocityTemplates/motu.vm
- security (folder with the configuration elements for cas authentification)
- the produced data files folder outputed on /opt/motu/data/download/public
Other folders for logs and Apache configuration are optionnals:
- the log folder to mount to /opt/motu/log
The produced logs (errors.log, logbook.log, wrnings.log and motuQSlog.xml) will be available in this folder. - the apache tomcat log folder to mount to /opt/motu/tomcat-motu/logs
In this folder the host-manager, catalina, and access logs will be created. - the tomcat configuration folder to mount to /opt/motu/tomcat-product/conf
If this volume is not mounted, the default Apache configuration is used. In that case, the downloads have to be handled by another element of the installation, to which the ddownloadHttpUrl property of the motuConfiguration file will refer.- logging.properties
- server.xml For example to set the Apache HTTPd "Context" directive to handle the result files downloading with the MOTU server.
- web.xml
- other files that could be used for specific needs (configuring catalina, jaspic or the tomcat users)
Configuration files are similar to standard Motu configuration files with some exceptions to take into account:
- Motu is installed under /opt/motu (and not /opt/cmems-cis/motu)
- The download directory is under /opt/motu/data/download/public
The following command:
docker run -d -v /data/shared/motu/result:/opt/motu/data/download/public -v /home/motu/motu/config_qt/motu_config:/opt/motu/config -v /home/motu/motu/config_qt/apache_config:/opt/motu/tomcat-motu/conf -v /home/motu/motu/config_qt/log_motu:/opt/motu/log -v /home/motu/motu/config_qt/log_apache:/opt/motu/tomcat-motu/logs -p 8080:8080 -p 8443:8443 -p 8009:8009 -p 8005:8005 registry-ext.cls.fr:443/motu/motu/motu-distribution
Will run the Motu server. Logs (from logbook, warning and error MOTU files) can be accessed using:
tail /var/log/syslog
If a reverse proxy for downloading the results is to be added in the platform where MOTU server docker image is deployed, use:
docker run -d -v /data/shared/motu/result:/var/www -v /home/motu/motu/config_qt/motu_config:/opt/motu/config -p $NGINX_PORT:8070 -e NGINX_PORT=$NGINX_PORT -e MOTU_DOWNLOAD_PATH=/motu-web -e MOTU_URL=http://$(hostname):18080 -d registry-ext.cls.fr:443/motu/motu/motu-nginx