EDA Data Service

Public-facing service providing all EDA data to EDA client apps

docker-compose

preliminary requirements

To bring up the eda project via docker-compose, you’ll need a few things.

a functioning docker setup and docker-compose (https://www.docker.com/products/docker-desktop)
a functioning traefik setup. clone the https://github.com/VEuPathDB/docker-traefik project, and run docker-compose up -d in the clone, or follow the instructions in that repo. This will run traefik locally, which handles the container routing
a functioning sshuttle setup. If you need to access databases on internal networks, you’ll need to properly setup sshuttle so that your containers can reach them. Setting this up is out of scope for this document, to not expose unnecessary internal details publicly.

bring up the eda project

clone the stack-eda-services repo, and add any required environment variables to your .env file. There is an example env file in that repo.
docker-compose up in your clone of the repo.

When everything comes up successfully, you’ll be able to reach this service at:

https://edadata-dev.local.apidb.org:8443/

notes

all services use https, but the certs are not valid. You will need to accept them in a browser, or use the -k flag with curl

TODO: replace text below with EdaDataService-specific documentation

Example Containerized JaxRS Service

Running the Example

Locally

Prerequisites

Maven
NodeJS/NPM

Setup & Run

Environment/dependency setup for local development can be done via the command make install-dev-env. This will install the dependencies required to build the project.

After the environment has been set up, the project can be built, tested, and packaged using make jar.

Starting the server can be done by simply running java -jar build/lib/service.jar.

In Docker

Prerequisites

Docker

Setup & Run

The project container can be built using make docker. The first build of the container will take several minutes.

Running the built container can be done with docker run --rm demo-service.

Development

To use this project as a template for the development of a new service, use the "Use this template" button above to create a new project based off of this repo.

Quick Start

Clone your new project and cd into the cloned directory
Run make install-dev-env cleanup-example
Edit service.properties with your service’s details.
Run make fix-path to move the service code to the correct location
Edit api.raml with your API design
Create any necessary types in docs/schema
Run make gen-jaxrs to generate skeleton code
Implement the generated service interfaces.
Register your services in the Resources class.
Run make test to run any unit tests and verify that things compile.

Detailed Overview

Environment Prerequisites

NPM: Required to run raml2html
Maven: Required to build raml-to-jaxrs & FgpUtil

Configuration

Configuration for the build is primarily done through the service.properties file. Here you can configure the project’s name, version, container name, Java package structure, etc…

If additional dependencies are required, they will be added to the dependencies.gradle.kts file. For information about adding gradle dependency declarations see the Gradle docs.

API definition

The service api is defined in 2 places initially:

The api.raml file which contains the server endpoints.
The schema folder which contains type schema definitions.
These files must be valid Raml 1.0 Libraries.^[1]^[2]

`api.raml`

Including Type Definitions

In the api.raml file, there is a statement near the top of the file that declares the Raml file "uses" schema/library.raml. This is a generated file based on the contents of the schema library under schema. The library.raml file should not be edited directly.

api.raml: uses declaration

uses:
  err: .tools/raml/errors.raml
  lib: schema/library.raml

The uses keyword maps an import alias to the imported library. This import alias is used to access the types defined in that library. In the case of the above example, library types would be available using lib.{MyTypeName}

api.raml: Library type usage.

        body:
          application/json:
            type: lib.HelloResponse

Type Schema

Each raml library file under schema should define a root types object defining the types used by the API. The name of the types defined under the types object will be the name of the generated Java classes based on those types.

Example 1. Type definition

Schema

#%RAML 1.0 Library
types:
  MyType:
    properties:
      foo: string

Resulting Java Interface

package org.veupathdb.service.demo.generated.model;

import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;

@JsonDeserialize(
    as = HealthResponseImpl.class
)
public interface MyType {
  @JsonProperty("foo")
  String getFoo();

  @JsonProperty("foo")
  void setFoo(String foo);
}

Resulting Java Class

package org.veupathdb.service.demo.generated.model;

import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonPropertyOrder;

@JsonInclude(JsonInclude.Include.NON_NULL)
@JsonPropertyOrder({
    "foo",
})
public class HealthResponseImpl implements HealthResponse {
  @JsonProperty("foo")
  private String foo;

  @JsonProperty("foo")
  public String getFoo() {
    return this.foo;
  }

  @JsonProperty("foo")
  public void setFoo(String foo) {
    this.foo = foo;
  }
}

Generating A Service

Once your API spec is complete, you can begin development of Java code by running make gen-jaxrs. This will create a skeleton of the API in the generated source package located under the root package defined using the app.package values in service.properties.

The generated interfaces and types have the basic necessary annotations for use by both Jackson and Jersey.

Once you have implemented the interfaces defined under {source-package}.generated.resources they must be registered in the {source-package}.Resources class.

Run Your Service

Running your service locally can be done by following the same steps as defined above in the Running the Example section:

Run make build-jar
Run java -jar build/lib/service.jar

Running in Docker can be done by:

Run make build-docker
Run docker run <your-image-name>

Other Information

Authentication

The base service contains an authentication layer that will be enabled on any service class or method annotated with @Authenticated. This authentication will validate a WDK user session against the account database and append user profile information to the incoming request object.

Make Targets

compile

Generates code & docs if the API def has changed.
Compiles Java code if anything has changed.

test

Generates code & docs if the API def has changed.
Compiles Java code if necessary.
Runs unit tests.

jar

Generates code & docs if the API def has changed.
Compiles Java code if necessary.
Runs unit tests if necessary.
Packages a self-contained runnable jar.

docker

Runs docker build for the project.

install-dev-env

Checks for system prerequisites.
Downloads and builds raml-to-jaxrs.
Installs the Oracle JDBC components into the vendor directory.
Downloads, builds, and installs the FgpUtil project into the vendor directory.
Installs the required NPM packages

gen-jaxrs

Generates Java code from the Raml spec.

gen-docs

Generates API docs from the Raml spec.

fix-path

Migrates the source code from the demo package to the correct package as defined in service.properties

Build Process

Presently the build process is operated through a makefile which calls and sets up the necessary prerequisites. This is a temporary solution to be used until

FgpUtil is converted to a Gradle friendly form (via Java9 modularization and package publishing)
The build utils are converted into a gradle library
The team as a whole is familiar with Gradle and Docker

Generating Jax-RS Code

The make gen-jaxrs command is backed by the Mulesoft Raml-for-JaxRs library.

Hiccups and limitations

The RAML to JaxRS conversion library has the following known issues that are likely to impact use of the tool:

Enums

Enum generation creates types that do not allow access to the raw backing text, which may be desired when constructing complex responses.
Enum generation may cause the generator to fail with cryptic errors involving bad imports for the Java builtin type String

One alternative/workaround for this is defining the enum type in Java and typing the RAML as string with the possible values defined as examples.

Inheritance

Extending types can work out for trees with at most 1 parent depth, however going beyond that may cause things to generate in a way that has compile errors.

additionalProperties aka //:

Using a catchall block for maps with anything other than a simple value type will cause the java type to be Map<String, Object>. Additionally even a simple value type may have this result.

Hacks and Oddities

FgpUtil

Due to the fact that FgpUtil is not a Gradle project, in a Maven repo, or safe to use with standard imports (due to likelyhood of dependency conflicts), at present, this dependency is built separately from the rest of the project by one of the make targets and is included as a local, runtime dependency.

Due to this method of including FgpUtil, additional dependencies have been added to support it in addition to bridging compatibility issues.

The project root

The root directory of this project contains multiple standard files used by Gradle, Docker, Make, and GitHub, in addition to custom files created for developer convenience.

Table 1. Root Directory Files

File	For	Description
`.dockerignore`	Docker	Similar to `.gitignore` files, tells Docker to ignore specified file patterns when performing glob copies from the source directory into the container being built.
`.gitignore`	Git	Tells git to ignore files matching the specified patterns.
`api.raml`	RAML	API endpoint definition.
`build.gradle.kts`	Gradle	Standard Gradle build script definition.^[3] This particular build script uses the Kotlin DSL for gradle.
`dependencies.gradle.kts`	Gradle	Custom separation of the project’s dependencies into a specialized file to help keep the build script clean. Imported by `build.gradle.kts`
`Dockerfile`	Docker	Docker container definition file.^[4]
`gradlew`	Gradle	Standard, autogenerated Gradle script that allows the use of Gradle without requiring the host machine itself to have Gradle installed.
`makefile`	Make	Temporary wrapper for the build utils until the required tasks are performed which will allow simplifying the build. See Make Targets for usage.
`readme.adoc`	GitHub	Asciidoc readme file (the source backing this page).
`service.properties`	Gradle Build	Custom file added to simplify configuring the project build. This file is read by the Gradle build and controls basic config details about this project, such as the project name and version, generated Docker image name, Java entrypoint class, etc…

Table 2. Root Directory Dirs

Directory	For	Description
`gradle/`	Gradle	Contains the backing source for the `gradlew` script in the root directory. Keeping this directory in your repository is standard practice for Gradle.
`schema/`	RAML	Contains RAML type definitions imported by the `api.raml` file in the root directory. The `schema/library.raml` file is autogenerated based on the other files present in the schema directory.
`src/`		Java source root

1. Modular Raml Guide 1

2. Modular Raml Guide 2

3. Gradle Basics

4. Dockerfile Reference

VEuPathDB/service-eda

EDA Data Service

docker-compose

preliminary requirements

bring up the eda project

notes

TODO: replace text below with EdaDataService-specific documentation

Example Containerized JaxRS Service

Running the Example

Locally

Prerequisites

Setup & Run

In Docker

Prerequisites

Setup & Run

Development

Quick Start

Detailed Overview

Environment Prerequisites

Configuration

API definition

api.raml

Including Type Definitions

Type Schema

Generating A Service

Run Your Service

Other Information

Authentication

Make Targets

Build Process

Generating Jax-RS Code

Hiccups and limitations

Hacks and Oddities

FgpUtil

The project root

`api.raml`