Network file service with anti-virus checking

A file service prototype with a replication and anti-virus checking. It is mostly a study project but it is possible to use it after some "production" tuning.

A replacement for amqpav and first requirements were from the old service. Actual network file service with anti-virus checking supports JMS, AMQP, Kafka and REST for transport. AV-checker project was integrated as the checker submodule, then completely rewritten and the old code was removed.

From version 0.6 it is mostly a file service with an AV checking. Currently it is still a bit experimental feature. So if you need the AV checking only you can use the maintained 0.5 version.

Contribution

Possibilities: Quality assurance (QA), Front-end for REST services (statistics and file storage), Linux admin for better deployment and distribution, Wiki mage, Server for testing, or we can find something else for you

Release notes
Features
Planned features
Overview
Used components
How to send a message
Installation
Run services
AMQP checker
JMS checker
Legacy checker
Load tests

Release notes

Release NOTES

Features

AMQP 0.9.1 support
JMS support
Kafka support
REST support
Message tracking DB service (PostgreSQL or Solr)
Statistics module
File service with network replication
Highly scalable architecture
Robust design

Planned features

Separate REST app for better scaling
REST load balancer
AMQP 1.0 support

Overview

Remote REST configuration with file replication

Broker connection configuration with file replication

Local REST connection configuration

Used components

ClamAV - open source anti-virus engine
RabbitMQ - open source message broker used for AMQP
ActiveMQ - open source message broker used for JMS
Kafka - open source message broker used for Kafka
PostgreSQL - open source DB
Solr - open source enterprise search platform

Frameworks and libraries

Spring
Spring Boot
Spring REST - REST
Spring Security - REST security
Spring AMQP - AMQP
Spring JMS - JMS
Spring Kafka - Kafka
Spring Data - message logging and file saving
Ehcache 3 - caching
Spock - testing
Log4j 2 - logging API
Logback - logging

Code quality

Checkstyle
FindBugs
PMD
JaCoCo
SonarQube

Build

Gradle - build
Docker - environment preparation

Development

Elastic Stack (Elasticsearch, Logstash, Kibana) - logs analyzing

Spring profiles

Application is configurable and it's possible to run many configurations. Here is a list with all profiles and descriptions.

profile	description	from version	to version
amqp	enable AMQP infrastructure
amqp2jms	bridging AMQP to JMS		0.6
amqp-checker	AMQP checker		0.6
amqp-file-server	AMQP file server	0.6	0.6
amqp-rest	AMQP infrastructure for REST	0.5	0.6
amqp-server	AMQP server		0.6
bridge	enable bridge	0.7
check	enable AV checking	0.7
checker	enable checker	0.7
client	enable client infrastructure	0.7
core	core functionality
db	use SQL DB for message logging
db-mem	use in-memory SQL DB for message logging	0.8
db-solr	use Solr for message logging
file-client	enable file client	0.7
itest	profile for integration testing	0.7
jms	enable JMS infrastructure
jms2amqp	bridging JMS to AMQP		0.6
jms-checker	JMS checker		0.6
jms-rest	JMS infrastructure for REST	0.5	0.6
jms-server	JMS server		0.6
ldap	enable LDAP infrastructure	0.7
no-db	disable message logging
performance	profile for performance testing	0.8
replication	enable replication infrastructure	0.7
replication-test	enable replication testing infrastructure	0.7
rest	enable REST
rest-amqp	REST to AMQP	0.5
rest-jms	REST to JMS	0.5
rest-local	REST with direct connection	0.5
server	enable server infrastructure	0.7
stats	enable statistics	0.5
stats-solr	statistics in Solr	0.5
storage	enable file service	0.6
storage-check	profile for storage testing	0.8
to-amqp	bridging to AMQP	0.7
to-jms	bridging to JMS	0.7

Here will be the best combinations for various usages soon.

You can find all profiles for a concrete code base with the script findAllSpringProfiles.sh in the tools directory.

How to send a message for anti-virus check

AvCheckExample class:

package dvoraka.avservice.client.example;

import dvoraka.avservice.client.AvMessageFuture;
import dvoraka.avservice.client.configuration.ClientConfig;
import dvoraka.avservice.client.service.AvServiceClient;
import dvoraka.avservice.common.util.Utils;
import dvoraka.avservice.common.data.AvMessage;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;

/**
 * Anti-virus checking example.
 */
public final class AvCheckExample {

    public static void main(String[] args) throws InterruptedException {
        // initialize client context
        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
        context.getEnvironment().setActiveProfiles("client", "amqp", "file-client", "no-db");
        context.register(ClientConfig.class);
        context.refresh();

        // get client
        AvServiceClient avServiceClient = context.getBean(AvServiceClient.class);

        // generate message
        AvMessage avMessage = Utils.genMessage();

        // send it and get response
        AvMessage response;
        try {
            AvMessageFuture futureResponse = avServiceClient.checkMessage(avMessage);
            response = futureResponse.get();
        } finally {
            context.close();
        }

        // raw output
        System.out.println("Response: " + response);
        // virus info
        System.out.println("Virus info: " + (response != null ? response.getVirusInfo() : ""));
    }
}

Installation

You can use Docker to prepare necessary services for a development. The Docker configurations are a good documentation for installing services without Docker. Everything is inside the docker/ directory.

Docker

Docker is recommended approach.

Services

RabbitMQ
ActiveMQ
ClamAV daemon
PostgreSQL
Solr
Kibana
Elasticsearch
Logstash
2 app replication nodes
SonarQube (optional)

Change your working directory to docker/:

$ cd docker/

And run docker-compose up command:

$ docker-compose up

It prepares services running on localhost.

Configuration:

For setting environment it is necessary to run a command from the root directory:

change to project root

$ cd ..

and

$ ./gradlew configureEnvironment

And everything should be prepared for application running.

Run service

You can run all services easily with Gradle.

AMQP

Anti-virus checking server:

$ ./gradlew runAmqpServer

File server with anti-virus checking:

$ ./gradlew runAmqpFileServer

JMS

Anti-virus checking server:

$ ./gradlew runJmsServer

File server with anti-virus checking:

$ ./gradlew runJmsFileServer

REST

It is a Spring Boot application. You can run it with Gradle or use an executable jar.

Gradle

$ ./gradlew rest:bootRun

Spring Boot

You need to build an executable jar:

$ ./gradlew assemble

And then the the jar is in rest/build/libs/ directory called avservice-rest-XXX.jar. You can run it with:

$ java -jar avservice-rest-0.9.jar 

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::            (v1.5.4.RELEASE)

AMQP checker

Utility for testing AMQP infrastructure. It sends a message and wait for a response.

Run

$ ./gradlew -q runAmqpCheck

JMS checker

Utility for testing JMS infrastructure. It sends a message and wait for a response.

Run

$ ./gradlew -q runJmsCheck

Check output logs

12:20:00.388 [main] INFO  d.avservice.client.checker.CheckApp - Checking... 
12:20:03.807 [main] INFO  d.avservice.client.checker.CheckApp - failed!
12:20:03.808 [main] INFO  d.avservice.client.checker.CheckApp - Check completed.

12:20:00.388 [main] INFO  d.avservice.client.checker.CheckApp - Checking... 
12:20:03.807 [main] INFO  d.avservice.client.checker.CheckApp - OK
12:20:03.808 [main] INFO  d.avservice.client.checker.CheckApp - Check completed.

Old checker

You can find old checker under legacy-checker release.

AMQP checker

Utility for testing AMQP infrastructure.

Run

$ ./gradlew -q amqpCheck

And output should be:

...
Test OK

Create jar with dependencies

$ ./gradlew :checker:shadowJar

And the full jar will be in checker/build/libs/ directory.

Run jar

Built jar is executable.

Properties:

host - message broker host
infected - infection flag (true/false)
appid - application ID string

$ java -Dhost=localhost -jar checker-0.1-SNAPSHOT-all.jar

And output should be:

...
Test OK

Load tests

Various performance testing: TESTS

dvoraka/av-service

Network file service with anti-virus checking

Contribution

Release notes

Features

Planned features

Overview

Remote REST configuration with file replication

Broker connection configuration with file replication

Local REST connection configuration

Used components

Spring profiles

How to send a message for anti-virus check

Installation

Docker

Run service

AMQP

JMS

REST

AMQP checker

Run

JMS checker

Run

Check output logs

Old checker

AMQP checker

Run

Create jar with dependencies

Run jar

Properties:

Load tests