This Processor is used for processing kafka events and backfills Informix with Challenge Resource data
- Processor
- NodeJS (v10)
- Kafka
- Docker
- Docker Compose
Configuration for the processor is at config/default.js.
The following parameters can be set in config files or in env variables:
- DISABLE_LOGGING: whether to disable logging, default is false
- LOG_LEVEL: the log level; default value: 'debug'
- KAFKA_URL: comma separated Kafka hosts for consumer to listen; default value: 'localhost:9092'
- KAFKA_GROUP_ID: Kafka consumer group id; default value: 'legacy-challenge-resource-processor-group'
- KAFKA_CLIENT_CERT: Kafka connection certificate, optional; default value is undefined;
- KAFKA_ERROR_TOPIC: The kafka error topic.
- MAX_RETRIES: the number of max retries; default value: 3
- RETRY_TIMEOUT: The timeout to retry processing the same message
- BUSAPI_URL: Bus API URL
if not provided, then SSL connection is not used, direct insecure connection is used;
- KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional; default value is undefined;
if not provided, then SSL connection is not used, direct insecure connection is used;
- CREATE_CHALLENGE_RESOURCE_TOPIC: Create challenge resource Kafka topic, default value is 'challenge.resources.notification.create'
- DELETE_CHALLENGE_RESOURCE_TOPIC: Delete challenge resource Kafka topic, default value is 'challenge.resources.notification.delete'
- CHALLENGE_API_V4_URL: Challenge V4 API URL, default value is 'https://api.topcoder-dev.com/v4/challenges'
- CHALLENGE_API_V5_URL: Challenge V5 API URL, default value is 'http://localhost:3001/v5/challenges', which is mocked for now since the V5 API is not deployed in Topcoder Dev environment yet.
- AUTH0_URL: Auth0 url for M2M token
- AUTH0_AUDIENCE: Auth0 audience for M2M token || 'https://www.topcoder-dev.com',
- TOKEN_CACHE_TIME: Cache time of M2M token, optional
- AUTH0_CLIENT_ID: Auth0 client id for M2M token
- AUTH0_CLIENT_SECRET: Auth0 client secret for M2M token
- IS_CREATE_FORUM: Should create forum resource or not
- CHALLENGE_ORIGINATOR: originator from challenge service
Also note that there is a /health endpoint that checks for the health of the app. This sets up an expressjs server and listens on the environment variable PORT. It's not part of the configuration file and needs to be passed as an environment variable
Configuration for the tests is at config/test.js. Following parameters need to be set via environment variables or directly in config file
- TEST_KAFKA_URL: Kafka URL pointing to Kafka test instance
- WAIT_TIME: wait time used in test, default is 1000 or one second
To install foreman follow this link To know how to use foreman follow this link
http://kafka.apache.org/quickstartcontains details to setup and manage Kafka server, below provides details to setup Kafka server in Mac, Windows will use bat commands in bin/windows instead- download kafka at
https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.11-1.1.0.tgz - extract out the downloaded tgz file
- go to extracted directory kafka_2.11-0.11.0.1
- start ZooKeeper server:
bin/zookeeper-server-start.sh config/zookeeper.properties - use another terminal, go to same directory, start the Kafka server:
bin/kafka-server-start.sh config/server.properties - note that the zookeeper server is at localhost:2181, and Kafka server is at localhost:9092
- use another terminal, go to same directory, create the needed topics:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic challenge.resources.notification.create
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic challenge.resources.notification.delete
- verify that the topics are created:
bin/kafka-topics.sh --list --zookeeper localhost:2181,
it should list out the created topics
- install dependencies
npm i - run code lint check
npm run lint, runningnpm run lint:fixcan fix some lint errors if any - start processor app
npm start - to start with production config
npm runt start:prod
To run the processor using docker, follow the below steps
- Navigate to the directory
docker - Rename the file
sample.api.envtoapi.env - Set parameters in the file
api.env - Once that is done, run the following command
docker-compose up- When you are running the application for the first time, It will take some time initially to download the image and install the dependencies
- TBD
Test configuration is at config/test.js. You don't need to change them.
The following test parameters can be set in config file or in env variables:
- DISABLE_LOGGING: whether to disable logging, default is false
- LOG_LEVEL: the log level; default value: 'debug'
- KAFKA_URL: comma separated Kafka hosts for consumer to listen; default value: 'localhost:9092'
- WAIT_TIME: wait time used in test, default is 1000 or one second
- Start Local services.
- Start Mock API.
- Various config parameters should be properly set.
To run unit tests alone
npm run testTo run unit tests with coverage report, you can check generated coverage report in coverage folder and coverage for src/services/ProcessorService.js is 100%.
npm run covTo run integration tests alone
Please note that E2E tests work against the real Topcoder Dev API, and the following challenge is used for testing :
- Go to accounts.topcoder-dev.com/member
- Login as TonyJ/appirio123
- https://software.topcoder-dev.com/review/actions/ViewProjectDetails?pid=30049360
npm run e2eTo run integration tests with coverage report
please note e2e tests will run with real api so e2e tests will not cover some error cases but most cases are modified from unit tests
npm run cov-e2e- TBD
Refer to the verification document Verification.md
Commit to force redeploy