This service filter sessions from a raw data file and returns response as JSON via api call
- Build system needs to have java and docker daemon running
- Raw data file is available on the system (mounted or local storage)
- Data is stored in ascending order of Date field (existing assumption)
- Response to the service is not bigger than max memory given to java process
- If the response is bigger, please update
-Xmx128m
- If the response is bigger, please update
- Java 14
- Spring Boot Framework
- Docker
- Service is a web based where api call is made and it returns JSON response
- Controller accepts the call, parse requested dates into valid Objects
- Controller sends request to Service where data processing happens
- Service reads single line at a time, starts storing the response, as soon as to time is reached, stops reading file.
- Service returns the response to Controller
- Controller returns JSON response (lombok is used to serialize Java Class in JSON Object)
Spring boot starter package contains maven wrapper which can test and build packages
To test unit and integration tests, run ./mvnw test
on command line
Build artifact is a docker image.
- First create package using
./mvnw package
- Build docker image
docker build . -t session:<image_tag>
- Run docker image created as part of build system
detached mode
docker run --rm --name session --publish 8080:8080 --volume <source_location_of_test_file>:<destination_location>:ro session:<image_tag> example: docker run --rm --name session --publish 8080:8080 --volume /Users/krupeshf/temp/processSession/sample3.txt:/app/sample.txt:ro session:latest
docker run --detach --rm --name session --publish 8080:8080 --volume <source_location_of_test_file>:<destination_location>:ro session:<image_tag> example: docker run --detach --rm --name session --publish 8080:8080 --volume /Users/krupeshf/temp/processSession/sample3.txt:/app/sample.txt:ro session:latest
- Request data
- Assumption here is the response size is smaller than java memory - this is to show that the full file is not loaded in the memory
- If we want response size bigger, please increase max memory given to java process
curl --location --request GET 'http://localhost:8080/?path=<destination_location>&fromDateTime=<from_time_in_ISO_format>&toDateTime=<to_time_in_ISO_format>' example: curl --location --request GET 'http://localhost:8080/?path=/app/sample.txt&fromDateTime=2000-01-01T13:13:53Z&toDateTime=2000-01-01T13:13:53Z'
- Assumption here is the response size is smaller than java memory - this is to show that the full file is not loaded in the memory
- Change the fetch time from
O(n)
toO(log n)
- Update linear fetch service to use binary search
- We can implement using
RandomAccessFile
- Some work has already been laid out using
FetchService
interface
- Use JIB for docker image creation
- This skips steps to install docker on the build system and thus making build system more secured
- Datadog to collect logs and metrics
- include datadog as part of maven dependency
- set java runtime arguments
-javaagent:dd-java-agent.jar
- Do not store response in memory
- Stream response using
StreamingResponseBody
- Stream response using
- Fetch file from a website or some storage bucket
Latest source code is available at krupeshf/rokt-session
All the commits and progress can be tracked