Its a generic open source development framework to
- write your complex business logic in scala but
- wrap it with spark to run your app logic in distributed cluster
- you can use the same business logic to be consumed by both batch or online.
- Use of lazy vals where-ever required so that the framework is more performant.
- Use of Spark SQL is thus limited as the business logic is mainly written in Scala.
At any point of time the branch main will be the most updated branch.
- On each release, the main branch will be updated with the Integration branch.
- If you wish to move your changes create a branch from the feature/Integration branch and then raise PR from your branch to Integraiton.
Atom -> Element -> Compound -> Nucleus
- Atom is the unit element which needs to be calculated,
- You will pass that function to calculate as input.
- Multiple Atoms can be grouped into an Element Object.
- Each Element object will be of a specific type (this can be your data source schema).
- Multiple Elements can be grouped together in a Compound Object.
- This will be useful when you have same datasource but you want to distinguish by logical name.
- Then finally all the Compounds will be coming under one singleton object Nucleus.
- To calculate all atom's value you can go to the entry point of Nucleus and call the calc method.
The architectural diagram of the framework will be added later.