WebSessionization
Spark Web sessionization project using scala.
A goal of this project is to process the access-log file and achieve time based sessionization.
Following is the outcome of this project.
- Sessionize the web log by IP. Sessionize = aggregrate all page hits by visitor/IP during a fixed time window.
Output is uploaded to /output1 Format of output1 is
(sessionId, ipAddress), (lowestStartTime, maximumEndTime)
- Determine the average session time
Output is uploaded to /output2
Format of the output2 is (Labeled Session ID, total duration in milliseconds spend within the session) - handling the average at the end of the program.
Also, supplying staging data in stageData folder used for computing sessionDuration for every IP in that session window.
Format of stageData
((sessionID, ipAddress), (lowestStartTime, maximumEndTime, sessionDuration)
- Determine unique URL visits per session. To clarify, count a hit to a unique URL only once per session.
Output is uploaded to /output3 Format (sessionID, uniqueURL)
- Find the most engaged users, ie the IPs with the longest session times Output is uploaded to /output4
Format (ipAddress, sumOfAllSessionsDuration) - most engaged user will be at the end of the file.
==================== Building
- Please refer to build.sbt
- Go to project folder in shell - type sbt
- run-main WebLogChallange
==================== Tools and Language
- Spark (1.x version)
- Scala
- IntelliJ IDE
- SBT