YScope Log Viewer

yscope-log-viewer is a tool that can be used to view log files that were compressed using CLP's IR stream format. The viewer can be used to navigate the log file, filter by log level, pretty print the logs and generate direct links to specific log events.

See the features in development section for upcoming features.

Online Demo

A demo of the log viewer can be found at https://yscope.com/log-viewer
The demo loads a Hadoop YARN log file from the hive-24hrs log dataset.
- More info on the dataset and other datasets can be found here.
To open an IR stream, drag and drop it onto the log viewer or use the open file dialog.

Generating IR Stream Logs

IR stream log files can currently be generated using these libraries:

Python Logging Library
Log4j Logging Library
Golang Logging Library (in development)

Providing Feedback

You can use GitHub issues to report a bug or request a feature.

Join us on Zulip to chat with developers and other community members.

Developing `yscope-log-viewer`

Clone the repo to get a copy of the code for development

git clone https://github.com/yscope/yscope-log-viewer.git
cd yscope-log-viewer

Install Node.js using a release or via nvm
Install the dependencies:
```
$ npm install
```
Run the development server:
```
$ npm start
```
The application should now be served at http://localhost:3010.

Distribute

To create a build, run the following command and the build will be placed in the dist folder:

$ npm run build

How does it work?

yscope-log-viewer is written using the ReactJS framework and uses the open source monaco-editor.

To open IR stream files, the viewer spawns workers to do the following:

Decompress the Zstd-compressed file
Build an index of log events
Paginate the indexed logs based on the number of log events per page
Decode the CLP-encoded log data as needed

Once the worker decompresses, decodes, and extracts logs, the viewer UI can be used to navigate the log. Tasks are passed to the worker as needed and changes are rendered to the UI.

Validation

Currently, there is limited automated testing to verify that changes don't cause bugs. While this is being developed, the following tests can be run manually:

Verify that the following features work:
- Changing the number of events per page
- Navigating to the first/last/next/previous page
- Loading a log file using the open file dialog and dragging & dropping
- Copying a link to a log event
- Change the log level
- Prettifying logs
- Using the keyboard shortcuts
Perform a build and verify that all features are functional

Running tests

We use jest as our testing framework. Tests can be run with:

npm test

Features in Development

Dashboard to visualize distribution of log types and log levels
Opening log files which are larger than 2 GB when uncompressed
Support for plain text, archived log files, and other requested formats
Search across file, multiple files or within provided time ranges
Infinite scrolling instead of using pagination
Log correlation with sync by timestamp across multiple editors
Automatic conversion of text log files to IR stream format in the browser
Deployment of components via NPM

jackluo923/yscope-log-viewer