/OdinCodeBrowserJdk17

Primary LanguageShell

Odin Code Browser

This project attempts to create a completely static website of your source code, that navigates all the way down to the depths of the JDK. The goal is to navigate as well as Intellij. Try it yourself by navigating from Apache Commons Text's StringEscapeUtils.

Motivation

I used grepcode at least once a week and I am sad to see it go. It's navigation was almost as good as an IDE. You could navigate from apache commons all the way down to the JDK. Occasionally, I would forget what I was originally doing and get lost in the depths of the JDK. I hope that I can create a spiritual successor by using this tool, that will live much longer since the source code is available, and the resulting static web pages are decentralized.

Design Philosophy

  1. Each page is static
  2. Works well without javascript

How Do I Use This

As a code browser

  1. Find a project you want to browse from: jdk8, jdk11, jdk17, Apache Lang3, Apache Collections, Apache Text, Github Java Parser OdinCodeBrowser
  2. Find your class (ex: A class Odin uses StringEscapeUtils)
  3. Start reading code and navigating by clicking on the links, which are underlined.

As a code owner

git clone git@github.com:josephmate/OdinCodeBrowser.git
cd OdinCodeBrowser
args="--inputSourceDirectory <path_to_your_source>"
args="$args --outputDirectory <output_directory>"
args="$args --webPathToCssFile https://josephmate.github.io/OdinCodeBrowser/css/styles.css"
args="$args --webPathToSourceHtmlFiles <directory_on_webserver>"
args="$args --languageLevel JAVA_16"
args="$args --urlToDependantIndexJson <url_of_dependencies_index_json_file>"
mvn install exec:java \
  -Dexec.mainClass=Main \
  -Dexec.args="$args"

Creating an Odin Navigator for my project as an example, with dependencies on github javaparser, apache commons, apache commons text, and JDK8 that I host on https://josephmate.github.io/OdinCodeBrowser/ .

args="--inputSourceDirectory src/main/java"
args="$args --outputDirectory docs/odin.code.browser"
args="$args --webPathToCssFile /OdinCodeBrowser/css/styles.css"
args="$args --webPathToSourceHtmlFiles /OdinCodeBrowser/odin.code.browser"
args="$args --languageLevel JAVA_16"
args="$args --urlToDependantIndexJson https://josephmate.github.io/OdinCodeBrowserJdk8/index.json"
args="$args --urlToDependantIndexJson https://josephmate.github.io/OdinCodeBrowserRepos/javaparser-core_3.23.1/index.json"
args="$args --urlToDependantIndexJson https://josephmate.github.io/OdinCodeBrowserRepos/commons-lang3_3.11/index.json"
args="$args --urlToDependantIndexJson https://josephmate.github.io/OdinCodeBrowserRepos/commons-text_1.9/index.json"
args="$args --urlToDependantIndexJson https://josephmate.github.io/OdinCodeBrowserRepos/commons-collections_4.4/index.json"
mvn install exec:java \
  -Dexec.mainClass=Main \
  -Dexec.args="$args"

Fun and games

  1. Start at a random class in jdk8
  2. Using links only, how fast can you get my favourite class: HashMap?

For example I did I18NImpl -> PropertyResourceBundle -> HashMap

However, sometimes you'll pick a class that does not have a path to HashMap :(.

Alternatives

Your Java IDE can do a much better job. As I write this project, I use Intellij to navigate from my project's code to the code of the projects I depend on. The experience is so much better in the IDE and you should use that. If you are already using your IDE, keep using it!

If you don't use an IDE then github provides some support for navigating code without any extra effort. Just commit your code as you normally do and github updates the navigation.

The two features together that sets Odin apart from an IDE and GitHub:

  1. sharing links to the code (can't do this in an IDE). Since it's a web page, this means you can use it on low memory devices like your phone!
  2. can link to the dependant sources as well (github can't do this)

Comparison

Dimension Odin IDE Github
Shares links ✅ ❌ ✅
Minimal CPU Usage ✅ ❌ ✅
Minimal Memory Usage ✅ ❌ ✅
Minimal Storage Usage ✅ ❌ ✅
Realtime ❌ ✅ ❌
Automatically Applied ❌ ✅ ✅
Static Pages ✅ ?
Works without javascript ✅ ❌
Navigate to dependencies ✅ ✅ ❌
Comment on code ❌ ✅ ✅
Navigation Complete ❌ ✅ ❌

Explanation of criteria:

  • Shares links: In Odin and GitHub you can share links to any line in the code. An IDE does not let you do that.
  • Minimal CPU/Memory/Storage Usage: Since Odin and GitHub are not realtime, no cpu, memory or disk space is needed by the client to efficiently calculate all the indexes.
  • Realtime As you make changes in an IDE, the indexes are updated and you can navigate to the new code changes or new dependencies. For GitHub you need to commit and push before the navigation updates. For Odin you need to run this tool and publish the generated html files.
  • Automatically Applied: By rebuilding in the IDE, new code is recognized. By committing and pushing your changes to GitHub, the navigation is updated. For Odin, you must manually build and publish the static web pages. You unfortunately need to manage this. GitHub and an IDE manage this for you with no effort on your part.
  • Static Pages: Odin's output is hosted as static web pages that can be efficiently distributed. This comparison doesn't make sense for an IDE.
  • No javascript: Odin doesn't use javascript while GitHub needs to.
  • Navigate to dependencies: IDE and seamlessly navigate to your dependencies. Odin plans to navigate to dependencies when you build. It will not be as convenient as an IDE. GitHub only lets you navigate within the same repository.
  • Navigation Complete: anything you could possibly want to navigate to or from in an IDE works. Odin is working on matching an IDE and might reach the same level one day. GitHub navigation is okay. It gives you a list of options as a pop. Odin tries to be like an IDE and take you directly to the definition of the variable or method without asking you which one. For now, Odin sometimes makes mistakes due to method overloading.
  • Comment on code: GitHub is probably the best for commenting and discussing code. You can also do this in an IDE with plugins, but it's not as convenient as GitHub. Since Odin pages are static without javascript, it's impossible to have commenting.

Other Alternatives

These are alternatives that I do not use on a daily basis so I do not feel qualified in giving a thoughtful review. However, I'm going to help you find what you're looking for.

woboq : Looks exactly what I'm trying to achieve, but for C++ instead of Java. If you're working on open source C/C++ , I recommend checking it out.

Opengrok: Provides much better searching than what Odin can provide. However, as a result some components cannot be hosted as static files. If you host Opengrok for your project, I recommend it.

Sourcegraph: The search is really good. It like a super powered Github search. However, they aren't indexing OdinCodeBrowser yet.. I'm not sure what triggers the indexing because some of my old side projects are there. Also, it does not support cross repo navigation.

How it Works

  1. I use Github's JavaParser to visit nodes in the Java AST.
  2. Through visiting, I build indexes of classes, their methods, their fields, and their super classes.
  3. Through one more visit, I look to see if there's anything in the index I can create navigation links to and place these into a 'Rendering Queue'.
  4. I iterate over the code char by checking if I need to insert anything.
  5. All of these are saved as HTML files.
  6. The index is saved as a json file for other repos to use. That way they do not have to checkout the source code and build the index of all the dependencies. It also allows the repositories to independently host their source code and allows dependents to navigate from their code host on their servers to the code hosted on the dependency's servers.

Color Scheme

The color scheme is based on vim-dichromatic put together by Romain Lafourcade. I wanted a single color scheme that could be used by anyone, since it is really difficult for readers to change the styles. In order for someone to change the style, they will need to install a plugin for their browser and override Odin's css styles. Maybe in the future there will be some javascript to select from a list of styles that writes to your browser's local storage.

Future Work

Below is a checklist of features Odin needs.

  1. Line numbers with links
  2. Code is easily copy and pastable
    1. Code is copy and pastable
    2. Copied code is exactly the same as pasted code
  3. Click on a Type to navigate to the definition of that type
    1. Type
      1. Interface
      2. Annotation
      3. Outer class
      4. Inner class
      5. Enum
      6. Record
      7. Generics
      8. Class inside function
      9. Anonymous class
    2. Type Usage
      1. Variable type
      2. Method return type
      3. Extends type
      4. Implement type
      5. Record
      6. Enum
      7. Generics
      8. Annotation (the implementation of the annotation)
      9. Import
      10. Types through wildcard imports (ex: List from java.util.*)
      11. Types within same package (ex: ThreadPoolExecutor when in package java.util.concurrent)
      12. Types from java.lang (ex: IllegalArgumentException)
      13. Types within same file
  4. Click on method to go the definition of that method
    1. Static and only one function with the name (ex: Objects.hashCode(key))
    2. Functions within the same file (ex: putMapEntries(m, true)
    3. Functions on this (ex: this.getCanonName())
    4. Object instance and only one function with the name and not in super class (ex: x.getClass())
    5. Object instance and only one function with the name but in some super class
    6. super.method() (super.clone() in HashMap)
    7. Chained method calls
    8. Functions are overloaded (different parameters)
      1. all arguments are variables (ex: String.lastIndexOf(char vs String))
      2. length heuristic for generics (ex: Pair.of(A,B) vs Pair.of(Map.Entry<A,B>))
      3. all args are variables or literals (ex add(a, 1))
      4. where any of the arguments are expressions (ex: add(add(1,2), add(a,b)))
      5. variable arguments (ex: int add(int... vals))
    9. import static functions
    10. Record getter functions
    11. Super constructors
    12. Overloaded constructors
    13. Scoping rules (if names are duplicated in multiple scopes, need to use the closest scope)
    14. String literal method calls (ex: "true".equals(blah))
    15. class literal method calls (ex: boolean.class.getName())
  5. Click on variable to the definition of that variable
    1. from function param
    2. local var
    3. from scope (for/while/if)
    4. field var
    5. static field var
    6. Enum value
    7. this.field
    8. record.value
    9. static variable (ex: System.out)
    10. variable chaining (ex: a.b.c.d)
  6. Click on method to get a list of implementations
  7. Click on Override takes you to nearest super class's method that was overridden
  8. File list in root directory /OdinCodeBrowserJdk8
  9. Click on class definition to get all usages
  10. Click on method definition to get all usages
  11. Click on variable definition to get all usages
  12. Click on class/interface definition to get all subtypes
  13. Click on method definition to get all overrides and impls
  14. List of functions and fields at the side of the page.
    1. List of functions within this class, grouped by visibility
    2. List of fields within this class
    3. List of functions and fields from super classes
  15. Multi repository support (ex: browsing guava but also linking to JDK8)
    1. Repository exposes it's index file as json
    2. Build loads index files, then builds it own index
    3. Create a demo using current project OdinCodeBrowser -> apache text -> apache commons -> JDK8
      1. Take a look at SourceHtmlRenderer
      2. In there I use apache commons text to handle the html escaping for me
      3. From SourceHtmlRenderer can navigate to the StringEscapeUtils.escapeHtml4 implementation
      4. escapeHtml4 uses CharSequenceTranslator.translate
      5. Ignore not being able to navigate to the overloaded method. Odin doesn't support that yet.
      6. Validate.isTrue which is in apache commons lang3
      7. Again ignore that it went to the wrong overload of Validate.isTrue
      8. Which throws IllegalArgumentException which allows you to navigate to jdk8!
  16. Nice syntax highlighting somehow without javascript!
    1. Make it look decent on mobile (IE: not tiny text)
    2. Some syntax highlighting
    3. A dark mode syntax highlighting on
      1. pick a colour scheme: vim-dichromatic
      2. comments
      3. literals
        1. strings
        2. numbers
        3. boolean
        4. enum values
      4. native types (boolean, char, int, etc.)
      5. keywords (public, static, record, class, etc)
        1. this
        2. super
        3. class + modifiers
        4. modifiers of constructors
        5. modifiers of fields
        6. modifiers of functions
        7. modifiers of params
        8. modifiers of locals
        9. for
        10. synchronized
        11. import
        12. switch case
        13. if
        14. while
        15. do
        16. return
        17. extends
        18. implements
      6. Types
      7. function calls
      8. variables
  17. Render javadoc as text like in Intellij
  18. Link to native code
  19. navigation comparison between Odin, IDE, and GitHub
  20. Figure out how to support repos with multiple source directories
  21. Figure out a better release process other than having a developer clone the project and use the mvn exec:java goal.
    1. Tried fatJar but does not work well with java 16