portal

A clojure tool to navigate through your data.

The portal UI can be used to inspect values of various shapes and sizes. The UX will probably evolve over time and user feedback is welcome!

For an in-depth explanation of the UX, jump to UX Concepts.

Demo

To get an overview of the Portal UI and workflow, checkout the following recording of a live demo I gave for London Clojurians.

Usage

To start a repl with portal, run the clojure >= 1.10.0 cli with:

clj -Sdeps '{:deps {djblue/portal {:mvn/version "0.22.1"}}}'

or for a web clojurescript >= 1.10.773 repl, do:

clj -Sdeps '{:deps {djblue/portal {:mvn/version "0.22.1"}
                    org.clojure/clojurescript {:mvn/version "1.10.844"}}}' \
    -m cljs.main

or for a node clojurescript >= 1.10.773 repl, do:

clj -Sdeps '{:deps {djblue/portal {:mvn/version "0.22.1"}
                    org.clojure/clojurescript {:mvn/version "1.10.844"}}}' \
    -m cljs.main -re node

or for a babashka >=0.2.4 repl, do:

bb -cp `clj -Spath -Sdeps '{:deps {djblue/portal {:mvn/version "0.22.1"}}}'`

or for examples on how to integrate portal into an existing project, look through the examples directory.

NOTE: Portal can also be used without a runtime via the standalone version. The standalone version can be installed as a chrome pwa which will provide a dock launcher for easy access.

API

Try the portal api with the following commands:

;; for node and jvm
(require '[portal.api :as p])

;; for web
;; NOTE: you might need to enable popups for the portal ui to work in the
;; browser.
(require '[portal.web :as p])


(def p (p/open)) ; Open a new inspector

;; or with an extension installed, do:
(def p (p/open {:launcher :vs-code}))  ; JVM only for now
(def p (p/open {:launcher :intellij})) ; JVM only for now

(add-tap #'p/submit) ; Add portal as a tap> target

(tap> :hello) ; Start tapping out values

(p/clear) ; Clear all values

(tap> :world) ; Tap out more values

(prn @p) ; bring selected value back into repl

(remove-tap #'p/submit) ; Remove portal from tap> targetset

(p/close) ; Close the inspector when done

NOTE: portal will keep objects from being garbage collected until they are cleared from the UI.

Portal Atom

In addition to getting the selected value back in the repl, the portal atom also allows direct manipulation of portal history. For example:

(def a (p/open))

; push the value 0 into portal history
(reset! a 0)

@a ;=> 0 - get current value

; inc the current value in portal
(swap! a inc)

@a ;=> 1 - get current value

Remote API

In addition to running portal in process, values can be sent over the wire to a remote instance.

In the process hosting the remote api, do:

(require '[portal.api :as p])
(p/open {:port 5678})

In the client process, do:

(require '[portal.client.jvm :as p])
;; (require '[portal.client.node :as p])
;; (require '[portal.client.web :as p])

(def submit (partial p/submit {:port 5678})) ;; :encoding :edn is the default
;; (def submit (partial p/submit {:port 5678 :encoding :json}))
;; (def submit (partial p/submit {:port 5678 :encoding :transit}))

(add-tap #'submit)

NOTE: tap>'d values must be serializable as edn, transit or json.

Options

Themes

There are currently three built-in themes:

Which can be passed as an option to p/open:

(p/open
  {:theme :portal.colors/nord})

Launcher

By default, when p/open is called, an HTTP server is started on a randomly chosen port. It is also given a default window title of the form portal - <platform> - <version>. To control this server's port, host, and window title, call the p/start function with the following options:

Option	Description	If not specified
`:port`	Port used to access UI	random port selected
`:host`	Hostname used to access UI	"localhost"
`:app`	Launch as separate window	true
`:window-title`	Custom title for UI window	"portal"

UX Concepts

The portal ux can be broken down into the following components:

Selection

A single click will select a value. The arrow keys, (⭠ ⭣ ⭡ ⭢) or (h j k l) will change the selection relative to the currently selected value. Relative selection is based on the viewer.

Multi-Select

To select multiple values, hold down either ⌘ or alt and single click a value. To un-select a value, simply click it again while holding down the multi-select key. The order of selection is important to the application of commands.

Viewers

A viewer takes a raw value and renders it to the screen. A single value can have many viewers. Most viewers have a :predicate function to define what values they support. A :predicate can be as simple as a type check to as complex as a clojure.spec.alpha/valid? assertion. The bottom-left dropdown displays the viewer for the currently selected value and contains all viewers for the value.

A default viewer can be set via metadata. For example:

^{:portal.viewer/default :portal.viewer/hiccup} [:h1 "hello, world"]

Commands

The bottom-right yellow button will open the command palette. Commands can have a :predicate function like viewers, so only relevant commands will be visible which is based on the currently selected value. They will be sorted alphabetically by name and can quickly be filtered. The (ctrl | ⌘) + shift + p or ctrl + j shortcuts can also be used to open the command palette.

The filter string is split by white space and all words must appear in a name to be considered a match.

To register your own command, use the portal.api/register! function. For example:

(portal.api/register! #'identity)

When multiple values are selected, commands will be applied as follows:

(apply f [first-selected second-selcted ...])

NOTES:

A very useful command is portal.ui.commands/copy which will copy the currently selected value as an edn string to the clipboard.
lambdaisland.deep-diff2/diff is a useful command for diffing two selected values.
Commands manipulating the UI itself will live under the portal.ui.commands namespace.
The cljs.core namespace will be aliased as clojure.core when using a clojurescript runtime.

History

The top-left arrow buttons (⭠ ⭢) can be used to navigate through history. History is built up from commands pushing values into it. For example, anytime a value is double clicked, clojure.datafy/nav is applied to that value and the result is pushed onto the history stack. All commands that produce a new value will do the same.

To quickly navigation through history, checkout the following commands:

portal.ui.commands/history-back
portal.ui.commands/history-forward
portal.ui.commands/history-first
portal.ui.commands/history-last

Shortcuts

To run a command without having to open the command palette, you can use the commands shortcut. They will be listed in the command palette next to the command. When multiple shortcuts are available, they are separated by a vertical bar.

NOTE: shortcuts aren't currently user definable.

Filtering

Like many concepts listed above, filtering is relative to the currently selected value. If no value is selected, filtering is disabled. When a collection is selected, the filter text will act to remove elements from that collection, similar to the command palette.

Datafy and Nav

There is one exception to the behavior described above for the UI, datafy and nav. They are extension points defined in clojure to support user defined logic for transforming anything into clojure data and how to traverse it.

For a great overview of datafy and nav, I recommend reading Clojure 1.10's Datafy and Nav by Sean Corfield.

The below will demonstrate the power of datafy and nav by allowing you to traverse the hacker news api! It will produce data tagged with metadata on how to get more data!

(require '[examples.hacker-news :as hn])

(tap> hn/stories)

An interesting use case for nav is allowing users to nav into keywords to produce documentation for that keyword. This really highlights the power behind datafy and nav. It becomes very easy to tailor a browser into the perfect development environment!

CLI Usage

Add a portal alias in ~/.clojure/deps.edn

:portal/cli
{:main-opts ["-m" "portal.main"]
 :extra-deps
 {djblue/portal {:mvn/version "0.22.1"}
  ;; optional yaml support
  clj-commons/clj-yaml {:mvn/version "0.7.0"}}}

Then do the following depending on your data format:

cat data | clojure -M:portal/cli [edn|json|transit|yaml]
# or with babashka for faster startup
cat data | bb -cp `clojure -Spath -M:portal/cli` -m portal.main [edn|json|transit|yaml]

I keep the following bash aliases handy for easier CLI use:

alias portal='bb -cp `clojure -Spath -M:portal/cli` -m portal.main'
alias edn='portal edn'
alias json='portal json'
alias transit='portal transit'
alias yaml='portal yaml'

and often use the Copy as cURL feature in the chrome network tab to do the following:

curl ... | transit

There is also the ability to invoke a standalone http server to listen and display data from remote client

   bb -cp `clojure -Spath -Sdeps '{:deps {djblue/portal {:mvn/version "LATEST"}}}'` \
      -e '(require (quote [portal.api])) (portal.api/open {:port 53755}) @(promise)'

Editor Integration

Emacs

If you are an emacs + cider user and would like tighter integration with portal, the following section may be of interest to you.

;; Leverage an existing cider nrepl connection to evaluate portal.api functions
;; and map them to convenient key bindings.

(defun portal.api/open ()
  (interactive)
  (cider-nrepl-sync-request:eval
   "(require 'portal.api) (portal.api/tap) (portal.api/open)"))

(defun portal.api/clear ()
  (interactive)
  (cider-nrepl-sync-request:eval "(portal.api/clear)"))

(defun portal.api/close ()
  (interactive)
  (cider-nrepl-sync-request:eval "(portal.api/close)"))

;; Example key mappings for doom emacs
(map! :map clojure-mode-map
      ;; cmd  + o
      :n "s-o" #'portal.api/open
      ;; ctrl + l
      :n "C-l" #'portal.api/clear)

;; NOTE: You do need to have portal on the class path and the easiest way I know
;; how is via a clj user or project alias.
(setq cider-clojure-cli-global-options "-A:portal")

VS Code

If you are using vs-code, try out the vs-code-extension. It allows launching portal in an embedded webview within vs-code.

For a more in depth look at customizing vs-code for use with portal, particularly with mauricioszabo/clover, take a look at seancorfield/vscode-clover-setup.

NOTE: The version of portal being run in the webview is still decided by the runtime in which (portal.api/open {:launcher :vs-code}) is run.

IntelliJ

Download the IntelliJ plugin from Releases and install it from the disk. A "Portal" button will appear on the right-hand bar.

Add a dependency on Portal to your project, start a Clojure REPL and inside that evaluate:

(do
    (def user/portal ((requiring-resolve 'portal.api/open) {:launcher :intellij}))
    (add-tap (requiring-resolve 'portal.api/submit)))

You can now tap> data and they will appear in the Portal tool window.

Principles

Support as much of clojure's data as possible
Independent of any particular editor but embeddable
Simple standalone usage without a clojure environment
Easy theming

Inspiration

Ideas for the Future

~~Diff Viewer~~
- Any vector pair can be diffed in portal via lambdaisland/deep-diff2
~~Markdown Viewer~~
- Any string can be viewed as markdown in portal via yogthos/markdown-clj
- Any hiccup data structure can also be viewed as html
~~Chart Viewer~~
- Initial chart viewers #31 using vega-lite.
Node+Edge Graphs Viewer

Development

Dependencies

To start the development server, make sure you have the following dependencies installed on your system:

java - for clojure runtime
- for osx, do brew install openjdk
babashka - for build scripting
- for osx, do: brew install borkdude/brew/babashka
- to list all build tasks, do: bb tasks
node, npm - for javascript dependencies
- for osx, do: brew install node

vim + vim-fireplace

To start the nrepl server, do:

bb dev

vim-fireplace should automatically connect upon evaluation, but this will only be for clj files, to get a cljs repl, do:

:CljEval (user/cljs)

emacs + cider

The best way to get started via emacs is to have cider start the repl, do:

M-x cider-jack-in-clj&cljs

.dir-locals.el has all the configuration variables for cider.

`user.clj`

The user.clj namespace has a bunch of useful examples and code for development. Take a peek to get going.

To launch the dev client version of portal, make sure to do:

(portal.api/open {:mode :dev})

Formatting

To format source code, do:

bb fmt

CI Checks

To run ci checks, do:

bb ci    # run all ci check

bb check # run just the static analysis
bb test  # run just the tests

E2E Testing

To run the e2e tests in the jvm, node and web environments, do:

bb e2e

NOTE: these aren't fully automated tests. They depend on a human for verification and synchronization but it beats having to type everything out manually into a repl.

Extensions

To build the vs-code and intellij extensions, do:

bb ext

Deployment

To deploy to a release to clojars, do:

bb tag
bb deploy

davidpham87/portal