typelead/eta

[etlas] Dhall package format

rahulmutt opened this issue · 16 comments

rahulmutt commented

This issue is to discuss the refinements to the etlas.dhall format to make it more concise and readable than it already is as we get closer to releasing Etlas v1.6.0.0 which will make etlas.dhall the default format.

Cc: @jneira - Maintainer of the Dhall integration in Etlas

As a starting point to this discussion, I'll suggest some things that I think might make it cleaner using this sample etlas.dhall as a reference for discussion as it is a non-trivial example of real-world usage.

  1. Observe

    let prelude =
      https://raw.githubusercontent.com/eta-lang/dhall-to-etlas/master/dhall/prelude.dhall sha256:47a62403bcd28107cb0a99b05fc4d8bc73f06277c422d059ba9efdd50307abbc

let types =
      https://raw.githubusercontent.com/eta-lang/dhall-to-etlas/master/dhall/types.dhall sha256:d36b3384c6e0dc1837f5740b14eb20b1c51de1cb16a8008ee577d7d8f7ea115e

    It makes sense to bind each version of Etlas to a certain version of dhall-to-etlas and have these two lines automatically added in by Etlas with perhaps flags to customize these since the Etlas code expects the Dhall data structures to be in a certain way when translating to the internal GenericPackageDescription type.

  2. Haskell2010 should be the default language so there should be no release to bind it. Once we release an Eta standard, we can make that the default.

  3. pkg and pkgVer should both be included inside of some standard library for Etlas package format that is auto-let binded similar to (1).

  4. Observe

    base = pkgVer "base" "4.5" "5"

    Note that we had to type in base twice! And I have to do this for every dependency I have. Does dhall have a way to traverse the the keys of a record generically so that we can make it a mapping from package name to version bounds? Also, perhaps we can use the Ivy notation for managing version bounds - it's compact and avoids the awkward && and comparison operations. (i.e. "[4.5, 5)" would mean >= 4.5 && <= 5.

  5. updateRepo should probably also be a built-in.

  6. Observe

      [ types.Extension.DataKinds True
  , types.Extension.TypeOperators True
  , types.Extension.TypeFamilies True
  , types.Extension.FlexibleContexts True
  , types.Extension.MultiParamTypeClasses True
  , types.Extension.OverloadedStrings True
  ]

    It is very rare that one decides to disable an extension. Perhaps we should abstract out those True's and make disabling happen through a function? Also, I'm not completely familiar with Dhall, but is there any language feature that lets you abstract out the long qualified names - like somehow shorten types.Extension.DataKinds to DataKinds in the context of extensions?

jneira commented

Good points! I will comment one by one:

  1. It makes sense to bind each version of Etlas to a certain version of dhall-to-etlas and have these two lines automatically added in by Etlas

Agree, they can be automatically added and set in scope for all config files. Moreover, if user want to shadow them with its own versions it will be able to do it cause let x=1 let x = 2 in x == 2

2. Haskell2010 should be the default language

5. updateRepo should probably also be a built-in.

👍

  • About dependencies i was thinking in create some default to let users set somethink like:
let commonDeps = [ 
   deps.default "base" "4.5" "5.0", 
   deps.any "dhall", 
   deps.strict "bytestring" "0.11" 
]

It would generate:

base >= 4.5 && < 5.0,
dhall -any
bytestring == 0.11

And for libs or executables you could do:

build-depends = commonDeps # [ deps.any "wai-servlet" ]

So no need for record keys (although you could use them). I am afraid that it is not a great improvement in conciseness though.

We even could put in scope "sets" of dependencies to match the more common use cases.
Or we could generate automatically a dhall file with a record with the more used dependencies fixed to the recommended bounds (so it would be a curated dep set) and user could simply pick them from there:

let deps= https://github.com/typelead/etlas-index/dhall/default-dependencies
in
....
build-depends: [deps.base, deps.bytestring, ... ]

The possibilities would be endless 😄

4. Does dhall have a way to traverse the the keys of a record generically so that we can make it a mapping from package name to version bounds? Also, perhaps we can use the Ivy notation for managing version bounds - it's compact and avoids the awkward && and comparison operations. (i.e. "[4.5, 5)" would mean >= 4.5 && <= 5.

I am afraid that for now dhall has no sense of text comparation. It is so by design, to enforce use of lists Records and Union types, more verbose but safer.
It neither can handle record keys in a generic way cause they are types and there is no reflection or another way to instrospect them. You at most could use associacion lists: { key = "base", value = ["4.5", "4" ]} and then use a function to search the key base (wrong, you can not search by text, we would have to make packages an union type to search in a list)

6. Perhaps we should abstract out those True's and make disabling happen through a function?

I think it would be realizable

6. Also, I'm not completely familiar with Dhall, but is there any language feature that lets you abstract out the long qualified names - like somehow shorten types.Extension.DataKinds to DataKinds in the context of extensions?

You always can use let: let Exts = types.Extension, i am afraid there is no way to make a "context" implicit without naming it in the dhall file itself (you could add them when reading the file in haskell, like prelude or types).
Like dependencies, or other fields (ghc args), we can add to prelude sets of more common used values.

jneira commented

A caveat about add implicit definitions is you cant use dhall executables to proccess the file.
Otoh, if user want to add an explicit prelude/types the implicit ones would be loaded too (so it would add some overhead and cant be used, cause dhall is strict).
So not sure if it is worth save the two lines imports...
Maybe we can try to simplify as much as possible using only imports and functions and see if it could be enough ergonomic.

jneira commented

Hi, i've changed the defaults in dhall-to-etlas to match the most common options for eta projects.
The following options are by default:

  • default-language: Haskell2010
  • default-extensions:
BangPatterns DataKinds DeriveFoldable
DeriveFunctor DeriveGeneric DeriveTraversable EmptyCase
ExistentialQuantification FlexibleContexts FlexibleInstances
FunctionalDependencies GeneralizedNewtypeDeriving MagicHash
MultiParamTypeClasses MultiWayIf LambdaCase OverloadedStrings
RankNTypes RecordWildCards StandaloneDeriving ScopedTypeVariables
TupleSections TypeFamilies TypeOperators
  • ghc-options: -Wall -fwarn-incomplete-uni-patterns -fwarn-incomplete-record-updates
  • New function GitHubTag-project that creates the repo info suggested in etlas index (mapSourceRepo is not needed)

To make dependencies more ergonomic i am thinking in create dhall files with the packages and their bounds and import them in etlas.dhall files:

  • A index.dhall file in https://github.com/typelead/etlas-index with the packages stored in it (like dhall-eta,wai-servlet) and the pref-ver data with the patched packages bounds
  • An index.dhall file for each suitable stackage lts with all their packages fixed to a version
  • Not sure if we could make something similar for hackage cause it is really, really big

Attached how looks the etlas.dhall and the equivalent cabal file:

jneira commented 
let prelude =
      https://raw.githubusercontent.com/eta-lang/dhall-to-etlas/etlas/dhall/prelude.dhall

let types =
      https://raw.githubusercontent.com/eta-lang/dhall-to-etlas/etlas/dhall/types.dhall sha256:a6c967e2f3af97d621c2ec058822f41527e10d4288bd45b191e3a79f2ab87217

let deps =
      https://raw.githubusercontent.com/eta-lang/dhall-to-etlas/etlas/dhall/dependencies.dhall sha256:e034fad0030e42d5fb1d21056ced132eaf9ded8adcd84f70bc8eaed0e60b1bfe

let v = prelude.v

let dep = prelude.Dependency.orLater-earlier

let any = prelude.Dependency.any

let project =
      prelude.utils.GitHubTag-project
      { owner = "eta-lang", repo = "dhall-eta", version = "1.0.0" }

in    project
    ⫽ { synopsis =
          "dhall-eta is a eta library that wraps the haskell implementation of dhall configuration language."
      , description =
          ""
      , category =
          "Language"
      , maintainer =
          "atreyu.bbb@gmail.com"
      , author =
          "Javier Neira Sánchez <atreyu.bbb@gmail.com>"
      , extra-source-files =
          [ "build.gradle"
          , "dhall-eta.cabal"
          , "dhall-eta.dhall"
          , "examples/build.gradle"
          , "examples/src/main/java/org/dhall/eta/example/*.java"
          , "gradlew"
          , "gradlew.bat"
          , "gradle/wrapper/gradle-wrapper.jar"
          , "gradle/wrapper/gradle-wrapper.properties"
          , "java/build.gradle"
          , "java/src/main/java/org/dhall/*.java"
          , "java/src/main/java/org/dhall/binary/*.java"
          , "java/src/main/java/org/dhall/binary/decoding/failure/*.java"
          , "java/src/main/java/org/dhall/common/types/*.java"
          , "java/src/main/java/org/dhall/common/types/either/*.java"
          , "java/src/main/java/org/dhall/common/types/functor/*.java"
          , "java/src/main/java/org/dhall/core/*.java"
          , "java/src/main/java/org/dhall/core/constant/*.java"
          , "java/src/main/java/org/dhall/core/expr/*.java"
          , "java/src/main/java/org/dhall/core/imports/*.java"
          , "java/src/main/java/org/dhall/core/imports/hashed/*.java"
          , "java/src/main/java/org/dhall/core/imports/types/*.java"
          , "java/src/main/java/org/dhall/core/imports/types/url/*.java"
          , "proguard.txt"
          , "README.md"
          , "settings.gradle"
          , "src/main/java/org/dhall/eta/*.java"
          , "src/test/resources/import/data/foo/bar/*.dhall"
          , "src/test/resources/import/success/*.dhall"
          ]
      , license =
          types.License.BSD3 {=}
      , license-files =
          [ "LICENSE" ]
      , library =
          prelude.unconditional.library
          (   prelude.defaults.Library
            ⫽ { build-depends =
                    [ deps.base
                    , deps.bytestring
                    , deps.containers
                    , deps.contravariant
                    , deps.cryptonite
                    , deps.dhall
                    , deps.eta-java-interop
                    , deps.megaparsec
                    , deps.memory
                    , deps.scientific
                    , deps.serialise
                    , deps.text
                    , deps.transformers
                    ]
                  # [ dep "dotgen" "0.4.2" "0.5"
                    , dep "lens-family-core" "1.0.0" "1.3"
                    , dep "prettyprinter" "1.2.0.1" "1.3"
                    ]
              , exposed-modules =
                  [ "Dhall.Eta"
                  , "Dhall.Eta.Binary"
                  , "Dhall.Eta.Context"
                  , "Dhall.Eta.Core"
                  , "Dhall.Eta.Core.Java"
                  , "Dhall.Eta.Import"
                  , "Dhall.Eta.Parser"
                  , "Dhall.Eta.Parser.Java"
                  , "Dhall.Eta.TypeCheck"
                  , "Dhall.Eta.TypeCheck.Java"
                  , "Eta.Types"
                  ]
              , hs-source-dirs =
                  [ "src/main/eta" ]
              , java-sources =
                  [ "@classes.java" ]
              , other-modules =
                  [ "Dhall.Eta.Map" ]
              }
          )
      , executables =
          [ prelude.unconditional.executable
            "dhall-eta-all"
            (   prelude.defaults.Executable
              ⫽ { build-depends =
                    [ deps.base, any "dhall-eta" ]
                , hs-source-dirs =
                    [ "examples/src/main/eta" ]
                , main-is =
                    "Main.hs"
                }
            )
          ]
      , test-suites =
          [ prelude.unconditional.test-suite
            "tasty"
            (   prelude.defaults.TestSuite
              ⫽ { type =
                    types.TestType.exitcode-stdio
                    { main-is = "Dhall/Eta/Test/Main.hs" }
                , build-depends =
                      [ deps.base
                      , deps.dhall
                      , deps.dhall-eta
                      , deps.directory
                      , deps.filepath
                      , deps.tasty
                      , deps.text
                      , deps.transformers
                      ]
                    # [ any "dhall-eta"
                      , dep "tasty-hunit" "0.9.2" "0.11"]
                , hs-source-dirs =
                    [ "src/test/eta" ]
                , other-modules =
                    [ "Dhall.Eta.Test.Common"
                    , "Dhall.Eta.Test.Import"
                    , "Dhall.Eta.Test.Normalization"
                    , "Dhall.Eta.Test.Parser"
                    , "Dhall.Eta.Test.TypeCheck"
                    ]
                }
            )
          ]
      }

Attaching the equivalent cabal file:
dhall-eta.cabal.txt

jneira commented

Also, perhaps we can use the Ivy notation for managing version bounds - it's compact and avoids the awkward && and comparison operations. (i.e. "[4.5, 5)" would mean >= 4.5 && <= 5.

I think we could change dhall-to-etlas to use that notation in addition to existing ones, cause versions already are opaque Strings and it will continue being valid dhall

  • 👍1
rahulmutt commented

The improvement with dependencies is very nice.

  1. Make the project bit a bit more concise.

    let project =
      prelude.utils.GitHubTag-project
      { owner = "eta-lang", repo = "dhall-eta", version = "1.0.0" }

in    project
    ⫽ { synopsis =

    to

    defaultProject { owner = "eta-lang", repo = "dhall-eta", version = "1.0.0",
                 synopsis = ...

    where defaultProject is a function that takes a record and does the equivalent behavior. Maybe a built-in?

  2. Make components a tad more concise.

          , executables =
          [ prelude.unconditional.executable
            "dhall-eta-all"
            (   prelude.defaults.Executable
              ⫽ { build-depends =
                    [ deps.base, any "dhall-eta" ]
                , hs-source-dirs =
                    [ "examples/src/main/eta" ]
                , main-is =
                    "Main.hs"
                }
            )
          ]

    to

          , executables =
          [ defaultExecutable
            "dhall-eta-all"
            { build-depends =
                    [ deps.base, any "dhall-eta" ]
             , hs-source-dirs =
                    [ "examples/src/main/eta" ]
             , main-is =
                    "Main.hs"
              }
          )
        ]

    where defaultExecutable is again a built-in. And a similar change for library and test components as well.

The names defaultProject and defaultExecutable don't really matter - make them what you wish. The basic idea is to factor out the common stuff and make it more streamlined.

Everything else looks good, now we have to decide how to handle built-ins. @jneira What's your plan on that front?

jneira commented

Yeah, i was thinking in make shorter that ones too, but you cant pass arbitrary records to a function, so you have to choose the set of fields to be filled... we can choose only the required ones or something, i'll take a look.
There is a open issue that is relevant dhall-lang/dhall-lang#382

I am currently adding the ivy notation to dhall-to-etlas

jneira commented

Ivy notation already added so the dependencies would be

let dep = prelude.Dependency.singleInterval
.....
build-depends =
                    [ deps.base
                    , deps.bytestring
                    , deps.containers
                    , deps.contravariant
                    , deps.cryptonite
                    , deps.dhall
                    , deps.eta-java-interop
                    , deps.megaparsec
                    , deps.memory
                    , deps.scientific
                    , deps.serialise
                    , deps.text
                    , deps.transformers
                    ]
                  # [ dep "dotgen" "[0.4.2,0.5)"
                    , dep "lens-family-core" "[1.0.0,1.3)"
                    , dep "prettyprinter" "[1.2.0.1,1.3)"
                    ]

I've moved dependencies.dhall to etlas-index, so it would be auto downloaded with etlas update. The next step would be change etlas to replace any remote import referring any repo from ~/etlas/config with local paths inside ~/etlas (emitting warnings)

  • 👍1
rahulmutt commented

Sounds good. Will you be submitting a PR to etlas-index with dependencies.dhall?

jneira commented

@rahulmutt submitted typelead/etlas-index#22!

Otoh i've made some progress in simplify the project and components: i've chosen the basic fields for build etlas project as a subset of actual cabal BuildInfo:

{ build-depends :
    List ./Dependency.dhall
, compiler-options :
    ./CompilerOptions.dhall
, hs-source-dirs :
    List Text
, java-sources :
    List Text
, maven-depends :
    List Text
, other-modules :
    List Text
}

It focuses int the fields more used by etlas projects but if you want to use another one you will have to switch to complete BuildInfo

With this, the etlas.dhall file is close to the equivalent .cabalfile:

let prelude =
      https://raw.githubusercontent.com/eta-lang/dhall-to-etlas/etlas/dhall/prelude.dhall

let types =
      https://raw.githubusercontent.com/eta-lang/dhall-to-etlas/etlas/dhall/types.dhall 

let deps =
      https://raw.githubusercontent.com/jneira/etlas-index/dhall-deps/dhall/dependencies.dhall 

let v = prelude.v

let dep = prelude.Dependency.singleInterval

let any = prelude.Dependency.any

let comp = prelude.utils.simpleComponent

in  prelude.utils.GitHubTag-simple-project
    { repo-owner =
        "eta-lang"
    , name =
        "dhall-eta"
    , version =
        "1.0.0"
    , synopsis =
        "dhall-eta is a eta library that wraps the haskell implementation of dhall configuration language."
    , category =
        "Language"
    , maintainer =
        "atreyu.bbb@gmail.com"
    , author =
        "Javier Neira Sánchez <atreyu.bbb@gmail.com>"
    , extra-source-files =
        [ "build.gradle"
        , .....................................
        ]
    , license =
        types.License.BSD3 {=}
    , license-files =
        [ "LICENSE" ]
    , library =
        comp.library
        (   prelude.defaults.SimpleBuildInfo
          ⫽ { build-depends =
                  [ deps.base
                  , deps.bytestring
                  , deps.containers
                  , deps.contravariant
                  , deps.cryptonite
                  , deps.dhall
                  , deps.eta-java-interop
                  , deps.megaparsec
                  , deps.memory
                  , deps.scientific
                  , deps.serialise
                  , deps.text
                  , deps.transformers
                  ]
                # [ dep "dotgen" "[0.4.2,0.5)"
                  , dep "lens-family-core" "[1.0.0,1.3)"
                  , dep "prettyprinter" "[1.2.0.1,1.3)"
                  ]
            , exposed-modules =
                [ "Dhall.Eta"
                , "Dhall.Eta.Binary"
                , "Dhall.Eta.Context"
                , "Dhall.Eta.Core"
                , "Dhall.Eta.Core.Java"
                , "Dhall.Eta.Import"
                , "Dhall.Eta.Parser"
                , "Dhall.Eta.Parser.Java"
                , "Dhall.Eta.TypeCheck"
                , "Dhall.Eta.TypeCheck.Java"
                , "Eta.Types"
                ]
            , hs-source-dirs =
                [ "src/main/eta" ]
            , java-sources =
                [ "@classes.java" ]
            , other-modules =
                [ "Dhall.Eta.Map" ]
            }
        )
    , executables =
        [ comp.executable
          (   prelude.defaults.SimpleBuildInfo
            ⫽ { name =
                  "dhall-eta-all"
              , build-depends =
                  [ deps.base, any "dhall-eta" ]
              , hs-source-dirs =
                  [ "examples/src/main/eta" ]
              , main-is =
                  "Main.hs"
              }
          )
        ]
    , test-suites =
        [ comp.test-suite
          (   prelude.defaults.SimpleBuildInfo
            ⫽ { name =
                  "tasty"
              , main-is =
                  "Dhall/Eta/Test/Main.hs"
              , build-depends =
                    [ deps.base
                    , deps.dhall
                    , deps.directory
                    , deps.filepath
                    , deps.tasty
                    , deps.text
                    , deps.transformers
                    ]
                  # [ any "dhall-eta", dep "tasty-hunit" "[0.9.2,0.11)" ]
              , hs-source-dirs =
                  [ "src/test/eta" ]
              , other-modules =
                  [ "Dhall.Eta.Test.Common"
                  , "Dhall.Eta.Test.Import"
                  , "Dhall.Eta.Test.Normalization"
                  , "Dhall.Eta.Test.Parser"
                  , "Dhall.Eta.Test.TypeCheck"
                  ]
              }
          )
        ]
    }
rahulmutt commented

@jneira This looks beautiful now, thanks! Can you commit these changes into etlas? And also somehow lock-in prelude and types for a particular version of Etlas.

jneira commented

@rahulmutt by default it would use the last version of dhall-to-etlas (1.4.0.0) instead master.
I would like to fix the existent tests and make some new ones before doing a pr.

jneira commented

@rahulmutt also, do you think that will be interesting to let users use directly dhall-to-etlaand etlas-to-dhall? We could:

  • Upload executables in the project and point out in docs
  • Download auto executables in the etlas update execution
rahulmutt commented

@jneira Perhaps we can add some etlas commands instead that can perform the conversion taking an input file/output file?

jneira commented

oh, yeah, using etlas project, no need to download it. Maybe i will upload the executables, just in case.

jneira commented

I've updated dhall support for 1.26.1 of dhall-to-etlas and etlas: typelead/etlas#108