Rethink key principles again

[artemyarulin]

Although current approach works rather well (absolutely custom cljs_rn_module is a good proof of that) we still have couple of issues which we should address:

• Figwheel approach isn's possible (and any other tools for live reload) as source files are copied to the other destination and no in place editing is possible (no auto-reload then). One approach is described #19
• buck run [target]-repl blocks other Buck tasks from running (plus sometimes it leaves running REPL behind after closing)
• Test targets should support additional modules and resources. Issue #21 (it's possible by creating new separate target)
• As Buck encourages to have many small independent modules REPL development became harder: Running and switching to different REPL session, for different modules is not convenient.

Let's take some hammock and solve all of those

[artemyarulin]

• Checked buck project, what it does - it simply generate (in place, meaning in a same folder with source) project files which you can open afterwards in your favourite IDE. Idea is nice, but we cannot use it as source file namespaces don't follow the paths, so :require wouldn't work, we need some pre-processing.
• buck-out/gen/[target] has a subfolder [target]__srcs which has links to all source file of the target. Looks exactly what we are looking for but:
  • Only direct source file, no dependencies
  • Internal thing in Buck, not reliable
• Going though the issue about REPL for bazel and though bazel scala repl docs it looks like a usual approach is to build REPL app and then start it manually, which is close to what we have now and in any case has no support of live reload.
• Magic command buck query "filter('clj.?$',labels(srcs,'//...'))" will find all CLJ/CLJS files in the repo which are used in any targets.

[artemyarulin]

Getting back to testing deps: We could solve by making test target a module as well: Meaning test target is just a module with their own dependencies where one of them is a code under tests. Here how it may look:

def clj_cljs_module(ext,project_file,builder,tester,name,src=None,modules=[],main=None,tests=[],test_modules=[],tester_args=[],resources=[],isTest=False):
    src = [name.replace('-','_') + '.' + ext] if src == None else ensure_list(src)
    modules,tests,resources= map(ensure_list,[modules,tests,resources])

    genrule(name = name,
            srcs = src,
            bash = 'mkdir -p $OUT/resources && ' +
                   ('&&'.join(map(lambda d: 'rsync -r $(location ' + d + ') $OUT/resources',resources)) if len(resources) else 'true') + '&&' +
                   ('&&'.join(map(lambda d: 'rsync -r --prune-empty-dirs $(location ' + d + ')/resources/ $OUT/resources',modules)) if len(modules) else 'true') + '&&' +
                   'echo "{name};{type};{main};$SRCDIR;$OUT;" > $OUT/info && '.format(name=name,type=ext,main=main or "") +
                   ('&&'.join(map(lambda d: 'echo "$(location ' + d + ')" >> $OUT/deps',modules)) if len(modules) else 'true') + '&& ' +
                   '$(location {0}) $OUT/info {1}'.format(builder,'test' if isTest else 'build'),
            out = 'build',
            visibility = ['PUBLIC'])

    if tests:
        clj_cljs_module(ext, project_file, builder, tester, '__' + name, src = tests, modules = test_modules, tester_args = tester_args, isTest = True)

    if isTest:
      sh_test(name = name,
              test = tester,
              args = ['$(location :{0})'.format('__' + name)] + ensure_list(tester_args),
              deps = [':__' + name])

[artemyarulin]

REPL implementation idea

Create one buck executable which will accept a list of targets as an argument. After run it will create a new folder (where? In /tmp I guess as buck-out in under control of Buck itself) and sym-link all the source files into that (we can find it using previously mentioned buck query "filter('clj.?$',labels(srcs,'//...'))"). As we cannot simply link it and we have to put it in a right place we pre-process each file by executable reusing the same code from build.cljs. There should be no problem with tests as we don't care if it's in the same src folder with actual source.

Issues

• Project file: It should be configurable, but CLJS config should suit fine as we can test CLJ code without lifting REPL to CLJS
• Resources: Resource basically is an export_file dependency, which means we can go through all dep tree where:
  • If it's a CLJ|CLJS module (= depends on //ext:org.clojure/clojure) then copy sources
  • Otherwise it's a resource - copy it to resources
  • Or actually can we say that everything that is not a *.CLJ.?file is a resource?
• New file | Removed file: we should be able to re-run executable in order to re-sym-link all the files and remove old links
• New deps: Once again re-running executable should update project file as well. After that we can use tools like alembic or clj-refactor for adding new dep into REPL session

Use cases

Considering following example command: buck run //RULES/clj-cljs-config:repl -- //[target]
Using Buck aliases like:

[alias]
  repl     = //RULES/clj-cljs-config:repl

we can simplify it till buck run repl -- //target which is nice. As don't want to block Buck the command itself will only build sym-links and take care about project.clj file creation without running actual REPL. We can (not sure yet) move to the project directory, so that we can do buck run repl -- //[target] && lein repl.

Here 3 use cases how REPL may be used:

• REPL to one module: buck run repl -- //[target]
• REPL to multiple modules: buck run repl -- buck query "//[target1] + //[target2]" or `buck run repl --`buck query "filter([query],//...)"
• REPL to all modules: buck run repl --buck query "//..."``

Configuration

We should follow the same approach like with whole project where configuration comes through the user config. Here home the it may look:

# RULES/clj-cljs-config/BUCK

clj_cljs_repl(name = 'repl', # Will create //RULES/clj-cljs-config:repl target. In case user may want to have multiple REPL profiles
              project_file = ':project-repl', # Just a target path to file, same as with project files
              output_folder = '.repl') # Relative from Buck root or absolute path where REPL folder would be created

Risks

• Does Clojure/ClojureScript works fine with sym-links?
• Does Figwheel live-reload works fine with sym-links?
• Does aliases supported when using buck run?
• How can we delete an existing sym-link?

[artemyarulin]

Answering risks:

Using hard links like ln repo/target/source.clj repl-folder/src/target/soruce.clj and using pooling mechanism for Figwheel like

:figwheel  {:hawk-options {:watcher :polling}}

makes live reloading happen. Soft links ln -s and default FS events based file watcher doesn't work.

Creating aliases works fine for Buck runnable as well.

As we cannot use soft links it means that whenever original file got deleted or renamed our REPL folder would still have a copy of it. The only thing we can do is to actually implement sync process which would go though all the files. Or as a shortcut - delete all the files in src folder and copy all the files into that folder again.

[artemyarulin]

Just to keep in mind: I was thinking that buck run repl --buck query "//..."`` would be the most popular choice for REPL but it wouldn't often work:

• One module could be targeting React Native/Node environment and use js/require in top form which will throw and stop the compilation

[artemyarulin]

Another problem is dependencies, while getting names is easy filter(ext,deps(//[target])) we don't have an access to version numbers.

One workaround could be: cat buck targets "//[target]" --show-output --verbose 0 | awk '{print $2}'/deps, but in this case we suppose that target is built already which could be too big assumption.

Better workaround could be building all the ext_dep files and getting exact version from it like:

buck build "//ext:" && cat buck targets "//ext:" --show-output --verbose 0 | awk '{print $2 "/deps"}'``

[artemyarulin]

OK, it's not possible to track resources with this approach.

We have to find another way. Question, why some files goes to src and others goes to resources?

$> buck query "labels(srcs,deps(//lib/react-native/react-native-externs:))"
# Everything here should be under src, even though it's not CLJ/CLJS files
lib/react-native/react-native-externs/src/core.cljs
lib/react-native/react-native-externs/src/deps.cljs
lib/react-native/react-native-externs/src/react/react.ext.js
lib/react-native/react-native-externs/src/react/react.native.ext.js

$> buck query "labels(srcs,deps(//kapteko/blog:blog))"
# All non CLJ/CLJS files should be under resources
kapteko/blog/blog.cljs
kapteko/blog/articles.clj
kapteko/blog/articles/2016-01-10-10-39:Hello_world!.md

[artemyarulin]

OK, it's not possible to have this approach in general: We are running buck commands (queries) while being inside command (buck run) which not stable with v2016.03.28.01 and doesn't work at all with further versions.

I still want to have buck run repl -- buck query "//..."`` as it's very convenient entry point for REPL tasks. One possible approach could be using the same principle but rather than executing a command (which will fail) we can output the command to execute, so that Buck finishes it's own part of generating a string and then we eval this string outside of Buck. In our example it could be something like that: $(buck run repl -- buck query "//..."`)`

[artemyarulin]

So, it tuns out it's a valid approach, it works fine, couple of comments:

1 As we didn't found a way to distinguish resource from source file we decided to copy non CLJ/CLJS files into both src and resources. One of my targets has index.html which got copied into src folder and it seems that Figwheel searching src folders first for index.html files, which pretty much breaks it.

2 It turns out errors like js/require is not defined or any other runtime errors doesn't cause Figwheel to stop, so it works just fine in this case

3 it's slow - for our whole monorepo it takes 2:34 in order to re-sync changes if "//..." query is used. It could be dramatically improved if we concat multiple queries into one

4 Code it ugly so far

Overall it's good result - finally we have a working approach which needs some polishing.

[artemyarulin]

Closing it as there is nothing left to rethink - yes we have some hardcoded things, some things are slow, but it's all the matter of other issues