Macros and DSLs
Opened this issue · 97 comments
A homoiconic programming language can parse itself to an AST expressed as native data that can be transformed within the language, evaluated, or isomorphically serialized to human readable syntax.
I don’t think the sacrifices in syntactical and grammatical clarity caused by reduction to minimum consistent grammar such as hierarchical lists of lists s-expressions, is necessary to achieve the homoiconicity which enables macros and DSLs. Although Lisp's low-level semantics distills to Seven Primitive Operators, which makes its AST eval simple to implement, the complexity of higher-level semantics which can be expressed (and thus potential semantic complexity of a macro's semantics) is not limited any more so than for any Turing-complete language.
Requirements:
- Self-hosted compiler.
- Bracketed quotes.
- Type inference localized to expressions.
- Everything-as-an-expression syntax so that we can select the parser based on the data type the expression is assigned to.
- Incremental AST compilation whilst parsing so the compiler knows the data type for #4.
For macros we’d only need #1 - 3, because macros take some or all of their arguments as AST references. We wouldn’t want macros to depend on wide-scale inference as this could result in live or dead locks on inference decidability. K.I.S.S.. For DSLs such as a SQL query language, we need to select a customized parser. If we know the data type of the expression being parsed is for example SQL.Query and if that type has associated a customized parser and separately compiler, then that parser is run to create the customized AST of the said expression (and if it is not a macro then the associated compiler is run). There are cases where the expression determines its own type (i.e. the expression has side-effects and it is not assigned or the type of the reference it is assigned to it inferred from the expression's type), so the current parser is presumed. This strategy would not work with any wide-scale type inference, which is yet another reason I am against non-localized type inference.
A customized parser and compiler could invoke another parser and compiler for an expression within its grammar (perhaps using a similar strategy of selecting by target data type or by explicit delimiters, e.g. the use of curly braces in React JSX), and this could even be recursive.
For macros we only need #1 - 3
I don't want macros :-) I don't want a second weaker version of application in a language.
What you are basically saying is that the data format for expressing objects combined with the AST is the language syntax. I am not sure I want this - as the AST currently features things like application nodes which you would not want in the language syntax. I think homoiconicity is over-rated, and I would rather have a nice user friendly syntax without it compromising the AST, and vice versa.
I don't want macros :-)
Many programmers want them. Also DSLs.
a second weaker version of application
Huh?
as the AST currently features things like application nodes
Huh?
I would rather have a nice user friendly syntax without it compromising the AST, and vice versa
Me too. I didn't see any flaw in my proposal for macros which would prevent that. Perhaps you misread the OP.
Macro's are bad and are included in languages because normal function application is not powerful enough. If you want a function write a function.
Making things too powerful is often worse than a proper separation-of-concerns.
I doubt you can achieve the same things that macros and DSLs can achieve without making the type system so powerful and obtusely convoluted (and with mind bending and numbing type inference) that nobody can understand the code any more, not even the person who wrote it. K.I.S.S..
Macro's provide an escape hatch to bypass the type system (in some languages), this is a bad thing. Messing about with the program code itself using macro splicing is a bad idea, just write a function to do the thing in the first place. With generics you should never need a macro anyway.
Macros also make debugging harder as they are applied at compile time. Any errors reported at runtime will be in the post macro processed code, so the line numbers and context cannot be found in the program source. I have yet to see one compelling use of macros in the last 20 years (and this comes from someone that wrote their own macro-assembler and used to think that macros were all you needed to build a compiler and a high level language).
Maybe you can persuade me, if you can provide an example of a macro that cannot be more cleanly implemented in another way?
Macros enable editing the source code of the arguments before they are evaluated. A functional lazy evaluation of those arguments will not enable the transformation of their source code. Macros can thus perform some automated operations based on the structure of the source code within its arguments, not just the evaluated value of the arguments. That is turtles all the way down recursively.
For example, if we have list of functions we want to call with different numbers of arguments, they each have different types, and we want to add new common argument to the end of each of these calls, and collect the results in a list. Much easier to call a macro to do that, as it doesn't need to fight with the type system, as the type checking will be performed after the macro has done its code transformation. This example could also be done with partial application without a macro, except the typing can still get unwieldy, because those functions might have different type signatures for remaining default arguments or we have to manually specify the default arguments. Macros are basically a way the programmer can rescue himself from the fact that the language development moves too slowly and can't always do everything that you and I could imagine it might do one day in the distant future.
Macros are basically useful any where you need to do transpiling, so you don't have to write your own whole program compiler. And there have been many times I wanted to transform code, e.g. to convert the if as an expression to a lamba function compatible with TypeScript. If TypeScript had powerful macros, I might already be done with some of the features I wanted for my transpiler! Instead I will have to reinvent the wheel.
Also much of the functionality needed for macros is also needed for evaluating and running code sent over the wire, i.e. dynamic code compilation and modularity. We can't expect to link everything statically in this era of the Internet.
DSLs enable friendlier syntax such as SQL queries that are checked grammatically by the compiler.
Edit: a possible new idea for the utility of AST macros: #11 (comment)
Macros enable editing the source code of the arguments before they are evaluated.
I know what macro's are, I am saying there is always a better way to do the same thing, without having source-to-source rewriting going on in the program code, which obfuscates errors and results in another case of write-only-code.
there is always a better way to do the same thing
Someday. But someday can be too many years (even decades) too late. Refresh.
I have not written a macro for about 20 years, so that someday was decades ago :-)
I don't limit my analysis to only your anecdotal limited needs. I’m interested in mine and those of others.
I think DSLs are the most important thing that can't be done reasonably without transforming code. I explained to enable the "macros" (parser + compiler transformation of code) for those based on detecting the type of the expression.
Show me how to write a SQL query DSL (without any noise in the syntax that isn't in the normal SQL syntax!) that is grammatically correct at compile (evaluation) time, that can be done without code transformation. Scala can sort of finagle it by use infix function names as keywords, but that is not compile-time parsing of the SQL specific syntax issues.
You do not need macro's to create DSLs.
Show me how to write a SQL query DSL
Sure I did it in Haskell back in 2004 :-)
Sure I did it in Haskell back in 2004 :-)
I am not interested in your obtuse HList gobbledygook.
A DSL should be elegant and easy to comprehend and work correctly. I asked you to show me how to do this, not asking you to brag.
What's obtuse about this:
moveContaminatedAnimal :: DAnimalType -> DCntdType -> DFarmId -> Query ()
moveContaminatedAnimal animalType contaminationType newLocation = do
a1 <- table animalTable
a2 <- restrict a1 (\r -> r!AnimalType `SQL.eq` animalType)
c1 <- table contaminatedTable
c2 <- restrict c1 (\r -> r!CntdType `SQL.eq` contaminationType)
j1 <- join SqlInnerJoin a2 c2 (\r -> r!AnimalId `SQL.eq` r!CntdAnimal)
j2 <- project j1 (CntdAnimal .*. HNil)
doUpdate animalTable
(\_ -> AnimalLocation .=. toSqlType newLocation .*. HNil)
(\r -> r!AnimalId `SQL.elem` relation j2)That doesn't look like unadulterated SQL to me! And you are filling up my thread with (obtuse Haskell and braggart) noise!
You have no chance in hell of creating anything popular...
...if you think that gobbledygook is more readable than SQL syntax for the person who wants to express SQL.
That's relational algebra :-)
Here's an example:
R1 = join(CSG,SNAP)
R2 = select(CDH,Day="Monday" and Hour="Noon")
R3 = join(R1,R2)
R4 = select(R3,Name="Amy")
R5 = join(R4,CR)
R6 = project(R5,Room)
Taken from here: https://www.cs.rochester.edu/~nelson/courses/csc_173/relations/algebra.html
You have no chance in hell of creating anything popular.
C# LINQ seems pretty popular and its based on exactly the same ideas.
That's relational algebra :-)
It's not elegant. It's gobbledygook.
I don't care! Its not elegant.
Its very elegant. Relational algebra avoids several problems that SQL has as a language. SQL is not a proper algebra, its got a lot of ad-hoc elements that are pretty bad.
It's not SQL.
I asked you to show me how to do unadulterated SQL without a code transformation.
Its better then SQL. Why would I recreate a flawed query model along with all its problems. I took the opportunity to so something better.
In any case you are not going to be able to parse SQL syntax in the host language with macro's either because you cannot represent SQL in the languages AST.
Whether SQL is good or not is not germane to the point of this thread, which is whether it is possible to model the unadulterated syntax of another language without a code transformation. Yes or no?
In any case you are not going to be able to do that with macro's either because you cannot represent SQL in the languages AST
Incorrect. Re-read the OP.
No, and its not possible to model it with code transformation either.
its not possible to model it with code transformation either.
Incorrect.
prove it :-)
prove it :-)
Re-read the OP. It is explained there.
That's not a proof, I want code :-)
Its not possible without custom parsers that can be implemented for new types of literals.
Its not possible without custom parsers that can be implemented for new types of literals.
Which is exactly what I wrote in the OP.
Back on examples where macros can’t be done with functions, there is no way to write a function will execute its argument expressions in the lexical scope of the caller and provide the name of the variables as a separate argument. There is no way to form that functional closure with the replacement lexical binding without a code transformation.
Paul Graham’s anaphoric macro eliminates the boilerplate of lambdas. It couldn’t be done without boilerplate nor macros with Scala _ shorthand for lambdas that take only one argument if the it is within another lambda.
Another example is when you want to declare numerous types and/or functions (in the lexical scope of the caller) based on some pattern (that would be an absolute pita to manually enter from the keyboard). A function can’t declare new types and functions. You can only do this with code generation.
There are instances where only code transformation will suffice.
there is no way to write a function will execute its argument expressions in the lexical scope of the caller and provide the name of the variables as a separate argument.
And why would you ever want to do that?
There is no way to form that functional closure with the replacement lexical binding without a code transformation.
Which is again the kind of trick I don't want to allow. If I wanted that kind of thing I could provide first class environments, but its just makes code harder to understand for no real reason.
What algorithm do you need the above to implement? Does it help me implement a sort function?
Another example is when you want to declare numerous types and/or functions (in the lexical scope of the caller) based on some pattern
Needing to do this is a symptom of a lack of sufficient abstractive power in the language itself. The 'numerous functions' should be a single generic function.
Generics remove the need for macros in the language by providing a type safe way to abstract code over parameters.
And why would you ever want to do that?
What algorithm do you need the above to implement?
To eliminate repetitive boilerplate.
The 'numerous functions' should be a single generic function.
That doesn’t apply to function calls and data instance declarations. To eliminate repetitive boilerplate.
I agree that macros create a new syntax, which leads to less readability (i.e. less people who readily understand the syntax). But it is not for me to decide when and where the scenarios dictate the value of their usage. Note however though that typing systems and function application can become so complex also to reason about in order to facilitate certain coding patterns (e.g. that Scala and Haskell gobbledygook), that it is same effect of essentially a new syntax (although I understand that tools can perhaps understand and perhaps aid, but not the higher-level semantics). Nevertheless, code transformation macros are a lower priority for me than DSLs. DSLs are also code transformation but they employ a custom parser and AST (before they’re transpiled); whereas, macros (as I’ve envisioned them) reuse the AST and parser of the main language.
And as for DSLs such as SQL have in some cases very wide appeal and readability.
Any errors reported at runtime will be in the post macro processed code, so the line numbers and context cannot be found in the program source.
This is not a problem that can’t be rectified with a @pragma in the generated code.
C# LINQ seems pretty popular and its based on exactly the same ideas.
It is not germane to this thread whether someone can invent an API and/or new syntax which is popular among some programmers. Rather it is germane whether we can also offer for example unadulterated SQL syntax. Nothwithstanding that SQL is afaik much more widely known (and thus readable) than LINQ.
and put an arrogant smirky face 😏 on it to rub it in the wound you've opened
I can see you interpret that differently, from "friendly disagreement".
I just wish you’d organize yourself better and not be so flippant.
I'm not writing a formal journal paper. This is a friendly informal discussion (at least i think it is), with the only restriction being to try and keep on topic. I think my position on macros is fairly clear, and i had assumed that because you had not yet mentioned macros in any of the other discussions, that this was not something you felt was really core to the language. I had previously posted a manifesto for the language that did not get big objections, so i thought we were at least interested in the same kind of things, buy were having difficulties settling on syntax, which is understandable because i an not even sure what i want for syntax in all areas, there are so many compromises. Macros go against the principles of readability and directness in such a fundamental way, i was shocked that you brought them up, out of the blue at this late stage.
That doesn’t apply to function calls and data instance declarations. To eliminate repetitive boilerplate.
That's what generic modules are for.
But it is not for me to decide when and where the scenarios dictate the value of their usage.
It is when creating a new language. Python does not have macros, does that seem to limit the libraries and applications available?
Macros go against the principles of readability and directness in such a fundamental way, i was shocked that you brought them up, out of the blue at this late stage.
We actually briefly mentioned macros and the problems they create for readability in our past discussions on this issue threads, and I had agreed.
I am raising the issue of macros and DSLs now because I want to have a holistic analysis on the extent and manner of integrating other syntax, such as a markup language(s). This is relevant because React's JSX is very relevant to the JavaScript ecosystem right now. I received a private message this past week from a developer who is interested to code apps for my blockchain project, and he cited several JS frameworks he uses, one of which is React-like and employs JSX.
I can see you interpret that differently, from "friendly disagreement".
I could definitely detect your “shocked” as it hit me as defiance, combativeness, and a touch of arrogance.
Btw, I am not trying to influence (other readers or even you on) the direction of your project. I post in your project's current discussion forum (the issues threads), because it is the only place I can currently get peer review (predominantly from you). And as a continuation of all the discussion we did before, to try to have all my latest understandings/proposals/conclusions in one place for now (for a lack of a better place). I highly doubt that we will be working on the same language going forward. We'll go off on separate directions due to different priorities, and if one of makes vastly superior progress (especially in terms of adoption), then perhaps the other will abandon/merge his fork. (e.g. you want to explore typing research such as improved row polymorphism and I don't have time for that right now)
Besides, neither of us is going to get any community momentum until we actually produce a tool that can be used (especially for real projects).
I have no problem with you stating your stance unequivocally and frankly, but I don't think it is necessary to make it so personally combative. I think it is possible to be cordial and help others feel good about working with you.
For example, you could have started your first post, “Frankly I am a bit shocked and maybe even a little bit put off that you would raise the issue of macros after we expended many months on what I thought was core concept of employing genericity, typing, and God smacked type inference to perfect static typing even though nobody else in the entire world has ever been able to figure out how to do so without gobbledygook and avoid needing to break out of the typing system. But I am a God and Superman so just listen to me and STFU”.
At least then I could have just laughed, because you would have been honest.
If there is one thing that really turns me off, that is (incorrect!) brow beating and an ambiance of "holier than thou" stench. As I said, unless the other side is off the rails, then we'd need to be absolutely certain in order to be justified to be brow beating the other about some technical nirvana. I don't think there is any absolute technical nirvana. When technology becomes a religion, I am gone.
It just felt like you were trying to say I was completely wrong and silly for making the thread. And I think you've just admitted that is what you felt. So at least you are being honest and I am too.
Bouncing ideas off colleagues can either be bright, airy ambiance of an open-minded discussion or it can be a dark, bloody, depressing religious holy war.
I am somewhat interested in exploring better static typing. But with trepidation.
I am also contemplating the likelihood that by being more cautious about adding extensively expressive strict (as contrasted with TypeScript's sometime heuristic, e.g. subtyping variance) static typing, I am probably more aligned with the philosophy of JavaScript's ecosystem than someone who wants a better Haskell + C = Rust and is contemplating Rust + PureScript = ZenScript formula. But neither of us knows yet what the sweet spot should be for the next great move forward in mainstream programming languages. I am trying to find my way forward by keeping my eyes and ears open as I navigate the design process simultaneous with an urgency to complete a real world project. I am trying to keep up also with your (and @skaller's) far ranging research level design interests, but to some extent I can't entirely do that while also doing the former.
That's what generic modules are for.
The devil is in the details of that. The level of complexity that will be required to express most of the variants that could be coded with a macro and still you will NEVER be able to express all possibilities. NEVER.
There is more than one perspective or way to attack a problem set.
From my perspective, it is better to have a discussion about the tradeoffs of something, so that those on either side of the issue can refer to the enumeration of the issues. That is superior to "STFU and go away" or any demeanor or ambiance which contributes to demoralizing those who just want to have a open-minded inspection and thus understanding.
Your perfect static typing heaven might be another person's hell. I don't even know how all these extensive typing ideas of yours will work out. All I know is you can't have a total order on typing and thus there will ALWAYS be cases where the programmer needs to break out of the typing system. ALWAYS. Mark my word.
A typing system is at best a set of hints and partial checks on correctness. It is never completely consistent. There will always be cases where one has to do something outside the statically checked type system.
Don't forget Godel incompleteness theorems, that any complete system of axioms is inherently inconsistent. Total orders don't exist.
In short, I think maybe you are chasing perfect rainbows. You probably thought macro-assemblers were the holy grail 20+ years ago, then Haskell and pure functions in the prior decade, and now on to the next failed attempt to attain absolute nirvana...
...I am just trying to understand attributes of paradigms and relate to use cases. And understand what programmers want. Understand what is popular and why. Etc...
And mostly I am just trying to understand what if anything can accelerate and help my current programming needs. And integrating markup language is probably one of those high priority items for me, so I wanted to summarize how it fits into holistically with DSLs (and macros relate to DSLs).
Python does not have macros
https://www.quora.com/Does-Python-have-macros-If-so-how-do-they-work
http://stackoverflow.com/questions/15669921/adding-macros-to-python
and i had assumed that because you had not yet mentioned macros in any of the other discussions, that this was not something you felt was really core to the language
Again I didn't realize in the past that for example type inference might interfere with how to best integrate a markup language. That is the next issue I want to discuss in this thread.
The prior discussion has caused me to more fully understand the duality of a total order of compile-time typing versus the partial orders of run-time typing.
But I am a God and Superman so just listen to me and STFU”.
I would never claim that. I know i am just as likely to be wrong as the next man. Everything is my current opinion, and could change at any time.
you will NEVER be able to express all possibilities. NEVER
You already can, that's what Turing completeness is about. We already know you can write any program that could ever be written in a Turing complete language. We are discussing differences in readability, structure and style here. Nothing s as black and white as you suggest.
In short, I think maybe you are chasing perfect rainbows. You probably thought macro-assemblers were the holy grail 20+ years ago, then Haskell and pure functions in the prior decade, and now on to the next failed attempt to attain absolute nirvana
I don't think so, because i have spent that time actually writing code in those paradigms. As such I have already been writing programs using the model i am proposing, although through a layer of annoying boilerplate. Even HList stuff is elegant if you don't need the boilerplate you need in Haskell. My interest in language design started after the HList paper when i saw the amount of boilerplate Haskell needed, and things became clearer for me after reading Stepanov and seeing the amount of boilerplate C++ needed to do things. To do HList stuff in C++ required two layers of extensive boilerplate, one at the Haskell level of abstraction, and one at the C++ level of abstraction. What i have been trying to achieve hasn't really changed since the OOHaskell paper.
so is static typing (e.g. lower-level dependent typing divide-by-zero or higher-level semantic errors).
You can use static refinement types, so that the type of a division is:
div : (int, x : int | x != 0) -> Int
Now of course the question is how do you turn an int into an int | x != 0, and the answer is with path dependent types like this:
if x != 0 then
// x has type (int | x != 0)
else
// x has type (int | x == 0)
So using the div function forces you to add a zero test, but only one zero test, so type inference and type propagation mean you can force the caller of a function to do the test before calling the function.
So using the div function forces you to add a zero test
Yes of course I also wrote about that 7 years ago for my Copute language research, but the point I made upthread is that is becomes too noisy, so instead we prefer run-time typing and generate an exception on divide-by-zero or overflow conditions. There is a trade-off consideration, and static typing is not always the chosen optimum.
It is important to relate this to the next comment from me about unbounded nondeterminism. If we do model all divide-by-zero and overflows as types, then we have to figure out how to maintain a meaningful semantics with those errors, otherwise they are still exceptions. And exception is something that puts the program in an indeterminate state.
You already can, that's what Turing completeness is about.
Incorrect.
Bounded nondeterministic Turing-completeness can’t model unbounded non-determinism (aka Hewitt’s indeterminism), i.e. the unbounded entropy of permutations in the universe. I had already explained that to you in the Concurrency issue thread and I provided a link to Carl Hewitt’s video where he explained that.
The point is what we can not model all of unbounded permutations of composition. Sometimes we will have to punt and accept run-time exceptions (where the program is in an indeterminate state outside of the safety of the static type system). We can not statically type everything.
Additionally, expression should not violate SPOT (single-point-of-truth) and solve the Expression Problem and otherwise not require factoring. To state we can express something in a non-SPOT, refactoring hell, is not that germane to PL design.
In short, I think maybe you are chasing perfect rainbows. You probably thought macro-assemblers were the holy grail 20+ years ago, then Haskell and pure functions in the prior decade, and now on to the next failed attempt to attain absolute nirvana
I don't think so, because i have spent that time actually writing code in those paradigms. As such I have already been writing programs using the model i am proposing, although through a layer of annoying boilerplate. Even HList stuff is elegant if you don't need the boilerplate you need in Haskell. My interest in language design started after the HList paper when i saw the amount of boilerplate Haskell needed, and things became clearer for me after reading Stepanov and seeing the amount of boilerplate C++ needed to do things. To do HList stuff in C++ required two layers of extensive boilerplate, one at the Haskell level of abstraction, and one at the C++ level of abstraction. What i have been trying to achieve hasn't really changed since the OOHaskell paper.
And this is apparently why complete type inference appears to be such a critically important priority to you.
I think I am onboard the benefits of typeclasses, but I still need to understand fully the ramifications of higher-ranked callbacks and typeclasses.
I still need to try to quantify better at what high-level of semantics we might need (or prefer) dynamic typing. At the low-level, I think the programming and hardware design community long ago decided to prefer runtime typing and thus exceptions for divide-by-zero and overflow errors.
What's obtuse about this:
moveContaminatedAnimal :: DAnimalType -> DCntdType -> DFarmId -> Query () moveContaminatedAnimal animalType contaminationType newLocation = do a1 <- table animalTable a2 <- restrict a1 (\r -> r!AnimalType `SQL.eq` animalType) c1 <- table contaminatedTable c2 <- restrict c1 (\r -> r!CntdType `SQL.eq` contaminationType) j1 <- join SqlInnerJoin a2 c2 (\r -> r!AnimalId `SQL.eq` r!CntdAnimal) j2 <- project j1 (CntdAnimal .*. HNil) doUpdate animalTable (\_ -> AnimalLocation .=. toSqlType newLocation .*. HNil) (\r -> r!AnimalId `SQL.elem` relation j2)
It could still be made more elegant with a fully integrated DSL syntax transpiled seamlessly, which is what I proposed in the OP.
C# LINQ seems pretty popular and its based on exactly the same ideas.
LINQ has a DSL query syntax (at least in C#).
It could still be made more elegant with a fully integrated DSL syntax transpiled seamlessly, which is what I proposed in the OP.
When you go through the design permutations, i think you will find its hard to improve on. For example the semantics of equality are different in SQL from the host language, so we do not want to use the same symbol (although we could due to type-classes). Defining the restrictions and joins as functions on row parameters is more powerful than simply listing the columns (which again we could do). What you see is deliberately based on the mathematical notation for relational algebra. It was probably a bad example to give when you wanted SQL, but when i was creating a DSL for databases, i came up with this, and i think its better than SQL. It's also more powerful than LINQ.
The point is if we can define functions and operators, we can implement any DSL. We can even avoid the limitations on the use of functions and operators by having limited scopes.
I would argue the one thing you don't want to do in a DSL is change the fundamental language syntax, so function application and function definition should look the same everywhere. A domain specific language is not about embedding SQL into any other language, it is about creating a domain specific language within another language. As such the embedding of SQL in Haskell would look different to the embedding if SQL in C, etc.
What i want from a DSL is to preserve the overall flavour of the host language (so i don't get surprised by the syntax of function application suddenly changing) but that allows me to operate efficiently in a particular problem domain. A DSL should make it feel like the library/extension DSL was part of the original language design. So a database query DSL in Haskell should still look and feel like Haskell, but it should be as if relational queries were built into the language from the start, like for example sequential IO was.
Bounded nondeterministic Turing-completeness can’t model unbounded non-determinism
Actually it can, you just need a random number source. Just like we can model a quantum computer in a non-quantum computer.
For example i can put two packets into an ordered list sorted by a transport delay, where that delay is chosen randomly (and here we need a proper random number created from entropy sources like the keyboard, disk other IO timings). Then the recipient can pull the messages in 'simulated arrival order'. Using this we can model a complete actor system, complete with non-determinism on a single, non-parallel, classical computer running a simple sequential language like 'C'.
yo shelby i signed up here just to reach you and ask you where are you bringing your BitNet forum? damn they banned even the BitNet (official) account. sorry for the off-topic guys i really need to reach shelby i dont know where to look.
Actually it can, you just need a random number source.
Sorry but that is incorrect. Unbounded non-determinism means your program is in an indeterminate state. So to model it, you have to execute code instructions in random order, i.e. no coherent model.
What you mean to say is your program can interact through its I/O with the unbounded nondeterminism of the universe. Interaction is not a model. A program is a partial order. It is impossible to model the total order of the universe because to do so would mean everything is predictable, thus statically known and the past and future collapse into indistinguishable in time order.
And science doesn’t understand well quantum computers. We need to dive into relativistic quantum mechanics, which is not yet widely accepted. I am composing a blog about this and realistic time travel. The speed-of-light is just an imaginary limitation because we choose to retain our perspective of mass and forces, but information (i.e. Shannon entropy) is not limited to such a deterministic bound. Time is only a relativistic metric between mutual referential perspectives. For disjoint perspectives, there is no mutual information of elapsed time. Note maximum entropy (uncertainty aka uniformity of distribution of probability of possible outcomes) would be with maximum disjointness of perspectives, i.e. lack of mutual dependencies, information, and order.
Using this we can model a complete actor system, complete with non-determinism on a single, non-parallel, classical computer running a simple sequential language like 'C'.
Modeling an Actor system doesn’t model the unbounded nondeterminism of the universe, rather the interaction with the (bounded?) entropy of your random number entropy source. It is impossible to model the entropy of the universe, lest if you were such an omniscient God then you could time travel at-will to any place and time that every existed and will ever exist in the universe because the universe is statically known.
We can model quantum computers on normal computers. IBM has an API and a quantum simulator that runs on a normal CPU.
See:
http://research.ibm.com/ibm-q/qx/
https://github.com/corbett/QuantumComputing
IBM has a model of something but it isn’t a model of universal unbounded nondeterminism.
The definition of what is a quantum computer comes into play. Science doesn’t even agree or understand this well yet. As I said, relavistic quantum mechanics is rejected by most mainstream physicists at this time.
We are getting too far off topic.
You may as well say we don't understand a magic computer... We can only go with what is proven. In my opinion computers are only useful because they are deterministic. I really don't want to worry about my car not working because the engine-management computer is having a bad day and just split up with the entertainment system which it had been dating for a while...
You may as well say we don't understand a magic computer... We can only go with what is proven. In my opinion computers are only useful because they are deterministic.
So we don’t get too far off topic, please note the context of the limitations of static typing in which I brought up the point of unbounded nondeterminism:
The point is what we can not model all of unbounded permutations of composition. Sometimes we will have to punt and accept run-time exceptions (where the program is in an indeterminate state outside of the safety of the static type system). We can not statically type everything.
The prior discussion has caused me to more fully understand the duality of a total order of compile-time typing versus the partial orders of run-time typing.
Whether we can statically type everything depends on the type system. Dependent types can type everything as they can depend on values - however to my mind this removes the benefit of static types - which is we can check for correctness at compile time.
There is a tension between what you type statically and the degrees-of-freedom to accomplish composition without refactoring. The refactoring and typing complexity can become so onerous and the project so large, that everything grinds to a halt in rigor mortis.
Static typing can help us enforce some invariants but we should use some discretion and no get too ridiculous and try to type everything due to the downside stated.
When static typing is helping document it is very useful. When the types become so complex that no one can even comprehend them, the utility has been lost.
Note the conclusion I wrote in another thread:
Regarding recent upthread discussion of JSON vs. XML which began far upthread, after further thought I agree it would be better to model for example HTML with the full generality our programming language such as actual data types (e.g.
Div()for<div>); and maybe even we will eventually develop a better DOM (personally I’d like an improved markdown for text and using program language data types for an improved DOM). In terms of a textual serialization format, the advantage of JSON is a simplistic least common denominator with ad hoc sematics in order to simplify usage in varied environments. I would say we not try to improve on JSON for that use case. I think we should however consider having our language itself be a textual serialization format, if we think it is going to be that ubiquitous and if we’ve fixed the script injection attack holes. As for security, the code which is compiling and running code dynamically (i.e. at runtime) need only restrict the imports of libraries to cordon off the access permissions of the dynamically run code.
EDIT: @keean raising this point again in the Mutable Objects issues thread #32.
Another viewpoint:
Actually, the whole point of Lisp is that there is not one conceptual barrier to development, or a few, or even a lot, but that there are arbitrarily_many, and that is why you need be able to extend your language through syntactic_abstraction to build DSLs so that every abstraction layer can be written in a language that is fit that that layer. [Actually, traditional Lisp is missing the fact that DSL tooling depends on restriction as well as extension; but Haskell types and Racket languages show the way forward in this respect.]
That is why all languages without macros, even with AMM, remain “blub” to those who grok Lisp. Even in Go, they reinvent macros, just very badly, with various preprocessors to cope with the otherwise very low abstraction ceiling.
(Incidentally, I wouldn’t say that Rust has no AMM; instead it has static AMM. It also has some support for macros.)
Macro's are evil :-) proper generics are better. DSLs are great and probably where you want to be for abstraction, but you don't need macros to define a DSL. Operator overloading (using type-classes), parametric polymorphism, and type classes are all you need.
Well consider that any DSL is constructing an abstract syntax representation of the language, and then evaluating that syntax. Also consider that sensible operator reuse preserves the critical properties of the operators. For example '+' should always be associative and commutative (at least approximately - floating points break this rule, but they only give approximate answers anyway - the real number mathematics they approximate is associative and commutative).
So we can overload operators and define functions that build an abstract syntax representation, evaluation of the abstract syntax would be specific to the embedded language.
with Macros I rearrange the forms and create another form, which is what the compiler will find to work with
One of the arguments against macros is that it creates unreadable code. Code is easier to read when it follows a consistent regularity of syntax. When we have possibly 10s, 100s, or 1000s of new syntax in a program then the person who is new to the code cannot read it without learning all those new syntax.
IOW, regularity is good for readability.
Why not Clojure (or any LISP)?
@analyticbastard is a Clojure expert. Clojure is a variant of LISP which normally runs on the JVM (Java Virtual Machine). He tells me that he loves homoiconicity. I explained that homoiconicity is very good for the programmer who creates the code because homoiconicity allows them to create private syntaxes and macros which enable their code to do magical jumps through circus hoops and do more with less code.
But as we discussed up-thread, it is not so good for the readers of the code who have to try to learn all these new private syntaxes invented for each code base. IOW, it lacks the regularity that I wrote about in my prior comment on Apr 13.
But the counter argument is that private syntaxes are more readable than the cruft and boilerplate that would otherwise be required in a regularized PL without homoiconicity.
@keean argued up-thread that all the necessary cruft can be avoided with polymorphism (aka generics) and functions (along with typeclasses). I tried to find examples of macros that could refute @keean and I was unable to.
@analyticbastard can you present a macro that leverages homoiconicity and that refutes @keean?
What value for the human programmer does homoiconicity add?
I prefer declarative syntax. Make it easier for the human, not the computer. Readability, understandability, and maintainability are some of the most important things for source code.
I also don't like macros as manipulating the AST without type safety is like cracking a walnut with a sledge hammer. Generics and type-classes provide a much cleaner form of meta-programming. In fact I think even meta-programming might be bad. What I want is a language to express generic programming as done by Stepanov in "Elements of Programming".
Programming is about algorithms, and being able to clearly express algorithms is the most important thing. Languages like Lisp lack the power of generics and typeclasses and so cannot express algorithms in truly generic forms. For example f x y = x + y when truly generic can 'add' any two addable things together. We do not care about the representation of the things, we never need to even think about what the data structures representing those things are, they could be integers, complex numbers, or sets, etc. This is abstraction, and it's something macros don't do, because you are still manipulating the base data structures. Lisp gets its power from letting you manipulate code like it is data, which in my opinion is completely the wrong approach. Abstraction of data is a much better approach, as now we can use simple algorithms on complex data. We can have only one definition of 'sort' in the whole language and it works on anything (this is a simplification, but makes my point). Any coder who improves the universal sort algorithm benefits every programmer. You can have an online repository of algorithms that represent best practice, and anyone can improve them (subject to some peer review).
@keean wrote:
I also don't like macros as manipulating the AST without type safety is like cracking a walnut with a sledge hammer.
Lol :) I agree that bypassing the type system to do magic tricks seems to violate the point of having a typing system. I had forgotten that was one of your points up-thread. Does Clojure have a weak type system?
I prefer declarative syntax […] Lisp gets its power from letting you manipulate code like it is data
Turing-complete manipulation of the language syntax is not Principle of Least Power. It’s the principle of the kitchen sink.
Programming is about algorithms, and being able to clearly express algorithms is the most important thing. Languages like Lisp lack the power of generics and typeclasses and so cannot express algorithms in truly generic forms. For example
f x y = x + ywhen truly generic can 'add' any two addable things together. We do not care about the representation of the things, we never need to even think about what the data structures representing those things are, they could be integers, complex numbers, or sets, etc. This is abstraction, and it's something macros don't do, because you are still manipulating the base data structures.
This is a very persuasive argument. Abstraction and typed algorithms are two very strong points.
I don’t know if @analyticbastard is familiar with typeclasses (aka adhoc polymorphism), so he might not visualize how well abstraction works with them.
Lisp gets its power from letting you manipulate code like it is data, which in my opinion is completely the wrong approach.
It is more powerful in any use cases?
Abstraction of data is a much better approach, as now we can use simple algorithms on complex data.
Hiding the complexity from the consumer of a library. So the library author can deal with the complexity.
Any coder who improves the universal sort algorithm benefits every programmer.
Maximizing the division-of-labor for optimal economic leverage. This is what enabled the West to rise to be so dominant.
Before beginning, I want to make clear that I view programming as a matter of personal taste. Different brains work differently according to nature and training, so one size will never fit all in this case, and where some see wonderful abstractions, others see an impractical labyrinth of constructs far from the task to be done.
Regarding homoiconicity, my experience tells me it helps in keeping your data embedded in your language. This is the natural path Clojure has followed over the years. For instance, hiccup generates html from within Clojure, and garden generates CSS. Server routes can be defined as EDN (again, Clojure) and even configuration can be describe as EDN, while containing readers specifying behaviour. You simply cannot do these things in a non-homoiconic language. In general, this avoids templating, or needing to keep hooks outside of your language in a foreign add-on, which has proven inferior against non-templating over and over. Facebook React docs also support this.
React embraces the fact that rendering logic is inherently coupled with other UI logic: how events are handled, how the state changes over time, and how the data is prepared for display.
Instead of artificially separating technologies by putting markup and logic in separate files, React separates concerns with loosely coupled units called “components” that contain both. We will come back to components in a further section, but if you’re not yet comfortable putting markup in JS, this talk might convince you otherwise.
Obviously, JSX being non-homoiconic is in reality two different languages. In case of Clojure, it naturally avoids templating without merging two foreign languages.
I concur that macros are to be avoided whenever a function can be used, and that is a common Clojure practice. Useful macros can be found as utilities that cannot be implemented as functions. I myself made one for a common use case I had often come across
(zipmap key-collection value-collection)where key-collection and value-collection are obviously two collections with the same number of elements, the former for the keys in the map to be constructed, the later for the values. A common instance is to have something like
(zipmap [:a :b :c] [a b c])to build a map like
{:a 1
:b 2
:c 3}of what ever the values of these variables might have (:a is a keyword native Clojure type).
My macro:
(selfmapk a b c))does precisely this. You need to access the data the compiler is seeing (the symbols names) to make the structure that will later be filled with the symbols value at runtime, there is no other way to do it. As you can see, this is Clojure-y, nobody would have trouble understanding it once docs have been read.
In general, the syntax in Clojure collapses to a few flavours, and the DSL chaos one could imaging after reading the assertions on this thread has never come to be.
Having data and code in one language is beneficial to quickly read code having only a glimpse, allowing the programmer to focus on the task instead of on how to deal with the language particularities.
Clojure is untyped, and this is a big drawback until one gets used to deal with it. It puts pressure on the programmer, yes, but my experience tells me that you end up learning the codebase better to compensate for the compiler's work. This extra burden is alleviated by the succinct syntax and the previous argument, but it is obviously still present.
Polymorphism is achieved mainly by multi methods, which can dispatch a function by an arbitrary process of any of its arguments (be it the Java/underlying class/type, its value, or meta information). I have never sought anything else and I have never needed anything else)
In summary, under the vision that a programmer is a technical poet, perfection makes sense only under the particularities of the artist.
In summary, under the vision that a programmer is a technical poet, perfection makes sense only under the particularities of the artist.
Whilst people are all different, I think there are several large categories of "thinking types". Certainly Lisp languages have their followers, but they never produced the same "industrialisation" as the object oriented languages.
s-expressions work well when your only data structure is a list. They give special syntactic precedence to this concept.
If you want homoiconic objects and lists, then you get JSON. And yes, you can write an AST directly in JSON, but when you start to look at compiler annotations for type inference etc, you probably don't want to be typing this out yourself. If we want to add arrays, and other data types, we are going to start running out of syntax.
I have yet to see homoiconic type annotations? I am not even sure what they would be. It seems clear that this is a limited special case for lists, and when you extend a language beyond that, you can't get it to work.
Besides, Haskell's
f x y = PutStrLn (show (x + y))
Seems much more readable to me. You still have the parenthesised arguments, but why force definitions into this format for the sake of making the parser simpler?
I received feedback from another (very high IQ) “LISP expert”:
I read the thread a little. I don't have any great refutation, I think a lot of it comes down to preference on syntax. I agree with @keean that you can do almost anything with say operator overloading and polymorphism but I disagree this necessarily means the code is more readable than macros. It is possible to write both readable and unreadable code in any language. I guess in a lot of cases someone skilled in creating readable code in one language (to those reasonably skilled in that language) would find it hard to do so in another language without a similar degree of skill. And similarly I think people accepting of the value proposition of one model tend to accept its warts as a cost of doing business while declaring the (in reality similarly bad) warts of another model to be fatal.
@analyticbastard wrote:
Before beginning, I want to make clear that I view programming as a matter of personal taste.
Thank you for commenting here so that we can analyse the perspective of those who use LISP languages. And I commend you for making the strongest possible points justifying LISP. Also revisiting this thread was the most fruitful way to explain to you why I am not going to choose LISP for any project I am leading.
Tangentially I think the Urbit project is based on homoiconicity and they tout how terse their fundamental eval is. AFAIR, they argue that this makes their system more secure because there is less code to review.
However, I am going to lean towards disagreement against the proposition that programming is a matter of personal taste, although I would have also thought that in the past (if I had been instigated to think about). It can be a matter of personal taste for those who like to code on an island with their pet, toy projects, but it is dying paradigm from an economic analysis. The era of closed source has sailed. We live in an open source age where everyone must be able to read the code as easily as possible. So it means we must standardize around a few core programming languages.
There can be several flavors of taste, but for a general purpose programming language then all of the choices of taste need to popular enough or as @keean wrote, “industrialized.”
The problem with private (aka highly specialized) languages and private syntax is that it supports the Theory of the Firm which I wrote about in one of my recent blogs:
Because of the relative numeric asymmetry of authorship compared to consumers of content, there’s no sustainable Coasian Theory of the Firm (c.f. also 1, 2, 3, and 4) moat for corporations to extract parasitic rents from content!
Quoting myself (@TPTB_need_war) from that first link to Theory of the Firm:
The Theory of the Firm can be explained from one perspective with the erroneous theory that knowledge creation can be duplicated and redundant thus managers play an important role of making sure there are backup employees in case one gets sick, leaves, or otherwise fails.
We've needed corporations to aggregate work, because for example you don't build Mozilla Firefox with one programmer. You need a large team.
This is why I was working so hard on solving the Expression Problem for computer programming language (which I think I've solved and will be working on after I finish the crypto work), because with true modularity (no need to refactor), then programmers can work on their own smaller modules and then other programmers can combine modules into large programs. This is the Holy Grail of programming yet to be achieved.
Humanity’s goal with open source and even decentralized ledgers is to disintermediate parasites who exist because of impedance mismatches in the peer-to-peer free market economy.
An example of such mismatch is if the LISP experts leaves and no one is sufficiently skilled is available to pick it up. This is why companies saddled with LISP code bases have to pay high compensation for LISP experts and thus this is very lucrative for LISP experts. But this is especially troublesome with open source, because there’s so many competing open source projects competing for eyeballs and skilled programmers, so choosing a programming language of which only 1-in-10,000 programmers are highly skilled, makes that open source project not anti-fragile. Meaning it’s at high risk of suffering a premature death. Anecdotally, I guesstimate I was (as of 2001) perhaps in the top 10% of programmers in terms of broad metrics of skill such as algorithms, holistic design, etc (or maybe higher, yet certainly in the top 30%), but I am not proficient in LISP. I would not choose to pick up a LISP code base if I had other choices available.
Do companies gain anything by choosing LISP? IOW, is the high degree of specialization justified? I think not. They lose type checking, they suffer the aforementioned fragility, and even I will rebut your other points in favor of LISP. For comparison, the programming language R probably is justifiable specialization (although I’ve never used it so I don’t know except I see so many data mining engineers list it on their CV as PL they know).
Obviously, JSX being non-homoiconic is in reality two different languages. In case of Clojure, it naturally avoids templating without merging two foreign languages.
@keean and I had that detailed discussion. I started off making the same argument you have made. But by the end of it I realized that we should code HTML in our language. Which is in effect what you do in LISP.
Clojure is untyped, and this is a big drawback until one gets used to deal with it.
There’s no way to “get used to” the fact that an untyped language can’t do the implicit plumbing that makes typeclasses so amazingly efficient for coding and refactoring. And which automatically delay binding dictionaries to data at the call site, thus avoiding the problems with OOP classes.
I almost drank the LISP Koolaid too, but thankfully @keean corrected me.
It seems that those of us who work with a non OOP, homoiconic language tend to care less for the constructs of typed people. I fail to see where all the fuss is with type classes or how they would help me with something that I am struggling to do at present.
All the innovations from functional programming have been coming over to the mainstream languages, so typeclasses, immutability, and purity are next.
Typing declares invariants which thus minimize “spooky action at a distance” effects which can cause the dreaded Heisenbug. Typing also enables monomorphisation (and other compiler optimizations) for performance. Excessive typing is undesirable. Follows is a quote of what I recently wrote about a specific example of probably typing fatigue:
As for the issues raised by @rossberg about that typeclasses don’t provide all the features of modularity and the requirement for encapsulation of (i.e. making the type opaque for) modular data types, I have two rebuttals. First, I had already mentioned upthread which I have reiterated above that typeclasses provide a form of implicit modularity which explicit modules do not provide. Secondly, I simply think that hiding the type of the underlying data isn’t worth the cost of forsaking HKT
(or upthread was it the partial application of them that is lost?), sidestepping the impredicativity issues, and double-vision problem which makes 1ML so abstruse and limited (c.f. the comparison of doing HKT operations with functors versus 1ML in the 1ML paper and both are non-optimal, unreadable, and abstruse IMO). This issue is a prime example of what I am referring to where I wrote that type systems should not be designed primarily to catch errors or enforce higher-level semantics. Also obfuscation is not encapsulation, and encapsulation is not security. Security should be handled as a higher-level semantic concern.
Oh then when @keean and I finally make a programming language with typeclasses (and simplified modules) then you’re going to find out about something amazing which you can’t get in LISP. If we successfully bring typeclasses to a mainstreamable PL, then we will have achieved a very important paradigm for modern computing. Especially if we can also improve the parallelism and concurrency conundrum. There’s something potentially very important brewing in these issues threads which is hidden by the vast quantity of verbiage as we were sorting out our thoughts.
Here’s my recent comment about the advantages of typeclasses:
Yet typeclasses have the advantage (c.f. also and also) of making refactoring and coding much more automated and modular (as well as the late binding of dictionaries to data which I had claimed in our discussions can be with unions a partial solution the Expression Problem). As I wrote somewhere upthread, it means we don’t have to lift all the typeclass bounds for all the functions in a module up to the module inference (aka signature), because they are implicitly resolved. And as @keean had pointed out in the past (in other issues threads), this means that recursive typeclass bounds (i.e.
requiresclauses on the functions with in a typeclass) don’t have to be lifted. The automated plumbing is incredibly beneficial.So for global coherence without forsaking modularity then use a module which bundles the inviolable dictionaries with the data instance and track the type of such instances applicatively. Retain typeclasses for their elegance and modularity but discard global canonicity. And opaque encapsulation of data types in modules is probably not a good priority as explained below.
The typeclasses can still retain local canonicity which is inherently required for retaining the aforementioned benefits of the automated (implicit) plumbing.
@analyticbastard responded to my query about if Clojure’s persistant data structures are important:
Immutability helps a lot when developing software. Not having to care whether the current value of a symbol is the correct one frees your mind a lot. You are just sure they are right. Immutability promotes declarativeness. You have immutable data structures in typed languages, but they are implemented as classes and objects, but it feels artificial, attempts to modify can still be made and an exception (bad thing to have!!!) NEEDS to be raised.
Both @keean and I think immutability is important and we are looking at it. And of course we won’t follow the flawed designs of Java, C++, etc.. Luckily presuming he is not preoccupied, @keean will likely catch any such blunders in design because of his considerable experience with just about every programming language of significance. What I bring to the mix is my relative PL design ignorance[naivete] and creative juices. So I introduce new things that might be slightly different ways of looking at a PL design issue. I also bring a strength in economic analysis and attention to detail in holistic design (at least when my cognitive health allows me the energy to do so).
@keean wrote:
Whilst people are all different, I think there are several large categories of "thinking types". Certainly Lisp languages have their followers, but they never produced the same "industrialisation" as the object oriented languages.
I think it’s true that some programmers want to think in that model. And they may be very productive in that model. But it’s not a model that seems to be natural for the majority.
I was drinking the Paul Graham Koolaid which argues that the smartest programmers choose the best PL and that LISP offers some advantage. But I think you have successfully argued that it doesn’t offer advantages and has some limitations. Thus what LISP actually offers is a model that some few dudes prefer.
There’s a concision of generative essence, elegance, and eloquence to the model. The concept of homoiconicity is an interesting discovery. But it doesn’t really excel against the competition (of what we are designing for a PL).
s-expressions work well when your only data structure is a list. They give special syntactic precedence to this concept.
If you want homoiconic objects and lists, then you get JSON. And yes, you can write an AST directly in JSON, but when you start to look at compiler annotations for type inference etc, you probably don't want to be typing this out yourself. If we want to add arrays, and other data types, we are going to start running out of syntax.
This was your other major points up-thread which I forgot to repeat in my recent recapitulation.
Lists are actually a very inefficient data type for cache coherency.
The model is quite limited.
Also the lack of typing is a severe limitation, which I think the LISP experts don’t yet appreciate because of the bad experiences they’ve had with Java, C++, etc.. And Haskell which introduced typeclasses, apparently didn’t get the design quite right. The global canonicity of typeclass instances was a blunder. We’re discussing now in the Modularity thread #39 about how to rectify that by moving “global” coherence to a simplified module layer.
I received some additional feedback from that anonymous LISP expert I know:
But this is especially troublesome with open source, because there’s so many competing open source projects competing for eyeballs and skilled programmers, so choosing a programming language of which only 1-in-10,000 programmers are highly skilled, makes that open source project not anti-fragile. Meaning it’s at high risk of suffering a premature death.
That's going to be the case of any "good" language you choose or build!
Ironically an Oxford PhD ended up not working on my crypto project because (for one reason) I wouldn’t let him code it in Java (although I think that wasn’t the main reason and I will not elaborate). He has employed Java extensively in his contract work for banks.
It seems there isn’t a PL which the very awesome programmers want to use which I also accept. This is a dilemma. Do I bite the bullet and accept some hodgepodge mix of languages that different guru programmers want to use? Do I force them to all standardize on a choice such as C++ or Java? Frustrating!
What if we could instead create something new that attracted the interest of guru programmers?! Build a community. Nice fantasy?
If you don't want that you are pretty much stuck with Java, C/C++, Python, JavaScript. Maybe a few others, possibly Go. I can't think of many other languages where the language itself doesn't serve as a barrier to open source participation.
Programming languages are languages not just tools.
I was also thinking about this issue when I wrote what is quoted. I guess I didn’t address it because I was in a rush, because my post was already very long, and because my response will be overconfident and risky (might end up incorrect). Also because I am still thinking about this dilemma and what will be my decision. I still have some more analysis to do on specifics (especially of implementation realism and costs).
I think it’s plausible to create a popular PL which is adopts the next series of major innovations from functional programming:
All the innovations from functional programming have been coming over to the mainstream languages, so typeclasses, immutability, and purity are next.
@at the moment OOPish languages are up to two orders-of-magnitude more popular than FP languages, but the typeclass paradigm hasn’t yet arrived in an easy-to-use PL.
I think typeclasses are one of the key factors in making that happen, because they’re both so elegant and powerful. They’re not difficult conceptually. It’s some of the other stuff in Haskell which is so abstruse and thus typeclasses haven’t been appreciated by those who stumble when attempting to learn Haskell. Yet typeclasses move the bar far forward from those extant PLs you mentioned. For example we all know that templates make C++ a very complex and brittle language (and because the semantics aren’t declarative but rather Turing complete so that there’s no way to realistically reason about them and the only way to know what will happen is to compile it and observe the “spooky action at a distance” ramifications which can foster the dreaded Heisenbug). The C++ “frequently questioned answers” is a humorously slanted critique. Also my prior analysis of the C++ clusterfuck. @bcmills also posted his summary of some issues with C++ templates.
But typeclasses have problems on Haskell because of the requirement for global canonicity. They’ve been trying (e.g. Backpack) to figure out how to deal with the need for modularity and for example the fact that there’s more than one monoid for an integer (e.g. (0, +) or (1, ×)). It appears that neither ML nor Haskell resolved the tension between the dual approaches (typeclasses vs. modules) correctly. Both err on diametrically opposed extremes/dualities. Recently in the Modularity thread, I have proposed/posited an integrative, middle ground.
Also I think it is important to stay close to C-like or Python syntax (i.e. do not adopt the model that all functions take one argument not delimited with parens) and avoid overt use of abstruse (highly abstracted) concepts such as monads (and especially don’t advocate monadic modeling of effects!). We employ monads even without knowing it, and that is okay.
Yet there’s another factor that is involved with making a PL popular. And that is it needs backing from an entity with sufficient resources and which can drive its use in major projects. Well you already know about the ancillary project I am working on which I hope might provide that backing. Quite grand plans for a person who is ill. Any way, perhaps another significant company will pick up our open source PL project and run with it.
Also I’m contemplating a transpiler to Go initially in order to hopefully ease both implementation cost and uptake. There’s significant complaining over at their issues threads about Go not having generics.
As you know Python and JavaScript are not compiled nor strongly typed. Thus they are scripting languages, not serious programming languages. In terms of serious programming languages, we really don’t have any reasonable choices which are popular. C++ is a master of complexity (unless one avoids abstraction). And Go is lacking even basic generics. I won’t recapitulate all the discussions we’ve already had criticizing Java (and Scala). This is why I had to go on this PL design tangent. The status quo sucks!
For reasons already stated, I do not want to code in C++ with templates nor Boost. The complexity is off the charts when attempting to maximize abstractions. Nevertheless I can read the Steem(it) C++ code reasonably well because it appears to employ C++ reasonably amateurishly and conservatively. I want Go’s green threads. I don’t want Haskell’s forced purity, laziness, everything-is-a-one-argument-curried-function, monad hammer for every nail, and global canonicity without modularity. I don’t want Java’s numerous deficiencies (where is my unsigned data type!), where’s my green threads instead of mutex hell, where’s my typeclasses instead of that OOPish anti-pattern, and I’m still forced to run on JVM. Actually Java probably remains one of the most rational choices for a serious programming language, which is a sad statement about the progress of mainstream PL development over the past decade. JavaScript really sucked a lot of resources out of where they should have been. JavaScript is not a serious programming language and NPM is a clusterfuck of 300,000+ modules that are not canonical. The JavaScript ecosystem is too decentralized and chaotic on top of it being a very inadequate base of a language. Python is not for writing the most performant and well typed code.
Perhaps we should consider adding Swift to your list, which is apparently being open-sourced. It also has some form of typeclasses. @keean and I should probably recapitulate our analysis of that PL. My recollection is that is sort of kitchen sink approach to language features with some peculiarities such as the forced use of reference counting. I have proposed a new GC design which merges generational collector for maximum efficiency and RC for maximum control.
s-expressions work well when your only data structure is a list.
I'm not sure what @keean meant by that. Clojure (and most/all modern lisps) has other native types (sets, arrays/vectors, maps/records) that extend traditional (i.e. Lisp from the 50s) s-expressions and are directly supported by the reader/printer and in some cases used in the language itself (for example in Clojure function paramaters are a vector not a list). It is similar to JSON really (which is mentioned subsequently, so I don't understand the point here, unless keean doesn't know this about Lisp which I doubt) but with a simpler syntax.
Hope you’re not put off that I shared this comment publicly (and anonymously). I would also like to read @keean’s clarification on this issue.
I think perhaps @keean’s point is that the homoiconicity is limited to the list structure of s-expressions. Thus any data which should also be fully generalized as code, must adhere to a list structure. So perhaps I was mistaken to extrapolate it to being generally deleterious for cache coherency.
Nevertheless, I think the other points about lack of typing which impacts abstraction and the importance of declarative reasoning (which I have also related above to the weakness of C++) remain persuasive.
Maybe we should have a LISP as a DSL for writing DSLs though? I’m still unclear about the best way to handle DSLs. I think @keean’s points are persuasive for a popular general purpose PL though.
An s-expression is just JSON without objects :-) having names in the syntax allows named arguments etc.
We can easily define a homoiconic language using JSON, we make the function the first list element, as Lisp does so:
[ print [ add, 3, 4 ] ]Would print 7. The difference is Lisp uses braces and spaces as element separators, and JSON uses square brackets and commas.
But as I put above this is only for Lists, if we want an object (named properties) we need extra syntax, like the curly bracket objects in JSON.
But why is the list-structure undesirable when considered separately from the other limitations (lack of typing, non-declarative paradigm) which can be argued to be undesirable?
@shelby3 there is nothing wrong with having a literal syntax for lists, the questions is whether lists should be the only literal syntax? Why should we not have a literal syntax for objects, maps, arrays etc?
Why should we not have a literal syntax for objects, maps, arrays etc?
Then we come back to why not just use the language syntax?
Obviously, JSX being non-homoiconic is in reality two different languages. In case of Clojure, it naturally avoids templating without merging two foreign languages.
@keean and I had that detailed discussion. I started off making the same argument you have made. But by the end of it I realized that we should code HTML in our language. Which is in effect what you do in LISP.
Clojure programmer wrote:
I wrote:
Clojure programmer wrote:
I wrote:
Clojure programmer wrote:
I wrote:
There is just no way we can popularize LISP. If we code in LISP then other developers won’t be able to follow our examples.
If this Zer0 syntax ends up looking very similar to C, JavaScript and Python, then it will be easier for devs to pick up.
S-expressions are weird.
yet it's going to be another language, isn't that going to scare away people?
why are they weird?
because they do not look like C
yet it's going to be another language, isn't that going to scare away people?
Maybe. I need to see what it looks like once I finish the syntax.
that's like saying "chinese is weird cause it does not resemble English"
yet there are more than 1b people who don't find it weird
There are orders of magnitude more C, JavaScript, Java, C++, C#, and Python programmers than s-expression programmers.
Also I want a typed PL.
And Clojure does not have typeclasses.
I really want typeclasses.
Clojure programmers think differently.
Implicit resolution of nested typeclasses is very convenient.
How do you do polymorphism in Clojure?
How can I have a function that accepts any type and automatically selects the correct interface for that type, based on its type? Impossible in Clojure.
Clojure does not have types and it is impractical to force them in, after this week I can give a precise problem description
I have not written a macro for about 20 years, so that someday was decades ago :-)
Maybe you don't really need macros because of lazyness in Haskell?
Lazy evaluation offers, in some sense, macros which are restricted to take valid expressions and are required not to transform these expressions.
Lazyness offers also a form for macros to be first class, treating them as function accepting values of the computed type of an expression and returning values of the same type.
I'm discordant about macros, they have advantages but they have clearly three disadvantages:
1.) They allow for syntax constructs exhibiting no relation to the hosting language itself.
2.) Too, the definition syntax for macros is also ugly and hard o follow
3.) Error projection to original macro code is bad or nearly impossible.
I'm strongly favoring macros allowing only to consume or produce valid host language expressions as well as macro definitions written in the host language itself and the ability to optionally type tag macros as they are functions though they operate on a different type system.
But I would be also ok when a "statically typed" language decide against macros as I understand their impact.
Maybe you are interested how kotlin provides DSL's without to use macros.
@sighoya Kotlin seems to provide a kind of DSL by a sort of 'notation hack' defining a lambda as '{ args -> }' which means that zero argument lambdas look just like C blocks { ... }. The second part us calling the argument of the constructor in the context of the constructed object.
Although Kotlin's approach is nice, I think it suffered from having 'objects' because it makes it harder to store and retrieve state. I am moving towards not allowing code to be referenced from data, instead the 'type' of the data selects the implementation, which is what happens with type-classes.
Regarding macros, I think all legitimate uses of macros can be replaced with generic functions.
Tikhon Jelvis opines that Haskell would benefit from macros for syntax sugar to clean up some of the syntactic noise:
it provides ad hoc syntax sugar instead of a flexible, disciplined macro system, which can lead to a bit of a syntactic mess.
I would love to see something that mostly looks and feels like Haskell but with a macro system and no language-level syntactic sugar.
Any reactions?
I would love to see something that mostly looks and feels like Haskell but with a macro system and no language-level syntactic sugar.
Personally I would find that a terrible idea. I like that once I have learned the syntax and semantics I understand the language and can read anyone's code. Macro's would take that away. There are other reasons, but I think this is such an important one we don't need to go I to the others. Haskell has a great system for generics using type-classes, and even allows operator overloading. You can create your own DSL using the built in syntax, so why would you want macro's, apart from to make it not look like Haskell, which would be bad.
If you have optional lazyness combined with inlining you can cover a wide range of use cases for macros.
"Laziness by default is absolutely not a mistake."
Wrong, and I never understood the argument of "writing code in any order". There is always in order in code evaluation, it depends what you want to have first for evaluation.
When you implement lazyness with coroutines then you will see there is an order.
@sighoya Order matters, it's why you cannot write programs (apart from proofs) in Prolog without 'cut'. Infact where it really shows is that some programs work in Prolog without 'cut' because they are logically correct, but will not terminate in your lifetime without it :-)
Well guys I argued in the Why Not Haskell? that we want to code as much as possible so that order does not matter, e.g. the Actor paradigm.
Yes order matters sometimes, but when it does not need to matter, then we shouldn’t fail to take advantage of that increased degrees-of-freedom.
@sighoya why did you thread jack? This thread is not about purity. We have a thread for that. You should have posted your comment in the correct thread.
Sorry my German perfectionism can’t handle this chaos.
Let's just start writing randomly in any thread okay? 😵
I think those who are sloppy in their communications will also write sloppy code.
Why not Clojure (or any LISP)?
@analyticbastard is a Clojure expert. Clojure is a variant of LISP which normally runs on the JVM (Java Virtual Machine). He tells me that he loves homoiconicity. I explained that homoiconicity is very good for the programmer who creates the code because homoiconicity allows them to create private syntaxes and macros which enable their code to do magical jumps through circus hoops and do more with less code.
But as we discussed up-thread, it is not so good for the readers of the code who have to try to learn all these new private syntaxes invented for each code base. IOW, it lacks the regularity that I wrote about in my prior comment on Apr 13.
But the counter argument is that private syntaxes are more readable than the cruft and boilerplate that would otherwise be required in a regularized PL without homoiconicity.
@keean argued up-thread that all the necessary cruft can be avoided with polymorphism (aka generics) and functions (along with typeclasses). I tried to find examples of macros that could refute @keean and I was unable to.
@analyticbastard can you present a macro that leverages homoiconicity and that refutes @keean?
I wrote the following in an email to @analyticbastard
Originally I wanted to target Go, but that just seems impossible to achieve right now from any high-level PL with strong typing.
Hickey is incorrect in some of his claims about the value of typing because he is for example as follows thinking of classes and not type classes:
https://lispcast.com/clojure-and-types/#types-as-concretions
https://youtu.be/2V1FtfBDsLU?t=1141
A typeclass is just a requested interface. There is not concrete abstraction, because additional interfaces can be requested by any function. Whereas classes bind interfaces to data at instantiation.
That is why we will utilize typeclasses and afaik only Haskell, Scala, Rust and Swift offer typeclasses at this time.
I do agree with him (and I disagree with Keean) about the futility of attempting to type check semantics at runtime. Much of our logic in the semantics, so types are not as important as some people think they are. But types are also useful when not abused.
https://youtu.be/2V1FtfBDsLU?t=2097
(Place Oriented Programming)My answer to this is not just immutability but also an ALP Arena paradigm, because otherwise it’s not going to scale to massively multicore due to for example one GC for all threads.
https://youtu.be/2V1FtfBDsLU?t=2266
(Information)Hickey may not realize he is championing typeclasses on this point.
https://youtu.be/2V1FtfBDsLU?t=2605
(Clojure and Information)I do not fully grok this point. Why do we need names to be “first class” instead of just using strings for keys of a map?
https://clojure.org/guides/faq#why_keywords
https://youtu.be/2V1FtfBDsLU?t=2724
(Brittleness and Coupling)This I believe is mitigated to great extent with degrees-of-freedom with typeclasses. Also Scala and JS both support named-parameters.
https://youtu.be/2V1FtfBDsLU?t=3055
(Language Model Complexity)I agree with him that reducing the language model complexity is an important goal. We will be using only a subset of Scala and Java’s model, e.g. we will not use classes nor subtyping of classes (i.e. subclassing).
https://youtu.be/2V1FtfBDsLU?t=3164
(Parochialism - names)He constructs a strawman. The culprit is not static typing per se.
https://youtu.be/2V1FtfBDsLU?t=4008
(Joy of Types rebuttal)Typeclasses are names (in my design, not structural as in Haskell)! Thus they do specify an interface. Typeclasses enable the degrees-of-freedom Hickey laments don’t exist with classes. Problem is Haskell forces a structural match on typeclasses instead of names (which I think is a design error) and Haskell forces a consistent algebra for each typeclass, which I think is also an unobtainable delusion (aka unobtainium). I have disagree with both Keean and Skaller about for example proving axioms for typeclasses. That’s taking it too far into the the sort of useless and counterproductive activity that Hickey laments. The only real, scalable utility of the typeclasses is a named interface that can be bound at the function site. And yes extensive unit testing can not be replaced by static typing except in small-world, toy examples.
In summary of all his points, yes a keyed-map is a lingua franca and code-as-data with dynamism enables more capabilities. And yes maybe some programs want to include a LISP interpreter library. But static typing is still important, especially for high-performance optimization, space-efficient data structures and cryptographic algorithms.
Note I agree with the advantages in the following linked, of pulling from a pool of enthuiastic, knowledgeable, and pragmatic (i.e. not dogmatic, pious) programmers, as well not having to wait for slow compiles and the flexibility of dynamic typing when needed:
https://lispcast.com/what-is-the-business-value-of-clojure/
https://clojure.org/about/rationale
Apparently there’s a Clojure to C++ compiler which in theory could be compiled to ASM.js with Emscripten:
There’s typed Clojure:
Although it’s apparently heresy:
https://stackoverflow.com/questions/5110292/why-is-clojure-dynamically-typed/5110459#5110459
Also:
https://m.oursky.com/why-i-chose-clojure-over-javascript-24f045daab7e
@shelby3 if you want to see how much help types are, you only have to compare programming JavaScript with programming TypeScript. In production code TypeScript is much more reliable, especially if you use strict null checking.
I also think a complete separation between code and data is a good idea. Data should not be executable, and code should be static and unchanging. In this was the correctness of code can be verified before delivery to the user.
I think the kind of dynamic LISP like environments that mix code and data are great where the programmer is also the end user.
I think macro's, staged compilation and even code generators are not good ideas in general. I think syntax should have a consistent meaning, and datatype generic programming allows similar conciseness without the same problems. I do like the idea of allowing operator overloading and custom operators, but I think it's important that operator semantics are consistent (ie predictable by the programmer without consulting the implementation). I think code should be readable in isolation, that means I should be able to read and understand the code in a module without looking at other modules, just by understanding the syntax of the language. This does mean applying axioms sometimes, for example when overloading '+' we should be able to check it's associative and commutative. (Should floating point addition use the '+' operator as it's not always associative?)
Hickey points out that performance optimization may be the most important use of static typing.
I also think a complete separation between code and data is a good idea. Data should not be executable, and code should be static and unchanging. In this was the correctness of code can be verified before delivery to the user.
I think the kind of dynamic LISP like environments that mix code and data are great where the programmer is also the end user.
I think macro's, staged compilation and even code generators are not good ideas in general. I think syntax should have a consistent meaning, and datatype generic programming allows similar conciseness without the same problems.
I will reserve judgement until I gain more experience. But Hickey is correct that eventually at some level of complexity typing runs out of usable runway. I may find that I need the additional flexibility and power that would otherwise be intractable with typing.
I think perhaps you are being too dogmatic. Don’t throw out the baby with the bath water. But again I will wait until I have real world use cases from experience to discuss with you. Until then, I dunno.
I do like the idea of allowing operator overloading and custom operators, but I think it's important that operator semantics are consistent (ie predictable by the programmer without consulting the implementation). I think code should be readable in isolation, that means I should be able to read and understand the code in a module without looking at other modules, just by understanding the syntax of the language. This does mean applying axioms sometimes, for example when overloading '+' we should be able to check it's associative and commutative. (Should floating point addition use the '+' operator as it's not always associative?)
Yes strive for that but don’t even attempt to prove everything and don’t require a global, total order because remember I already debated you in the Subtyping thread #8 that universal, total orders do not exist. I will not repeat that mathematical and philosophical debate again.
You simply can’t hold the world fully ordered in the palm of your hand Keean. Don’t let the impossible quest for perfection be the enemy of good, and accept it doesn’t entirely fit into your ideals. We can’t reconcile such a rigid idealism, and still flex, with unbounded nature.
Note I do appreciate the points you have made in this thread and others. I learned a lot. Thanks.
if you want to see how much help types are, you only have to compare programming JavaScript with programming TypeScript. In production code TypeScript is much more reliable, especially if you use strict null checking.
Hickey’s valid point is that types are only really valuable for very low-level, low value “typos” level of checks. The orders-of-magnitude more important semantics can’t be checked with types without abusing types so much that programming become refactoring hell and rigor(ous) mortis. Unit tests are much more important.
I am not sure that everything runs out of runway. A Turing machine can execute any computable program. The end of the runway is really the limit of computability, you want something more, you need a quantum computer. So if the limit of computability is fixed, what is left is expressive power. Generics have coped with the needs of mathematics for thousands of years (whether mathematicians understood that or not is a separate question). Stepanov's "From Mathematics to Generic Programming" presents programming as part of mathematics thousand year history.
What more flexibility and power could you need than the whole of mathematics? Even an intractable example would not overthrow thousands of years of mathematical progress. That would be like creationists claiming one fossil could "dis-prove" evolution... It would not, it is more likely something is wrong with the fossil (mis-dated, a fake etc).
If universal total orders do not exist, then what is the periodic table? Here we have a rigid ideal that not only allows us to "flex" with nature, it let's us find new elements, it let's us discover things we would no be able to do without it.
In any case, nobody is talking about being able to prove everything. Some things are unprovable within any system (see Godel). However operator overloading can lead to cryptic programs if '+' can stand for any binary function. Of course people can write critic programs using any function, so 'add' could divide for example, so we can't stop every obfuscation. Instead I suggest we focus on stopping the easy ones to stop, and in that way we reduce the cognitive load on the programmer allowing them to focus on the truly complex problems.
We can see this cognitive load principle with JavaScript Vs TypeScript. When I code in JS, I often find it takes a long time to get all the typos out of the code. After getting the program working, I am never totally sure it's correct in all circumstances, and yes it needs thorough testing, but this means running the program and making sure we get 100% code coverage in tests. This takes a long time and just the process of doing this takes up time that could be used validating higher level properties. With TS the program would not even compile for a lot of typos, saving a huge amount of time going round the develop/test loop. So even if types only prevented typos (they don't) they would still be worth it.
An example where types help semantically: the other day I was trying to debug a graphics rendering problem - it took a long time to find the cause, which was not obvious from the symptoms. It turns out that for one particular type of render object a Uint8Array was being passed instead of an ArrayBuffer. Without typing this would have been impossible to spot - I have written similar sized JS programs and after 3 or 4 years of continuous development you just have no idea what types are being passed around. Here is a case where testing found the problem, but it was no help in solving the problem. You simply have test 'X' fails. You look at the bitmap comparison, and yes the rendered image is distorted compared to the reference.
Test driven development is a great idea when you know what it is you are developing. When writing a compiler tests are where I would start. A lot of software is more experimental though, and you can't write the tests before you know what you are testing. But my point above is that the develop/build/test cycle slows down development. It's a lot quicker if the compiler points out the error than if you have to build and run the whole test suite. It's even quicker if your IDE points out the error. For example I have started using the eslint "compat" plugin that test you set target browser versions and will lint and functions that are not supported in any of the selected browsers and versions, this saves a lot of cross-browser testing time. Of course you still do a full test across a range of platforms when releasing software, but you don't want this burden in your main development loop. Running a complete set of tests across different platforms can take days for large software projects, especially when looking for resource starvation bugs like memory leaks. So when the IDE tells you the type error as you code, you can save a lot of time.
Types also facilitate independent testing of modules because they specify the interface precisely, which allows modules to be tested independently, again resulting in an exponential (in number of modules) speed up in testing.
In summary I find developing without types tediously slow, because you have to keep interrupting coding to run tests, which takes longer and longer as the software grows.
What more flexibility and power could you need than the whole of mathematics?
Think about the intractability of composing Monads with Monad transformers. Then explode your brain by imagining needing to abstract that to composing Monad transformers.
I hope you understand the runway is indeed limited. And mathematics is highly limited. As I had argued in the Subtyping thread even the proofs are becoming computational and thus the Halting problem has now invaded mathematics.
If universal total orders do not exist, then what is the periodic table?
I am not going to reopen that discussion/debate we engaged in the Subtyping thread. At least not now. Too busy. The periodic table is not a total order — it’s aliasing error at best (it’s one myopic attempt to summarize nature). Again I will not reply if you attempt to rebut that. That that would be a thread jack anyway.
So even if types only prevented typos (they don't) they would still be worth it.
I did not claim typing is useless. I claimed we should not abuse typing because otherwise we will end up in (refactoring, modularity and composability) rigor(ous) mortis.
Examples of abusing typing are probably Monad compositions and perhaps even GADTs. It eventually becomes an unwieldy hairball and if pushed far enough a full blown dumpster fire.
I am currently enthusiastic about typeclasses and I think types may be important for certain performance optimizations. I do not disagree with your points about type also add some information. But I also agree with Hickey that types often say very little about the semantics if anything at all.
It's a lot quicker if the compiler points out the error than if you have to build and run the whole test suite. It's even quicker if your IDE points out the error.
Somehow you must have missed my statement that typing can’t check all semantic errors. I was speaking about what typing can’t realistically do and thus testing is the ONLY way to catch those errors anyway.
Types also facilitate independent testing of modules because they specify the interface precisely, which allows modules to be tested independently, again resulting in an exponential (in number of modules) speed up in testing.
Semantic errors can leak across typed APIs and still type check.
@shelby3 so I watched the whole clojure video and I liked a lot of what he said. I don't agree with him on types, and think he is missing something by only thinking about values. In Stepanov's introduction to "Elements of Programming" he talks about entities (values), species, and genus. Clojure is only operating at the level of the first of these.
I agree with a lot of things in that video, and I agree that we need to consider the "wire" protocol, and that "classes" don't work. If all he had said was that types (as in C++) don't work, then I could agree, but he also says algebraic datatypes don't work (as in Haskell), yet goes on to claim that being an algebra (composition etc) is important.
There's a lot of good stuff in that video, and the Epochal timeline is exactly what I was proposing about using state-machines.
I think taking Stepanov's model, I would argue there is a lot of value in adding Species(types) and Genus(typeclasses) to a language. The comment that (int,int,float,string) is semantically meaningless is true, but I see it more as an attack on product types Vs Maps than types themselves. I would see this as more a strong argument for nominal types Vs structural types. But I would also suggest that using species/types correctly a product can contain information, for example (Dog, Sheep) it's important to know which is the sheep and which the sheep-dog, even if we don't know anything about that particular set of sheep and dog. Further Genus, also provides semantic information like Four legged a => (a, a, a)
I think he's completely correct, if you look at types from a certain point of view, but I think it completely missed the point about Entity, Species and Genus.
Think about the intractability of composing Monads with Monad transformers. Then explode your brain by imagining needing to abstract that to composing Monad transformers.
I totally agree, been there, done that, I don't think it's the right way. Note that Monads only are important when you approach combining functional purity with static types in a certain way. "Clean" is a statically typed pure functional language with no 'Monads'...
Algebraic affects seem a better approach that is compostable.
Somehow you must have missed my statement that typing can’t check all semantic errors. I was speaking about what typing can’t realistically do and thus testing is the ONLY way to catch those errors anyway.
I am not saying you don't need to test, I am saying that catching as many errors as possible, as early as possible, saves you time and makes you more productive. In my experience programming in TypeScript is a lot more enjoyable, productive and results in a lot less errors in shipped code than programming in JavaScript (because no testing is perfect), and here you have a good comparison because the value level syntax and semantics of the languages are the same.
Also note that "spec" is practically a type system:
https://clojure.org/guides/spec
Which is designed for specifying things like "wire protocols", but also note that in the video he says (paraphrased) we should write the internals of our applications like wire protocols.
So we should be using "spec" to internally define APIs and interfaces, which is really an ad-hoc type system by another name. To me it seems like "type systems are bad" has become a box he can't break out of, and rather than admit he was wrong, he's going to great lengths to make "spec" not a type system... Or maybe he just has a limited view of what a type system is.
By the way I agree with him about academics :-) databases are important, wire protocols are important. I would add that specifying interfaces both wire and internal are important (especially with multi-person dev teams, working with libraries, even geographically distributed teams).
I would see this as more a strong argument for nominal types Vs structural types.
[…]
I think he's completely correct, if you look at types from a certain point of view, but I think it completely missed the point about […typeclasses]
That is what I wrote in my first post about this yesterday.
So we should be using "spec" to internally define APIs and interfaces, which is really an ad-hoc type system by another name. To me it seems like "type systems are bad" has become a box he can't break out of, and rather than admit he was wrong, he's going to great lengths to make "spec" not a type system... Or maybe he just has a limited view of what a type system is.
Yeah but the “we can type everything” purists have the analogous dilemma from the other side of the coin.
@shelby3 Also I think libraries are a big problem, and by his own admission unsolved by clojure. For me types and generics are an important part of the library problem because they allow interfaces to be specified (and a library interface should be treated just like a wire protocol).
Generics solve his problem of having a string at one point and at string somewhere else, without all the intervening code needing to know about the string, and needing to be refactored if the string changes type. By generics here I mean the combination of types (Species) and typeclasses (Genus).
@shelby3 Also I think libraries are a big problem, and by his own admission unsolved by clojure. For me types and generics are an important part of the library problem because they allow interfaces to be specified (and a library interface should be treated just like a wire protocol).
Generics solve his problem of having a string at one point and at string somewhere else, without all the intervening code needing to know about the string, and needing to be refactored if the string changes type. By generics here I mean the combination of types (Species) and typeclasses (Genus).
I agree with that but Hickey also made the point that even composable typing can become unwieldy to compose requiring refactoring etc.. I think when we were discussing how to modularize typeclassing so we can multifurcate the abstract algebra we realized that there could be the Cambrian explosion that Hickey implicitly alludes to.
So I can understand that to be philosophically why he may be trying to avoid adding strong typing.
I agree with that but Hickey also made the point that even composable typing can become unwieldy to compose requiring refactoring etc..
I find typing helps with refactoring, because when I change the type, the type checker finds all the code I need to change so I don't miss any. I find my refactorings in TypeScript are a lot easier and quicker than refactoring the same JavaScript. I think this is slightly missing the point, as he probably means refactorings are a symptom of the problem, and I agree, but generics with typeclasses and row-polymorphism is one answer to this.
I agree with his points on composition, but I find they are not actually opposed to types, but more opposed to structural types.
For one I would strongly be I favour of having relations as a built in type, along with relational algebra. I think this answers a lot of his points about composibility without reducing everything to a Typeless map. His points about composing data, that's a relational Join, and subsets, that's a relational Restrict. Relations are typed, and yet maintain all the properties of Maps that he values, as far as I can tell. His problem with this would be needing to know the complete types of relations, but again generics and row polymorphism come to the rescue. I can specify a function that inputs a relation that contains a Name type , but I don't care about the rest and it just gets passed along.
I find typing helps with refactoring, because when I change the type, the type checker finds all the code I need to change so I don't miss any.
I believe that is a facet of his point. That you actually have to change them all. That’s the Cambrian explosion. Type inference should not be employed for APIs.
I think this is slightly missing the point, as he probably means refactorings are a symptom of the problem, and I agree, but generics with typeclasses and row-polymorphism is one answer to this.
I believe that is another facet of his point. To compose is analogous to Robert Harper’s criticism on a the monolithic abstract algebra for Haskell’s typeclasses:
https://existentialtype.wordpress.com/2011/04/16/modules-matter-most/
“As a consequence, using type classes is, in Greg Morrisett’s term, like steering the Queen Mary: you have to get this hulking mass pointed in the right direction so that the inference mechanism resolves things the way you want it to.”
Hickey would prefer we can just pretend the APIs compose fully and use testing to make sure we did not violate any of our assumptions about facets of the API our callers don’t actually engage.
I agree with his points on composition, but I find they are not actually opposed to types, but more opposed to structural types.
I agree that structural compared to nominal typing increases the lack of specificity which widens the Cambrian explosion or mass of the hulking typing morass.
I think I am inclined to be clever about how I modularize and structure APIs to minimize the issue. I do think he has a point that at some level we punt to untyped semantics and employ more testing.
@shelby3 I prefer type inference. If you can write code that looks untyped, without the need extra clutter, but it catches some problems, then this is only a benefit.
Then at the module/interface/wire-protocol level the type system forms a specification.
If the wire protocol spec, and the type system are the same, then you get rid of any impedance matching required.and therefore boilerplate types/code.
@keean I think you are just entirely missing the point that requiring everything to have a type whether it is inferred or not can become too heavy to lift.
[Inserted edit: I am sympathetic to Hickey’s point, although I fall more on the side of @keean’s preference for strong typing and at least type classes. Where I may differ from @keean is that I want to punt to not trying to type the semantics — I want to draw the line somewhere in terms of how much typing tsuris I would tolerate. Which is why I don’t like Rust, because I think the benefits of manual lifetime typing don’t outweigh the tsuris and overly complex, brittle type system.]
You think you have a typing solution for everything, e.g. row polymorphism, typed relational algebra, etc, but that belies Hickey’s point. As you well know that the more features you add to a typing system, the more byzantine it becomes due to the myriad of ways that type features interact.
I think he has a very strong point. Perfection is the enemy of good. Since we need to write unit tests anyway, then typing at some level of abuse that you seem to be advocating is just more pita than it is actually beneficial to any great extent.
Finding the right balance is what I am trying to figure out. Whereas, you seem determine to remain a type system zealot aka purist. I say actually try to build your ADA-on-steriods and see how it works out in reality.
@keean please correct me if I am mistaken as follows. And please feel free to discuss this.
I am trying to determine whether Clojure offers a viable option for my need to write code both for JavaScript clients and performant, highly scalable servers. Currently I am thinking the optimal solution for my needs would be Scala 3 compiled to ScalaJS and perhaps to Go if I create a Scala compiler plugin for transpiling (a subset of Scala that I use) to Go.
-
Clojure lacks implicit interface instance resolution at call sites, i.e. type classes, which you and I think is an essential feature for genericity, composability and extension.
-
Clojure allows sharing of references (either immutable or software transaction memory mutables) between threads, c.f. also Agents. This is not a paradigm for a Pony-esque (or my contemplated ALP-like partitioning) thread-local garbage collection (JVM doesn’t support it also) and eliminating hardware cache coherency — and thus it can’t feasibly scale to massively multicore which will become a reality soon (c.f. the Parallelism #41 and Mutable Objects #32 threads).
However, Clojure’s
pmap,futureand (for Go-like channels)Core.asyncoffers asynchronous processing which could hypothetically be employed with only copied sharing between threads. In any case, this is not even at par with Go’s elegant, lightweight green threads because Clojure runs on the dreaded JVM.(Note sharing mutable state via STM or only sharing immutable state is an orthogonal issue to insoluble fact that in any conceivable programming language, the sharing of state machines between threads, even via STM or Actors, will be susceptible to transaction conflicts)
-
Ditto as is the case for ScalaJS, ClojureScript doesn’t support (c.f. also) native JavaScript
async/awaitand instead advocates portability (e.g. to JVM on server) attained along with the less byzantine code achieved when employingcore.async.However, I have proposed to modify the ScalaJS compiler (perhaps with a Scala compiler plugin) to implement portable
Promiseinteropt that would also hypothetically compile (without requiring compile-time, output target case logic, i.e. write-once and fully portable) to both native JavaScriptasync/awaitand Go’s green threads model. I do not envision how instead to accomplish such a feat with Clojure although apparently it’s possible with ClojureScript, and I think it would be nearly impossible to source-to-source code transpile Clojure to Go because of Clojure’s the lack of strong typing.Imo much more elegant to have native
async/awaitin JavaScript code, such as for when debugging, especially given the special compilation model I proposed which I posit will fix the long-standing bifurcation composeability problem with JavaScript’sasync/await. And Go’s green threads model is superior on the server.
I replied on Quora:
I am not currently actively coding in Scala. I am reasonably familiar with the language and have been observing the progress for the Dotty Scala 3 development, which is expected to be stable within a few months perhaps. Scala 3 adds some interesting capabilities and fixes some long-standing soundness issues present in Scala 2.
Ostensibly the main distinction between Scala and Clojure is the former possessing a powerful, strong typing system with higher-kinded types and even Haskell-like type classes. Typeclasses offer more degrees-of-expression and extension than subclassing (aka OOP) which Scala also offers but which I and others think is an antipattern. Higher-kinded types enable abstracting generics for example such as Map, Reduce, Fold over any type of collection. Clojure achieves flexibility by eschewing typing but this comes with some drawbacks and advantages which we are discussing:
Macros and DSLs · Issue #31 · keean/zenscript
I have also been studying the design proposal for generics in Go:
WD-40 Issues thread was accidentally deleted by @keean · Issue #49 · keean/zenscript (#49 (comment))
I could get into a long discussion about why Scala and other PLs such as Clojure, Kotlin, etc are stuck at less than 1% popularity.
I think one critical error for ScalaJS is not supporting JavaScript’s
asyncandawaitnatively. ClojureScript currently also has this deliberate limitation, and it instead supplies its own portable (to all compile targets) API as does Scala. Thus many people just use the excellent TypeScript instead which is a superset of JavaScript and thus supports JavaScript’sasyncandawaitnatively. But TypeScript also doesn’t have higher-kinded typing which is longstanding ignored request that is heavily upvoted on their issues tracker.I looked at Elm in 2018 and was not impressed at all. Popularity does not indicate level seriousness nor revenues. So lots of script kiddies like it but some of us write serious stuff with millions of lines of code.
Higher-kinded types: the difference between giving up, and moving forward
Ironic that I have never heard of Reason. Indicates how out-of-touch I have become to a lot of the new stuff that flows out. Everytime I look at the new script kiddie flavor-of-the-month I always conclude I wasted that time. So I guess I just tuned out.
If I am not mistaken the optimization with the Google Closure compiler is optional, not required:
Clojure Developer use mostly persistent data structures with log2 performance.
Ref is only for critical sections of mutable transitions or you can leverage just Java for it.
Some points about Clojure:
- It is dynamic
- is tied to the JVM
- has immutable data structures
- has a REPL
- typed clojure is just a plugin similar to mypy, external tools are afflicted with asynchronism to their inspected target.
- has macros
- no type classes but at least some kind of Interfaces (Protocols)
- It has not the best performance because of persistent data structures
- It has horrible error message, they were improved in 1.10, but I don't know if this holds for any case, e.g. debugging
- Standard implementation of simple functions is hidden behind Core Java functions, probably because of performance.
@sighoya good to see that you’re still alive. I was worried when you went silent. I hope we can both find solutions for our health. I replied to your email about that.
Clojure Developer use mostly persistent data structures with log2 performance.
[…]
Some points about Clojure:
- has immutable data structures
Edward Kmett’s 2009 post is how I first became of aware of that immutable data structures have a logarithm performance penalty for algorithms that require random access writes.
- typed clojure is just a plugin similar to mypy, external tools are afflicted with asynchronism to their inspected target.
Strong typing is ill-fit to Lisp.
- no type classes but at least some kind of Interfaces (Protocols)
I criticized that:
- Clojure lacks implicit interface instance resolution at call sites, i.e. type classes, which you and I think is an essential feature for genericity, composability and extension.
A detailed follow-up discussion about the tradeoffs with static typing.
And a new idea that requires static typing for achieving its posited benefits.