JavaScript standard library contents

[littledan]

The JavaScript standard library is very minimal. Is this how it should be? What should we add?

Note, the purpose of this repository is to develop the infrastructure for built-in modules, not the contents. However, I know many of you have ideas for the contents as well, so let's brainstorm those in this issue exclusively.

[Mouvedia]

enhanced promises

• no leaking promises-aplus/promises-spec#179
• propagated cancelation
• splitting then into different methods

@littledan is this issue a dump for proposals? If that's the case I don't think we will ever be able to close it.

[ljharb]

Changing something that’s already in the language would need its own proposal; i think the standard library would be for adding new things.

[Mouvedia]

Changing something that’s already in the language would need its own proposal

@ljharb If the change is too radical or breaking, it has to be externalized: an alternative with another name to avoid confusion; future comes to mind in my case.

[littledan]

I meant this as a brainstorming thread. What we talk about here might graduate into proposals. As @ljharb says, let's try to focus on additions of new features, rather than changes to existing things (it's not clear how to expose what you're talking about as a built-in module). If this thread gets too unwieldy, we can close it.

[Mouvedia]

Composable?

[jasnell]

WHATWG URL should be normative.
await JSON.parse() and await JSON.stringify() or similar

[NarHakobyan]

Would be nice to have promise cancelation

[Mouvedia]

uuid?

[Mouvedia]

Anything that validates ECMAScript internal stuff.
e.g. https://mothereff.in/js-variables

[littledan]

@Mouvedia Would uuid make sense as part of WebCrypto? w3c/webcrypto#215

[littledan]

@mathiasbynens 's Unicode RegExp properties should help with making that checker more efficient!

[Mouvedia]

That validator is just an example. My point is that the language should expose its own validators/assertors.

That would permit to have a basis for other interfaces (like the DOM).
e.g. I had to make my own class/id assertors (html/css)

[Mouvedia]

@Mouvedia Would uuid make sense as part of WebCrypto? w3c/webcrypto#215

I don't know. Last time I checked the polyfill was enormous—it was almost impractical.

[littledan]

@Mouvedia Well, the idea would be that you just use uuid if you don't have the built-in one. (uuid actually uses WebCrypto in browsers today.)

[Mouvedia]

The idea of futures is something that should be pursued; in the same way that XMLHttpRequest was retroactively redefined in terms of fetch, we could have Promise be redefined in terms of Future where futures are more granular and hence give more control to the user.

@NarHakobyan #16 (comment)

[JonForest]

It would be great to be able to access object properties without having to do something along the lines of

const prop2 = obj && obj.prop1 && obj.prop1.prop2

Something like the lodash _.get function, or Angular's elvis operator in the templates (obj?.prop1?.prop2?)
I'm not super-keen on _.get because it's turning the property path into a string, which is not great from a typing and build checking point of view, but I use it as an example of desired functionality.

[decadecity]

My two longstanding bugbears with the JS standard library (going back to my pre-jQuery days) are:

1. A better interface for dealing with cookies (i.e. doesn't involve directly manipulating a string - something like Web Storage).
2. A parser/constructor for URLs like Python's urllib.parse (other languages' standard library methods for dealing with URLs are also available).

For a language that spends a lot of its time running in a browser these seem like obvious candidates for a standard library.

[littledan]

@decadecity I'm wondering if the solutions at the web level, currently shipping in browsers, are enough for your use case.

1. Does Web Storage solve the issue for you, or are there remaining problems?
2. Does the WHATWG URL API solve this issue for you?

[littledan]

@JonForest Have you see the optional chaining proposal? It's attempting to add that operator.

[decadecity]

@littledan

1. Web Storage is fine for client side storage but doesn't do cookies 😛 Directly manipulating the document.cookies string is a terrible API, I'd like to see something like the extension cookies API.
2. The URL API looks like it's about right, thanks 👍

[littledan]

@decadecity Work is ongoing to evolve the cookie API itself at https://github.com/WICG/cookie-store . TC39 is probably the wrong place to advance work in this area, as we focus on things that don't involve I/O. Cookies don't mean much outside of the Web.

[anotheredward]

In terms of what we can see people using, lodash is the most depended upon library that would be potentially appropriate to exist in core:
https://www.npmjs.com/browse/depended

It's a bit harder to break down which functions are used the most, but you can get an idea here:
https://www.npmjs.com/search?q=lodash&ranking=popularity

There is also a minimal version of the library:
https://github.com/lodash/lodash/wiki/Build-Differences

[StoneCypher]

No, the JS standard library should not be minimal. Part of the reason there's so many variations on the underscore theme is that there's a lot of really basic stuff missing from the language (like deep copy.)

Unfortunately it's not clear where feature requests belong. While I was writing out a flood, I was told "not here. Maybe on issue 16."

There's dozens of easy one-liners that would massively improve the language if gatekeepers would permit them

[StoneCypher]

I'll copy pasta a bunch right now if someone tells me where to put them

[littledan]

@StoneCypher I'm glad you came here to do additional brainstorming for contents. For JavaScript one-liners, maybe could you make your own repository or gist for development and link to it from here?

[martinheidegger]

Somehow I feel like the input from @kgryte could be useful to this discussion ;)

[StoneCypher]

Would be really nice if there was an official place to give people recommendations for a more complete standard libary

[StoneCypher]

There's a whole lot of donation languishing in repos that gatekeepers don't look for

JS could be an amazingly easier language with very little work on anyone's part if the gatekeepers would facilitate donation

[StoneCypher]

In erlang, for example, there is a bug tracker where I can go, add a ticket with some code and some appropriate tags, and if my code isn't garbage and makes sense, there's a good chance it'll show up in the next standard library under some very different name

Recommendations I gave to Ian Hicks almost 20 years ago and gave code for are being invented this year and treated as important and ground breaking ☹️

Javascript should have a process for letting people who aren't friends with standards maintainers be helpful too

[StoneCypher]

I'm just going to go ahead and put what I wrote yesterday into a ticket

If I put it in my own repo nobody will ever look at it and it will die on-site

[jhpratt]

I'm surprised no one has brought this up yet — the much-proposed range generator function would be a great addition, and presumably without much opposition. Following Python's semantics would make sense.

[ljharb]

@jhpratt Array.build was proposed long ago, and might be a candidate for picking up again.

[jhpratt]

@ljharb I think it would be better to have it as a generator to avoid memory constraints, no? Though having a map function would be useful in certain cases, I imagine.

[lachlanhunt]

More useful functions for working with generators and iterables should be included.

Standard sequence generators:

• counter(start = 0, step = 1) - Simply generates values starting from the given start value, incrementing by step. This one is most useful when combined with

• range(start, end, step) - generates a sequence from start to but not including end, incrementing by the value given in step (default 1 or -1 depending on whether start < end)

  for (let i of range(0, 10)) {
       ...
  }
  

• random() - yields pseudo random values. Though this one seems to be covered by an existing proposal https://github.com/tc39/proposal-seeded-random

Generator modifiers:

Some of these are based on existing Array or lodash/underscore methods for Arrays, which are also useful for generators so that they can be run lazily as items are pulled from the generator, rather than first converting to an array.

• map(it, mapFn)

• forEach(it, fn)

• take(it, n) - Wraps a generator to limit the number of items that can be taken from it. This is particularly useful with generators that produce infinite sequences.

  [...take(map(random(), mapToInteger(0, 10)), 10)] // Creates array of 10 random values between 0 and 10. (mapToInteger function not illustrated, but assume it converts a random float to an integer)

I've implemented these, and more, in my own library. I've listed just some of the most useful for inclusion in a standard library.

https://github.com/lachlanhunt/generator-utilities/
(Not everything in there would be appropriate for inclusion, though)

Edit: Fixed syntax error in code sample.

[viktor-ku]

If you have general range it isn't clear what type it's gonna produce: array, maybe set, maybe weak set or any other current or future iterable? Why is it worse in your opinion than Array.range() or Set.range() hence adding methods to Array or Set directly without having to do this in standard lib? The same goes with len or take or map.

Question then arrives isn't it better to rename this proposal to enhance js builtin-ins?

[StoneCypher]

It's really unfortunate that Dan keeps closing my tickets without taking the time to understand them first, even though the ones that got talked over had community support :(

I am not alone in feeling that gatekeeping in JS is out of control

[ProdigySim]

1. Get data from NPM and github looking at package.json dependencies and subdependencies
2. Find the most depended-on libraries
3. Filter by human to determine which ones would fit in a standard library

I think a Standard library for JS should seek to alleviate npm dependency issues.

Having a standard implementation for basic functionality that works in both Node and Browser environments would remove much of the need for simple NPM packages that increase the number of potential security events just by nature of distributed ownership.

[lazarljubenovic]

A bunch of things missing from Array's prototype:

• flat, flatMap (although it's a stage 3 proposal now)
• chunk(count: number): T[][], makes [1, 2, 3, 4, 5] into [[1, 2], [3, 4], [5]] with argument 2
• remove(index: number)
• takeUntil(predicate: (t: T => boolean))
• takeWhile(predicate)
• dropUntil(predicate)
• dropWhile(predicate)
• the right version of the above four, which are used for pruning from either side of the array until or while a condition holds for sequential elements
• partition, like a filter which returns two disjoint arrays
• unique, uniqueBy to remove duplicates
• filterIndex would be to filter what findIndex is to find\
• scan, a reduce which reports partial solutions on each iteration

Set operations are missing too (edit: been notified of an existing proposal). Since these are n-ary (n > 1), instead of methods like a.op(x).op(y) or a.op(...xs), static functions might make more sense:

• Set.union(...sets)
• Set.union(...sets)
• Set.difference(set, ...sets)

Not sure if Array deserves these too.

---

Sorting with the default coercion to string is ridiculous; [111, 22, 3] is already sorted by default because strings are lexicographically ascending. Maybe some global common comparison functions? Eg. for numbers, [111, 22, 3].sort(Compare.number) where Compare.number is (a, b) => a > b ? -1 : 1).

Probably a sortBy would also come in handy so we don't have to arr.sort((a, b) => a.x.y.z > b.x.y.z ? -1 : 1) and instead do arr.sortBy(a => a.x.y.z).

---

Add more data structures: Queue, Stack would be a good starting point, where implementations could choose more efficient storage strategies (eg. linked lists) instead of working with arrays and push/pop/unshift/shift.

Of course, LinkedList, Tree and Graph would be great addition as well.

The constructor of linked lists could tweak the exact type, with switches like circular and/or doubly-linked (or just introduce more constructors). Then we get things like get head, get tail, add to head, add to tail, remove from head, remove from tail, add after, add before.

Trees are interesting because the traversal of the DOM has improved a lot and it's quite pleasant to work with with all the things you can do to nodes. A lot of these operations have little to do with DOM-specific API and a lot to do with trees in general (parent, children, insertBefore, insertAfter, remove, contains, etc). They would be a starting point for more things to add to standard library such as BinarySearchTree, BTree, BPlusTree, etc, which would have all the operations you expect them to have.

Similar with graphs, lots of common operations to add, including things like loading and serializing to a matrix, etc.

---

By the way, I think we shouldn't talk about a single "standard library", but instead several of those. I mean, aren't Map, Set, Math, URL, etc. already "standard libraries"? They provide utilities for common tasks. Even Array, if it wasn't so tied to the low-level working of [] would be just a utility for working with arrays -- you could still write a for loop to Array#reverse.

[ljharb]

For set operations, see https://github.com/tc39/proposal-set-methods

[joelgallant]

Is the intention to bring any existing standard library into this import style?

I like this idea, but it would feel inconsistent if a subset of lodash, say, were to be added to the language in a different (import reserved path) way than others, like Math.

[lazarljubenovic]

I like this idea, but it would feel inconsistent if a subset of lodash, say, were to be added to the language in a different (import reserved path) way than others, like Math.

We could add the import style for Math as well and alias it back to global's Math?

[emilianobovetti]

There are so many implementations of Maybe/Option and Either/Result monads that I feel a standard library could be perfect fit for those.

[renatoagds]

@littledan Immutable API? I think we could take a look in Immutable.js and bring some ideas from there. We'll have a crash between Map and Set that's already implemented in JS, but could be a good point of start.

[jbreckmckye]

Like @emilianobovetti, I'd like better ways to handle non-existent values, perhaps using a Maybe monad.

Consider an optional object with recursively optional fields:

// Could be one of three things:
// 1. undefined
// 2. {name: 'Joanna', age: undefined}
// 3. {name: 'Joanna', age: {years: undefined}}
const user = await getUser();

const yearsOld = user.age.years; // will break 

If user was wrapped in a Maybe, we could query and manipulate its fields with no fear of typeErrors:

const maybeUser = await getUser();
const yearsOld = maybeUser
    .map(_ => _.age)
    .map(_ => _.years)
    .getOrElse('Not known');

This may become less compelling if tc39/proposal-optional-chaining advances from Stage 1.

[bmarkovic]

@jbreckmckye I don't see why optional chaining syntax couldn't be employed on "whatever-optional-type-is-chosen" to behave like similar operator does in Rust, and to behave as proposed on non-monadic objects (ie. return undefined). It would be semantically consistent, even if perhaps not fully transparent to new users at first, but then, it can't be less transparent to new users than optional types and Promises.

Speaking of Promises I personally also liked the idea of the ? operator returning undefined and discarding rejection when applied to awaited Promises i.e.

// if Promise is rejected result = undefined, move along
let result = await someFnThatReturnsPromise()? 

[tristan-shelton]

I would love to see deep get/set/copy as part of the standard library.

[lazarljubenovic]

The problem with deep copy is that it's difficult to determine how deep and what exactly you want to do. How do you "deep copy" a function? Do you carry the prototype into the copy? What about cloning things like DOM nodes? Not to mention circular structures -- graphs? trees? doubly linked lists? The only "proper" way to deep copy is having something along the lines of a copy constructor for everything (including authored classes).

[maple3142]

How about having a function like assignDeep? (It will work like Object.assign, but deeper one.)

[ljharb]

I think a “structured clone” protocol might pave the way for “deep copy” operations without having to bake in answers for those questions.

[tristan-shelton]

@lazarljubenovic While I do agree those issues should be thought through I think that any model you pick would be less confusing for most developers than the shallow copy that we get as part of the standard library today.

There are also many reference implementations of deep copy/clone out there in the wild being used by legions of developers so those would probably be a good place to look for answers to some of those questions.

[lachlanhunt]

@viktor-ku wrote:

If you have general range it isn't clear what type it's gonna produce: array, maybe set, maybe weak set or any other current or future iterable? Why is it worse in your opinion than Array.range() or Set.range() hence adding methods to Array or Set directly without having to do this in standard lib? The same goes with len or take or map.

Having a range() function as a generator means that values can be pulled from it and used lazily, rather than having to populate a whole collection first. For use cases that need the values in an Array or Set, it's trivial to do:

new Set(range(10)) // Returns a Set with ten values from 0 to 9
[...range(1, 6)] // Array with five values from 1 to 5

[gynet]

Trees support is definitely one thing we want!

[chicoxyzzy]

• exposed SameValueSameValueZero comparison algorithm
• fixed Sets/Maps which values/keys compared using SameValue

[littledan]

Object.is?

[ljharb]

It wouldn't be web compatible to "fix" Sets and Maps to differentiate positive and negative zero, but with the rekey proposal, you'd be able to make your own Map and Set instances that did this differentiation.

[chicoxyzzy]

@littledan sorry I meant SameValueZero

[chicoxyzzy]

It wouldn't be web compatible to "fix" Sets and Maps to differentiate positive and negative zero

@ljharb yes, but it's probably possible to add new fixed versions of Set and Map to stdlib

[chicoxyzzy]

Also it would be web compatible to have old broken Set and Map available globally and new fixed Set and Map available via protected namespace

[ljharb]

It would; but I think it would be very confusing to have two things named "Map" that behaved differently.

[Mouvedia]

but I think it would be very confusing to have two things named "Map" that behaved differently.

Number.isNaN vs isNaN comes to mind.

[ljharb]

Indeed, a good example to avoid repeating :-)

[js-choi]

Indeed, Map and Array.prototype.map also come to mind too.

[Mouvedia]

Here's a summary of the most popular proposals based on the 👍:

• a bunch of new methods for existing data structures
• new data structures
• a module based on RFC4122
  • proposal
• a range generator
  • related proposal
• the tryptic Maybe/Just/Nothing

[littledan]

Good summary, those all seem like good ideas (though I'm skeptical of the last one).

[Mouvedia]

though I'm skeptical of the last one

If partial application ever reaches stage 4, you will have to bow down to our new functional overlords.

[ljharb]

Certainly a Result would work very well in Promise.allSettled, and in the proposed Set/Map find methods, and a number of other places.

[evanplaice]

I kinda wish Promises never happened in JS core. They're eager evaluated and miss valuable use cases (ex async vs waterfall vs parallel) that some of the more popular Promise libs (ex bluebird, async.js) provide out of the box.

I like the idea of Futures as a 'do over' to provide a more useful alternative to promises.

• make it compatible with async/await
• make it lazy loaded so it's possible to compose futures in a functional manner
• add missing common functionality (ex async, waterfall, parallel)
• make it so the API doesn't assume implicit knowledge (ex Promise.all 👎)
• make it compatible with the proposed pipeline operator (ie and other related proposals)

[Jack-Works]

Hello, for anyone who wants a range in JS, here is an independent proposal about range https://github.com/Jack-Works/proposal-Number.range

[Mouvedia]

If you are adding a static method for numbers then Id expect the analogue on String to generate range of characters:

/* String.range(range: String, charset = 'UTF-8': String): String */

String.range('∏-»', 'machintosh'); // ∏π∫ªºΩæø¿¡¬√ƒ≈∆«»
String.range('Ø-é', 'ISO-8859-1'); // ØÙÚÛÜÝÞßàáâãäåæçèé

[Jack-Works]

If you are adding a static method for numbers then Id expect the analogue on String to generate range of characters:

/* String.range(range: String, charset = 'UTF-8': String): String */

String.range('∏-»', 'machintosh'); // ∏π∫ªºΩæø¿¡¬√ƒ≈∆«»
String.range('Ø-é', 'ISO-8859-1'); // ØÙÚÛÜÝÞßàáâãäåæçèé

The range for String is not in the goals of Number.range, you can build your own proposal, I'll focus on numbers and BigInts only 😉

[Mouvedia]

I was using the generic you i.e. I wasn't talking to you :)
What I meant is that a generic range module would have to support characters and numbers.

[obedm503]

@Mouvedia
This might be derailing the conversation, but what is the purpose/use case of a string range generator?

[Mouvedia]

@obedm503 you will have to come up with the use cases yourself.

ruby: https://ruby-doc.org/core-2.2.0/Range.html
PHP: http://php.net/manual/fr/function.range.php
groovy: http://groovy-lang.org/operators.html#_range_operator
swift: Range/ClosedRange
nim: https://nim-lang.org/docs/system.html#..%2CT%2CU
prototype: http://prototypejs.org/doc/latest/language/ObjectRange/
etc.

[kaizhu256]

Trees support is definitely one thing we want!

a builtin sqlite3 (like python) would be far more practical and useful in web-applications than a low-level tree-library.

[kaizhu256]

i don't know how it would be implemented, but some sort of full-text-search primitives would greatly benefit the industry.

[evanplaice]

@kaizhu256 a builtin sqlite3 (like python) would be far more practical and useful in web-applications than a low-level tree-library.

Local storage can be used for quickly/easily persisting smaller amounts of data. Also, keep in mind that this would be for JS as a whole; so probably needs to be useful in bith the browser and node.

[kaizhu256]

i may be wrong but i think what industry really wants is a "standardized" sqlite3 in browser with full-text-search capabilities (ala websql, which never seems to have truly died [1]). indexeddb is just terrible in terms of practical usability/performance (joins need to be done in userland-javascript? really?).

adding a low-level tree-library so users can reinvent even crappier versions of indexeddb is pointless.

i also don't mind a sqlite3-library to be memory-only, so it can be encapsulated for security reasons. one can always [infrequently] persist it to indexeddb as a blob.

speaking of indexedb, its sorely in need of an appendFile feature. without it, popular browser-db's like nedb are impractical for persisting largish datasets (> 20mb) [2].

[1] current websql browser-support
https://caniuse.com/#search=websql

[2] nedb emulates appendFile by re-saving its entire database as a blob to indexedb whenever a row insert/delete/update is performed.
https://github.com/louischatriot/nedb/blob/v1.6.0/browser-version/browser-specific/lib/storage.js#L49

[littledan]

I'd recommend making feature requests for IndexedDB in their GitHub repository; you're not likely to reach the right folks here.

[Mouvedia]

related: https://github.com/samchon/tstl

[tarjei]

leftpad?

[lazarljubenovic]

leftpad?

It exists.

[ege-lmobile]

SQL is everywhere, and I think some sort of supported SQL library built in on top of indexedDB without downloading a megabyte of asm.js generated code would be amazing.

We also need generic versions of the whole array and string -prototype functions. For extra functionality, see python for inspiration. For example: https://docs.python.org/3/library/textwrap.html A universal print function! That must be in. Math and date functions. Especially date functions for dealing with time-zones and all. Some way to work with complex numbers in the math library?

Just look here: https://docs.python.org/3/library/
Let's tackle all the relevant ones!

[tarjei]

@lazarljubenovic I stand corrected! Thanks :)

[kaizhu256]

• @kripken, @mapbox on feasibility of a sqlite3 standard-library.

there already exists asm.js-compiled sqlite3 at http://kripken.github.io/sql.js/GUI/.
curious on feasibility of having something similar (or wasm-variant) incorporated as standard-library?

[maetl]

• utils for string scanning and tokenizing
• immutable DateTime and TimeRange/Duration objects with timezones, arithmetic
• Queue and PriorityQueue
• 2D and 3D vector points and matrix math
• Trie
• deep copy
• immutable versions of Array, Object(?), Hash, Set, etc
• money and currency

[bfattori-TDA]

The stdlib should be similar in functionality to that of other languages. As such, I would suggest taking something like lodash or underscore and pairing it down to its most useful functions. For example, JS has never had a simple way to deep-extend an object - but the two aforementioned libraries do. They are essentially the same thing and probably the most ubiquitous libs for basic operations.

Suggestion: Start from Lodash or Underscore and pair back, then extend with any missing functionality but only those that affect the built-in objects.

[evanplaice]

@bfattori-TDA IMO, lodash/underscore are a step in the right direction in terms of providing functional transforms for data.

Unfortunately, if the pipeline-operator proposal passes, we'll likely see their functionality re-incarnated in a new library that fully leverages native pipes. Adding them in their current form would be premature.