Create an example without depending on TypeScript, Webpack, Node.js, npm
Closed this issue · 57 comments
This should be possible with without depending on TypeScript, Webpack, Node.js, npm.
+1 a build could be served on a gh-pages branch via github.io.
I'd like to confirm whether I understand both of you correctly. You'd like to have a simpler way to download a pre-built version of the app, right? Or is the issue that the app itself is too complex, and you would prefer to also have a separate repo that contains a super simple test app that doesn't require Webpack / TypeScript / etc. to build it?
I don't generally install Node.js. I fetch the Node.js nightly archive and get rid of everything except the node executable and run that from the directory I downloaded to. npm doesn't work out of the box when Node.js is not installed. For npm to work without Nde.js installed several files in the Node.js download folder have to be modified, however, I normally don't even write files other than node executable to my file system.
Never tried Webpack. I have no use for TypeScript. I do want to experiment with DirectSockets, I will start by using txiki.js to create a local TCP socket.
I think you are making the assumption that npm is installed, TypeScript is installed. The example should be as generic as possible. Or, create a generic example where the user can plu-and-play and use whatever programming language they select for the socket.
The intention of this repository is to demonstrate how an Isolated Web App could be built using a typical developer workflow. For that we chose Webpack / Typescript but of course there are others. Modifying this example to remove all its dependencies would defeat the purpose.
using a typical developer workflow
I think that is your preference based on the context of your environment.
Some developers don't use or depend on external libraries, package managers.
Just JavaScript as specified by Ecmascript; Web API's, DOM methods, HTML shipped in the given browser.
Modifying this example to remove all its dependencies would defeat the purpose.
How so?
You can easily create a different example without dependence on thord-party libraries.
Not everybody installs Node.js. Node.js, or Deno (V8) JavaScript engines. QuickJS, txiki.js, Bun, etc. do exist and are used by modern web developers.
Not everybody uses TypeScript or Webpack.
At the bare minimum you can provide detailed explanation of what
your local server.
in README is expected to look like. I will first use a QuickJS server module https://github.com/guest271314/webserver-c/tree/quickjs-webserver, because Node.js is too expensive (40MB to run, over 80MB executable) to justify using in a temporary file system running on RAM, or as an embedded application.
There are quite a few repositories on GitHub that are using QuickJS compiled to WASM as a JavaScript runtime in the scope of a "worker" server - nobody that I am aware of has even tried to compile V8 to WASM to embed in a system, let alone Node.js.
Ok.
The intention of this repository is to demonstrate how an Isolated Web App could be built using a typical developer workflow.
Just peeking at https://github.com/GoogleChromeLabs/telnet-client/blob/main/webpack.wbn.js you are still using require()!
That is a Node.js artifact. We are using Ecmascript/HTML import and export now.
That is one issue with writing source code in Node.js.
I use .mjs extension and no package.js or node_modules at all when I use node executable https://github.com/guest271314/native-messaging-nodejs. The first thing I would do is get rid of those require() calls. That is, if we are talking about modern web development, and not continuing to use Node.js exclusive artifacts.
I think I can implement the front-end. I just need to know what the server looks like at a low-level, so I can implement this myself using C, C++, QuickJS, txiki.js, Deno, etc.
To me TypeScript is just a hyped-up linter; an entirely different language from JavaScript, that just happens to compile to JavaScript. I can write JavaScript without errors without TypeScript. In fact, I enjoy doing so. If I'm going to compile something it's gonna be C, C++, or Rust - not TypeScript, without any compelling reason to do so other than
a typical developer workflow
I'm not typical, nor a follower. Just describe, in detail, the technical process in your README and I'll implement an example myself, using various programming languages and JavaScript engines for a local server.
To me it makes no sense to download Node.js (the last time I fetched the Nightly build it was ~100MB) just to build a local server, when we can do so in C for less than 1MB executable and ~5MB running. That's fine if you think Node.js, TypeScript, Webpack are required for your environment. I think the public example should be far more generic, so developers in the field can just plug-and-play whatever server they want to or have developed other than Node.js (using `require()!). I think that's fair.
You'd like to have a simpler way to download a pre-built version of the app, right?
This was my desire though now I realise it's different from the original poster's.
You'd like to have a simpler way to download a pre-built version of the app, right?
This was my desire though now I realise it's different from the original poster's.
I think on the front-end all we have to do is create an Isolated Web Application, or per DirectSockets repository, a Progressive Web Application, include some Chromium/Chrome flags, then go.
On the server side I think we can use a Unix socket, not just telnet. This is what I am asking for. The bare bones, low-level explanation, without assuming the developer has Node.js, TypeScript, Webpack installed and depending on those applications, languages, and libraries just to create a local socket that we can connect to using Direct Scckets API. We should be able to use nc or netcat on Linux to connect to from the browser, without necessarily installing anything.
@reillyeon Alright, I downloaded the repository and launched the app with npm run start. What is the expected result in the app window once connection is establish and Local echo option is checked?
xterm.js is logging errors
Looking at reports of this issue on the xterm.js project, it seems like this behavior is expected and in a real interactive terminal application the backspace key needs to be handled by the application, not the terminal. Enabling local echo simply duplicates keyboard input back to the terminal itself.
That's why I said this example is way too over-complicated. After downloading Node.js, webpack, and packages this should just work.
More importantly this should work without Node.js, webpack, etc. The only thing DirectSockets does is enable socket connections, which we can do using Linux built-ins.
You continue to miss the point of why this example project exists. It's goal is to be just complicated enough to prove that complex projects can be built into an Isolated Web App.
If you want to build something simpler you are free to do so. I will be closing this issue. There are other developer resources that are being developed to explain how to build an IWA that will be published when this work is closer to launching that may be helpful to you.
So this is about Isolated Web Apps and not DirectSockets. Still, the dependencies and code in the repository is unnecessarily complicated, for no apparent reason, and the code doesn't work anyway,
Folks over on DirectSockets are being referred to this example when it doesn't even work!
Looking at WICG/direct-sockets#46 (comment), the reduced Direct Socket API example seems to be https://direct-sockets-ntp.glitch.me/.
I'll try that over the next few days. Thanks.
@tomayac There is no 'Isolated App Origins' filed in chrome://flags on Chromium 117.0.5891.0 (Developer Build) (64-bit) Revision f8142065670523d0c5a8b91ba26e8cdc09e9933f-refs/heads/main@{#1170895}.
Listen to the people. This is too complicated (for no good reason) - to not work after jumping through all the different hoops that keep changing just to try to experiment with Direct Sockets.
It is true that the 'Isolated App Origins' flag no longer exists, so the Direct Socket demo's instructions are unfortunately outdated. The only way to demo Direct Sockets at the moment is by creating an IWA, like the telnet client, and installing that.
The only way to demo Direct Sockets at the moment is by creating an IWA, like the telnet client, and installing that.
That doesn't work. See above where errors are thrown. Have you folks tested this?
Why all the unnecessary dependencies? The socket connection is user-defined. We should be able to do this using Bash, C, whatever.
I'm alright with using an Isolated Web App. Unfortunately, Direct ockets has been tied in to Isolated Web Apps. But y'all keep moving the goal-posts on installation requirements in Chromium.
I know I am far more interested in Direct Sockets than IWA. I will do what it takes to get to experiment to perhaps eventually test Direct Sockets.
Since you are now the custodians or access to Direct Sockets, kindly describe precisely what I need to do to create an Isolated Web App where TCPSocket, et al. will be exposed and defined globally, and I can take it from there - without necessarily using TypeScript or installing Node.js.
This is an experimental technology where the complete design is not yet available. This is why the requirements for using the API are shifting. Once it has been enabled by default the requirements will be stable.
I have asked the owner of https://direct-sockets-ntp.glitch.me/ to update the instructions for running it.
@GrapeGreen A step-by-step documentation on how to get Direct Sockets exposed - without dependence on Node.js, TypeScript, Webpack, npm will be very helpful. The goal posts keep moving. We can use a Bash shell script of txiki.js for a TCP socket so we don't need to install a whole bunch of packages just to use Direct Sockets. Can you kindly put together such a step-by-step outline?
Just a sidenote here that using the wbn-sign npm package to bundle your IWA requires Node anyway, because JavaScript's Web Crypto API doesn't support Ed25519 yet and thus Node's crypto library must be available in order to use the tools. If you don't want to have Node installed, then the alternative would be to use the Golang tool for bundling and signing your code.
If you don't want to depend on Webpack / Rollup (still requiers Node tho), you can use the CLI tools of the wbn & wbn-sign libraries (latter requires Node).
@GrapeGreen A step-by-step documentation on how to get Direct Sockets exposed - without dependence on Node.js, TypeScript, Webpack, npm will be very helpful. The goal posts keep moving. We can use a Bash shell script of txiki.js for a TCP socket so we don't need to install a whole bunch of packages just to use Direct Sockets. Can you kindly put together such a step-by-step outline?
We have now updated the instructions at https://direct-sockets-ntp.glitch.me/ - let me know if it works for you :)
GET https://direct-sockets-ntp.glitch.me/manifest.json 404 manifest.json:1
Manifest: Line: 1, column: 1, Syntax error. manifest.json:1
let me know if it works for you :)
No, it doesn't. There is a 404 error then a syntax error when requesting the manifest.json file.
let me know if it works for you :)
No, it doesn't. There is a 404 error then a syntax error when requesting the manifest.json file.
I'm curious -- why do you need to fetch the manifest directly? The installation flow from url doesn't require that -- have you taken a look at the instructions?
If that's a must, call GET on https://direct-sockets-ntp.glitch.me/manifest.webmanifest
I'm notifying you the link you shared evidently does not load the manifest.json file. Maybe your code needs to make a GET request for manifest.webmanifest instead of manifest.json because right now the Web site does not prompt the user to Install anything.
I'm notifying you the link you shared evidently does not load the manifest.json file. Maybe your code needs to make a GET request for manifest.webmanifest instead of manifest.json because right now the Web site does not prompt the user to Install anything.
Ah, thanks for that, fixed.
The website is not supposed to prompt you to install an IWA -- have you checked the provided instructions? It should be installable from the command line, not from the UI.
I don't know what to tell you. Doesn't work under any variation of CLI flags or using chrome://apps
../launch_chromium.sh
[118130:118130:0805/102051.223319:ERROR:policy_logger.cc(154)] :components/enterprise/browser/controller/chrome_browser_cloud_management_controller.cc(163) Cloud management controller initialization aborted as CBCM is not enabled.
[118130:118130:0805/102051.253021:ERROR:startup_browser_creator.cc(1078)] Command line switches to install IWAs are incompatible with the Profile Picker. If you have multiple profiles, consider using the --profile-directory switch to select a profile (it accepts the name of a profile directory in /home/user/.config/chromium, such as 'Default').
Alright, got it working.
The app launcher looks like
/home/user/chrome-linux/chrome --profile-directory=Default --app-id=<ID>
the browser launcher
#!/bin/sh
$HOME/chrome-linux/chrome \
--profile-directory="Default" \
--install-isolated-web-app-from-url="https://direct-sockets-ntp.glitch.me"
--enable-features=IsolatedWebApps,IsolatedWebAppDevMode
@GrapeGreen Now the question is how do we create a completely local version where the source code can be modified and launched from localhost?
When I create a local TCP server with e.g., txiki.js I'm getting an error
#!/usr/bin/env -S /home/user/txiki.js/build/tjs run
const encoder = new TextEncoder();
const decoder = new TextDecoder();
async function doEchoServer(server) {
const conn = await server.accept();
if (!conn) {
return;
}
const buf = new Uint8Array(4096);
while (true) {
const nread = await conn.read(buf);
console.log(decoder.decode(buf));
if (nread === null) {
break;
}
conn.write(buf.slice(0, nread));
}
}
const server = await tjs.listen('tcp', '0.0.0.0', '8000');
console.log(server.localAddress);
doEchoServer(server);
sudo netstat -plnt | grep tjs
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 127914/tjs
var socket = new TCPSocket({localAddress: '0.0.0.0', localPort: '8000'}, {});
undefined
m7h56ge27i75c25ycuv2…oralc35zffnqaaac/:1 Uncaught (in promise) DOMException: Hostname couldn't be resolved.
The netstat report uses 0.0.0.0 as a special IP address that means "listening on all IPv4 addresses". You need to specify a valid IP address or hostname (e.g. 127.0.0.1 or localhost) for the server.
It's a little weird that you get "hostname couldn't be resolved" instead of something more specific like "invalid peer address".
Alright got it working with this in txiki.js
const server = await tjs.listen('tcp', '127.0.0.1', '8000');
and this at DevTools console of the Isolated Web App
var socket = new TCPSocket('127.0.0.1', 8000);
var {writable, readable} = await socket.opened;
var writer = writable.getWriter();
readable.pipeThrough(new TextDecoderStream())
.pipeTo(new WritableStream({
start() {
console.log('Stream started');
},
write(value) {
console.log(value);
},
close() {
console.log('Stream closed');
}
})).catch(console.dir);
VM782:4 Stream started
Promise {<pending>}
await writer.write(new TextEncoder().encode('yo'))
undefined
VM782:7 yo
var encoder = new TextEncoder();
undefined
await writer.write(encoder.encode('test'))
undefined
VM782:7 test
Now do I modify the client.js, register-sw.js, and manifest.webmanifest locally? Where are the files stored? In Chromium/Chrome configuration folder?
Ah... Turn off the Internet and this doesn't work. This (DirectSockets and to the extent applicable Isolated Web Apps) needs to work offline, completely locally.
When this is enabled outside of development IWAs are going to be installed from Signed Web Bundles, which you can test today with the --install-isolated-web-app-from-file flag. In that mode the app is served from the specified bundle and so it works offline.
The --install-isolated-web-app-from-url flag installs the app in what we call "proxy mode" which is useful for development purposes but still relies on the upstream server. This is designed for the case where you are coding against a local test server or are installing an app served by your development machine on a test device. Constantly copying around a bundle would be annoying during development, which is why we support this mode.
As a workaround, you can make an app installed this way work offline by adding a Service Worker and implementing offline functionality in the same way you would for any web app.
This should be enabled in any environment the developer decides to use Direct Sockets.
I'll dive in to creating a minimal Web Bundle.
Full disclosure: My ultimate use case for doing so is to embed an <iframe> in an arbitrary document or open an offscreen document or window using open() or a browser extension pointing to the Isolated Web App URL, then transfer the readable to the Web page, so that I can implement full duplex streaming on any Web page using Direct Sockets.
What you are intending to do is intentionally unsupported.
Why?
I don't see a reason for all of this elaborate process. You can't really prevent the use case I laid out from happening.
I've done this multiple ways already, both using Web API's and browser extension code, e.g., https://github.com/guest271314/requestNativeScripts.
We already have Native Messaging over IPC, WebSocketStream and fetch() require a server.
I'm basically going to implement Native Messaging using Streams and Direct Sockets. See https://bugs.chromium.org/p/chromium/issues/detail?id=1214621.
What you are intending to do is intentionally unsupported.
Keep in mind what I described is already possible. Because the Isolated Web App is a window, which any developer can get a handle on then establish cross-origin messaging using postMessage(). With an extension we can get a handle on all tabs and windows.
Direct Sockets should just be enabled by flag with origins specified the user wants to expose the feature. That vastly simplifies the process.
which you can test today with the
--install-isolated-web-app-from-fileflag.
Is there an example of that process? The below doesn't appear to work.
--install-isolated-web-app-from-file="file:///home/user/hello_b1.wbn"
The command line flag throws:
../launch_chrome.sh
[190829:190829:0812/080446.257874:ERROR:install_isolated_web_app_from_command_line.cc(274)] Isolated Web App command line installation failed: Invalid path provided to --install-isolated-web-app-from-file flag: 'file:///home/user/hello_b1.wbn'
@reillyeon Alright, I got the --install-isolated-web-app-from-file flag work, and am launching with --app-id=<ID> then moving the standalone window to a tab for testing. We can send messages to and from the Isolated Web App through browser extension messaging to start, pass messages, and stop the TCPSocket().
Are you sure Direct Sockets is working correctly?
I tested txiki.js TCP server https://github.com/saghul/txiki.js/blob/master/tests/test-web-streams.js and Deno TCP server https://deno.land/manual@v1.36.1/examples/tcp_echo.
When we do something like
var socket = new TCPSocket('0.0.0.0', '8000');
var abortable = new AbortController();
var {signal} = abortable;
var {readable, writable} = await socket.opened;
socket.closed.then(console.log).catch(console.warn);
readable.pipeThrough(new TextDecoderStream()).pipeTo(
new WritableStream({
write(v) {
console.log(v);
}, close() {
console.log('Socket closed');
},
abort(reason) {
console.log({reason});
}
})
, {signal}).then(()=>console.log('pipeThrough, pipeTo Promise')).catch(console.log);
var encoder = new TextEncoder();
var enc = (text) => encoder.encode(text);
var writer = writable.getWriter();
then call writer.close() the TCP server does not observably close. The same with controller.abort().
That's why this shouldn't be gated behind Isolated Web Apps and Web Bundles. Not so easy to test. We have to jump through a few hoops to finally find this out.
It looks like to close the connection we have to call writer.close() then abortable.abort('reason').
It looks like to close the connection we have to call
writer.close()thenabortable.abort('reason').
The draft spec for Direct Sockets contains information about the closing behavior, maybe they can help? E.g.: https://wicg.github.io/direct-sockets/#close-method
It looks like to close the connection we have to call
writer.close()thenabortable.abort('reason').The draft spec for Direct Sockets contains information about the closing behavior, maybe they can help? E.g.: https://wicg.github.io/direct-sockets/#close-method
Using AbortController works.
Where and how to we modify or remove the default (CSP, COEp, CORP) headers in the plugin before signing the Web Bundle?
If you are using the Webpack plugin, you can override headers by specifying the headerOverride and isIwa options. See here for more details: https://github.com/GoogleChromeLabs/webbundle-plugins/tree/main/packages/webbundle-webpack-plugin
I think I tried that and it didn't work.
To be clear is it possible to remove the CSP, COEP, COOP headers and still have Direct Sockets exposed?
opener is null in the IWA. I'm trying to have opener be what it is supposed to be when open('isolated-app://<ID>/') is called.
I think I tried that and it didn't work.
To be clear is it possible to remove the CSP, COEP, COOP headers and still have Direct Sockets exposed?
No, this is what Reilly has been saying above. A key requirement to expose this API safely is to have these headers set and being in the context of an IWA.
Well, I've already gotten around that a couple ways, it's just minimally tedious.
You can't really prevent users from working around these things.
We can still communicate with an extension and from the extension communicate with any arbitrary Web page via IPC messaging; and we can create a WebRTC connection between the IWA and the arbitrary Web page, among other approaches.
The Direct Sockets thing should really be exposed as an extension, so developers can simply write in a manifest which origins they want that feature exposed on.
@cmfcmf E.g., there is really no way for you to prevent this from happening https://github.com/guest271314/offscreen-webrtc in an IWA.
The Direct Sockets shouldn't be gated behind IWA so folks don;t have to try to find ways to break your your gear and expose said breakage.