Arlen22/tiddly-chrome-app

Support for TiddlyWiki Classic

YakovL opened this issue · 16 comments

YakovL commented

As far as I understand, it's not. As an active user of TWc and a developer, I can help you with supporting TWc, too (but I have to understand how TiddlyChrome works).
Best regards,
Yakov.

Arlen22 commented

It currently is not, as I do not know enough about it. The relevant file is linked below. You can see some of my comments concerning TWC.

https://github.com/Arlen22/tiddly-chrome-app/blob/master/src/tiddlyChromeFoxer.js

I have a few questions about TWC.

  1. Which functions does it need to save the file?
  2. Does it use these functions only for saving the main TWC file or also for other things like exports?
  3. What would I need to account for or how would I know if it is the main TWC file that is being written or another file?
YakovL commented

Well, the situation is the following:

  • there's a somewhat long chain of functions that causes saving, but one of the most important is window.saveFile 1. It is used for saving TW itself, backups, RSS, used by SaveAsPlugin (tiddlytools.com/#SaveAsPlugin) etc. It's used to save any text files and its implementation, as you can see, contains browser-dependent functions
    ** it is desirable in some cases to have that flexibility to save files even outside the current folder (esp. when using SaveAsPlugin), so I hope you won't restrict that or at least provide such an option
  • if you want to start from a more restricted mode, you probably are looking for the saveMain 2 and saveBackup 3 functions. But taking into account also those RSS and other features, I'd say it would be impractical to implement those separately (and all those use saveFile)
  • another thing to note is that current implementation of saving functions is syncronous which prevents users from using autoSave much, and async saving would be much more useful since it doesn't "freeze" TW for saving (I've implemented that in MicroTiddlyServer and it works nicely and improves workflow greatly), so I strongly suggest to implement it that way (actually for both TWc and TW5). Supporting async saving with messages like the ones you may see in saveBackup (about successful or insuccessful saving) would cost some additional tweaking of code, but not much really

Due to the contemporary browser restrictions, TW is also limited in loading files which is harmful for including (see SharedTiddlersPlugin and some other things, so if you implement the loadFile 4 method, it will be extra cool. Moreover, if you make loadFile work with remote sites, too, you will enable currently disfunct core upgrading mechanism 5 and open many other possibilities which are missing in TiddlyFox. Of'course async loading will be useful, too.

YakovL commented

Sorry, I've missed an important bit about async saving. The thing is, TWc is saved by "updating original" meaning that the original TW is loaded first and then the saving is done. The root of the process is the saveChanges function. It uses loadOriginal which calls loadFile; then it uses saveMain where the file contents text is updated via updateOriginal and then saveFile is used. So to make saving actually async we have to rewrite this

  • saveChanges
    • loadOriginal
    • saveMain
      • updateOriginal
      • saveFile

chain in an async manner (otherwise user has to wait until loading is done anyway, although that would take less time). I can help with doing that if you like, and this is quite probably something to put to the core.

YakovL commented

So the first step would be to rewrite the loadOriginal + stuff after inside saveChanges in the async fashion:

function loadOriginal(localPath,callback)
{
	var content = loadFile(localPath);
	if(!content) content = window.originalHTML || recreateOriginal();
	if(callback)
		return callback(content);
	return content; // for backward compability
}

and

function saveChanges(onlyIfDirty,tiddlers)
{
	if(onlyIfDirty && !store.isDirty())
		return;
	clearMessage();
	var t0 = new Date();
	var msg = config.messages;
	//# Get the URL of the document
	var originalPath = document.location.toString();
	//# Check we can save this file
	if(!window.allowSave()) {
		alert(msg.notFileUrlError);
		if(store.tiddlerExists(msg.saveInstructions))
			story.displayTiddler(null,msg.saveInstructions);
		return;
	}
	var localPath = getLocalPath(originalPath);

	// Load the original file and do updating and saving
	var onLoad = function(original)
	{
		if(original == null) {
			alert(msg.cantSaveError);
			if(store.tiddlerExists(msg.saveInstructions))
				story.displayTiddler(null,msg.saveInstructions);
			return;
		}
		//# Locate the storeArea div's
		var posDiv = locateStoreArea(original);
		if(!posDiv) {
			alert(msg.invalidFileError.format([localPath]));
			return;
		}
		var co = config.options; //# abbreviation
		config.saveByDownload = false;
		config.saveByManualDownload = false;
		saveMain(localPath,original,posDiv);
		if (!config.saveByDownload && !config.saveByManualDownload) {
			if(co.chkSaveBackups)
				saveBackup(localPath,original);
			if(co.chkSaveEmptyTemplate)
				saveEmpty(localPath,original,posDiv);
			if(co.chkGenerateAnRssFeed && saveRss instanceof Function)
				saveRss(localPath);
		}
		if(co.chkDisplayInstrumentation)
			displayMessage("saveChanges " + (new Date()-t0) + " ms");
		return original;
	}
	return loadOriginal(localPath,onLoad);
}
YakovL commented

Next step would be to do the same stuff with window.loadFile and stuff inside loadOriginal:

function loadOriginal(localPath,callback)
{
	var onLoad = function(content)
	{
		if(!content) content = window.originalHTML || recreateOriginal();
		if(callback)
			return callback(content);
		return content; // for backward compability
	}
	return loadFile(localPath,onLoad);
}

or, to make it shorter,

function loadOriginal(localPath,callback)
{
	return loadFile(localPath,function(content)
	{
		if(!content) content = window.originalHTML || recreateOriginal();
		if(callback)
			return callback(content);
		return content; // for backward compability
	});
}

and

window.loadFile = window.loadFile || function(fileUrl,callback)
{
	var r = mozillaLoadFile(fileUrl);
	if((r == null) || (r == false))
		r = ieLoadFile(fileUrl);
	if((r == null) || (r == false))
		r = javaLoadFile(fileUrl);
	if(callback)
		return callback(r);
	return r;
}

And after that, window.loadFile should be extended to the async version using what Chrome Extensions can do. And saveFile should be extended as well (in fact, it doesn't have to be async for saving to be async!)

Arlen22 commented

mozillaLoadFile and mozillaSaveFile is the ones that I would like to implement. But I have to know how to tell whether the wiki is trying to save itself or another file. Because from what you're saying, it uses it for everything, right? So I need to know what string I will be seeing. Also, keep in mind that the wiki is loaded via a blob URL in TiddlyChrome, IIRC..

Arlen22 commented

I could implement other functions, if necessary.

YakovL commented

Hi, sorry for delay,
I guess you can overwrite window.loadFile and window.saveFile as a whole (anyway it is to be altered so that mozillaLoadFile is called with a callback as an argument); that's is, right.
To know whether the wiki is trying itself, you may check the path, I think: as you can see, it is calced this way:

var originalPath = document.location.toString();
... // window.allowSave() check if originalPath starts from file: or not
var localPath = getLocalPath(originalPath);

I'm not a security expert, but AFAIK document.location can't be altered without browser getting somewhere else, so if the url suggested to save to coinsides with getLocalPath(document.location.toString()), we're saving the wiki itself. Though, if one wants to trick this test, they can rewrite getLocalPath to always return the TW's path, so if you want to implement a somewhat strict restriction, you should copy/adapt getLocalPath into TiddlyChrome itself (the drawback would be if getLocalPath gets changed in the core, it should be changed in TC as well, but I wouldn't expect that happen any time soon). Anyway, saving functionality of TW is usually used by a user themselves, so usually using getLocalPath from the core should be enough (though, I don't know if a Chrome extension can utilize JS of the page).

keep in mind that the wiki is loaded via a blob URL in TiddlyChrome, IIRC..

Have to admit, I'm not sure if this has anything to do with implementation of what we're discussing :)

Arlen22 commented

Actually, nothing needs to be asynchronous, I don't think. Unless load file really needs it but I think all that happens is the message in the top right? Or am I missing something?

The savefile would be asynchronous most likely, but it doesn't really matter if it returns early? Or does it? Maybe it would break dirty checking. In any case, we could always see how TiddlyFox handles it.

Arlen22 commented

Actually, looking over your comments again, you seem to know TWC quite well. Hey, if you give me the code we can work together on getting it in. You will have to make one minor change and that is allow blob: URLs as we as file: URLs. Or maybe handle Blob differently. We can do whatever we want if the code is in our plugin anyway.

But if it makes it easier, loadfile can be synchronous because it is already stored in the memory. It won't necessarily be reloaded from disc. And it won't be async anyway, I'm pretty sure. Although I would have to check for sure. I guess making it async won't hurt anything. It will still work both ways.

Arlen22 commented

Also, I currently have no way of loading external local files. HTTP with CORS should work fine, though.

YakovL commented

Well, I'm not sure about the mechanism, but syncronous saving is bad since TW hangs until the saving is finished, which is somewhat annoying at times (and the bigger TW is the longer this delay becomes). And the point is I've implemented async saving in MicroTiddlyServer (I'm planning to release it after fixing one issue) and it improved workflow greatly. (MTS is a potential solution to many issues, but it requires a PHP server and also for now limits in some aspects using TWs outside the working folder of the PHP script, so making other saving solutions will be quite valueable.)

Loading original looks somewhat excessive for now, but actually I'm thinking of making use of it: remember original on load, get it again before saving and notify if they do not coinside (useful for me when I use TWs from my usb stick on different devices but, more importantly, will be useful for working with one TW for multiple authors; moreover, this mechanic can be extended not only to notify but to actually merge non-conflicting edits on the go).

if you give me the code we can work together on getting it in. You will have to make one minor change and that is allow blob: URLs as we as file: URLs. Or maybe handle Blob differently. We can do whatever we want if the code is in our plugin anyway.

Sorry, I'm still missing something...

wiki is loaded via a blob URL in TiddlyChrome

could you reference the piece of code that implements that? I'm not getting what "allow blob: URLs as we as file: URLs" means.

Also, I currently have no way of loading external local files. HTTP with CORS should work fine, though.

Right, let's get saving work first, then explore other stuff.

Arlen22 commented

This is the line that sets the webview src. Basically it's the URL that the tiddlywiki sees.

https://github.com/Arlen22/tiddly-chrome-app/blob/master/src/window.js#L107

Arlen22 commented

And this code that you quoted above. window.allowSave apparently contains the required code.

        if(!window.allowSave()) {
		alert(msg.notFileUrlError);
		if(store.tiddlerExists(msg.saveInstructions))
			story.displayTiddler(null,msg.saveInstructions);
		return;
	}

Also, you can find the appropriate instance of the app to debug by first opening the app, then opening a new tab in a regular chrome window and navigating to chrome://inspect, then select the app or its webview.

YakovL commented

Hi Arlen, sorry, we've got a very instensive month before a crucial deadline at work, but now I'd like to resume this. Let's clarify where are we so far:

  • you say that a TiddlyWiki is loaded via a blob URL
  • (I haven't used blobs in JS and) I don't understand whether the address in document.location is changed to that blob, or there's still usual stuff like "file:///D:/some/path/tw.html" in it. In the first case document.location.toString can't be helpful in checking whether we're trying to save current TW or some other file, I guess
  • you propsed that I implement this change: "allow blob: URLs as we as file: URLs. Or maybe handle Blob differently." But I still don't understand what do you mean. This may be useful, as well as disappointing, but I'm not sure
  • you said that "This is the line that sets the webview src. Basically it's the URL that the tiddlywiki sees. https://github.com/Arlen22/tiddly-chrome-app/blob/master/src/window.js#L107" Unfortunately, I'm not familiar with what webview is and don't understand what do you mean by "tiddlywiki sees"

You asked the following questions:

  1. Which functions does it need to save the file? → looks like I've answered that
  2. Does it use these functions only for saving the main TWC file or also for other things like exports? → same
  3. What would I need to account for or how would I know if it is the main TWC file that is being written or another file? → depends on the answer to my question above, I guess
    You rephrased that one as "how to tell whether the wiki is trying to save itself or another file?" and added "I need to know what string I will be seeing." I'm not sure which string are you asking about?
  4. The savefile would be asynchronous most likely, but it doesn't really matter if it returns early? Or does it? Maybe it would break dirty checking. In any case, we could always see how TiddlyFox handles it. → There shouldn't be any issues if proper callbacks are passed. Setting dirty to false should be done in the callback in the case of success. As for TiddlyFox, it is not async now.

What'd say you? I guess I need some links to descriptions of concepts I'm not very good with: loading via blob URLs and WebView (well, I've already found some about blobs). It seems like loadOriginal will require HTTP with CORS anyway (and after that they should be turned into async requests to prevent "hanging" of the interface). Do you still have any questions?

Arlen22 commented

I guess I would rather not commit the time to explore the TiddlyWiki Classic code base and figure out how this should work, mostly due to other things requiring my time. However I would be glad to accept a pull request for the required code.

I can provide whatever window functions are needed, but saving can only be asynchronous. It will return before the file get's written, but after it get's "posted" for saving. I can also do a callback to confirm whether or not the file was saved.

So if you write up the code, I will be happy to include it. The code will be injected using <script src=""></script> and I will inject it using document.body.appendChild. It would then be executed immediately. I don't know if it can be added to the head before the body runs, but I would rather add it to the end of the body.

Let me know if this works for you. Your location.protocol will be "blob:". And the location.origin will be chrome-extension://somechromeextensionid.