watch-fs - dangerous disk-filling loop

[xeor]

There is a strange bug I noticed yesterday, and again today.

Sometimes, if watch-fs sees an .md file, it reads it and writes it back, (as an .md), but in a wrong way.
The result is a file with a json list containing an object, with metadata, including text, but with wrong escaping.

So

• Saving a file on fs using .md ext
• watch-fs sees it and tries to make it into an .tid?
• It misses on the format, including escaping of " and \ characters
• repeats, making the file bigger and bigger since it is escaping all the \ it just made.

If I'm not quick on stopping tiddlywiki, it will continue making very big files.
Here is an example of a file originally named aaa.md and the content

# test
abc

after a couple of seconds, it looks like this:

[
    {
        "title": "/tiddlywiki/data/tiddlers/aaa.md",
        "text": "[\n    {\n        \"title\": \"/tiddlywiki/data/tiddlers/aaa.md\",\n        \"text\": \"[\\n    {\\n        \\\"title\\\": \\\"/tiddlywiki/data/tiddlers/aaa.md\\\",\\n        \\\"text\\\": \\\"[\\\\n    {\\\\n        \\\\\\\"title\\\\\\\": \\\\\\\"/tiddlywiki/data/tiddlers/aaa.md\\\\\\\",\\\\n        \\\\\\\"text\\\\\\\": \\\\\\\"[\\\\\\\\n    {\\\\\\\\n        \\\\\\\\\\\\\\\"created\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"20200915124727593\\\\\\\\\\\\\\\",\\\\\\\\n        \\\\\\\\\\\\\\\"title\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"/tiddlywiki/data/tiddlers/aaa.md\\\\\\\\\\\\\\\",\\\\\\\\n        \\\\\\\\\\\\\\\"text\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"# test\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\nabc\\\\\\\\\\\\\\\",\\\\\\\\n        \\\\\\\\\\\\\\\"type\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"text/x-markdown\\\\\\\\\\\\\\\",\\\\\\\\n        \\\\\\\\\\\\\\\"tmap.id\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"1b38e09e-755f-478a-b53b-300d4ad532cc\\\\\\\\\\\\\\\",\\\\\\\\n        \\\\\\\\\\\\\\\"modified\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"20200915124727593\\\\\\\\\\\\\\\"\\\\\\\\n    }\\\\\\\\n]\\\\\\\",\\\\n        \\\\\\\"type\\\\\\\": \\\\\\\"text/x-markdown\\\\\\\"\\\\n    }\\\\n]\\\",\\n        \\\"type\\\": \\\"text/x-markdown\\\"\\n    }\\n]\",\n        \"type\": \"text/x-markdown\"\n    }\n]",
        "type": "text/x-markdown"
    }
]

If I add aaa.md.meta and put a title: aaa inside, there is no problem.

[linonetwo]

This is funny, yes, I haven't tested the case like this, creating a non-tiddler file directly without metafile.

The solution may be quite simple, Maybe I should automatically create a metafile before handle this, I think I'm just simply treating anything as .tid file now.

Hopefully, I add a 500ms debounce to the listener function, otherwise, this situation would be hazardous.

[xeor]

500ms debounce helped a little. But it escalates quicly when it first starts wanting those backslashes :p

The solution seams simple as you say. The only thing I added to the meta-file was a title, then it added type and everything else itself.

I was kinda baffled when this hit me, since I thought for sure others where just creating eg .md files manually. Are you creating .tid files when creating from eg vscode? Isn't that a pain when it comes to stuff like language detection inside vscode?

[xeor]

About this case and #8. Do they have the same fix you think?
Is there any more info/tests I can do here, or did you have an idea already how to solve it?

[linonetwo]

I'm working on this, now that chokdir give us "create" event, I'm adding a logic that creates a meta file for newly created non-tiddler file.

[linonetwo]

I managed to get it to work!

Now you can create md file from vscode or drag one into the tiddler folder.

[xeor]

Strange that this didnt work for me. I'm using poll instead of inotify, so maybe it's a race-condition?
Here are my test:

• made a3.md with "test" content
• waited
• this log output

a3.md add
Adding meta file /tiddlywiki/data/tiddlers/a3.md.meta using mime type text/x-markdown
tiddlersDescriptor {
  "filepath": "/tiddlywiki/data/tiddlers/a3.md",
  "type": "text/x-markdown",
  "tiddlers": [
    {
      "text": "test\n",
      "type": "text/x-markdown",
      "caption": "a3",
      "created": "20201005093614317",
      "modified": "20201005093614317",
      "title": "a3.md"
    }
  ],
  "hasMetaFile": true
}
getting new tiddler.title a3.md
get new addTiddler fileDescriptor.tiddlerTitle undefined tiddler.title a3.md
canSync is now false
 syncer-server-filesystem: Dispatching 'save' task: a3.md
a3.md.md add
tiddlersDescriptor {
  "filepath": "/tiddlywiki/data/tiddlers/a3.md.md",
  "type": "text/x-markdown",
  "tiddlers": [
    {
      "text": "test\n",
      "type": "text/x-markdown",
      "caption": "a3",
      "created": "20201005093614317",
      "modified": "20201005093614317",
      "title": "a3.md"
    }
  ],
  "hasMetaFile": true
}
getting new tiddler.title a3.md
deepEqual with existed tiddler fileDescriptor.tiddlerTitle undefined
canSync is now false
a3.md unlink
existedTiddlerResult {
  "fields": {
    "text": "test\n",
    "type": "text/x-markdown",
    "caption": "a3",
    "created": "2020-10-05T09:36:14.317Z",
    "modified": "2020-10-05T09:36:14.317Z",
    "title": "a3.md"
  },
  "cache": {}
}
trying to delete /tiddlywiki/data/tiddlers/a3.md
canSync is now false
 syncer-server-filesystem: Dispatching 'delete' task: a3.md
a3.md add
a3.md ignored due to mutex lock
a3.md.md unlink
existedTiddlerResult undefined
file already deleted by wiki /tiddlywiki/data/tiddlers/a3.md.md
canSync is now false
canSync is now true

• a3.md got deleted and content lost
• wrote "test2" in a3.md, this triggered a3.md.meta to be created, but no new logs
• waited
• this came as the output

a3.md change
tiddlersDescriptor {
  "filepath": "/tiddlywiki/data/tiddlers/a3.md",
  "type": "text/x-markdown",
  "tiddlers": [
    {
      "text": "test2\n",
      "type": "text/x-markdown"
    }
  ],
  "hasMetaFile": true
}
getting new tiddler.title undefined
get new addTiddler fileDescriptor.tiddlerTitle undefined tiddler.title undefined
canSync is now false
canSync is now true

• a3.md got to keep it content
• a3.md.meta is blank

[linonetwo]

Okay, I will try if it works in poll mode

That a3.md.md add didn't happened on my side last week.