watch-fs - dangerous disk-filling loop
xeor opened this issue · 7 comments
There is a strange bug I noticed yesterday, and again today.
Sometimes, if watch-fs sees an .md
file, it reads it and writes it back, (as an .md
), but in a wrong way.
The result is a file with a json list containing an object, with metadata, including text
, but with wrong escaping.
So
- Saving a file on fs using
.md
ext - watch-fs sees it and tries to make it into an .tid?
- It misses on the format, including escaping of
"
and\
characters - repeats, making the file bigger and bigger since it is escaping all the
\
it just made.
If I'm not quick on stopping tiddlywiki, it will continue making very big files.
Here is an example of a file originally named aaa.md
and the content
# test
abc
after a couple of seconds, it looks like this:
[
{
"title": "/tiddlywiki/data/tiddlers/aaa.md",
"text": "[\n {\n \"title\": \"/tiddlywiki/data/tiddlers/aaa.md\",\n \"text\": \"[\\n {\\n \\\"title\\\": \\\"/tiddlywiki/data/tiddlers/aaa.md\\\",\\n \\\"text\\\": \\\"[\\\\n {\\\\n \\\\\\\"title\\\\\\\": \\\\\\\"/tiddlywiki/data/tiddlers/aaa.md\\\\\\\",\\\\n \\\\\\\"text\\\\\\\": \\\\\\\"[\\\\\\\\n {\\\\\\\\n \\\\\\\\\\\\\\\"created\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"20200915124727593\\\\\\\\\\\\\\\",\\\\\\\\n \\\\\\\\\\\\\\\"title\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"/tiddlywiki/data/tiddlers/aaa.md\\\\\\\\\\\\\\\",\\\\\\\\n \\\\\\\\\\\\\\\"text\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"# test\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\nabc\\\\\\\\\\\\\\\",\\\\\\\\n \\\\\\\\\\\\\\\"type\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"text/x-markdown\\\\\\\\\\\\\\\",\\\\\\\\n \\\\\\\\\\\\\\\"tmap.id\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"1b38e09e-755f-478a-b53b-300d4ad532cc\\\\\\\\\\\\\\\",\\\\\\\\n \\\\\\\\\\\\\\\"modified\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"20200915124727593\\\\\\\\\\\\\\\"\\\\\\\\n }\\\\\\\\n]\\\\\\\",\\\\n \\\\\\\"type\\\\\\\": \\\\\\\"text/x-markdown\\\\\\\"\\\\n }\\\\n]\\\",\\n \\\"type\\\": \\\"text/x-markdown\\\"\\n }\\n]\",\n \"type\": \"text/x-markdown\"\n }\n]",
"type": "text/x-markdown"
}
]
If I add aaa.md.meta
and put a title: aaa
inside, there is no problem.
This is funny, yes, I haven't tested the case like this, creating a non-tiddler file directly without metafile.
The solution may be quite simple, Maybe I should automatically create a metafile before handle this, I think I'm just simply treating anything as .tid file now.
Hopefully, I add a 500ms debounce to the listener function, otherwise, this situation would be hazardous.
500ms debounce helped a little. But it escalates quicly when it first starts wanting those backslashes :p
The solution seams simple as you say. The only thing I added to the meta-file was a title, then it added type
and everything else itself.
I was kinda baffled when this hit me, since I thought for sure others where just creating eg .md
files manually. Are you creating .tid
files when creating from eg vscode? Isn't that a pain when it comes to stuff like language detection inside vscode?
About this case and #8. Do they have the same fix you think?
Is there any more info/tests I can do here, or did you have an idea already how to solve it?
I'm working on this, now that chokdir give us "create" event, I'm adding a logic that creates a meta file for newly created non-tiddler file.
I managed to get it to work!
Now you can create md file from vscode or drag one into the tiddler folder.
Strange that this didnt work for me. I'm using poll instead of inotify, so maybe it's a race-condition?
Here are my test:
- made a3.md with "test" content
- waited
- this log output
a3.md add
Adding meta file /tiddlywiki/data/tiddlers/a3.md.meta using mime type text/x-markdown
tiddlersDescriptor {
"filepath": "/tiddlywiki/data/tiddlers/a3.md",
"type": "text/x-markdown",
"tiddlers": [
{
"text": "test\n",
"type": "text/x-markdown",
"caption": "a3",
"created": "20201005093614317",
"modified": "20201005093614317",
"title": "a3.md"
}
],
"hasMetaFile": true
}
getting new tiddler.title a3.md
get new addTiddler fileDescriptor.tiddlerTitle undefined tiddler.title a3.md
canSync is now false
syncer-server-filesystem: Dispatching 'save' task: a3.md
a3.md.md add
tiddlersDescriptor {
"filepath": "/tiddlywiki/data/tiddlers/a3.md.md",
"type": "text/x-markdown",
"tiddlers": [
{
"text": "test\n",
"type": "text/x-markdown",
"caption": "a3",
"created": "20201005093614317",
"modified": "20201005093614317",
"title": "a3.md"
}
],
"hasMetaFile": true
}
getting new tiddler.title a3.md
deepEqual with existed tiddler fileDescriptor.tiddlerTitle undefined
canSync is now false
a3.md unlink
existedTiddlerResult {
"fields": {
"text": "test\n",
"type": "text/x-markdown",
"caption": "a3",
"created": "2020-10-05T09:36:14.317Z",
"modified": "2020-10-05T09:36:14.317Z",
"title": "a3.md"
},
"cache": {}
}
trying to delete /tiddlywiki/data/tiddlers/a3.md
canSync is now false
syncer-server-filesystem: Dispatching 'delete' task: a3.md
a3.md add
a3.md ignored due to mutex lock
a3.md.md unlink
existedTiddlerResult undefined
file already deleted by wiki /tiddlywiki/data/tiddlers/a3.md.md
canSync is now false
canSync is now true
- a3.md got deleted and content lost
- wrote "test2" in a3.md, this triggered
a3.md.meta
to be created, but no new logs - waited
- this came as the output
a3.md change
tiddlersDescriptor {
"filepath": "/tiddlywiki/data/tiddlers/a3.md",
"type": "text/x-markdown",
"tiddlers": [
{
"text": "test2\n",
"type": "text/x-markdown"
}
],
"hasMetaFile": true
}
getting new tiddler.title undefined
get new addTiddler fileDescriptor.tiddlerTitle undefined tiddler.title undefined
canSync is now false
canSync is now true
- a3.md got to keep it content
- a3.md.meta is blank
Okay, I will try if it works in poll mode
That a3.md.md add
didn't happened on my side last week.