macOS Monterey?
Closed this issue · 34 comments
Hi! I'm really interested in this script. Is this functional on Monterey? Homebrew installs in opt/homebrew from MacOS 12 and I wonder if it all works smoothly on it.
BTW, How do we add or modify rules? For example for Python projects I don't use requirements.txt but Pipfile. Can see that file to trigger the exclusion of .venv folders too?
Thanks!
Hy! Sorry for the late reply. No idea if it works on Monterey, I'm on 11.5.2 right now, but I think they didn't modify the timemachine. The script cp -R $(brew --prefix heptapod)/rules ~/.heptapod should get the os specific prefix, so I hope everything in the readme works as is.
Adding a new rule for python is as simple as;
- copy this file to the rules folder (with a new name); https://github.com/tg44/heptapod/blob/main/rules/python-venv.yaml
- rewrite the file trigger to "Pipfile" or any other file (you want to exclude .venv so the excludes are good as it is) and the name to something meaningful
- test it with dryrun
I've tried the app in my older Mac (MacOS Catalina) and it works really great!
I have two questions:
- The command
heptapod lseshows the excludednode_modulesfolder, but not the.venvfolders. Is it possible that's because they are hidden folders? - Is it possible to run
heptapod runautomatically, for example every day at a certain time, or depending on some events, for example before a time machine backup? Or for example every couple hours, or periodically? That'd be great!
I have to find a bit of courage to run it on my new MacBook Pro with Monterey (I don't want to mess up Time Machine since you never tested it there!). Any big risks of things going wrong?
Thank you!
I've installed Heptapod on my Monterey Mac, and before running I did heptapod -v run -d
The output is very different from my Catalina Mac. In my older mac I get a long list of all excluded paths.
But this is what I get in my terminal in Monterey:
2021/12/23 18:31:47 (runner-4) walk on ~ took 1.3699695s
2021/12/23 18:31:47 walker run took 1.3700215s
There's no list of folders to be excluded. I have some already excluded through the Time Machine GUI, like Applications and photo library. but nothing is showing up (see screenshot).
Is this the expected behavior?
I can't get to my mac in the next couple of days to check everything below.
If I remember well you can list all of the current time machine excludes with this tool (check the -h flag for the commands). Also, if I remember well it can revert all the changes it done (listing all the manually added excludes to a file, and can include them again later). So you can check what was prev. added to the excludes, and what is added after the run.
As I remember the dryrun should list all the folders (even the nonexisting ones). If you list the enabled rules, do you see all the rules you want to run? Also all of the runs starts from your home folder by default, if your project dirs are not under ~ it will be missed out.
About the auto runs; not yet. It is on my todo list, maybe between Christmas and New Year I will have some time to adress this.
Thank you for your reply!
Yes, on my iMac (MacOS Catalina) the app works well and the command heptapod -v run -d gives the list of all excluded directories.
However, on my MacBook Pro (MacOS Monterey) the dry run gives me just this, without any folder to be excluded:
The rules are being listed:
And the command heptapod lse also shows long output:
It's the dry run the one not working, so I think that the command heptapod run will not work either.
I hope I'm clear, and thank you!
I will do a better debug log for dry-run, bcs right now it is really hard to tell what is the actual problem. Also, I will add your python rule for convenience.
Looking forward to the updates, this tool should become the missing time machine setting and will be very appreciated by all the community once it's functional on Monterey!
Can you run an update, and run the following commands?
mkdir -p ~/.heptapod/rules
cp -R $(brew --prefix heptapod)/rules ~/.heptapod
heptapod ls
heptapod -v run -d
I'm interested if the new python-pip is added as a new (enabled) rule or not. And if you get new outputs in the verbose output or not.
If not;
mkdir -p ~/test
touch ~/test/.venv
touch ~/test/Pipfile
touch ~/test/package.json
heptapod -v run -d
This should print at least something :)
Hey mate, so glad you're working on this!! ❤️
heptapod ls -a gives me the rules
But, regrettably, heptapod-v run -d still gives me empty output!
I have already quite a few .venv and node_modules folders that should have been listed.
Okay I detected that Heptapod can exclude when directly in ~/ but not when nested.
So I created ~/test/.venv and ~/test/pipfile and those are added.
Hoewever, the folders in ~/Code/test/.venv/ are not added. Like it's just looking for a depth 1 or something. I hope I'm clear.
Maybe I explain better if I say that it doesn't search in nested folders.
To try to be clearer:
When the folders are directly in ~/ heptapod works fine:
But when it's nested deeper, it's not added:
Note that this new .venv inside ~/Code/delete_me/ is not added. Only the one from ~/test is present, and I have quite a few virtual environments and node_modules folders inside ~/Code/ and no one was added.
I get it, but seems super strange.
Can you try 'mkdir -p ~/test/test/test/Pipfile', and check if the newly created file/dir is excluded or not?
I think it is most likely some privilege problem, I will extend the verbose mode further.
You're actually right, a nested test/test/.venv gets added!
But my ~/Code/ directory was created like any other, what could be the issue?
Could it be related to the folder starting with a capital "C"?
If I create a "code2" folder, heptapod works fine with it, but if I add a folder "Code3" (capital c) it doesn't.
With a folder "DeleteMe" it doesn't work either. I'm starting to assume it's related to the capital letter in the name of the folder, but it's very strange, because it works fine with a "Test" folder with a capital "T"!
I'm very confused.
I can confirm that if the folder starts with a capital letter from A to L, heptapod doesn't work with them.
But It does work fine with folders starting with all capital letters from M to Z.
It doesn't seem to work with folder names starting with a number either.
Strange!
That is really strange indeed. I played with it on my machine, and tried to debug this, but it works correctly on my os version;
mkdir -p ~/test_hepta/4test
mkdir -p ~/test_hepta/Abc
mkdir -p ~/test_hepta/Lol
mkdir -p ~/test_hepta/Mooo
touch ~/test_hepta/4test/package.json
touch ~/test_hepta/Abc/package.json
touch ~/test_hepta/Lol/package.json
touch ~/test_hepta/Mooo/package.json
mkdir -p ~/test_hepta/rules
cat << EOF > ~/test_hepta/rules/test-rule.yaml
name: "test-node"
enabled: true
searchPaths: ["~/test_hepta"]
ignorePaths: []
ruleType: "file-trigger"
ruleSettings:
fileTrigger: "package.json"
excludePaths:
- "node_modules"
EOF
heptapod -r ~/test_hepta/rules -v run -d
rm -rf ~/test_hepta
Can you test this both on your machines and report back that this is only broken on Monterey?
If it is a macos version problem, I will upgrade, after I traveled home (maybe 1 week), and continue to debug.
The strangest thing is that I use filepath.Join everywhere. So I have no idea where I concat/join paths with dirname faulty. It is either golang or os bug I think. After an hour trial and error I have really no idea what can cause this.
Yes, but I meant to say the the problem is with folders starting with a Capital letter immediately in the root.
Could you please try mkdir ~/Code (capital C in the root of the user folder) and see if that works for you?
Capital letter folders after the parent folder is lowercase are fine
Please use a letter from A to L. After the letter M it seems to work fine.
Strange, no?
I could reproduce this, and I just wanna cry :( This was a really silly mistake, when we get the first error instead of "continue" the loop I used "break", which stopped the execution. And bcs I did a tree-like walk, the first error with Pictures, or Library broke the loop. Also, the folders come with lexicographical older, and I reverse insert them so everything is really logical at this point :D
I fixed this, but also I need to add a default ignore list bcs now we parse the .Trash, Library and other not so useful folders too... My runtime increased a lot.
Yay!! Good news no??
And yes, please, ignore the Library and also all the dot folders or otherwise it will take ages to scan and add all the node_modules of (for example) .yarn/cache every time!!
I just pushed a release. Can you check it? Also, I reorganized the CLI, so some commands will not work and others will appear.
We should create a curated list of "recommended ignores" that I will add to the default rules. If you have knowledge about global folders like ~/.yarn, or ~/.mvn that we probably want to either ignore from TM all together, or globally leave out from the execution that would be also helpful. Any idea or contribution is welcomed.
That's great, I'll try it right away!
Regarding the ignore list, how about a file in the .heptapod folder with custom folders that append to your default list of ignores?
That would add a lot of flexibility to the app!
Is that an idea?
Mmm I'm getting an incredibly long output, and heptapod is scanning all the Library folder and quite a few others that should be ignored...
Additionally I'm getting a lot of these errors:
(I had to abort or it would go on for quite long!)
I think it's mostly Library and some dot folders like .vscode, .yarn, .npm and .nvm.
I think all dot folders should be ignored in the search.
It looks like heptapod is searching exactly in the folders that should be ignored, because my ~/Code folder is ignored but Library and the dot folders are added. Maybe you forgot a negation somewhere and it's doing the opposite of what it should?
You can try heptapod rules ignoreAdd ~/.Trash ~/.yarn ~/.npm ~/.nvm ~/Library. The ignored folders where the walker don't go in. You can change the walking route in each rule (manually right now) from ~ to ~/Code which is also a good feature idea.
I didn't want to add a global ignore list, bcs it can mess up things, this is the reason I added the new command to easily add ignores to all the rules. I think the field handling from the cli is generally a more flexible solution, and also it is more self explain. I'm thinking about a three-way switch for the .yarn like folders where you either want it to excluded from tm, or ignored by heptapod, or walked by rules as the rules want. My main concern with this is that it will still mess up the self-explain thing (it will go above the rules), and will generate a "global ignore list" like thing. So I'm still hesitant with this.
Ah I'm sorry I didn't realise heptapod has now this new command. I will try that out. But maybe having a list of the folders you want to ignore in a file in the .heptapod/ folder isn't a bad idea because it's easy to see what you've added to it.
Regarding changing the rule: yes I tried that before when ~/Code wasn't walked and it worked, but I'd need to duplicate a lot of rules if for example y save code in different folders, not just ~/Code.
Let me play with the new command. I'll run the help command on heptapod to see what's new! 😊
Thank you so much, I'm getting a much cleaner and now correct output!
If you allow me I'll ask a couple questions:
- Where are the
ignoressaved? It would be great if one could be able to see all the folders added to the list. Maybe append them to a file of "Ignored folders" or similar? - I think that the verbose is now too verbose, if I may say so. As an example, here are the first lines of the dry-run:
I don't understand why we see all the list of the ignored folders on each iteration of the rules. Maybe it's now too much information?
Again, thank you so much for all the work and I'm sorry if I'm too annoying with questions.
I'm afraid I'm a curious "beta-tester" 😉 and like to try all things possible.
I really believe this should become an extremely valuable tool, and I'm really interested in it being just perfect. It's really something any developer using Time Machine will find of great benefit! Great work!
EDIT:
I see that you add now the ignored folders to each of the active rules individually. That's nice!
But what would happen if I add a new rule? I'll need to manually add those ignores to that file I suppose.
I still think that the verbose is too verbose, though. But the app is looking great!
Yapp, verbose is definitely too verbose, I should make levels of it or something like that :D
If you run the tool with run, it logs all the excluded folders to a file in ~/.heptapod/logs With prune you can revert back bcs of these logs. If you run dry without -v, it logs all the files (or folders) it will try to exclude (heptapod run -d). This is a much bigger list than the real exclude list, bcs it contains nonexisting folders, and already excluded ones too (but the exclude logs will only contain the added excludes).
I think I will add some more tm commands like "checkpoint", "remove all excludes", "rollback to checkpoint", to make it safer to play with.
Your questions and findings are not annoying, it helps me a lot. It is not the same when you start to support actual users and the tool is not just a toy anymore :)
I'm really happy to read you and see how this is such an ambitious project.
Next step after you're really happy with it is to make the publicity it deserves!
Keep up the great work!
I can imagine an interesting command, something like heptapod clean to delete from the Time Machine list all folders that don't exist anymore.
Like if I delete a folder containing a .venv that was previously added to the Time Machine excludes, the clean command would be able to check if that still exists and delete it if it doesn't
Just an idea! :)
And, of course, a way to automate heptapod! 😀
Just another (hopefully the last!) little issue:
I ran heptapod run and all went smoothly.
But when I do heptapod tm ls I don't see the .venv folders in the exclusions. Is that because they are hidden or they are just not added?
Okay, some of my ideas will not work, I made a deeper dive into how TM really works (we can't list the excluded folders even if we want to, if we really really really want to we would possibly can, but that would be slow as hell so I put this on ice RN).
heptapod clean is not necessary, read the new section from TM in the readme :)
Nonappearing .venv folders also documented in the new readme section (and I added a new command which prints the basic informations why we can't run the function).
Automation will be in on other thread later.
Verbose is now leveled 1 will show speed, 2 basic info, 3-4 extended debug-like info.
I think I close this now.
I'll wait to the new release to fully understand. I think I'm not understanding completely now: so we will need to run tmutil commands besides heptapod commands?
Nope. heptapod runs tmutil commands. There are two types of add. The first one is adding a path to a list, this is the list that you also see on the TM UI in the OS settings. The second one is adding metadata to the folder/file. The metadata moves and getting deleted with the folder/file. But we only can list the flagged files if we traverse all the folders. heptapod uses the second method. So we don't need to cleanup deleted folders, the os doing this for us.
Ah, that's nice! So that means that when I delete my test/node_modules for example it will be automatically removed and not present anymore when I run heptapod tm ls? That's cool.
tm ls using mdfind which caches, so idk if it is removed from that list or not (or when it is removed), but it should be removed at least after a while. It is out of my control :) The core takeaway, that we don't leave trash anywhere, it is not our responsibility to manage those things, it should be happening at os level.