FLAP directory structure (proposal for installable FLAP package)
Closed this issue · 13 comments
It only took me 4hours to figure out that the problem is not with my python path but with the fact that there is a flap/flap directory. This included reinstalling Anaconda, spyder, mdsplus etc.
I know this has been highlighted before. Also, I didn't have this problem before I update every repo to an actual clone of the GITHUB repos. If you simply clone the flap github repo into the flap directory it doesn't work.
I think it is intuitive to put the git repository directly into the python PATH with the same name as the repo:
Suggestion:
- Remove the flap directory and move everything into the root of the repo. The other flap repos use this convention. Also, it works out-of-the-box. Only the flap directory needs to be renamed in the PYTHONPATH. For now, I am using this solution as I can add the mlampert/repos directly into the PYTHONPATH.
- Rename the flap directory to src, move init.py to the root and rename the imports in the init.py. This is an elegant solution.
- Put a line in the README file so people wouldn't spend this much time on figuring out that there is nothing wrong with their environment .
The second is not working as I expected. Something is still wrong. Sometimes the module is working, sometimes it doesn't.
I don't really understand this whole module concept. I thought if there is a flap directory with a init.py file, then that's the flap module. But it should't be within a flap named directory, because that confuses python.
I tried the following:
flap master: ./repos/fusion_flap
all the others: ./repos/...
Didn't work.
right now the only setup which is working is ./repos/flap is in the path and all the other flap_* repos are in the flap directory. This is awful but works. Please let me know what is the correct setup. Once I believe I could set it up in a way, where all the files from ./flap/flap were in ./flap (in contrary to the github repo) and every repo was in the repos directory.
Na most már tényleg befejeztem:
https://dev.to/codemouse92/dead-simple-python-project-structure-and-imports-38c6
Nekem az az érzésem, hogy át kellene struktúrálni az egész FLAP-et egy csomaggá, ha azt akarjuk, hogy ebből valami nagy és messzire mutató fejlesztés legyen.
Az amcsik csinálnak egy OMFIT nevű dolgot. Ott minden egyben van és jó sokan használják elvileg. Nekem is mondták, hogy használjam, de csak nagyon alap fluktuációs dolgok vannak benne. Ha érdekel, akkor megnézhetem, hogy pontosan mi a struktúra. De ez így nekem valahogy nem áll össze, hogy órákat kell játszanom azzal, hogy hogyan tudom beimportálni a csomagokat.
Úgy gondolom, hogy ami magától értetődő, azt úgy kéne megírni, hogy tényleg úgy is működjön ( ez alatt arra gondolok, hogy berántom a github-ról a csomagot, berakom a PATH-ba és egyből megy az import flap attó függetlenül, hogy mit csinálok)
Ami mondjuk furcsa, hogy a numpy is úgy van struktúrálva, hogy van benne numpy/numpy könyvtár, de az nem dobál hibát. (Bocs a magyarért, kezd késő lenni én meg kicsit ideges vagyok, hogy ennyi idő elment az importálgatással).
Még nem vagyok K.O. úgyhogy angolul válaszolok :)
I believe @sandorzoletnik replied to your question in a different issue: #37 (comment)
The reason the flap directory is structured the way it is, is because I read it a long time ago here that it is the standard: https://docs.python-guide.org/writing/structure/ . So I believe that that is generally OK, although it would make more sense to rename flap/flap folder to something like flap/code or whatever.
However, the idea is to make the package modular, as e.g. you probably do not need the flap_w7x_abes module and if we have had only one repo for the whole thing, than that would soon become huge with the branches and all. I think it is a generally good idea to somehow keep the whole thing modular.
Nevertheless, originally we also had a completely different setup: There was an additional folder in the main flap repo: modules. This folder was not tracked with git and one could check out the flap modules one needed to this folder. (There was a readme and a very basic script which did this for users not familiar with git.) Then one could import the modules as flap.modules.bestmoduleever in their codes. With this method one needed to add only the main flap folder to the pythonpath and things worked out of the box. If one wanted to work on a module, than that could be done directly in the flap/modules/bestmoduleever folder without the commits interfering with the core flap repo. There is another method for making stuff like this: git submodules. However, I played around with that and it seemed to be cumbersome to work with and was not recommended on the blogs I read. This structure was dropped when we moved to github to the one we have now.
The way I have it set up now is: there is repos/flap where the main repo is. Then there are folders repos/modules/flap_w7x_abes, repos/modules/flap_apdcam, etc. for the modules I add the repos/flap folder and the repos/modules folder to the pythonpath and it works.
I agree that the one we have now may not be the best configuration. I was OK with the way it was before, however that may also be unintuitive. So I am open to changes
I think I understand the concept somewhat.
I believe the solution is to make flap as an installable package. It is not the best to add folders to the PYTHONPATH and non of the packages do that. You just say conda install or pip install and it gets the job done. Since this package is well in the development status, it is not a huge problem, but later on, it is going to be better to have the installation possibility.
Also, it would be advisable to reserve the flap namespace in the conda cloud so it could be installed from conda later on without messing with git checkouts.
About the submodule part:
I don't think it is a bad idea to have submodules (actually modules, flap is supposed to be a package). I don't know about git submodules probably I wouldn't bother. As long as there are clearly tested releases from master, it shouldn't matter how many commits are in the development. The users should be using the master package in general.
There is this OMFIT package at DIII-D. Just to compare it to ours:
It has 25099 commits, 186 releases and 85 contributors. It has been developed for 6years now.
The core of the package is stored in the OMFIT folder while all the submodules are in the modules folder. I cannot really share more as it is a closed project, but I think if we want something professional, we need to define a package and not just individual modules.
There is also a confusion here about modules. Flap is not a module, it is a package. All of these directories/repos are packages, with only a single module inside.
In summary: I believe flap needs to be restructured into an installable package with modules (flap, w7x_abes, nstx_gpi, etc.)
I can dedicate some of my time into this but for now I need to produce some results for APS. Please let me know if this is something you are interested in.
I tried some other directory structures, but only this one worked properly:
- flap core (fusion-flap/flap) is checked out to a directory called flap_repos (cannot be checked out to a directory named flap under any circumstance).
- all the other "modules" are checked out into flap_repos
- flap_repos is added to the PYTHONPATH in spyder.
EDIT1:
Only flap was working this way, the other modules are not. Something is also wrong with my setup as sometimes it is working sometimes it doesn't.
EDIT2:
I still don't quite understand, but sometimes it is working, sometimes it is not. I have had an issue when the directory was named flap_repos and then I didn't when I renamed to repos. Or maybe the problem was having the repo/repo/flap_* structure.
My current enhancement idea still remains useful. This would never happen with an installable package.
EDIT3:
It turned out that the Sourcetree git client was braking everything. if a repo was included in its configuration, it was not importable by Spyder. Too bad, it was working pretty well otherwise. I am going to learn how to deal with the git command line.
PS: Sorry for the spams if you are not concerned.
Again, sorry for the spam, but this information could be relevant.
It turns out, that I cannot import any package if its directory is opened in either Commander One (Totalcommander like software on macOS) or Sourcetree. There is no problem if I open it in Finder (My Computer in Windows). Also, if I do the same with e.g. matplotlib, that can be imported.
It could be that there is a problem with the Google Drive File Stream which I am using for backup up my data.
Here is the related stackoverflow question (and my answer to it): https://stackoverflow.com/questions/58207355/cannot-import-my-own-module-in-python-if-its-folder-is-opened-in-another-applica/58208192#58208192
Is anyone interested in making FLAP an installable package at some point?
Wow, that sounds super frustrating :D Anyway, yes I'd be interested in making the code installable for end users. Next week we could discuss with @sandorzoletnik and the others who are interested and get to it. Also, I think it is a good idea to come to an agreement relatively fast, as restructuring flap will only get more complicated the more we wait.
I had problems originally with path and modules. What turned out is that when one changes the PYTHONPATH in spyder it is not applied, one needs to restart spyder. Now I have as strucure as both of you write: flap_development for the flap rep and flap_modules for all the modules. I have flap_development and flap_modules in the PYTHONPATH and had no problem since a long time.
So we agree to rename the flap repository to flap_core and then all problems are solved?
According to some Python documentation it is deliberately not descending into subdirectories in PYTHONPATH. You should list all directories. If flap_core and the modules subdir is listed it will work.