[feature] allow west to report if any of the cloned repositories is out of sync with the manifest
xnlcasad opened this issue · 41 comments
In order to ensure that a Zephyr application is always built from the exact same sources it would be good if west could report if any of the west-handled repositories is out of sync with the manifest file. Much like a git superproject git status would report a dirty working tree if any of the submodules' HEADs point to a sha different from the one recorded in the superproject.
$ west forall -c 'git status'
=== running "git status" in manifest (nrf):
HEAD detached at v1.7.0
nothing to commit, working tree clean
=== running "git status" in zephyr (zephyr):
HEAD detached at v2.6.99-ncs1
$ cd -
~/work/zephyr-upstream/zephyr
$ git checkout HEAD^
$ cd -
~/work/zephyr-upstream/nrf
$ west forall -c 'git status'
=== running "git status" in manifest (nrf):
HEAD detached at v1.7.0
nothing to commit, working tree clean
=== running "git status" in zephyr (zephyr):
HEAD detached at 6bfc48ef0b
So basically I rolled back zephyr one commit, but west would not tell me that zephyr's HEAD is pointing to a different commit than whatever is on nrf/west.yml, which is:
- name: zephyr
repo-path: sdk-zephyr
revision: v2.6.99-ncs1
@carlescufi suggested that a new parameter to west list could be used. I don't have anything against that, I would only suggest the interface to be automation friendly, so that it could be invoked from a build script. If I take git as an analogy, git diff --exit-code would return 1 if anything has been changed from the working tree versus the HEAD revision.
Since commit eff0e1b, west status stopped reporting projects with "clean" git status, even when they're not on manifest-rev!
I think that was a mistake. I mean from a west perspective a "clean" status should be 1. a clean git status AND 2. being on manifest-rev.
west status, as a wrapper around git status, reports status relative to HEAD, not manifest-rev. So its output is not related to the manifest file, unless of course if you've just run west update and HEAD is the same commit as manifest-rev.
it would be good if west could report if any of the west-handled repositories is out of sync with the manifest file.
We get requests like this from time to time. The devil is in the details, as usual. For a manifest that does not use import: that only uses SHAs in its revision: values, like upstream zephyr/west.yml, the request seems clear. But it's less obvious what the request is when the manifest contains import: and/or non-SHA revision: values, especially when it contains both.
The issue is that revision: is really a revision on the remote project, not the local tree. And since branches are allowed, the remote commit can change without west knowing about it. And then when you mix imports in with non-SHAs, you can actually end up with a final resolved manifest with a totally different project list, because the remote got some new projects added in the manifest you're importing from.
The best I've come up with in such situations is to compare the working tree (or the HEAD commit) with manifest-rev.
But what is in manifest-rev for each project is not the same thing as "the manifest file" for reasons explained above.
Given this, can you refine your proposal a bit? Is comparing with manifest-rev OK?
We get requests like this from time to time. The devil is in the details, as usual.
See for example zephyrproject-rtos/zephyr#23025
The devil is in the details, as usual.
It appears complicated because west update does too many things at once with no way or no easy way to go step by step. Something I already reported (for a different reason) in #338 .
The issue is that revision: is really a revision on the remote project, not the local tree.
Right. The apparent complexity here is IMHO because west update:
- (online) downloads new
revisions from the network, including new versions of branches - (offline) moves
manifest-revs - (offline) resets projects to
manifest-rev.
... all in one go. Git also has commands that do too many things at once but there's always a slower, step by step fallback for non trivial cases.
So ignoring manifest changes (that's for zephyrproject-rtos/zephyr#23025) manifest-rev represents the "last known remote state". People with a tiny bit of version control experience (unless it's Perforce...) are aware that the "last known remote state" is a constantly moving target when they're not alone working on the project. It's unfortunate west update tries to make the critical refresh of this last known remote state look like a low level implementation detail buried in the middle of a larger sequence. It's not a low level implementation detail.
Given this, can you refine your proposal a bit? Is comparing with manifest-rev OK?
Obviously yes for me.
The route I figured I'd end up taking:
-
force west to manage all the dependencies of my applications to be imported into a directory somewhere within the root directory of my applications git repository
-
create a plugin for west which checks git hashes and as a sanity check manages a standard-ish file containing a signature for all non ignored files within the top level directory
-
If possible inject a warning when a build is attempted in a directory that doesn't match the signature
For me this is an absolute requirement in a dependency and package management tool.
I've been searching for years and only 1 or 2 solutions exist.
West is close to being capable as a universal solution.
Given this, can you refine your proposal a bit? Is comparing with manifest-rev OK?
Even after having read https://docs.zephyrproject.org/latest/guides/west/workspaces.html#the-manifest-rev-branch I have to acknowledge that I don't completely understand how is this branch used by west, so please bear with me if I somehow state/ask what might be obvious to others.
What I can try to do is to expose our use case, so that it might trigger a possible discussion/solution.
We'd need to have many freestanding applications, with most of them using the exact same version of Zephyr and NCS, we'd use the freestanding model to avoid having n-copies of the exact same NCS on disk, and to reduce cloning time.
We need to make sure that when any developer builds a final image of the (app + NCS bundle) we can programatically verify that the sources used were exactly the expected ones, so that the final image can be bit by bit reproducible in any subsequent build.
We would expect the build system to inform the developer if something is being built that is not exactly what was originally committed to git. This is also valid when going back in history (or bisecting, etc)
On nrfconnect/sdk-nrf#5797 (comment) @tejlmand wrote a proposal to use cmake's find_package( EXACT ) to achieve this purpose, but in our case this would not go all the way, essentially because package versions are manually updated.
Maybe an analogy with git submodules could better explain what I would ideally expect: If any of the superprojects' submodules is not at the expected revision, or has modified files in the working tree, then running git diff --exit-code will let me know I'm not building with a clean tree.
We get requests like this from time to time. The devil is in the details, as usual. For a manifest that does not use
import:that only uses SHAs in itsrevision:values, like upstream zephyr/west.yml, the request seems clear. But it's less obvious what the request is when the manifest containsimport:and/or non-SHArevision:values, especially when it contains both.
Here's a suggestion from my side:
How about we add a new status formatting option in west list that is able to produce the following output:
dirty: if the tree is dirtysynced: if the revision matches themanifest-revand the revision is not a branchunsynced: if the revision does not match themanifest-revunknown: if the revision is a branch and not a tag or a SHA
Even after having read https://docs.zephyrproject.org/latest/guides/west/workspaces.html#the-manifest-rev-branch I have to acknowledge that I don't completely understand how is this branch used by west,
manifest-rev is simply where the "parent" expect the child git repo to be. It's conceptually the exact same information that git submodules track, expect the things are more complicated for west because west can do things that submodule can't. First of all, a west manifest allows pointing at a moving branch, something submodules don't support. Then as @mbolivar-nordic mentioned west supports imports that can override each other. Also, with west that information is in multiple places that can fall out of sync with each other depending on what you just did: The manifest-rev ref., the revision: field in the manifest working copy, the same field in the checked in manifest (think imports), etc.
So, instead of being where the child repo should be, manifest-rev is "where the child repo should be the last time I checked" and resolved the manifest.
add a new status formatting option in west list that is able to produce the following output
That seems useful for interactive use but for simplicity and to avoid scripts having to parse output I'd prefer west status to be fixed and return nothing when everything matches manifest-rev. These don't seem mutually exclusive.
I don't consider a moving branch revision to be a build reproduction problem in practice because some build system proudly offering an "exact" reproduction of a build must attach the manifest to the offered build. Then you can easily and instantly caught it red-handed.
synced: if the revision matches the manifest-rev and the revision is not a branch
I suspect @mbolivar-nordic will find some corner case showing that synced is not that easy to define. Maybe something with tricky imports where the revision is a branch in some manifest(s) but not other(s)? Now combine this with different users having different import filters (#543) in their config files: game over.
I'd prefer
west statusto be fixed and return nothing when everything matchesmanifest-rev
@marc-hb : the current behavior does not need to be "fixed" in my opinion. It is working exactly as documented:
$ west help status
usage: west status [-h] [-a] [PROJECT ...]
Runs 'git status' for each of the specified projects.
[...]
Changing this to be relative to manifest-rev would be a breaking change. I use west status as-is in order to see what local uncommitted changes I have in my workspace. This is what I want when I am working on something that spans multiple repositories in the workspace.
I'm not against something like we are discussing, but let's please drop the idea of forcing this into west status. As I wrote earlier, that is always relative to HEAD because it is meant as a wrapper around git status.
What I can try to do is to expose our use case, so that it might trigger a possible discussion/solution.
@xnlcasad indeed, that is very useful. I think that, like issue zephyrproject-rtos/zephyr#23025 which I mentioned earlier, you may be looking for something that actually belongs in the Zephyr build system, not west itself.
We'd need to have many freestanding applications, with most of them using the exact same version of Zephyr and NCS, we'd use the freestanding model to avoid having n-copies of the exact same NCS on disk, and to reduce cloning time.
You are of course free to use multiple freestanding applications, but I just want to point out that you can have as many applications as you want inside of a single workspace; there's nothing preventing that. Anyway, back to the main discussion...
We need to make sure that when any developer builds a final image of the (app + NCS bundle) we can programatically verify that the sources used were exactly the expected ones, so that the final image can be bit by bit reproducible in any subsequent build.
In my opinion this (very reasonable) requirement indicates a solution in the build system. The existence of ZEPHYR_MODULES and ZEPHYR_EXTRA_MODULES could push additional code into a local build that stands outside of anything in the manifest file, for example.
We would expect the build system to inform the developer if something is being built that is not exactly what was originally committed to git. This is also valid when going back in history (or bisecting, etc)
I think you can solve this if the build system can report status for each of the modules it used and put the output in the build directory. This is the real ground truth for what's in your binary, after all. Maybe it's time for the build/zephyr_modules.txt file to be a more robust format (like YAML, JSON, or pickle) that could contain additional data like whether the build was done from a dirty tree for each module, what the git describe output was, etc.
We could include some basic flags for common needs like these, but also potentially add support for running an arbitrary CMake function or shell script on each module.
Incidentally, have you looked at the west spdx extension that's provided with Zephyr? It is intended to address a superset of this use case, as far as I know:
@marc-hb : the current behavior does not need to be "fixed" in my opinion. It is working exactly as documented:
Sorry about the wrong terminology.
Changing this to be relative to manifest-rev would be a breaking change.
I agree it would be an interface change but unless you have some example I think the term "breaking" is just as wrong. I don't think it would "break" much more than eff0e1b already did but maybe I'm missing something.
I use west status as-is in order to see what local uncommitted changes I have in my workspace.
... and you would of course still see that.
Runs 'git status' for each of the specified projects. [...] As I wrote earlier, that is always relative to HEAD because it is meant as a wrapper around git status.
When using submodules, running git status does a bit more than just iterating on submodules and I suspect similar tools do the same.
I'm not against something like we are discussing, but let's please drop the idea of forcing this into west status.
Fine but generally speaking I don't think users should expect commands of the same name to automatically be simple forall loops (see also #337 on that topic), I don't see anything "obvious" about that.
The existence of ZEPHYR_MODULES and ZEPHYR_EXTRA_MODULES could push additional code into a local build that stands outside of anything in the manifest file, for example.
There is no doubt that environment variables and many other things can affect the final output even when all git versions are exactly the same. There's a good list at https://reproducible-builds.org/docs/ Before any of these however, the very first thing to check is version control and -dirty repositories. Without such a solid starting point there's not much point looking at anything else. So it is CMake's or west's job to run git describe/status/diff for that pure version control purpose? Of course there's always some west forall "nuclear" option on the west side if everything else falls short.
Going out on a limb here, but I've always felt git should be doing more to solve this problem.
Why git modules are based on non human readable entries into the index versus human readable configuration files tracked like .gitignore I'll never understand.
I assume we are all under the assumption that getting git to change is not a better path?
I have not taken a look at the development community surrounding git but I'm always down to help make a case if one can be made.
It's a digression but in interesting one (and no: git is not going to totally change how submodules work just for us, you can forget about that)
If usability is not a top concern[*], I can see the submodules appeal of not mixing up data (editable text files) with metadata (children SHAs) because that helps avoid some dilemma. Take for example surepos of subrepos, I mean three levels of git repos. Now imagine the manifest at the middle level has been edited. So there are now two different manifests at the middle level: the edited one and the committed one. Which one should git use? The edited one because you should never ignore user input? Or the committed one because that's what the upper level refers to? Do you know what west does here? With submodules no need to ask yourself this question because you cannot "edit" the pointer, there's no file where you can do that. That extra degree of freedom is not present.
[*] https://www.ted.com/talks/linus_torvalds_the_mind_behind_linux/transcript, search "island"
How about we add a new
statusformatting option inwest listthat is able to produce the following output:
But you still have the use-case where a sub-project is tacking a branch and using import, like this:
- name: zephyr
revision: main
import: true
and where the manifest in Zephyr contains only SHAs.
Now looping all those imported projects will tell you they are SHAs, and thus appear correct, but the Zephyr state will be unknown.
As you don't know if the Zephyr manifest is up-to-date, then imho it would be wrong to mark any of those imported projects synced, even though they are using SHAs.
We need to make sure that when any developer builds a final image of the (app + NCS bundle) we can programatically verify that the sources used were exactly the expected ones, so that the final image can be bit by bit reproducible in any subsequent build.
In my opinion this (very reasonable) requirement indicates a solution in the build system. The existence of ZEPHYR_MODULES and ZEPHYR_EXTRA_MODULES could push additional code into a local build that stands outside of anything in the manifest file, for example.
but we need to remember, there might be projects in the west manifest that are not Zephyr modules, and thus they won't be part of build/zephyr_modules.txt.
But I do think there could be value in the CMake build system to support generating a meta file of:
- Zephyr modules in use, and there revision if the module is also a git repository
- west projects available if
west listis used as part of the build process.
I think I would prefer to have projects duplicated in both west and zephyr module if they are both in west but also a Zephyr module.
Cause Zephyr modules themselves can only specify the SHAs and dirty.
But west may contain more project, and also have an expected revision.
Let me try to prototype, and you can provide some feedback.
On nrfconnect/sdk-nrf#5797 (comment) @tejlmand wrote a proposal to use cmake's find_package( EXACT ) to achieve this purpose, but in our case this would not go all the way, essentially because package versions are manually updated.
yes, but this is to ensure you build against something known, which is different than telling what you actually built against.
The latter doesn't care what you built against, it will built against anything (or break in the process).
It's a digression but in interesting one (and no: git is not going to totally change how submodules work just for us, you can forget about that)
If usability is not a top concern[*], I can see the submodules appeal of not confusing data with metadata because that helps avoid some dilemma. Take for example surepos of subrepos, I mean three levels of git repos. Now imagine the manifest at the middle level has been edited. So there are now two different manifests at the middle level: the edited one and the committed one. Which one should git use? The edited one because you should never ignore user input? Or the committed one because that's what the upper level refers to? Do you know what west does here? With submodules no need to ask yourself this question because you cannot "edit" the pointer, there's no file where you can do that. That extra degree of freedom is not present.
[*] https://www.ted.com/talks/linus_torvalds_the_mind_behind_linux/transcript, search "island"
I debated taking this to a separate github discussion since yes it is a digression and maybe still will as it continues to turn into a thought experiment. In the case referenced I might actually prefer git throw up it's hands and say "sorry, you touched something that should only be touched in isolation"
So essentially prevent resolving and pulling the dependency change until the change to the manifest is committed upstream, to a branch (which would force a change in the manifest of the top level), or till the mid level manifest change is stashed?
But you still have the use-case where a sub-project is tacking a branch and using import, like this:
I would argue in that case all imported projects are to be tagged as unknown
But I do think there could be value in the CMake build system to support generating a meta file of:
Whether it fixes the problem(s) reported here or not, let's please discuss CMake work in https://github.com/zephyrproject-rtos/zephyr/issues and not here. There are people trying to use west without Zephyr.
So it is CMake's or west's job to run git describe/status/diff for that pure version control purpose?
"A bit of both" could be an acceptable answer BTW. Generally minor build performance issues aside, having "too much" version information has never hurt and the overlap is never exactly 100%, I mean there are always corner cases where one fails or is missing and then it's great to fall back on the other zephyrproject-rtos/zephyr@654a2f245d655f
There are people trying to use west without Zephyr.
Which was exactly the reason I wrote: if west list is used as part of the build process.
I don't see the harm in taking up a build infrastructure discussion in west, if it turn out to actually meet what the users need.
Of course, if it do turn out that this indeed should be fixed in the build system then a new discussion can be opened there.
Please find a PR here zephyrproject-rtos/zephyr#39382 for creation of a build meta file as part of the build process.
Note, that PR is simply a snapshot of what SHAs was actually in use when building and no relation to branches, tags, or similar.
Whether it fixes the problem(s) reported here or not, let's please discuss CMake work in https://github.com/zephyrproject-rtos/zephyr/issues and not here. There are people trying to use west without Zephyr.
The other issue I keep mentioning (zephyrproject-rtos/zephyr#23025) started out life in the west issues page. I moved it to zephyr following the discussion. That is basically what I was proposing here, but I didn't want to unilaterally make the move without further discussion.
I agree it would be an interface change but unless you have some example I think the term "breaking" is just as wrong. I don't think it would "break" much more than eff0e1b already did but maybe I'm missing something.
I mean... it would be a backwards incompatible change which is inconsistent with the current documented behavior. That is generally what people mean when they say "breaking". What am I missing?
But you still have the use-case where a sub-project is tacking a branch and using import, like this:
I would argue in that case all imported projects are to be tagged as
unknown
I think making this happen would be a big complexity increase with unknown benefit since the use case really seems to be about Zephyr builds and not west itself, as least as far as I can see.
Please find a PR here zephyrproject-rtos/zephyr#39382 for creation of a build meta file as part of the build process.
Thanks for this - I will be curious to see what @xnlcasad thinks of this.
I mean... it would be a backwards incompatible change which is inconsistent with the current documented behavior. That is generally what people mean when they say "breaking". What am I missing?
I think "breaking" implies some computer program failing. So far you've only mentioned interactive use and humans are more flexible and less binary than computers - especially when they perceive the "backwards incompatible" change as progress. I believe that's why earlier comit eff0e1b was not advertised as "breaking" anything either.
In fact this very issue is about making west status_or_something usable by a computer which demonstrates it's not there yet.
I believe that's why earlier comit eff0e1b was not advertised as "breaking" anything either.
It wasn't advertised as "breaking" because it omits printing useless information instead of changing the information printed.
I'm starting to come down on "this isn't well specified enough for upstreaming into west; I think people who are interested in this should experiment with the west APIs and come back with code before we can proceed."
Something I want to remind us all about is that west is optional and supposed to be conservative about adding features (https://docs.zephyrproject.org/latest/guides/west/why.html#design-constraints).
And I have personally screwed up in the past by getting things merged and having to issue a breaking change later -- the initial implementation of project groups is a good example of that.
But I'm really not sure we've got anything near to a compelling argument that there is really a missing feature in west for getting project status, given that:
- it's not obvious (at least to me) that we have defined how this feature should behave in a way that is clear and addresses the use case
- it seems like the use case actually belongs in a higher layer
- it isn't hard to write west extensions or even standalone python scripts to do simple tests, to get experimental data
West's Python APIs are documented here:
https://docs.zephyrproject.org/latest/guides/west/west-apis.html#west-apis
Here is a relatively simple script you can run from your workspace, for example:
(edit: pasted a buggy version the first time, sorry)
import subprocess
import sys
from west.manifest import Manifest, ManifestProject
for p in Manifest.from_file().projects:
commit = 'HEAD' if isinstance(p, ManifestProject) else 'manifest-rev'
try:
subprocess.run(f'git diff --quiet {commit}', shell=True, check=True, cwd=p.abspath)
except subprocess.CalledProcessError as e:
sys.exit(e.returncode)This exits 0 if there is a nonempty diff from manifest-rev for all your projects and your manifest repository also has a clean working tree. Otherwise, it bubbles up the return code from git diff.
It seems easy enough to me to run something like that in a Zephyr application's build system and do what you need to with it. When we've got some experimental validation that people's scripts do in fact need to be upstreamed into west itself because they solve a general enough problem, then it will be straightforward to do that.
If you're not comfortable in Python and you need some help using the APIs, feel free to ask for support and I'll do my best here.
eff0e1b wasn't advertised as "breaking" because it omits printing useless information instead of changing the information printed.
A clean git status on a repo away from manifest-rev is:
- something that commit eff0e1b removed from
west statusoutput, and - exactly what is requested in the description of this issue.
If you think this was "useless information" then I'm not sure why we (and especially you) spent so much time and effort discussing here.
thanks @mbolivar for such a complete answer. I agree that given all the use cases that west has to deal with it seems non-trivial to specify a no-surprise, straightforward behaviour to this 'feature'. I agree with your suggestion of letting the community experiment and come back with a proposal that has proved itself to work in the trenches.
Just fyi, at the moment, we are hardcoding the repos sha1s in the freestanding application's CMakeLists.txt, like this:
macro(use_ncs_sdk_repo_at_revision path sha1)
set(repo_${path} ${path} ${sha1})
list(APPEND repo_list repo_${path})
endmacro()
function(validate_ncs_sdk list_of_repos clean_sdk)
set(clean_sdk YES PARENT_SCOPE)
foreach(a ${list_of_repos})
list(GET ${a} 0 mod_path)
list(GET ${a} 1 expected_mod_sha)
message(STATUS "Checking repo: ${mod_path}, expected rev: ${expected_mod_sha}")
execute_process(COMMAND git -C ${NCS_DIR}/${mod_path} rev-parse HEAD
OUTPUT_VARIABLE mod_sha OUTPUT_STRIP_TRAILING_WHITESPACE)
if (NOT mod_sha STREQUAL expected_mod_sha)
message(WARNING " repo: ${NCS_DIR}/${mod_path} is not at the expected revision.\n"
" expected revision: ${expected_mod_sha}\n"
" actual revision : ${mod_sha}\n")
set(clean_sdk NO PARENT_SCOPE)
endif()
execute_process(COMMAND git -C ${NCS_DIR}/${mod_path} diff --exit-code HEAD
RESULT_VARIABLE mod_dirty OUTPUT_QUIET ERROR_QUIET)
if (mod_dirty STREQUAL 1)
message(WARNING "repo: ${NCS_DIR}/${mod_path} has modified local files")
set(clean_sdk NO PARENT_SCOPE)
endif()
endforeach(a)
endfunction()
use_ncs_sdk_repo_at_revision(nrf d29f1ddec99b81a66af93b7f0882a7bd0dfa30ee)
use_ncs_sdk_repo_at_revision(zephyr f19ebe4b50405b2eb174199b3635f24d741c9bf4)
use_ncs_sdk_repo_at_revision(nrfxlib a79fdc3dc432e40a13d9c631582aa5f83ea93635)
use_ncs_sdk_repo_at_revision(bootloader/mcuboot b514a3190821b7721dd6ffcd9940474ebe7a8832)
set(clean_sdk NO)
validate_ncs_sdk("${repo_list}" ${clean_sdk})
if (NOT IGNORE_NCS_SDK_STATE AND NOT clean_sdk)
message(FATAL_ERROR "you are using an sdk that is not synced to this application")
endif ()
This is enough for our current needs. It has the caveat that this runs only at configure time, so we will need to rework it (maybe reusing or getting inspired from https://stackoverflow.com/a/4318642/2748456) so that configure is (re)triggered on git project status change.
However, I would still be interested in understanding how would west achieve R4 in https://docs.zephyrproject.org/latest/guides/west/why.html#requirements because I dont't see how to link a freestanding application with a specific version (down to the sha1) of the ncs sdk and its dependencies. This lets me wondering how I would go about bisecting the app's history accurately.
Then, another part of the problem is that we usually include the sha1 and the dirty status information within the final binary itself, so as to be able to rebuild from the exact same sources any binary found on a final product any time (even years) after the binary was produced. In other words this sha1+dirty information acts like a signature over all the source code used to build the application. I would have expected that west could have provided a similar feature to what git submodules provide, but as it looks, it is a non-trivial endeavour, due to the many other use cases that west has to handle.
IMO, we've done a good effort on brainstorming around this, and we see that there are some custom solutions that can be used to achieve this purpose somehow. This is enough for our needs, so I would be OK if you'd decide to close this issue now. Thanks!
A clean git status on a repo away from manifest-rev is:
- something that commit eff0e1b removed from
west statusoutput, and- exactly what is requested in the description of this issue.
No, that is just not true. I wrote this at the beginning of this discussion:
west status, as a wrapper around git status, reports status relative to HEAD, not manifest-rev
The commit you are referencing did not remove the output this issue is asking about from anything.
If you think this was "useless information" then I'm not sure why we (and especially you) spent so much time and effort discussing here.
I don't think it's useless information. I think you are confused about what west status does and always did do.
However, I would still be interested in understanding how would west achieve R4 in https://docs.zephyrproject.org/latest/guides/west/why.html#requirements because I dont't see how to link a freestanding application with a specific version (down to the sha1) of the ncs sdk and its dependencies. This lets me wondering how I would go about bisecting the app's history accurately.
This requirement is met by west only for 'workspace applications'. If you're using a freestanding application, west isn't going to help you as much and you have to use the build system. This is because by definition a freestanding application is apart from the workspace, which is what west manages.
IMO, we've done a good effort on brainstorming around this, and we see that there are some custom solutions that can be used to achieve this purpose somehow. This is enough for our needs, so I would be OK if you'd decide to close this issue now. Thanks!
OK, thanks for confirming! I will close this then, but please feel free to reopen if you (or anyone else) feels they have a general solution backed up by experience and testing.
The commit you are referencing did not remove the output this issue is asking about from anything.
By chance this is the output of west status I can see right now on one of the systems I don't use often:
...
=== status of tflite-micro (modules/lib/tflite-micro):
HEAD detached at refs/heads/manifest-rev
nothing to commit, working tree clean
=== status of tinycbor (modules/lib/tinycbor):
HEAD detached at refs/heads/manifest-rev
Untracked files:
(use "git add <file>..." to include in what will be committed)
foo
nothing added to commit but untracked files present (use "git add" to track)
=== status of tinycrypt (modules/crypto/tinycrypt):
HEAD detached at refs/heads/manifest-rev
nothing to commit, working tree clean
=== status of TraceRecorderSource (modules/debug/TraceRecorder):
HEAD detached at 36c577727642 # == manifest-rev
nothing to commit, working tree clean
=== status of trusted-firmware-m (modules/tee/tfm):
HEAD detached at c74be3890c9d # == manifest-rev
...
This is with very recent west version 38e656b and git version 2.7.11.
I think you are confused about what west status does and always did do.
I honestly don't know why in this workspace git status displays manifest-rev most of the time and other times the manifest-rev SHA1 instead. But for sure the output above looks very much like what this issue requested.
Even when git status does not show manifest-rev, it provides the same information indirectly. I mean comparing a full west status output with the manifest is/was probably the most efficient way to work around the lack of the feature requested here. So for sure dropping repos from that list did not help.
I honestly don't know why in this workspace
git statusdisplaysmanifest-revmost of the time and other times the SHA1 instead.
... because git status (and therefore west status) is displaying status relative to HEAD. Which can be manifest-rev, or it can be any other random commit, as your own example shows.
But for sure the output above looks very much like what this issue requested.
No, again, HEAD != manifest-rev
You missed the point. In the example I gave HEAD == manifest-ref in ALL repos whether git status shows manifest-ref or not. I added some comments to make that clearer.
I honestly don't know why in this workspace git status displays manifest-rev most of the time and other times the manifest-rev SHA1 instead
I had a look at git's source code to get to the bottom of this. This depends on the reflog. I didn't expect "git status" to look at the reflog but strace confirmed that.
In git/wt_status.c, wt_status_get_detached_from() scans the reflog. Depending on what it finds, it makes wt_status_state->detached_from non-zero which changes the output of both git status and git branch: it makes these two commands stop displaying SHA1s and show "Detached from..." or "Detached at..." instead, pointing to the branch it found in the reflog.
After an initial west init + west update, git status shows HEAD detached at refs/heads/manifest-rev in every repo. Every time the manifest changes, the corresponding repos will for some reflog reason switch to displaying a SHA instead.
Tests and analysis based on git 2.7.1, reproduced with git version 2.31.1
You missed the point. In the example I gave
HEAD == manifest-refin ALL repos whethergit statusshowsmanifest-refor not.
OK, I see, in this particular situation you are saying they are the same; I get it. Thanks for explaining. But my point remains that they are not the same in general, though.
But my point remains that they are not the same in general, though.
?
Of course they are not in general, that's exactly why this issue was filed: to highlight the repos where they are not.