"cookie-composer update" did not updated the template layers of a project created from an old version of the template composition

Question

"cookie-composer update" did not updated the template layers of a project created from an old version of the template composition

Closed this issue 10 months ago · 9 comments

cookie-composer version: 0.11.0
Python version: 3.11.0
Operating System: Linux 631c6f7e8441 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:14:14 UTC 2023 x86_64 GNU/Linux

Description

First of all, I'd like to congratulate Calloway's team for providing this tool to the Python community. I believe it will be very useful to keep the many projects out there with less effort in the boilerplate part of the code base.

Now, about the issue... I'm testing the tool before using in a real case. My test has three steps:

Create a project with the template composition of https://github.com/coordt/cookiecomposer-templates, using the old version from commit 6a6d3c7cef1d5b2daabe9f749057dd6fe2875f67;
Update the project with the latest version of the template composition.
Since, I create the project with an old version of the template composition, I expected cookie-composer to update the template in step 2, but it said that all layers were up-to-date.

What I Did

I've executed the commands as follows:

user@631c6f7e8441:/tmp/teste$ curl https://raw.githubusercontent.com/coordt/cookiecomposer-templates/master/package-composition.yaml > package-composition.yaml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   603  100   603    0     0   1477      0 --:--:-- --:--:-- --:--:--  1481
user@631c6f7e8441:/tmp/teste$ cookie-composer create --checkout=6a6d3c7cef1d5b2daabe9f749057dd6fe2875f67 package-composition.yaml 
You've downloaded /home/user/.cookiecutters/cookiecomposer-templates before. Is it okay to delete and re-download it? [yes]: 
project_name [python-boilerplate]: 
project_slug [python_boilerplate]: 
friendly_name [Python Boilerplate]: 
project_short_description []: 
version [0.1.0]: 
author [Who am I?]: 
email [whoami@existential-crisis.doom]: 
github_user [whoami]: 
You've downloaded /home/user/.cookiecutters/cookiecomposer-templates before. Is it okay to delete and re-download it? [yes]: 
You've downloaded /home/user/.cookiecutters/cookiecomposer-templates before. Is it okay to delete and re-download it? [yes]

The .composition.yaml file was generated with the correct commit, as follows:

user@631c6f7e8441:/tmp/teste$ cat python-boilerplate/.composition.yaml 
checkout: 6a6d3c7cef1d5b2daabe9f749057dd6fe2875f67
commit: 6a6d3c7cef1d5b2daabe9f749057dd6fe2875f67
context:
  _copy_without_render:
  - .github/**/*.jinja
  _dev_requirements:
    bump2version: '>=1.0.1'
    generate-changelog: '>=0.7.6'
    git-fame: '>=1.12.2'
    pip-tools: ''
  _prod_requirements:
    environs: '>=9.3.5'
  _test_requirements:
    black: '>=19.10b0'
    coverage: '>=6.1.2'
    flake8: '>=4.0.1'
    pre-commit: '>=2.15.0'
    pytest: '>=6.0.0'
    pytest-cov: '>=3.0.0'
  author: Who am I?
  email: whoami@existential-crisis.doom
  friendly_name: Python Boilerplate
  github_user: whoami
  project_name: python-boilerplate
  project_short_description: ''
  project_slug: python_boilerplate
  version: 0.1.0
directory: cookiecutter-boilerplate
merge_strategies:
  '*.json': comprehensive
  '*.yaml': comprehensive
  '*.yml': comprehensive
no_input: false
overwrite: []
overwrite_exclude: []
password: null
skip_generation: []
skip_hooks: false
skip_if_file_exists: false
template: https://github.com/coordt/cookiecomposer-templates
---
checkout: 6a6d3c7cef1d5b2daabe9f749057dd6fe2875f67
commit: 6a6d3c7cef1d5b2daabe9f749057dd6fe2875f67
context:
  _copy_without_render:
  - docsrc/**/*.rst
  _dev_requirements: {}
  _docs_requirements:
    Sphinx: '>=4.3.0'
    furo: ''
    ghp-import: ''
    linkify-it-py: ''
    myst-parser: ''
    sphinx-autodoc-typehints: ''
    sphinx-click: ''
    sphinx-copybutton: ''
  _prod_requirements: {}
  _test_requirements:
    black: '>=19.10b0'
    coverage: '>=6.1.2'
    flake8: '>=4.0.1'
    pre-commit: '>=2.15.0'
    pytest: '>=6.0.0'
    pytest-cov: '>=3.0.0'
  author: Who am I?
  email: whoami@existential-crisis.doom
  friendly_name: Python Boilerplate
  github_user: whoami
  project_name: python-boilerplate
  project_short_description: ''
  project_slug: python_boilerplate
  version: 0.1.0
directory: cookiecutter-package
merge_strategies:
  '*.json': comprehensive
  '*.yaml': comprehensive
  '*.yml': comprehensive
no_input: false
overwrite: []
overwrite_exclude: []
password: null
skip_generation: []
skip_hooks: false
skip_if_file_exists: false
template: https://github.com/coordt/cookiecomposer-templates
---

However, when I ran the update command, cookie-composer gave the feedback that all layers were up-to-date.

user@631c6f7e8441:/tmp/teste/python-boilerplate$ cookie-composer update 
You've downloaded /home/user/.cookiecutters/cookiecomposer-templates before. Is it okay to delete and re-download it? [yes]: 
You've downloaded /home/user/.cookiecutters/cookiecomposer-templates before. Is it okay to delete and re-download it? [yes]: 
You've downloaded /home/user/.cookiecutters/cookiecomposer-templates before. Is it okay to delete and re-download it? [yes]: 
All layers are up-to-date.

Answer 1 · 2023-09-06T11:59:11.000Z

Sorry for the delayed response. I swear GitHub makes it hard to notice when issues are raised.

I'll schedule this work soon (probably this weekend, Sep 9/10).

Answer 2 · 2023-09-11T11:48:10.000Z

Just a quick update: I've found what was causing the issue. It was always comparing itself to the same commit on the template.

That said, the method you used above to test would not work exactly. The "checkout" flag is usually meant for branches, in that when the repo template is cloned, it check out the reference of "checkout". You would have to empty it from .composition.yaml file after you first generated the new repo.

That said, I hit another snag. The UI experience. With the way that cookiecutter handles things, the update process will ask you all the prompts, with the appropriate answers as the default. I think it is better if it only prompts for new things.

So I'm doing a bit of a refactor.

Answer 3 · 2023-09-12T09:18:42.000Z

Hi @coordt , thanks for the feedkback!

Regarding what you said about "checkout" parameter, I've forked https://github.com/coordt/cookiecomposer-templates, created the "test" branch, and adjusted package-composition.yaml to test again.

Once you are done with that changes, I'll follow the steps below.

pip install --upgrade cookie-composer
mkdir /tmp/test
cd /tmp/test
curl https://raw.githubusercontent.com/dinaldoap/cookiecomposer-templates/test/package-composition.yaml > package-composition.yaml
cookie-composer create --checkout=test package-composition.yaml
# Update template by adding a new section into https://github.com/dinaldoap/cookiecomposer-templates/blob/test/cookiecutter-package/%7B%7Bcookiecutter.project_name%7D%7D/README.md
cd python-boilerplate
cookie-composer update

Please, keep me noticed if you find another misuse of "cookie-composer".

Answer 4 · 2023-10-22T13:57:18.000Z

@dinaldoap Thanks for your patience on this issue. It was a bigger undertaking than expected.

I believe things are fixed, but I created a dev release so you might test it out.

Let me know if it works and I'll merge the PR.

Answer 5 · 2023-10-22T17:41:08.000Z

Hi, @coordt . Thanks for your fix! It worked like a charm when I referred to the template on the master branch.

I'v done the following steps to test that dev release:

pip install cookie-composer==0.11.0rc52.dev56
mkdir /tmp/test
cd /tmp/test
curl https://raw.githubusercontent.com/dinaldoap/cookiecomposer-templates/master/package-composition.yaml > package-composition.yaml
cookie-composer create package-composition.yaml
# Update template by adding a new section into https://github.com/dinaldoap/cookiecomposer-templates/blob/master/cookiecutter-package/%7B%7Bcookiecutter.project_name%7D%7D/README.md
cd python-boilerplate
cookie-composer update

The output was as expected and README.md was updated:

(venv) user@docker:/tmp/test/python-boilerplate$ cookie-composer update
Attempting to apply patch with 3-way merge.
Patch applied successfully.

\o/

Answer 6 · 2023-10-22T17:52:33.000Z

Hi, @coordt . Your fix solves the main problem and I'm able to move forward to integrate cookie-composer to my template. However, I believe I should give an additional feedback here and wait for your thoughts. I don't if this is a topic for another issue...

I've tested that dev release with the same template in another branch (https://github.com/dinaldoap/cookiecomposer-templates/tree/test). The used command and the traceback are as follows:

(venv) user@docker:/tmp/test$ cookie-composer create --checkout=test package-composition.yaml
  [1/8] project_name (python-boilerplate): 
  [2/8] project_slug (python_boilerplate): 
  [3/8] friendly_name (Python Boilerplate): 
  [4/8] project_short_description (): 
  [5/8] version (0.1.0): 
  [6/8] author (Who am I?): 
  [7/8] email (whoami@existential-crisis.doom): 
  [8/8] github_user (whoami): 
Traceback (most recent call last):
  File "/workspace/cookiecutter-python-vscode-github/.venv/bin/cookie-composer", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/rich_click/rich_command.py", line 126, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/cookie_composer/cli.py", line 127, in create
    create_cmd(
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/cookie_composer/commands/create.py", line 66, in create_cmd
    rendered_layers = render_layers(composition.layers, output_dir, no_input=no_input, accept_hooks=accept_hooks)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/cookie_composer/layers.py", line 370, in render_layers
    rendered_layer = render_layer(
                     ^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/cookie_composer/layers.py", line 280, in render_layer
    with layer_config.template.repo.render_source(commit=commit) as repo_dir:
  File "/usr/local/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/cookie_composer/templates/types.py", line 114, in render_source
    with temp_git_worktree_dir(
  File "/usr/local/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/cookie_composer/git_commands.py", line 219, in temp_git_worktree_dir
    repo.git.worktree(
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/git/cmd.py", line 736, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/git/cmd.py", line 1316, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/cookiecutter-python-vscode-github/.venv/lib/python3.11/site-packages/git/cmd.py", line 1111, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git worktree add /tmp/tmpnv7_qulm test
  stdout: 'Preparing worktree (checking out 'test')'
  stderr: 'fatal: 'test' is already checked out at '/home/user/.cookiecutters/cookiecomposer-templates_test''

What do you think?

Answer 7 · 2023-10-22T21:52:33.000Z

Your issue is related to the changes I made. There must be an issue when creating a project with a checkout option. The new process for generating and merging layers uses git worktrees. I'll put in a test for this and get it fixed. It shouldn't take much time (famous last words...)

Answer 8 · 2023-10-22T22:45:27.000Z

@dinaldoap Try https://pypi.org/project/cookie-composer/0.11.0rc52.dev57/

Answer 9 · 2023-10-23T10:01:57.000Z

Hi, @coordt . I've executed the same tests and they all succeeded. 👍

Thanks for your attention!