Adding `composer.lock` in VCS ?
drupol opened this issue · 4 comments
Hi there,
I'm writing to suggest an enhancement for this project to improve its integration and ensure long-term reliability, particularly for package managers like apt, rpm, yum, nix, etc etc.
The primary goal here is to provide full reproducibility of the exact build of the PHAR released, down to the last bit. This is crucial, especially when distributing it as a binary for reasons of reliability and security.
For example, without composer.lock, running two composer install at different date and time could potentially create 2 different vendor directories. This is what we strive to avoid at all cost.
To achieve full reproducibility, I propose adding a composer.lock file to this repository. This file will ensure that gdpr-dump is reproducible on any system, regardless of environmental differences, by locking down specific versions of dependencies. Its absence currently hinders the ability to precisely replicate the builds you provide in your GitHub releases, introducing potential variability.
For example, in Nix, we have to provide our own composer.lock for each update, as seen in this PR: NixOS/nixpkgs#286809 (I'll keep that PR in draft until I have some feedback about this issue).
Having a composer.lock file in your responsibility would transfer the responsibility of providing something stable and reproducible to this project instead of delegating it to anyone else.
For reference, here are some projects that have recognized the value of versioning the composer.lock file:
- sebastianbergmann/phpunit#5576
- bobthecow/psysh#767
- phpro/grumphp#1095
- NixOS/nixpkgs#255253
- https://github.com/phpstan/phpstan-src/blob/1.11.x/composer.lock
If for any reason there are concerns about including the composer.lock file in the main repository, could it be an option to add it to the release artefacts on GitHub along with the PHAR file? This middle-ground compromise would still greatly benefit reproducibility and consistency in dependency management.
Thank you for considering this enhancement. I look forward to your response.
Hi,
To achieve full reproducibility, I propose adding a composer.lock file to this repository.
As already explain in some of the other issues you created, adding a composer.lock file in the repo of a composer package is more a hindrance than a benefit (i would even say it's a bad practice).
Contrary to a production project (e.g. a drupal or magento website), a php package is compatible with multiple php versions, and there is no need to have a development environment that matches a production environment.
Not commiting the lock file ensures that the vendor dir always matches the version of the PHP runtime (e.g. for people who work on a PR, or for a CI pipeline that runs unit/functional tests, such as https://github.com/Smile-SA/gdpr-dump/blob/main/.github/workflows/tests.yaml).
This file will ensure that gdpr-dump is reproducible on any system
Currently it would not be enough.
If you want to create exact copies of our released phars, you have to make the same actions as our github workflow that creates the releases (notably the update of the constant the stores the app version, cf. https://github.com/Smile-SA/gdpr-dump/blob/main/.github/workflows/release.yaml#L45). To keep things as simple as possible, this change is not committed anywhere.
We can probably solve this issue by using the vendor/composer/installed.php file to determine the version (I think it's what phpstan does).
If for any reason there are concerns about including the composer.lock file in the main repository, could it be an option to add it to the release artefacts on GitHub along with the PHAR file? This middle-ground compromise would still greatly benefit reproducibility and consistency in dependency management.
It would not be complicated to implement (just an additional step in the github workflow), but I fear that a lot of people would be confused by this ("What the hell is this file? What do I use it for?" 😅).
I fully understand. I wouldn't say it's bad practice, but I understand that it might be a change of habits and therefore be considered as a hindrance.
Currently it would not be enough.
I can attest that this is really enough to make it reproducible, and this PR is actually the proof (NixOS/nixpkgs#286809). Nix will create identical copies of gdpr-dump thanks to that file (I just did it, it works fine!).
❯ nix build .#gdpr-dump -L
gdpr-dump> Running phase: unpackPhase
gdpr-dump> unpacking source archive /nix/store/fy6swavv78b262nsx2n2fw1rv78hi5rv-source
gdpr-dump> source root is source
gdpr-dump> Running phase: patchPhase
gdpr-dump> Running phase: updateAutotoolsGnuConfigScriptsPhase
gdpr-dump> Running phase: configurePhase
gdpr-dump> Executing composerInstallConfigureHook
gdpr-dump> Validating consistency between composer.lock and /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/composer.lock
gdpr-dump> Finished composerInstallConfigureHook
gdpr-dump> Running phase: buildPhase
gdpr-dump> Executing composerInstallBuildHook
gdpr-dump> Local repository manifest "packages.json" has been successfully created in /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository
gdpr-dump> Loading composer repositories with package information
gdpr-dump> Updating dependencies
gdpr-dump> Nothing to modify in lock file
gdpr-dump> Writing lock file
gdpr-dump> No security vulnerability advisories found.
gdpr-dump> Finished composerInstallBuildHook
gdpr-dump> Running phase: checkPhase
gdpr-dump> Executing composerInstallCheckHook
gdpr-dump> Finished composerInstallCheckHook
gdpr-dump> Running phase: installPhase
gdpr-dump> Executing composerInstallInstallHook
gdpr-dump> Installing dependencies from lock file
gdpr-dump> Verifying lock file contents can be installed on current platform.
gdpr-dump> Package operations: 25 installs, 0 updates, 0 removals
gdpr-dump> - Installing psr/log (3.0.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/psr/log/3.0.0
gdpr-dump> - Installing psr/cache (3.0.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/psr/cache/3.0.0
gdpr-dump> - Installing doctrine/event-manager (2.0.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/doctrine/event-manager/2.0.0
gdpr-dump> - Installing doctrine/deprecations (1.1.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/doctrine/deprecations/1.1.3
gdpr-dump> - Installing doctrine/cache (2.2.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/doctrine/cache/2.2.0
gdpr-dump> - Installing doctrine/dbal (3.8.1): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/doctrine/dbal/3.8.1
gdpr-dump> - Installing druidfi/mysqldump-php (1.0.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/druidfi/mysqldump-php/1.0.3
gdpr-dump> - Installing symfony/deprecation-contracts (v3.4.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/deprecation-contracts/v3.4.0
gdpr-dump> - Installing psr/container (2.0.2): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/psr/container/2.0.2
gdpr-dump> - Installing fakerphp/faker (v1.23.1): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/fakerphp/faker/v1.23.1
gdpr-dump> - Installing justinrainbow/json-schema (v5.2.13): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/justinrainbow/json-schema/v5.2.13
gdpr-dump> - Installing symfony/polyfill-ctype (v1.29.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/polyfill-ctype/v1.29.0
gdpr-dump> - Installing symfony/polyfill-mbstring (v1.29.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/polyfill-mbstring/v1.29.0
gdpr-dump> - Installing symfony/filesystem (v7.0.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/filesystem/v7.0.3
gdpr-dump> - Installing symfony/config (v6.4.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/config/v6.4.3
gdpr-dump> - Installing symfony/polyfill-intl-normalizer (v1.29.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/polyfill-intl-normalizer/v1.29.0
gdpr-dump> - Installing symfony/polyfill-intl-grapheme (v1.29.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/polyfill-intl-grapheme/v1.29.0
gdpr-dump> - Installing symfony/string (v7.0.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/string/v7.0.3
gdpr-dump> - Installing symfony/service-contracts (v3.4.1): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/service-contracts/v3.4.1
gdpr-dump> - Installing symfony/console (v6.4.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/console/v6.4.3
gdpr-dump> - Installing symfony/var-exporter (v7.0.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/var-exporter/v7.0.3
gdpr-dump> - Installing symfony/dependency-injection (v6.4.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/dependency-injection/v6.4.3
gdpr-dump> - Installing symfony/finder (v6.4.0): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/finder/v6.4.0
gdpr-dump> - Installing symfony/yaml (v6.4.3): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/symfony/yaml/v6.4.3
gdpr-dump> - Installing theseer/tokenizer (1.2.2): Mirroring from /nix/store/mzkwksdj5z0krg3z98vl8hznandjrjcf-gdpr-dump-4.0.3-composer-repository/theseer/tokenizer/1.2.2
gdpr-dump> Generating autoload files
gdpr-dump> 18 packages you are using are looking for funding.
gdpr-dump> Use the `composer fund` command to find out more!
gdpr-dump> Finished composerInstallInstallHook
gdpr-dump> Running phase: fixupPhase
gdpr-dump> shrinking RPATHs of ELF executables and libraries in /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3
gdpr-dump> shrinking /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/bin/gdpr-dump
gdpr-dump> checking for references to /build/ in /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3...
gdpr-dump> patching script interpreter paths in /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3
gdpr-dump> /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/share/php/gdpr-dump/bin/compile: interpreter directive changed from "#!/usr/bin/env php" to "/nix/store/a41yfg0vzgmnklj7frgp3c1qr1v6xrhr-php-with-extensions-8.2.15/bin/php"
gdpr-dump> /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/share/php/gdpr-dump/bin/gdpr-dump: interpreter directive changed from "#!/usr/bin/env php" to "/nix/store/a41yfg0vzgmnklj7frgp3c1qr1v6xrhr-php-with-extensions-8.2.15/bin/php"
gdpr-dump> /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/share/php/gdpr-dump/vendor/bin/validate-json: interpreter directive changed from "#!/usr/bin/env php" to "/nix/store/a41yfg0vzgmnklj7frgp3c1qr1v6xrhr-php-with-extensions-8.2.15/bin/php"
gdpr-dump> /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/share/php/gdpr-dump/vendor/bin/yaml-lint: interpreter directive changed from "#!/usr/bin/env php" to "/nix/store/a41yfg0vzgmnklj7frgp3c1qr1v6xrhr-php-with-extensions-8.2.15/bin/php"
gdpr-dump> /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/share/php/gdpr-dump/vendor/bin/doctrine-dbal: interpreter directive changed from "#!/usr/bin/env php" to "/nix/store/a41yfg0vzgmnklj7frgp3c1qr1v6xrhr-php-with-extensions-8.2.15/bin/php"
gdpr-dump> /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/share/php/gdpr-dump/vendor/symfony/yaml/Resources/bin/yaml-lint: interpreter directive changed from "#!/usr/bin/env php" to "/nix/store/a41yfg0vzgmnklj7frgp3c1qr1v6xrhr-php-with-extensions-8.2.15/bin/php"
gdpr-dump> /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/share/php/gdpr-dump/vendor/doctrine/dbal/bin/doctrine-dbal: interpreter directive changed from "#!/usr/bin/env php" to "/nix/store/a41yfg0vzgmnklj7frgp3c1qr1v6xrhr-php-with-extensions-8.2.15/bin/php"
gdpr-dump> /nix/store/gg62c3fyrdr5dxrfamxhjz56vlyahpjv-gdpr-dump-4.0.3/share/php/gdpr-dump/vendor/justinrainbow/json-schema/bin/validate-json: interpreter directive changed from "#!/usr/bin/env php" to "/nix/store/a41yfg0vzgmnklj7frgp3c1qr1v6xrhr-php-with-extensions-8.2.15/bin/php"
❯ ./result/bin/gdpr-dump --version
GdprDump 4.0.3
~/C/N/nixpkgs > init/gdpr-dump +5 [!] ❯
As a middle ground solution, would you be OK to publish the composer.lock used to create the PHAR file for each release? It wouldn't require more work from you since this can be automated and this would definitely allow anybody to make a reproducible PHAR file and would help us for not having to maintain a composer.lock in the Nix repository.
I can attest that this is really enough to make it reproducible, and this PR is actually the proof (NixOS/nixpkgs#286809). Nix will create identical copies of gdpr-dump thanks to that file (I just did it, it works fine!).
I meant to say adding a composer.lock file is not enough, as you also need to update the app version before creating the phar.
But it's now a thing of the past, I just pushed a commit that automatically detects the version (same logic as phpstan).
As a middle ground solution, would you be OK to publish the composer.lock used to create the PHAR file for each release? It wouldn't require more work from you since this can be automated and this would definitely allow anybody to make a reproducible PHAR file and would help us for not having to maintain a composer.lock in the Nix repository.
As said in my previous message, I fear that it could confuse users who want to download the phar and don't know what this file is for. Do you really need this file? Not using a lock file would ensure that your phar file has the most up-to-date dependencies (which is good for security).
I meant to say adding a composer.lock file is not enough, as you also need to update the app version before creating the phar.
And what I was saying is that that particular step is optional since I was already able to get something reproducible.
But it's now a thing of the past, I just pushed a commit that automatically detects the version (same logic as phpstan).
This commit assumes multiple things to be true now. It expects that the git command is available on the host AND that the .git directory is also available in the src directory. In some cases, when building software in isolation, that command and that very particular directory are not available (one of the reason is that because .git is not reproducible). I was much more preferring the old version... but hey, you're deciding here, I'm just giving my POV.
Here's a script example where this would fail (the .git directory is missing):
rm -rf main.zip gdpr-dump-main
echo "Getting a snapshot of the main branch..."
wget -q https://github.com/Smile-SA/gdpr-dump/archive/refs/heads/main.zip
echo "Unzipping..."
unzip -qq main.zip
cd gdpr-dump-main
echo "Running composer install..."
composer install --no-dev --quiet
echo "Compiling the PHAR..."
php -d phar.readonly=0 bin/compile
echo "Print a PHAR checksum..."
sha256sum build/dist/gdpr-dump.phar
echo "Check the PHAR version..."
php build/dist/gdpr-dump.phar --versionAs said in my previous message, I fear that it could confuse users who want to download the phar and don't know what this file is for. Do you really need this file? Not using a lock file would ensure that your phar file has the most up-to-date dependencies (which is good for security).
Sad thing, I'll close both issues then.