Add support for browser testing when using .buildpacks

[walkafwalka]

According to https://devcenter.heroku.com/articles/heroku-ci-browser-and-user-acceptance-testing-uat, the proper way for browser testing is to set app.json to the following:

{
  "environments": {
    "test": {
      "buildpacks": [
        { "url": "heroku/nodejs" },
        { "url": "https://github.com/heroku/heroku-buildpack-google-chrome" }
      ]
    }
  }
}

However, herokuish does not support app.json. It does, however, support .buildpacks.

When I add the NodeJS and Google Chrome buildpacks, herokuish uses the multi buildpack and fails to test because it is not using the Node.js buildpack.

[walkafwalka]

/cc @josegonzalez

[josegonzalez]

I don’t use the testing functionality in Dokku, but also this was contributed by folks using it for Gitlab. I’d certainly take a pull request to add support for app.json when detecting buildpacks for deploys and testing.

Also, you don’t need to CC a project maintainer, we’ll see the notification :)

[walkafwalka]

Sorry, I cc'd you specifically cause I know you had commented on a related issue.

[josegonzalez]

One thing that might make this difficult is that there isn't a binary that parses json we can use which we ship with Herokuish, though that can certainly be done in golang...

[walkafwalka]

It still is not clear to me why app.json is required. According to the Heroku documentation, adding the google-chrome buildpack to the app.json file is all that is required. Theoretically, all that would need to be done is adding the google-chrome build (in addition to the original buildpack) to .buildpacks.

I think there is a fundamental difference between the way app.json and .buildpacks work that I do not understand.

[josegonzalez]

We run the test for the last selected buildpack, which chrome in this case.

[walkafwalka]

Sorry, accidentally closed.

I have not seen this behavior. Whether I include the nodejs or google-chrome buildpack, I am still getting the following:

...
+ title 'Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.'
----->' 'Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.'
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
+ indent
+ read -r line
++ sed -e 's:/tmp/buildpacks/[0-9][0-9]_buildpack-::g'
+ echo 'Detected buildpacks: multi nodejs'
+ [[ Detected buildpacks: multi nodejs == --* ]]
+ [[ Detected buildpacks: multi nodejs == ==* ]]
      ' 'Detected buildpacks: multi nodejs'
       Detected buildpacks: multi nodejs
+ read -r line
+ [[ 2 -gt 0 ]]
+ selected_path=/tmp/buildpacks/00_buildpack-multi
++ unprivileged /tmp/buildpacks/00_buildpack-multi/bin/detect /tmp/build
++ setuidgid u28246 /tmp/buildpacks/00_buildpack-multi/bin/detect /tmp/build
+ selected_name=Multipack
+ [[ -n /tmp/buildpacks/00_buildpack-multi ]]
+ [[ -n Multipack ]]
+ title 'Multipack app detected'
----->' 'Multipack app detected'
-----> Multipack app detected
+ [[ ! -f /tmp/buildpacks/00_buildpack-multi/bin/test-compile ]]
+ echo 'Selected buildpack does not support test feature'
Selected buildpack does not support test feature
+ exit 1

[josegonzalez]

Ah, that seems wompy. Seems like we'd want to expand the buildpacks if multi is selected.

Can you show what this looks like on Heroku? I've actually never used this sort of functionality and it was contributed to the project, so there are bound to be small bugs like this.

[walkafwalka]

I am having difficulties setting up testing in Heroku. Something I did notice is that Heroku says the multi buildpack is deprecated.

[MichalBryxi]

Just bumped into this myself. Trying to make GitLab auto devops work and the test phase constantly fails on me, because herokuish code does not know that I need chrome for my tests to run (quite common requirement in JS projects). So trying to figure out whether there is any way to work around this?

[josegonzalez]

Ask the Gitlab folks? If they want to support this sort of functionality, they can either contribute the code or sponsor the work?

[MichalBryxi]

Just for the documentation if anyone is interrested. A stop-gap solution for my problem described above: https://gitlab.com/gitlab-org/gitlab-ce/issues/53073#note_197564726

[rugk]

Hm, is this really the best solution? This looks quite hackish… can't we get proper support in the main project here? 😃

BTW new issue link is: https://gitlab.com/gitlab-org/gitlab/issues/24843
(Please upvote, if you also want to see this happen in GitLab AutoDevOps)

[josegonzalez]

@rugk Yes, thats the best solution you'll get. Myself and the other maintainers aren't using this sort of functionality at the moment, and aren't really interested in building it for free.

If you're interested in seeing this for the project in general, feel free to email me for consulting (email is in my github profile). If you think the Gitlab folks should support this in their platform, it's worth talking to them about their platform.

Keeping this issue open in case anyone decides they'd like to work on this :)

[guzmanoj]

A working solution that (hopefully) can save you from hours of trial and error :)
https://github.com/guzmanoj/herokuish-install-chrome
The script will resolve the chrome dependencies so that CHROME_BIN will be defined in the host OS.

[josegonzalez]

This isn't a bug in herokuish, but rather a feature request that makes another platform work slightly better, so closing for now.