HTTPError: Response code 404 (Not Found)

[ericcornelissen]

I was trying this tool out and found that I always get a 404 error when using it. Both when installed using npm install -g or when using it with npx, see logs below. Oddly enough the tool seems to work fine when I clone the repository and run src/cli.js...

# with npm install -g
$ wallace projectwallace.com
HTTPError: Response code 404 (Not Found)

# with NPX
$ npx wallace-cli projectwallace.com
[...]
HTTPError: Response code 404 (Not Found)

[bartveneman]

Oddly enough the tool seems to work fine when I clone the repository and run src/cli.js...

There's a few updates in the normalize-url package, so perhaps the fix is to release a new version of wallace-cli.

[ericcornelissen]

I don't think that's causing the problem. normalize-url 4.5.1 is the latest version that will be installed for this package, given this project's package.json. This is the same version that's in this project's package-lock.json. So the normalize-url version should be consistent between all three options I listed, which seems to be true based on local testing.

[bartveneman]

Thank you for investigating and clarifying. I'm a little short on time, so it may take a while before I can solve the actual problem. In the meantime, a PR to fix this would be much appreciated.

[ericcornelissen]

The API that v2.4.0 of this packages uses to extract the CSS from a page is used incorrect (possibly it was updated). The master branch no longer uses this API and uses extract-css-core instead. I didn't yet check the implications of just release the current master as a new version. But...

Solution

If you just want to hotfix v2.4.0, you can update src/get-css.js to the following so that it uses the API correctly.

const got = require('got')
const normalizeUrl = require('normalize-url')
const isUrl = require('./is-url')

module.exports = async (input, options = {}) => {
	if (!isUrl(input)) {
		return input
	}

	const url = normalizeUrl(input, {
		stripWWW: false,
		stripProtocol: false
	})

	const {body} = await got(`https://extract-css.now.sh/api/extract-css?url=${url}`, {
		responseType: 'text',
		resolveBodyOnly: true,
		headers: {
			'User-Agent': options.userAgent
		}
	})

	return body
}

Which has the following diff with this file on v2.4.0:

  const got = require('got')
  const normalizeUrl = require('normalize-url')
  const isUrl = require('./is-url')

  module.exports = async (input, options = {}) => {
    if (!isUrl(input)) {
      return input
    }

    const url = normalizeUrl(input, {
      stripWWW: false,
-     stripProtocol: true
+     stripProtocol: false
    })

-   const {body} = await got(`https://extract-css.now.sh/${url}`, {
+   const {body} = await got(`https://extract-css.now.sh/api/extract-css?url=${url}`, {
      responseType: 'text',
      resolveBodyOnly: true,
      headers: {
        'User-Agent': options.userAgent
      }
    })

    return body
  }

Explanation

v2.4.0 makes requests as https://extract-css.now.sh/${url} where url is stripped of its protocol string, so for the command

$ wallace https://projectwallace.com

this will be https://extract-css.now.sh/projectwallace.com, which, as you can see by clicking the link, results in a 404. So, then I went to https://extract-css.vercel.app/ and clicked one of the examples, which rerouted me to https://extract-css.vercel.app/api/extract-css?url=https://www.google.com. From this I realized that API request should be updated:

-   const {body} = await got(`https://extract-css.now.sh/${url}`, {
+   const {body} = await got(`https://extract-css.now.sh/api/extract-css?url=${url}`, {

Then, I tested the API without providing the protocol, e.g. https://extract-css.vercel.app/api/extract-css?url=www.google.com, which, as you can see by clicking the link, results in a 400. So, then I knew the protocol should be included so I changed:

-     stripProtocol: true
+     stripProtocol: false

[bartveneman]

I have released current master as 2.5.0, because I wanted to get rid of that endpoint. I've double-checked that npx wallace-cli projectwallace.com gives the correct output, since this is the only part of the app that isn't coverd by tests (see #55).

Thanks again for your very thorough analysis!

[ericcornelissen]

Works for me 👍 Thanks for publishing an update this quickly!