Incorrect handling of LTI launch URLs that include query params with percent-encoded chars in decoded value

[robertknight]

We ran into a bug with our LTI app which uses PyLTI where LTI launches would always fail if the launch URL contained a query parameter where the value, after parsing the query string into items, contained percent-encoded characters. The OAuth 1.0 signature base string was being generated incorrectly in this case.

I distilled the relevant part of our code into the following reproduction using PyLTI:

Steps to reproduce:

Run the following code in Python 3:

import time
from urllib.parse import urlencode

import pylti.common


consumers = {"TEST_CONSUMER": {"secret": "TEST_SECRET"}}
params = {
    "oauth_consumer_key": "TEST_CONSUMER",
    "oauth_signature_method": "HMAC-SHA1",
    "oauth_timestamp": int(time.time()),
    "oauth_nonce": "foo",
    "oauth_signature": "DUMMY_SIGNATURE",
}
query = urlencode({"url": "https://en.wikipedia.org/wiki/G%C3%B6reme_National_ark"})
url = f"https://example.com/lti_launch?{query}"
pylti.common.verify_request_common(
    consumers=consumers, url=url, method="POST", params=params, headers={}
)

This will fail with an error similar to:

oauth2.Error: Invalid signature. Expected signature base string: b'POST&https%3A%2F%2Fexample.com%2Flti_launch&oauth_consumer_key%3DTEST_CONSUMER%26oauth_nonce%3Dfoo%26oauth_si
gnature_method%3DHMAC-SHA1%26oauth_timestamp%3D1562586532%26url%3Dhttps%253A%252F%252Fen.wikipedia.org%252Fwiki%252FG%25C3%25B6reme_National_ark

If you extract the part of the message starting at "url%3D", and decode it, you'll get the query string param:

"url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FG%C3%B6reme_National_ark"

Decode this again to get the URL value and you'll get:

"url=https://en.wikipedia.org/wiki/Göreme_National_ark"

Which doesn't match the original "url" param from the query string above. The "ö" character should still be percent-encoded.

If on the other hand the url query param value is passed only in the params:

import time

import pylti.common


consumers = {"TEST_CONSUMER": {"secret": "TEST_SECRET"}}
params = {
    "oauth_consumer_key": "TEST_CONSUMER",
    "oauth_signature_method": "HMAC-SHA1",
    "oauth_timestamp": int(time.time()),
    "oauth_nonce": "foo",
    "oauth_signature": "DUMMY_SIGNATURE",
}
params.update({"url": "https://en.wikipedia.org/wiki/G%C3%B6reme_National_ark"})
url = f"https://example.com/lti_launch"
pylti.common.verify_request_common(
    consumers=consumers, url=url, method="POST", params=params, headers={}
)

Then the error produced is:

oauth2.Error: Invalid signature. Expected signature base string: b'POST&https%3A%2F%2Fexample.com%2Flti_launch&oauth_consumer_key%3DTEST_CONSUMER%26oauth_nonce%3Dfoo%26oauth_si
gnature_method%3DHMAC-SHA1%26oauth_timestamp%3D1562587085%26url%3Dhttps%253A%252F%252Fen.wikipedia.org%252Fwiki%252FG%2525C3%2525B6reme_National_ark'

And if you extract the part from "url%3D" and double-decode it as before, you get the correct output:

"url=https://en.wikipedia.org/wiki/G%C3%B6reme_National_ark"

Related bug:

If a query param appears in both the URL and the "params" dict, the generated signature base string includes the param only once. My reading of the spec, and what oauthlib does, is to include both versions.

Downstream workaround:

Our downstream workaround was to strip the query string from the url argument and pass query params in the params dict instead. See hypothesis/lms#764