How to clone repositories with an account? Account in URL results in errors
zilti opened this issue · 5 comments
We're trying to use Buildbot with Fossil for our project. Now I've run into a problem: the after-receive hook does not work at all, but neither does the polling. I decided to add the account information into the URL, like this:
c['change_source'].append(changes.FossilPoller(
repourl="https://buildbot:********@fossil.example.com/reponame",
pollInterval=60,pollAtLaunch=True
))But this results in the following error:
[-] <buildbot_fossil.changes.FossilPoller object at 0x806470a60>: while polling for changes
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/buildbot_fossil/changes.py", line 285, in _json_get
response = yield self._http.get(path, params=kwargs)
File "/usr/local/lib/python3.9/site-packages/buildbot/util/httpclientservice.py", line 228, in get
return self._doRequest('get', ep, **kwargs)
File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1947, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1857, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status, _copy_context())
--- <exception caught here> ---
File "/usr/local/lib/python3.9/site-packages/buildbot_fossil/changes.py", line 203, in poll
changes = yield self._fetch_json()
File "/usr/local/lib/python3.9/site-packages/buildbot_fossil/changes.py", line 219, in _fetch_json
payload = yield self._json_get("timeline/checkin", files=True)
File "/usr/local/lib/python3.9/site-packages/buildbot_fossil/changes.py", line 285, in _json_get
response = yield self._http.get(path, params=kwargs)
File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks
result = context.run(gen.send, result)
File "/usr/local/lib/python3.9/site-packages/buildbot/util/httpclientservice.py", line 223, in _doTReq
res = yield getattr(treq, method)(url, **kwargs)
File "/usr/local/lib/python3.9/site-packages/treq/api.py", line 23, in get
return _client(kwargs).get(url, headers=headers, _stacklevel=4, **kwargs)
File "/usr/local/lib/python3.9/site-packages/treq/client.py", line 161, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python3.9/site-packages/treq/client.py", line 267, in request
d = wrapped_agent.request(
File "/usr/local/lib/python3.9/site-packages/twisted/web/client.py", line 1480, in request
deferred = self._agent.request(method, uri, headers, bodyProducer)
File "/usr/local/lib/python3.9/site-packages/twisted/web/client.py", line 1573, in request
deferred = self._agent.request(method, uri, headers, bodyProducer)
File "/usr/local/lib/python3.9/site-packages/twisted/web/client.py", line 1352, in request
d = self._agent.request(method, uri, headers, bodyProducer)
File "/usr/local/lib/python3.9/site-packages/twisted/web/client.py", line 1148, in request
endpoint = self._getEndpoint(parsedURI)
File "/usr/local/lib/python3.9/site-packages/twisted/web/client.py", line 1132, in _getEndpoint
return self._endpointFactory.endpointForURI(uri)
File "/usr/local/lib/python3.9/site-packages/twisted/web/client.py", line 1003, in endpointForURI
connectionCreator = self._policyForHTTPS.creatorForNetloc(
File "/usr/local/lib/python3.9/site-packages/twisted/web/client.py", line 359, in creatorForNetloc
return optionsForClientTLS(hostname.decode("ascii"), trustRoot=self._trustRoot)
File "/usr/local/lib/python3.9/site-packages/twisted/internet/_sslverify.py", line 1258, in optionsForClientTLS
return ClientTLSOptions(hostname, certificateOptions.getContext())
File "/usr/local/lib/python3.9/site-packages/twisted/internet/_sslverify.py", line 1124, in __init__
self._hostnameBytes = _idnaBytes(hostname)
File "/usr/local/lib/python3.9/site-packages/twisted/internet/_idna.py", line 31, in _idnaBytes
return idna.encode(text)
File "/usr/local/lib/python3.9/site-packages/idna/core.py", line 360, in encode
s = alabel(label)
File "/usr/local/lib/python3.9/site-packages/idna/core.py", line 258, in alabel
ulabel(label_bytes)
File "/usr/local/lib/python3.9/site-packages/idna/core.py", line 297, in ulabel
check_label(label_bytes)
File "/usr/local/lib/python3.9/site-packages/idna/core.py", line 250, in check_label
raise InvalidCodepoint('Codepoint {} at position {} of {} not allowed'.format(_unot(cp_value), pos+1, repr(label)))
idna.core.InvalidCodepoint: Codepoint U+003A at position 9 of 'buildbot:@fossil' not allowed
Is this a bug? Or do I have to do something differently?
Interestingly though, the Fossil Step does work; I can trigger it, and it will clone the repository as it should.
The poller is using the JSON API over HTTP so it doesn't depend on having Fossil installed on the buildbot server. It uses the anonymous login method to do this. At the moment, there is no support in buildbot-fossil for logging in with user credentials to fetch the timeline.
The Fossil build step uses fossil clone, so it would be able to authenticate with SSH keys at least. I personally use it with an http:// URL without any credentials. This gives it "nobody" access to the repository which is enough to clone.
Does the anonymous user have "h" privileges in your repository?
Looking at this a bit more, I can see how a http://user:pass@example.com/ URL would work for the build step. It is simply passed verbatim to fossil clone and fossil pull. I wonder if it saves the password in the repo so that fossil update will also work? There is a --save-http-password option we're not passing.
I haven't had a need for any authentication support myself since all my repos provide sufficient access for the nobody/anonymous users. I worry if your passwords are going to show up in log files and the web UI when you put them in the URL like that. I know that buildbot has a facility for obfuscating password in log files, but I don't know anything about it.
the after-receive hook does not work at all
The after-receive hook simply pokes to poller so it checks for new checkins immediately instead of waiting for its poll interval. When you get the poller working, the hook should work too.
Maybe the HTTP library used cannot handle having login information as part of the URL? I try to avoid enabling unauthenticated access to our repositories, so I removed all privileges for "nobody". I added h for now though.
The HTTP library is treq, and you are probably right that it doesn't support passwords in URLs. Even if it did, I don't think it would work because Fossil doesn't use HTTP authentication — you have to POST username+password to the /login endpoint and get a session cookie back. The code is already there since buildbot-fossil needs to do the same thing to log in as "anonymous". See the _json_login method. It looks pretty simple to support username+password authentication.
If you are running your Fossil server on a network that can have spiders and bots, make sure to read Defense Against Robots. The "h" privilege is given to "anonymous" and not to "nobody" to prevent spiders from mindlessly downloading all versions of all sources in the repo.
I'm going to leave this issue open and mark it as a feature request. It'pretty easy to add authentication support, but testing this plugin is a bit of a nightmare. I wrote it a couple years ago, and every time I come back to it, something new is broken upstream. Buildbot exposes a testing API that is needed to write plugin tests. It's not a stable API, unfortunately, and things keep moving around.