gamps-web-sync doesn't work on MacOS
Closed this issue · 14 comments
This plugin does not work with MacOS, tested with self hosted Web Gramps Backend and identical Configuration on MacOS and Windows: It does work with Gramps on Windows, fails with "Fehler beim Zugriff auf den Server.".
There is a discussion on this Issue here: https://gramps.discourse.group/t/debugging-connection-issue-with-gramps-web-sync-on-mac/4604 but the discussion quickly derailed on how this bug could be debugged with no solution in sight.
Versions used to reproduce:
MacOS Sonoma 14.3.1 with Gramps 5.1.6, Plugin 1.1.1: Does not works
Windows 10 with Gramps 5.1.6, Plugin 1.1.1: Works
Gramps Web: Gramps 5.1.6, Gramps Web API 1.6.0, Gramps Web Frontend 24.2.0, locale: en, multi-tree: false, task queue: true
I've been using Gramps Web and Web Sync on macOS without issues for the past 3 months on my local Gramps app. Therefore, I disagree with the claim that the app doesn't work on macOS in general.
However, I am experiencing the issue described in the linked discussion when trying to sync to the same app accessed through a Cloudflare tunnel using a DNS record for the tunnel endpoint. This is despite the app having an SSL certificate (potentially related to issue #24?). Localhost syncing works perfectly fine.
I believe that – at least in my case – the issue is originating from line 64 with urlopen(req) as res: in webapihandler.py.
Versions:
- macOS: Sonoma 14.2.1
- Gramps: 5.2
- Plugin: 1.0.3 (and developer version)
Thanks @andreaiorio, that's valuable information.
Did you check whether the fix suggested in #24 works for you?
Unfortunately, that trick does not work in my case. It still returns a generic urllib.error.HTTPError: HTTP Error 403: Forbidden.
But 403 means it's a permissions error, it can't be due to connection issues. Perhaps you mistyped the password or don't have owner permissions?
The password is correct, but I am not sure about the owner's permissions. Trying with this webapihandler.py:
def fetch_token(self) -> None:
"""Fetch and store an access token."""
data = json.dumps({"username": self.username, "password": self.password})
req = Request(
f"{self.url}/token/",
data=data.encode(),
headers={"Content-Type": "application/json"},
)
try:
_LOG.info("Before sending request: %s", req.full_url)
with urlopen(req, context=ctx) as res:
_LOG.info("Request successful. Reading response.")
res_json = json.load(res)
except (UnicodeDecodeError, json.JSONDecodeError, HTTPError) as e:
_LOG.info("Error during request: %s", e)
if "/api" not in self.url:
self.url = f"{self.url}/api"
_LOG.info("Retrying with updated URL: %s", self.url)
return self.fetch_token()
raise
self._access_token = res_json["access_token"]
_LOG.info("Access token fetched successfully.")The log returns this for the remote url:
2024-02-25 14:21:04.351: INFO: webapihandler.py: line 72: Before sending request: https://remote_url.com/token/
2024-02-25 14:21:04.528: INFO: webapihandler.py: line 77: Error during request: HTTP Error 403: Forbidden
2024-02-25 14:21:04.528: INFO: webapihandler.py: line 80: Retrying with updated URL: https://remote_url.com/api
2024-02-25 14:21:04.528: INFO: webapihandler.py: line 72: Before sending request: https://remote_url.com/api/token/
2024-02-25 14:21:04.686: INFO: webapihandler.py: line 77: Error during request: HTTP Error 403: Forbidden
and this for localhost:
2024-02-25 14:20:14.327: INFO: webapihandler.py: line 72: Before sending request: http://localhost/token/
2024-02-25 14:20:14.365: INFO: webapihandler.py: line 74: Request successful. Reading response.
2024-02-25 14:20:14.365: INFO: webapihandler.py: line 77: Error during request: Expecting value: line 1 column 1 (char 0)
2024-02-25 14:20:14.365: INFO: webapihandler.py: line 80: Retrying with updated URL: http://localhost/api
2024-02-25 14:20:14.366: INFO: webapihandler.py: line 72: Before sending request: http://localhost/api/token/
2024-02-25 14:20:14.535: INFO: webapihandler.py: line 74: Request successful. Reading response.
2024-02-25 14:20:14.536: INFO: webapihandler.py: line 84: Access token fetched successfully.
EDIT: in my case it seems to be a server-side issue, as the request does not pass at all, so it is probably not related to Gramps web.
I encountered two issues:
- Cloudflare was blocking requests until I added a User-Agent header:
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)". It's unclear to me if this is a viable solution or if it should be handled on the server-side. - Requests are still failing to verify the SSL certificate, even though it's valid. Disabling SSL checks, as suggested in issue #24, would resolve this. This issue might be related to a known problem with Python on macOS, where it requires a post-installation step to validate SSL connections and could potentially affect all macOS users who are not working on localhost.
Well, after the comments from @andreaiorio I tried a couple of things:
- Installed the newest Python Version and ran the SSL Certificate fix. (Does Gramps even use a System installed Python or does it bring its own?)
- My Main Mac is on Gramps 5.2 since it was released, so I did fresh install on a second Mac (also Sonoma 14.3.1) with Gramps 5.1.6 and a fresh Python install with the SSL Certificates Fix. Did not change the Problem.
- Again, as the Sync is working from Windows on a VM on the first Mac with the same credentials I think I can rule out any network problem or password typo
Is the anything I could do to help debug this? I am not a Python programmer, but I some one can provide a version of the plugin with debug / logging statement I'm happy to try it out
@jgtimm I'm afraid that Gramps uses its own Python interpreter and dependencies, and not the system one, so you don't see any change.
Providing more insights:
import ssl
print(ssl.get_default_verify_paths())It prints:
DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Users/john/Development/gramps-tarball-11-arm64/inst/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Users/john/Development/gramps-tarball-11-arm64/inst/etc/ssl/certs')I don't know where SSL is retrieving the openssl_cafile and openssl_capath locations (John, reveal yourself!), but they are not mine, and hence SSL fails verifying certificates.
Adding this at the beginning of webapihandler.py:
def create_macos_ssl_context():
import subprocess
import ssl
import tempfile
ctx = ssl.create_default_context()
macos_ca_certs = subprocess.run(
[
"security",
"find-certificate",
"-a",
"-p",
"/System/Library/Keychains/SystemRootCertificates.keychain",
],
stdout=subprocess.PIPE,
).stdout
with tempfile.NamedTemporaryFile("w+b") as tmp_file:
tmp_file.write(macos_ca_certs)
ctx.load_verify_locations(tmp_file.name)
return ctx
ctx = create_macos_ssl_context()Then using the ctx context in urlopen solves the issue for me. I'm not sure if this solution is the best, but it has been recommended here. I feel this SSL problem could be quite widespread, like also here.
Thanks @andreaiorio for this detailed reply! Now the connection seems to work, but other things fail. I'm on Gramps 5.2.0 on my main Mac. Will reinstall 5.1.6 and test again...
Ok: Reinstalled 5.1.6., still didn't work. Now I do see two lines in the logs on the server side:
"POST /token/ HTTP/1.1" 200 ...
"POST /api/token/ HTTP/1.1" 200 ...
but that's it. I give up for today.
There is no 5.2.0 compatible version of Gramps Web yet!! Please stick with 5.1.6 for the moment.
I also tested with Gramps 5.1.6 and plugin version 1.0.3, and I confirm to have no issues on Mac after implementing my fix.
@jgtimm a 200 status is indeed okay. What other issues are you experiencing? I'm happy to try to help!
@DavidMStraub should I open a PR with my edits?
Hi, I would be very happy about a PR, the only thing I didn't quite understand is, wouldn't this affect users on other architectures? So we would only apply the fix based on platform.system()?
Hi, today I was able to test the new versions of Gramps (5.2.0), Grampsweb (API 2.0.0, Frontend 24.3.0) and gramps-web-sync (1.1.1) on my Mac Intel with Sonoma 14.3.1. The Synchronization now works flawlessly!
Awesome! I guess we can close this then 🎉