searx/searx-docker

adding searx to firefox preview/nightly on Android is impossible due to block by filtron

unixfox opened this issue · 5 comments

unixfox commented

On Firefox preview/nightly for Android it's impossible to add a Searx instance behind searx-docker due to filtron which is blocking the request when adding the instance as a search engine.

Screenshot_20200725-143048_Firefox_Nightly_1_1

It sends these headers when trying to add Searx as a search engine:

User-Agent: MozacFetch/50.0.20200712190143
Accept: */*
Accept-Language: */*
Accept-Encoding: gzip
Connection: keep-alive

Its request is blocked by these two rules:

How to resolve the issue?

Should we whitelist only the specific user agent MozacFetch in order to solve the issue?
Or should we just whitelist the keyword 1 (?q=1) which is hardcoded into the fenix source code: https://github.com/mozilla-mobile/fenix/blob/master/app/src/main/java/org/mozilla/fenix/settings/search/SearchStringValidator.kt#L18?

Related issue: mozilla-mobile/fenix#11332

dalf commented

Where to download this version?

I have this one:
Screenshot_20200725-150644_Firefox Nightly

But the user agent is Mozilla/5.0 (Android 8.0.0; Mobile; rv:80.0) Gecko/80.0 Firefox/80.0.

[EDIT] Sorry for the big screenshot.

unixfox commented

Just use whatever version of new Firefox on Android that you want. The bug is present on all of them (except Firefox stable on Android) including the one that you sent (Firefox nightly).

dalf commented

Should we whitelist only the specific user agent MozacFetch in order to solve the issue?

An unique filter about the user agent would give an easy access to the bots.

Or should we just whitelist the keyword 1 (?q=1) which is hardcoded into the fenix source code

Why not combine the keyword and the user agent ?

It works with this rule after the bot forbidden rule:

            {
                "name": "Firefox android",
                "filters": ["Param:q=1", "Header:User-Agent=MozacFetch/", "Header:Accept=\\*/\\*", "Header:Accept-Language=\\*/\\*"],
                "stop": true,
                "actions": [{"name": "log"}]
            },
  • I've initially added the filter"Header:Te=trailers", and then removed it:
    • 1 still makes a query to bing, google, etc. but not very useful for the bots.
    • we know this browser sends this header with HTTP/2 but a middlebox may change that...
  • Firefox Android Beta: ok
  • Firefox Android Nightly: I can add searx, but it crashes when I select it later on.
unixfox commented

Why not combine the keyword and the user agent ?

That's a very good idea although I wouldn't add too much filters like Accept and Accept-Language because those could change in the near future. Here is someone seeking for changing the Accept-Language header: mozilla-mobile/fenix#10605 (comment)

I think it's already a very good filter with the combination of q=1 and the user agent mozfetch.

  • I've initially added the filter"Header:Te=trailers", and then removed it:

Just don't do that, an HTTP request is not always in HTTP2 even if the browser support it, there could be some cases where the client fallback to HTTP1.

  • Firefox Android Nightly: I can add searx, but it crashes when I select it later on.

Does it make an additional request when selecting the searx search engine?

dalf commented

Does it make an additional request when selecting the searx search engine?

no additional request, so it is related to Fenix not Searx. It seems related to mozilla-mobile/fenix#12929 (same stack trace)