uBlockOrigin/uBlock-issues

json-prune.js bug ( Response.json() ) in 1.56.0 & 1.56.1b

Procyon-b opened this issue · 7 comments

Procyon-b commented

Prerequisites

  • I verified that this is not a filter list issue. Report any issues with filter lists or broken website functionality in the uAssets issue tracker.
  • This is NOT a YouTube, Facebook or Twitch report. These sites MUST be reported by clicking their respective links.
  • I performed a cursory search of the issue tracker to avoid opening a duplicate issue.
  • The issue is not present after disabling uBO in the browser.
  • I checked the documentation to understand that the issue I am reporting is not normal behavior.

I tried to reproduce the issue when...

  • uBO is the only extension.
  • using a new, unmodified browser profile.

Description

Something has changed between v1.49.2 (no bug) and v1.56.0 (bug).

gmx.net webmail inserts ads directly in the listing of mails when the list is fetched from their servers. Some time ago I wrote a set of filters to discard them on the fly.

3c-bap.gmx.net,3c.gmx.net##+js(json-prune, mailListElements.2, mailListElements.2.rawData.promotext)
3c-bap.gmx.net,3c.gmx.net##+js(json-prune, mailListElements.7, mailListElements.7.rawData.promotext)
3c-bap.gmx.net,3c.gmx.net##+js(json-prune, mailListElements.8, mailListElements.8.rawData.promotext)
3c-bap.gmx.net,3c.gmx.net##+js(json-prune, mailListElements.15, mailListElements.15.rawData.promotext)
3c-bap.gmx.net,3c.gmx.net##+js(json-prune, mailListElements.23, mailListElements.23.rawData.promotext)

Recently I've replaced my main computer with a spare one (hardware problems). Computer # 1 was still using an older uBO version (don't ask me why, too long to explain), so I haven't noticed the problem earlier (that and the 3+ months in hospital). While I was already using computer # 2 before, it wasn't updated with all the fine-tuning for gmx. So I've updated it yesterday.

To my surprise the ads where still appearing in the listing. 3c-bap.gmx.net,3c.gmx.net,gmx.net##+js(json-prune) was added to check what was happening. Nothing appears in relation of the specific lisiting fetch (the structure and names are recognizable). I first thought that gmx found a way to avoid pruning (eg. their own parse engine) and decided to catch the fetch calls with a userscript. It worked and with the next step I also noticed that .json() was indeed called.

The following userscript is the result of that. It is also a fix. It intercepts every .json() and calls JSON.stringify and JSON.parse to expose the payload to uBO.

// ==UserScript==
// @name         fetch
// @namespace    https://github.com/Procyon-b
// @version      0.1
// @description  
// @author       Achernar
// @match        https://3c-bap.gmx.net/*
// @match        https://3c.gmx.net/*
// @grant        none
// @run-at  document-start
// ==/UserScript==

(function() {
"use strict";

var of=window.fetch;

window.fetch = async (...a) => {
  console.info("fetch:", a);
  const r = await of(...a);
  /*r.clone()
   .json()
   .then(d => {
      console.log("fetch response:", d);
      })
   .catch(e => console.error(e));/**/

  const json = () =>
    r.clone()
      .json()
      .then((d) => {
          console.info('json-ed',{d});
          d=JSON.parse(JSON.stringify(d));
          console.info('json-ed 2',{d});
          return d;
          } );

  r.json = json;

  return r;
};
})();

Test where done with:

  • chrome + uBO v1.56.1b13
  • chrome + uBO v1.56.0 (2 profiles)
  • FF + uBO v1.56.1b15
  • chrome + uBO v1.49.2 (then updated to v1.56.0)

v1.49.2 works as expected. It was the version available in a test profile that wasn't opened for a long time.
I can do more tests with different versions if needed. Thanks to one of my own extensions I have a copy of all versions of the extension packages (and they are available here too of course), except for the time I was in the hospital.

A specific URL where the issue occurs.

gmx.net webmail:
  bap.navigator.gmx.net (frame: 3c-bap.gmx.net)
  navigator.gmx.net (frame: 3c.gmx.net)

Steps to Reproduce

  • If you can get a @gmx.net email account, log into their webmail
  • Open a mail folder with many email, and see the 3-5 ad messages added to the lisiting
  • add the 5 filters above.
  • reload the page (or lisiting frame). The ads shouldn't appear but they do in v1.56 (they don't in v1.49.2)
  • add my userscript and reload the page
  • now the ads don't appear anymore

Expected behavior

json-prune should work on Response.json()

Actual behavior

In version 1.56 (at least) json-prune doesn't work anymore on Response.json() in the test case.

Configuration

This is from the profile used to report. Other profiles used for testing have a more basic uBO config.

uBlock Origin: 1.56.1b13
filterset (summary):
 network: 142543
 cosmetic: 41301
 scriptlet: 25328
 html: 0
listset (total-discarded, last-updated):
 added:
  FRA-0: 23036-2970, 1d.21h.13m
  NLD-0: 2772-29, 6d.21h.33m
  block-lan: 57-0, 24d.36m
  fanboy-social: 15828-11220, 6d.21h.33m
  ublock-annoyances: 6909-34, 19m Δ
 default:
  user-filters: 486-8, never
  easylist: 82264-16404, 19m Δ
  easyprivacy: 50352-254, 19m Δ
  plowe-0: 3779-822, 6d.21h.33m
  ublock-badware: 7858-30, 19m Δ
  ublock-filters: 37465-649, 19m Δ
  ublock-privacy: 754-4, 19m Δ
  ublock-quick-fixes: 175-1, 19m Δ
  ublock-unbreak: 2244-27, 19m Δ
  urlhaus-1: 7825-34, 16h.15m
filterset (user): [array of 483 redacted]
switchRuleset:
 added: [array of 3 redacted]
userSettings:
 advancedUserEnabled: true
 ignoreGenericCosmeticFilters: true
 showIconBadge: false
 suspendUntilListsAreLoaded: true
 tooltipsDisabled: true
 webrtcIPAddressHidden: true
hiddenSettings:
 filterAuthorMode: true
 popupFontSize: 13px
 popupPanelHeightMode: 1
supportStats:
 allReadyAfter: 742 ms (selfie)
 maxAssetCacheWait: 204 ms
 cacheBackend: indexedDB
gorhill commented

json-prune should work on Response.json()

Ability to prune Response.json was split from json-prune in 1.52.0, see gorhill/uBlock@749cec0f09.

stephenhawk8054 commented

json-prune now is only used to deal with JSON.parse. For Response.json, use json-prune-fetch-response or json-prune-xhr-response, depending on the method.

Procyon-b commented

OK, thanks. I missed that. I was expecting to have missed something. :)
I relied on the wiki and didn't see any mention of that.
Sorry

Procyon-b commented

@stephenhawk8054 , if you have used the scriptlet can you help me a little?

For the past 45 minutes I'm trying to make at least one filter work.
Say 3c-bap.gmx.net,3c.gmx.net##+js(json-prune-fetch-response, mailListElements.7.rawData, , log, match) with or without a matching element. All is does is wipe the entire object. When I look in the logger [3c-bap.gmx.net][json-prune-fetch-response ⁝ mailListElements.7 ⁝ ] Pruned displays an empty block when "hovered". What am I missing here? Can't it handle array elements anymore?

gorhill commented

[3c-bap.gmx.net][json-prune-fetch-response ⁝ mailListElements.7 ⁝ ]

This means that your filter is ##+js(json-prune-fetch-response, mailListElements.7, ...), not ##+js(json-prune-fetch-response, mailListElements.7.rawData, ...).

Procyon-b commented

Yes, I know, It's the last iteration of the original filter where I try to only remove a sub-part of the object. I've pasted the wrong filter of the 2, and then got back too far in the logger to copy the info.
Anyway, in all my tests the properties of the object seems to be all deleted.

Procyon-b commented

I have found the reason why the filter breaks the website.. json-prune-fetch-response has a different behaviour than json-prune. At least in the default behavior.
When deleting an element of an array:

  • json-prune deletes the element. You can see a gap in the index number
  • json-prune-fetch-response replaces the element with null

Here is a test case: https://xiii200.tripod.com/json/tmp.html
The filters to use for the test are displayed on the page and are:

xiii200.tripod.com##+js(json-prune-fetch-response, mailListElements.7)
xiii200.tripod.com##+js(json-prune, mailListElements.7)

Open the javascript console and execute the 2 functions as described. An xhr and a fetch will be done to recover the json file. It is then parsed with eval() and displayed (unfiltered object), parsed with JSON.parse() and with Response.json(), also displayed.
You can then see the difference with the 7th element.