Aggregation Service with Private Aggregation API

[michal-kalisz]

When an aggregatable report is created by sendHistogramReport() (i.e. called inside reportWin function) it contains shared info without attribution_destination nor source_registration_time. This seems to be logical as these keys are strictly related with attribution logic. Example:

"shared_info": "{\"api\":\"fledge\",\"debug_mode\":\"enabled\",\"report_id\":\"9ae1a0d0-8cf5-4951-b752-e932bf0f7705\",\"reporting_origin\":\"https://fledge-eu.creativecdn.com\",\"scheduled_report_time\":\"1668771714\",\"version\":\"0.1\"}"

More readable form:

{
 "api": "fledge",
 "debug_mode": "enabled",
 "report_id": "9ae1a0d0-8cf5-4951-b752-e932bf0f7705",
 "reporting_origin": "https://fledge-eu.creativecdn.com",
 "scheduled_report_time": "1668771714",
 "version": "0.1"
}

(note: version 0.1, values for: privacy_budget_key, attribution_destination, source_registration_time are missing)

In the same time Aggregation service expect to have both attribution_destination and source_registration_time for shared info.version==0.1 (since aggregation service version 0.4):
see SharedInfo.getPrivacyBudgetKey()

Tested on chrome:

• 108.0.5359.48
• 110.0.5433.0
• 107.0.5304.110
  And local aggregation service 0.4

The following exception were printed:
CustomMetric{nameSpace=scp/worker, name=WorkerJobError, value=1.0, unit=Count, labels={Type=JobHandlingError}}
2022-11-22 09:10:54:120 +0100 [WorkerPullWorkService] ERROR com.google.aggregate.adtech.worker.WorkerPullWorkService - Exception occurred in worker
com.google.aggregate.adtech.worker.JobProcessor$AggregationJobProcessException: java.util.concurrent.ExecutionException: java.util.NoSuchElementException: No value present
at com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.process(ConcurrentAggregationProcessor.java:400)
at com.google.aggregate.adtech.worker.WorkerPullWorkService.run(WorkerPullWorkService.java:145)
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:67)
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:103)
at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.util.concurrent.ExecutionException: java.util.NoSuchElementException: No value present
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:113)
at com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.process(ConcurrentAggregationProcessor.java:295)
... 4 more
Caused by: java.util.NoSuchElementException: No value present
at java.base/java.util.Optional.get(Optional.java:143)
at com.google.aggregate.adtech.worker.model.SharedInfo.getPrivacyBudgetKey(SharedInfo.java:161)
at com.google.aggregate.adtech.worker.aggregation.engine.AggregationEngine.accept(AggregationEngine.java:88)
at com.google.aggregate.adtech.worker.aggregation.engine.AggregationEngine.accept(AggregationEngine.java:49)

[csharrison]

Thanks for posting @michal-kalisz . As far as I know this is a known problem with the Aggregation Service's support for Private Aggregation API and the team is working on it. Let me cc @ghanekaromkar and ruclohani@ who might know more about a planned fix for this.

[ruclohani]

Thanks for posting @michal-kalisz. Aggregation support for Private Aggregate API is currently under development. The Private Aggregation API is available for testing in Chrome M107+ Canary and Dev with aggregation service support coming soon for testing.

[michal-kalisz]

Thanks for information @ruclohani
I was wondering if you know when we could expect Private Aggregation API in stable version in OT?
Currently, traffic volume in canary/dev is too low to perform wider tests.

[alexmturner]

Hi @michal-kalisz, we plan to roll the OT out to Beta as soon as we see some non-trivial usage on Canary/Dev. This is just to ensure that there are no major stability issues (i.e. crashes).

Chrome has a much higher stability bar for the stable channel, so we'd need to see some substantial testing before we're able to roll to Stable. While we don't have a specific timeline, we'd also like to do this as soon as we can.

[michal-kalisz]

Thanks Alex, below short summary from our tests:

We wanted to test the following scenarios:

• Calculate avg of bid value.
  and compare it with information retrieved by standard event level report sendReportTo
• Calculate number of auction that we participate in (how often generateBid is called)
  and compare it with forDebuggingOnly report

We added 2 histograms in both generateBid and in reportWin functions.

The results.
From 2022-12-08 till 2022-12-18

chrome_ver	uniq_reports	uniq_ips
Chrome/107.0.0.0	4348	61
Chrome/108.0.0.0	10490	127
Chrome/109.0.0.0	31564	324
Chrome/110.0.0.0	337536	2238
Chrome/111.0.0.0	15133	56

Let’s focus on Chrome/110 and single day:
We observed that the accuracy for bid is 99.9% , but for report win is 86.9%.
It seems that not all reports have been delivered. First - here we only checked debug reports (as scheduled/postponed upload introduces too much noise). Second - we checked the budget and it seems that it wasn’t exceeded for any user.
In the same time maybe we missed something and chrome behaviour is correct.

From a testing perspective, additional debug information would be very useful. Maybe something similar to ARA verbose debugging would be possible? Have you consider it?

Please let me know if have any questions regarding out tests. Of course I am also curious if you observe any crushes or others issues.

[michal-kalisz]

Hi @alexmturner ,
I don't know if you had time to read my previous comment. Maybe it is not the best place for it ?(if so please let me know :)

One more thing we observed: ratio debug reports vs "normal" reports can significantly vary:

In the chart above:

• debug is a number of debug reports from generateBid
• normal is a number of normal (scheduled/postponed) reports from generateBid
• bids is a number debug calls (forDebuggingOnly) from generateBid

[alexmturner]

Hi! Sorry for the delayed response -- I was out of office for the holidays.

Thanks for the feedback! Agreed that it seems odd the number of debug reports doesn't match the forDebuggingOnly reports. Let me follow up with the team to see if anyone has any thoughts. In the mean time, do let us know if you're able to reproduce such a case locally (e.g. by enabling the experiment flag, see https://developer.chrome.com/docs/privacy-sandbox/private-aggregation/#test-this-api).

[alexmturner]

Ah sorry I misinterpreted the chart. We would expect debug and bids to be near identical, which seems to be mostly true (but not sure what the discrepancy is on days like 26 December).

We do expect some discrepancy for 'normal' reports relative to debug reports. For example, if a user doesn't open Chrome for a while, their reports would have additional delay. Reports could also be lost in the case they delete their browsing history. How long are you waiting for normal reports in your experiment?

[michal-kalisz]

Hi @alexmturner , thanks for your reply.
I totally agree that it's possible, especially since it's canary/dev version (I assume, that more users do some experiments).
In this experiment - we don't reject any reports (even if we receive after few days from bid)

On previous chart the X axis was the day when raport was received.
Let's consider other perspective:
X axis is the day of bid (debugKey=bidTime, we match normal/debug report by reportID)
On the chart below we present all events received between 2022-12-23 -> 2023-01-02 for chrome/11[01]

• some normal reports were received after 23-12, but for bidding before 23-12 (which is fine and expected)
• What seems strange: between 2022-12-23 and 2023-01-02 there were cases (~7K) where we received ONLY normal report: without DEBUG.

I would treat it as an input to discussion if you notice some problems on the browser site.
If you haven't observed any - maybe you consider extend tests for beta chrome version?

[alexmturner]

Hi @michal-kalisz , thanks for the further detail!

Re normal reports without debug: I think it also expected that there is a small amount of debug reports that fail to be sent due to network failures or the user closing their browser at an inopportune time. Note that debug reports are not stored for later retries if they fail to send (unlike normal reports).

It does seem like a significant fraction of normal reports aren't ever being sent, though. As discussed, we do expect some reports to be lost and it's also possible that this number is inflated on Canary/Dev populations. So it might make sense to wait for broader testing before estimating this fraction.

Still if you're happy to share, I'm just curious for comparing results whether your testing population a random fraction of all users or if it restricted in some way? (Other than by Chrome version and Canary/Dev.) For example, do you test on only desktop or only mobile users?

Thanks again for sharing these results!

[michal-kalisz]

Hi @alexmturner,
Yes, we test on all chrome that support Fledge. I split chart by four device types (based on UA) and added (in bracket) number of uniq IP + UserAgent Pair

For Mac and linux - we can see, that small amount of user can have significant impact on this summary,

I also observed , that on 26.12 50% debug reports came from 6 IPs (if we remove them from the report still ratio debug/normal is low)

Please let me know if you need some more information from our site.

[alexmturner]

Hi @michal-kalisz ,

Just wanted to let you know that we've increased the Origin Trial population to include Beta users now (https://groups.google.com/a/chromium.org/g/blink-dev/c/Vi-Rj37aZLs/m/cCuQksVPAAAJ). I hope this helps with testing volume

Thanks for the break down! It looks like maybe the different platforms are actually not too different in the fraction of debug only. (Except for low volume data points.)

[dmdabbs]

Thank you for sharing PAA client OT progress to Beta!

@ruclohani ...with aggregation service support (for PAA) coming soon for testing.

With client advancing is service support soon to follow?

[alexmturner]

Sorry for the delay in responding -- we've just released a new version of the aggregation service local testing tool that support Private Aggregation API reports. More details available here: https://groups.google.com/a/chromium.org/g/shared-storage-api-announcements/c/Qabo00MwTXM.

Please let us know if you have any questions!

[dmdabbs]

Thank you @alexmturner. Will check it out.

[maybellineboon]

Hi @dmdabbs ,

Closing this out. Do let us know if you still have further questions on this.

Thanks!

[gota0]

Hi @maybellineboon, @alexmturner just wanted to check, is aggregation for reports from Private Aggregation such as protected-audience reports still supported only by the local testing tool and not in production?

[maybellineboon]

Hi @gota0 ,

Reports from Private Aggregation such as protected-audience should be supported in Production now.