lightswitch05/hosts

[False Positive] Xbox Live Achievement Domains blocked

Bryantdl7 opened this issue · 14 comments

Bryantdl7 commented

Thank you for using my hosts lists. I appreciate feedback on this project, but I do expect you to put effort into your request. At the end of the day, this is my list and I maintain it for my usage. I'll address reasonable requests, but I cannot make everyone happy. This means you might have to whitelist a few things on your own. A list that breaks nothing is a list that blocks nothing.

  • False positive?
    Yes - a few of the domains on your list are stopping Xbox One's from uploading achievement properly.

The second I unblocked these domains, the xboxes began notifying me of achievements from months ago (I have been using your list for a few months now)

Here's the domains I have it narrowed down to, I didn't check the list to see which ones arent present on your https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt list.

Here are the domains, this has been crazy to track down!:

v20.events.data.microsoft.com - on your list
watson.telemetry.microsoft.com - on your list
web.vortex.data.microsoft.com - on your list
v10.events.data.microsoft.com - on your list

=========================================
Domains not on your list, but may help some sorry soul having this problem in the future DuckDuckGo'ing the solution. Make sure all these domains are allowed if you use multiple lists:

attestation.xboxlive.comcert.mgt.xboxlive.com
ctldl.windowsupdate.comdef-vef.xboxlive.com
device.auth.xboxlive.comeds.xboxlive.com
help.ui.xboxlive.comlicensing.xboxlive.commicrosoft.com
notify.xboxlive.comsettings-win.data.microsoft.com
title.auth.xboxlive.comtitle.mgt.xboxlive.com
v10.vortex-win.data.microsoft.com
www.msftncsi.com
xbox.ipv6.microsoft.com
xboxexperiencesprod.experimentation.xboxlive.com
xflight.xboxlive.comxkms.xbolive.com
xsts.auth.xboxlive.com
v20.events.data.microsoft.com
watson.telemetry.microsoft.com
web.vortex.data.microsoft.com
v10.events.data.microsoft.com

lightswitch05 commented

I didn't check the list to see which ones arent present

So... I’m expected to do that for an issue that your having, which may or may not have anything to do with my list because you won’t look to see if they are in there or not?

As my issue template says: I do expect you to put some effort into your request.

Bryantdl7 commented
Bryantdl7 commented

Here are the domains present on your list, original post updated to note this as well:

v20.events.data.microsoft.com
watson.telemetry.microsoft.com
web.vortex.data.microsoft.com
v10.events.data.microsoft.com

Most of these domains are mentioned in the following forum post: https://answers.microsoft.com/en-us/xbox/forum/all/local-cache-is-full-stats-and-achievements-are-no/222d572a-56d3-45a1-97bc-b907c007e2dd

Additionally, Watson telemetry, in the world of windows is used to upload crash logs for when things like programs (games in this case) crash. That is helpful for developers to fix problems with their games, I am unsure how this relates to the overall problem of achievements other than that, but it was part of the long journey of slowly unblocking sites denied by my Xboxes.

Xbox has an annoying feature where you can not change a DHCPD-set IPv6 DNS server to a different value. I am certain others using your list will encounter this again in the future if they use pihole in DHCP mode, with IPv6 enabled, as the Xbox One prefers IPv6 when available.

Apologies for being vague in my original post.

  • 👍1
lightswitch05 commented

Thanks for the updated report @Bryantdl7 - I'm very busy at the moment, but I will look at this as soon as I have some time.

  • 🚀1
Bryantdl7 commented
XhmikosR commented

I'd be very careful with removing the aforementioned domains before extensive research.

Just my 2 cents.

Bryantdl7 commented
lightswitch05 commented

I added both *.telemetry.microsoft.com and *.events.data.microsoft.com on 2019-11-06 in 9dd786a8

Additionally, Watson telemetry, in the world of windows is used to upload crash logs for when things like programs (games in this case) crash. That is helpful for developers to fix problems with their games

Unfortunately diagnostic data like crash logs and tracking/analytics data is generally vacuumed up by the same tools and then separated out server side. Also, crash logs often contain sensitive information like PII. So my general rule of thumb is to block it all and let people whitelist whatever they decide is trustworthy. I prefer opt-in.

Just to see what some other list maintainers are doing, I searched the lists I have on my pihole for the given hosts, and came across these:

* v20.events.data.microsoft.com
    * https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
* watson.telemetry.microsoft.com
    * https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
    * https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
    * https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
    * https://raw.githubusercontent.com/michaeltrimm/hosts-blocking/master/_hosts.txt
    * https://raw.githubusercontent.com/vokins/yhosts/master/hosts
    * https://hosts-file.net/ad_servers.txt
    * https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
    * https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
    * https://raw.githubusercontent.com/jerryn70/GoodbyeAds/master/Hosts/GoodbyeAds.txt
* web.vortex.data.microsoft.com
    * https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
    * https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
    * https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
    * https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
* v10.events.data.microsoft.com
    * https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
    * https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt

The results are really all over the place. Given the auto-expand nature of my list, and that I have telemetry.microsoft.com blocked - its not surprising that I have some of the better sub-domain coverage compared to some of the other lists. But it also could be some other list maintainers have decided to whitelist some subdomains. Its interesting that Steven Black's list has v10c.events.data.microsoft.com in it, but not v10.events.data.microsoft.com.

Alright... so the reason I'm showing that a bunch of other lists block these too: If I chose to move them to the aggressive list, its very likely most people will still have to whitelist them. Particularly since Steven's list is a default PiHole list.

I hate that I'm breaking Xbox for you and that it took a long time for you to figure it out. I really appreciate you sharing your results back with me. I'm always saying that I appreciate feedback and it is true. I make a lot of mistakes. In this case however, we know that Microsoft is using these hosts to do tracking, data mining, segmenting, and then targeting people with that data.

At this time I'm going to leave the list blocked. But lets leave the ticket open to make it easier for people to find. I may even add a section to the README that links here.

Bryantdl7 commented
Deleted user commented

The domains are already listed on https://discourse.pi-hole.net/t/commonly-whitelisted-domains/212. How did this take you three months to figure out?

  • 👎1
beerisgood commented

Just to see what some other list maintainers are doing, I searched the lists I have on my pihole for the given hosts, and came across these:

Problem is that many people only copy domains from other, without think or read about it.

It doesn't make sense to use a OS which you don't trust. Also Windows provide disable sending personal info. At minimum level, only "needed" data are send.

Of course privacy "experts" think they're better then official documentation from Microsoft and telemetry is always bad, which is totally nonsense.
Sadly this would never change.

I already reporting a lot of these as false positive at different lists but i guess you know what most dev's answer. Only few make the right decision :(

lightswitch05 commented

Hey @beerisgood, the decision was made and I provided my line of thought. Your welcome to voice your opinion that the blocks are too aggressive, but leave it at that. There is no cause to try and belittle people, no one here is claiming to be an expert. If you do not like my list that I maintain for my own personal use then please do not use it.

@llacb47 some people have a harder time tracking down troublesome blocks then you do 😀

  • 👍2
  • 😄2
  • 🎉2
lightswitch05 commented

I added a section to the README about this: https://github.com/lightswitch05/hosts#common-issues

soredake commented

This also breaks authorization in "your phone" app from microsoft.

  • 👍1