derat/yambs

Support multi-country Tidal lookups

Closed this issue · 9 comments

derat commented

From @cherryblossom000 in #17 (comment):

I think a-tisket finds the countries Apple Music releases are available in by trying every country and seeing if that works (but obviously this takes time so this can be disabled). Maybe something similar could be done for Tidal. There doesn’t seem to be an official list of what countries Tidal is available in but I did find this:

Andorra
Argentina
Australia
Austria
Belgium
Brazil
Canada
Chile
Colombia
Cyprus
Czech Republic
Denmark
Dominican Republic
Estland
Finland
France
Germany
Greece
Hong Kong
Hungary
Iceland
Ireland
Israel
Italy
Jamaica
Latvia
Liechtenstein
Lithuania
Luxemburg
Malaysia
Malta
Mexico
Monaco
Netherlands
New Zealand
Norway
Peru
Poland
Portugal
Puerto Rico
Romania
Singapore
Slovakia
Slovenia
South Africa
Spain
Sweden
Switzerland
Thailand
Turkey
Uganda
UK
USA

Originally posted by @cherryblossom000 in #17 (comment)

derat commented

Thanks! I fall mostly in the camp of "streaming platforms add and remove countries all the time and it is hopeless to try to track them; MB should have an 'online' region that can be used in release events", but I'm open to adding features if people want to use them. :-)

Here are some potential concerns, though:

  1. I haven't been able to find any public documentation about the Tidal API, and I'm currently just using a random public token that's floating around. I can probably issue a bunch of requests for different countries in parallel, but I'm worried that I might run into rate limits or get blocked entirely. One option would only be to make this available in the command-line program (as opposed to the web instance that I'm running on GCP).
  2. The Tidal API seems to be pretty unreliable: I frequently get dropped connections and saw a bunch of TLS errors (before I configured my HTTP client to ignore them). I'm worried that the odds of having all of the requests complete successfully will be pretty low.
  3. I'm hesitant to add countries to a release without having an automated process that periodically updates them if/when they change. Do you know how this is handled when using a-tisket or other seeders?
  4. I'm not sure how to handle tracklist differences between countries. For example, when I query the API for https://tidal.com/browse/album/1588184's tracklist, I get 9 tracks for DE, JP, MX, and US, rather than the 16 shown in the web interface. A month ago, people in this Reddit thread wrote about seeing only 3 or 4 tracks. If there's a country for which all 16 tracks (or some other nonzero number) are available, how should the differences be represented? (Now I'm wondering if I should report an error when the /tracks API endpoint returns a different number of tracks than what the album claims to have.)
cherryblossom000 commented
  1. I would be happy with it only available in the CLI
  2. That’s annoying :/ I’m not sure what to do about this except retry the requests and hope they succeed…
  3. I’m still fairly new to MB, so take this with a grain of salt, but I think the release events are based on what countries the release is available in at the time of release. If later on, a release is made unavailable to particular countries (e.g. Russia) I don’t think the release events should be updated. As for the annotation with the list of countries, I’m not sure if that usually gets updated or not, but AFAIK there’s no automated process that updates this.
    For releases that are available almost everywhere except e.g. China, the British Indian Ocean Territory, and Russia, I agree that some sort of online/streaming country would be great. However, sometimes releases will be released under a different label for various countries, such as this release group, so I think for these cases knowing what countries a release is available in would be useful.
  4. I tried fetching that release with a countryCode of AU (because I’m in Australia), and I only got 9 tracks as well. Track 1 wasn’t returned in the response so I tried playing the preview for track 1 and got the error ‘Audio preview not available for your location’. Track 2 was returned in the response and the audio preview worked, so I assume this means that those 9 tracks are the only ones available in Australia, and I would consider this release no longer available in AU.
    I think if the number of tracks doesn’t match an error like you said would be useful, perhaps instructing the user to try a different country code?

I didn’t realise how many issues there were with Tidal so if you think it’s too difficult to support multi-country lookups that’s completely fine. It’s already great that there’s a way to seed Tidal releases

derat commented

Just to see how well this would work, I tried simultaneously asking the API for https://tidal.com/browse/album/1588184's tracklist for the full list of countries. It's worked more than half of the time that I've tried it (although sometimes one or more of the API calls seems to hang), and I haven't gotten blocked yet...

It looks like you need to be in either Norway or Sweden if you want to listen to the Step Up 2: The Streets soundtrack in its entirety:

AD: 9
AR: 9
AT: 10
AU: 9
BE: 9
BR: 9
CA: 9
CH: 10
CL: 9
CO: 9
CY: 9
CZ: 10
DE: 9
DK: 9
DO: 9
ES: 9
FI: 9
FR: 9
GB: 9
GR: 9
HK: 9
HU: 10
IE: 9
IL: 9
IS: 9
IT: 9
JM: 9
LI: 9
LT: 10
LU: 9
LV: 10
MC: 9
MT: 9
MX: 9
MY: 9
NL: 9
NO: 16
NZ: 9
PE: 9
PL: 10
PR: 9
PT: 9
RO: 10
SE: 16
SG: 9
SI: 10
SK: 10
TH: 9
TR: 9
UG: 9
US: 9
ZA: 9

I'm a bit concerned that in the 1.5-month old Reddit thread people reported only having access to 3 or 4 tracks, while I'm now seeing 9, 10, or the full 16. That suggests to me that availability might change frequently, and that the information at the time of the original release is unavailable.

cherryblossom000 commented

I think multi-country lookups would probably be more useful for new releases. This is the same limitation with a-tisket as it only tells you the countries that the release is available in at the time of seeding.

derat commented

Pasting a comment by tigerman325 from https://community.metabrainz.org/t/feedback-and-support-thread-for-the-pulsewidth-a-tisket-instance/578459/80:

If you could pull countries, that’d be great. I still think this info should go in the annotation, but not the release country lists.

Here's the annotation from https://musicbrainz.org/release/31765b9f-e969-4257-855f-c7ea1f657b2a (I'm not sure if this was generated by a-tisket or by some other tool):

== Countries excluded (Deezer + Spotify) ==
(As of 2020-08-31).

    * '''Eastern Africa''': 🇮🇴 British Indian Ocean Territory (io)

== Countries where available (Deezer + Spotify) ==
(As of 2020-08-31).

    * '''Northern America''': 🇨🇦 Canada (ca) 🇲🇽 Mexico (mx) 🇺🇸 United States (us)
    * '''Central America''': 🇨🇷 Costa Rica (cr) 🇬🇹 Guatemala (gt) 🇭🇳 Honduras (hn) 🇳🇮 Nicaragua (ni) 🇵🇦 Panama (pa) 🇸🇻 El Salvador (sv)
    * '''Caribbean''': 🇦🇬 Antigua and Barbuda (ag) 🇦🇮 Anguilla (ai) 🇧🇧 Barbados (bb) 🇧🇶 Bonaire (bq) 🇨🇺 Cuba (cu) 🇨🇼 Curaçao (cw) 🇩🇲 Dominica (dm) 🇬🇩 Grenada (gd) 🇯🇲 Jamaica (jm) 🇰🇳 Saint Kitts and Nevis (kn) 🇰🇾 Cayman Islands (ky) 🇱🇨 Saint Lucia (lc) 🇲🇸 Montserrat (ms) 🇸🇽 Sint Maarten (sx) 🇹🇨 Turks and Caicos Islands (tc) 🇻🇨 Saint Vincent and Grenadines (vc) 🇻🇬 Virgin Islands, British (vg) 🇻🇮 Virgin Islands, U.S. (vi) 🇩🇴 Dominican Republic (do)
    * '''South America''': 🇦🇷 Argentina (ar) 🇧🇴 Bolivia (bo) 🇧🇷 Brazil (br) 🇨🇱 Chile (cl) 🇨🇴 Colombia (co) 🇪🇨 Ecuador (ec) 🇫🇰 Falkland Islands (Malvinas) (fk) 🇬🇸 South Georgia and Sandwich Isl. (gs) 🇵🇪 Peru (pe) 🇵🇾 Paraguay (py) 🇺🇾 Uruguay (uy) 🇻🇪 Venezuela (ve)
    * '''Western Europe''': 🇦🇹 Austria (at) 🇧🇪 Belgium (be) 🇨🇭 Switzerland (ch) 🇩🇪 Germany (de) 🇫🇷 France (fr) 🇱🇺 Luxembourg (lu) 🇳🇱 Netherlands (nl) 🇱🇮 Liechtenstein (li) 🇲🇨 Monaco (mc)
    * '''Northern Europe''': 🇩🇰 Denmark (dk) 🇪🇪 Estonia (ee) 🇫🇮 Finland (fi) 🇬🇧 United Kingdom (gb) 🇮🇪 Ireland (ie) 🇮🇸 Iceland (is) 🇱🇹 Lithuania (lt) 🇱🇻 Latvia (lv) 🇳🇴 Norway (no) 🇸🇪 Sweden (se) 🇸🇯 Svalbard and Jan Mayen (sj)
    * '''Southern Europe''': 🇦🇱 Albania (al) 🇧🇦 Bosnia and Herzegovina (ba) 🇨🇾 Cyprus (cy) 🇪🇸 Spain (es) 🇬🇷 Greece (gr) 🇭🇷 Croatia (hr) 🇮🇹 Italy (it) 🇲🇪 Montenegro (me) 🇲🇰 Macedonia (mk) 🇲🇹 Malta (mt) 🇵🇹 Portugal (pt) 🇷🇸 Serbia (rs) 🇸🇮 Slovenia (si) 🇦🇩 Andorra (ad)
    * '''Eastern Europe''': 🇧🇬 Bulgaria (bg) 🇧🇾 Belarus (by) 🇨🇿 Czech Republic (cz) 🇭🇺 Hungary (hu) 🇲🇩 Moldova (md) 🇵🇱 Poland (pl) 🇷🇴 Romania (ro) 🇷🇺 Russian Federation (ru) 🇸🇰 Slovakia (sk) 🇺🇦 Ukraine (ua) 🇽🇰 Kosovo (xk)
    * '''Northern Africa''': 🇩🇿 Algeria (dz) 🇪🇬 Egypt (eg) 🇪🇭 Western Sahara (eh) 🇱🇾 Libyan Arab Jamahiriya (ly) 🇲🇦 Morocco (ma) 🇸🇩 Sudan (sd) 🇹🇳 Tunisia (tn)
    * '''Western Africa''': 🇧🇫 Burkina Faso (bf) 🇧🇯 Benin (bj) 🇨🇮 Cote D’Ivoire (ci) 🇨🇻 Cape Verde (cv) 🇬🇭 Ghana (gh) 🇬🇲 Gambia (gm) 🇬🇳 Guinea (gn) 🇬🇼 Guinea-Bissau (gw) 🇱🇷 Liberia (lr) 🇲🇱 Mali (ml) 🇲🇷 Mauritania (mr) 🇳🇪 Niger (ne) 🇳🇬 Nigeria (ng) 🇸🇱 Sierra Leone (sl) 🇸🇳 Senegal (sn) 🇹🇬 Togo (tg)
    * '''Eastern Africa''': 🇧🇮 Burundi (bi) 🇩🇯 Djibouti (dj) 🇪🇷 Eritrea (er) 🇪🇹 Ethiopia (et) 🇰🇪 Kenya (ke) 🇰🇲 Comoros (km) 🇲🇬 Madagascar (mg) 🇲🇺 Mauritius (mu) 🇲🇼 Malawi (mw) 🇲🇿 Mozambique (mz) 🇷🇼 Rwanda (rw) 🇸🇨 Seychelles (sc) 🇸🇴 Somalia (so) 🇹🇿 Tanzania (tz) 🇺🇬 Uganda (ug) 🇿🇲 Zambia (zm) 🇿🇼 Zimbabwe (zw)
    * '''Middle Africa''': 🇦🇴 Angola (ao) 🇨🇩 Congo, Democratic Republic (cd) 🇨🇫 Central African Republic (cf) 🇨🇬 Congo (cg) 🇨🇲 Cameroon (cm) 🇬🇦 Gabon (ga) 🇬🇶 Equatorial Guinea (gq) 🇸🇹 Sao Tome and Principe (st) 🇹🇩 Chad (td)
    * '''Southern Africa''': 🇧🇼 Botswana (bw) 🇱🇸 Lesotho (ls) 🇳🇦 Namibia (na) 🇸🇿 Swaziland (sz) 🇿🇦 South Africa (za)
    * '''Western Asia''': 🇦🇪 United Arab Emirates (ae) 🇦🇲 Armenia (am) 🇦🇿 Azerbaijan (az) 🇧🇭 Bahrain (bh) 🇬🇪 Georgia (ge) 🇮🇱 Israel (il) 🇮🇶 Iraq (iq) 🇯🇴 Jordan (jo) 🇰🇼 Kuwait (kw) 🇱🇧 Lebanon (lb) 🇴🇲 Oman (om) 🇵🇸 Palestinian Territory, Occupied (ps) 🇶🇦 Qatar (qa) 🇸🇦 Saudi Arabia (sa) 🇸🇾 Syrian Arab Republic (sy) 🇹🇷 Turkey (tr) 🇾🇪 Yemen (ye)
    * '''Central Asia''': 🇰🇬 Kyrgyzstan (kg) 🇰🇿 Kazakhstan (kz) 🇹🇯 Tajikistan (tj) 🇹🇲 Turkmenistan (tm) 🇺🇿 Uzbekistan (uz)
    * '''Southern Asia''': 🇦🇫 Afghanistan (af) 🇧🇩 Bangladesh (bd) 🇧🇹 Bhutan (bt) 🇮🇳 India (in) 🇮🇷 Iran, Islamic Republic of (ir) 🇱🇰 Sri Lanka (lk) 🇲🇻 Maldives (mv) 🇳🇵 Nepal (np) 🇵🇰 Pakistan (pk)
    * '''Eastern Asia''': 🇭🇰 Hong Kong (hk) 🇯🇵 Japan (jp) 🇰🇵 Korea,  Democratic People’s Republic of (kp) 🇰🇷 Korea (kr) 🇲🇳 Mongolia (mn) 🇹🇼 Taiwan (tw)
    * '''South-Eastern Asia''': 🇧🇳 Brunei Darussalam (bn) 🇮🇩 Indonesia (id) 🇰🇭 Cambodia (kh) 🇱🇦 Lao People’s Democratic Republic (la) 🇲🇲 Myanmar (mm) 🇲🇾 Malaysia (my) 🇵🇭 Philippines (ph) 🇸🇬 Singapore (sg) 🇹🇭 Thailand (th) 🇹🇱 Timor-Leste (tl) 🇻🇳 Viet‐Nam (vn)
    * '''Oceania''': 🇦🇺 Australia (au) 🇨🇨 Cocos (Keeling) Islands (cc) 🇨🇽 Christmas Island (cx) 🇭🇲 Heard Island & Mcdonald Islands (hm) 🇳🇫 Norfolk Island (nf) 🇳🇿 New Zealand (nz)
    * '''Polynesia''': 🇦🇸 American Samoa (as) 🇨🇰 Cook Islands (ck) 🇳🇺 Niue (nu) 🇵🇳 Pitcairn (pn) 🇹🇰 Tokelau (tk) 🇹🇴 Tonga (to) 🇹🇻 Tuvalu (tv) 🇼🇸 Samoa (ws)
    * '''Melanesia''': 🇫🇯 Fiji (fj) 🇵🇬 Papua New Guinea (pg) 🇸🇧 Solomon Islands (sb) 🇻🇺 Vanuatu (vu)
    * '''Micronesia''': 🇫🇲 Micronesia, Federated States of (fm) 🇬🇺 Guam (gu) 🇰🇮 Kiribati (ki) 🇲🇭 Marshall Islands (mh) 🇲🇵 Northern Mariana Islands (mp) 🇳🇷 Nauru (nr) 🇵🇼 Palau (pw)
    * '''Antarctica''': 🇦🇶 Antarctica (aq) 🇧🇻 Bouvet Island (bv)

I should probably read https://community.metabrainz.org/t/digital-releases-merging-long-country-list-just-worldwide/440381 again to see if any sort of consensus was reached. :-/

(To be honest, I'm still a bit fuzzy on the concept of streaming "releases" in MusicBrainz, since it seems like the data changes so frequently -- for example, this annotation still lists RU even though Deezer and I think also Spotify aren't currently available there. And since it's an annotation, it's probably only consumable by humans, and there are no files to tag since it's streaming...)

cherryblossom000 commented

Those annotations are created by a-tisket.

Unfortunately I don’t think a consensus has been reached, but I think for now we’re not including the countries in the release event (leaving the country in the release event blank). From tigerman325 on this edit:

Leaving it blank like you did on this was what they want now. If it's in the whole world, but Europe, or similar, I've been leaving those blank and just adding the full annotation from a-tisket. When they are in say less than 50 countries, I've still been adding, but there is no consensus. However, if all the countries are all in Europe, I'd add just the countries available in the annotation that a-tisket gives but put Europe for the release event.

derat commented

Thanks, agreed that setting countries or regions on release events is tricky and probably requires human oversight.

Here are some related threads (still going through the last one):

I think that this comment by chaban makes a good point:

Licensed territories can change at any time. Also licensing happens at track level. I’ve seen releases on Spotify of which only a handful are available at my location.

I'm leaning toward focusing on the case where the entire tracklist is only available in a limited number of countries (maybe 10 or fewer?) and adding a shorter annotation only then:

Regions with all tracks on Tidal (as of 2023-02-07):
Norway (NO), Sweden (SE)

I'm leaning toward not documenting partial tracklists or adding an annotation in cases where the album is available in all but a few countries. If the full tracklist isn't available in any country, seeding would fail.

derat commented

Okay, commands like yambs -country XW https://tidal.com/album/1588184 will query all countries in parallel now. I haven't created a new release of the command-line program yet, but let me know if that'd be helpful (or if you build it from source and notice any issues).

derat commented

Just to mention it somewhere, the annotations look like this:

Regions with all tracks on Tidal (as of 2023-02-08 UTC):

  • Norway (NO)
  • Sweden (SE)

(The country list is sorted alphabetically.)