youtube-dl is back
Closed this issue Β· 109 comments
As everybody already knows our dev repository has been reinstated earlier today. You can read the full story here.
We would like to thank @github for standing up for youtube-dl and making it possible to continue development without dropping any features. We appreciate @github for taking potential legal risks in this regard.
We would also like to thank @EFForg and personally @Mitch77 for invaluable legal help.
We would also like to heartily thank our main website hoster Uberspace who is currently being sued in Germany for hosting our essentially business card website and who have already spent thousands of Euros in their legal defense.
We also appreciate massive amount of support received lately and we are sorry we could not physically respond to everybody.
Finally, we would like to thank all youtube-dl users and contributors for using and improving youtube-dl.
Thank you all.
youtube-dl is back!
Long live the king!
We should look into backing up all the pull requests and issues in case this happens again.
Three cheers for youtube-dl! Glad it's back, RIAA can go sit in a hole.
pog
gg, riaa can suck ass
Hip hip hooray for the return of Youtube-dl
Whoohoo
I've never used youtube-dl personally but I've been following this closely. The community over at HackerNews has invested a lot of effort tracking this as well.
Thank you to the fine folks at EFF for sticking up for us. While I'm not personally the biggest fan of Github for unrelated reasons, this was a good act of showmanship on their part, too.
Thank you for pushing the boundaries of freedom of expression and the exercise of open technology. I'm glad to see you back.
Cheers!!
woot
WELCOME BACK BOIZZZZ
Welcome back!
Welcome Back. Keep youtube-dl alive :D
Welcome back!
aye π
Nice π
Welcome back! ππ really cool project
This program can be used for public benefits. No one can take it down ever again!
Do you consider switching to a self-hosted git server to prevent any possible takedowns?
Do you consider switching to a self-hosted git server to prevent any possible takedowns?
https://github.blog/2020-11-16-standing-up-for-developers-youtube-dl-is-back/
Read this. GitHub is pretty serious about protecting open source projects developers.
yay
Why not just move to a different git server?
GitHub is shit anyway.
Congrats guys! I just opened my email and got notification on this.
So glad this repo is reinstated again
πππ
Congratulations! You need an official mirror to prevent new problems in future.
Github played well.
okaeri !!!
Youtube-dl -> RCIA
WELCOME BACK!
We should look into backing up all the pull requests and issues in case this happens again.
From https://github.blog/2020-11-16-standing-up-for-developers-youtube-dl-is-back/:
Even after a repository has been taken down due to what appears to be a valid claim, we will ensure that repository owners can export their issues and PRs and other repository data that do not contain the alleged circumvention code, where legally possible.
To prevent further take downs maybe new rule NOT to use copyrighted test cases?
We should look into backing up all the pull requests and issues in case this happens again.
From https://github.blog/2020-11-16-standing-up-for-developers-youtube-dl-is-back/:
Even after a repository has been taken down due to what appears to be a valid claim, we will ensure that repository owners can export their issues and PRs and other repository data that do not contain the alleged circumvention code, where legally possible.
But contributors should have a backup of code with issues and pull requests, because rules of Github may change suddenly.
I would say we should automate the merging of PRs if it's only a site-support and not core change.
Worst case one site will fail. There are outstanding PRs dating 1year back. Then we can backup the remaining issues.
I used youtube-dl to archive channels that at the risk of being taken down either by the Turkish government, or youtube, or by 3rd parties with Content ID abuse. It is an amazing tool. Glad it's back.
Welcome back!
So this whole fiasco was due to a CI server downloading copyrighted material publicly available on YouTube during tests?
What a waste of everybody's time.
Happiest News of the day
So this whole fiasco was due to a CI server downloading copyrighted material publicly available on YouTube during tests?
No~t exactly ... Well, during the tests data is downloaded, but only enough that ytdl can verify it did exactly as ordered (I'm pretty sure that data's not even saved to disk during the test, instead it's in RAM and automatically gets dropped when that part of the test is done). That's 1 point GitHub/EFF made which RIAA described wrong.
So this whole fiasco was due to a CI server downloading copyrighted material publicly available on YouTube during tests?
No~t exactly ... Well, during the tests data is downloaded, but only enough that ytdl can verify it did exactly as ordered (I'm pretty sure that data's not even saved to disk during the test, instead it's in RAM and automatically gets dropped when that part of the test is done). That's 1 point GitHub/EFF made which RIAA described wrong.
You make it sound like the removing the tests wasn't even necessary.
So this whole fiasco was due to a CI server downloading copyrighted material publicly available on YouTube during tests?
No~t exactly ... Well, during the tests data is downloaded, but only enough that ytdl can verify it did exactly as ordered. That's 1 point GitHub/EFF made which RIAA described wrong.
Well, precisely, that is what I said.
So this whole fiasco was due to a CI server downloading copyrighted material publicly available on YouTube during tests?
No~t exactly ... Well, during the tests data is downloaded, but only enough that ytdl can verify it did exactly as ordered (I'm pretty sure that data's not even saved to disk during the test, instead it's in RAM and automatically gets dropped when that part of the test is done). That's 1 point GitHub/EFF made which RIAA described wrong.
You make it sound like the removing the tests wasn't even necessary.
And it wasn't, that was part of the reason for reinstating. The test didn't even download the whole thing (which I believe was his point, but rather enough to pass the test), even if it did it is fair use as you do the same to see if YouTube is working.
RIAA throw the clay at the wall, it didn't stick.
So this whole fiasco was due to a CI server downloading copyrighted material publicly available on YouTube during tests?
No~t exactly ... Well, during the tests data is downloaded, but only enough that ytdl can verify it did exactly as ordered (I'm pretty sure that data's not even saved to disk during the test, instead it's in RAM and automatically gets dropped when that part of the test is done). That's 1 point GitHub/EFF made which RIAA described wrong.
You make it sound like the removing the tests wasn't even necessary.
If the PDF is to be believed, it absolutely sounded like RIAA threw the DMCA just to show off. In other words they either knew they had no chance at winning but wanted to flex, or they had no idea and thought this'd be an A-B-C easy win.
Finally inner peace.... ππ
This has been hello of a journey. I learned to do scraping using Python because I thought this is how it is going to be from now on. While trying out, I had an idea, I thought that it will be better if I could use selenium to download videos so that it trigger less bot detection. I was trying to automatically use the selenium developer tool (using selenium) to download videos so that it resembles human more closely.
The solution I came so far is:
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import json
import requests
import time
caps = DesiredCapabilities.CHROME
caps['goog:loggingPrefs'] = {'performance': 'ALL'}
options = webdriver.ChromeOptions()
# options.add_argument("--window-size=1920,1080")
# options.add_argument('headless')
driver = webdriver.Chrome(desired_capabilities=caps, options=options)
driver.get('https://twitch.tv')
time.sleep(5)
driver.find_element_by_id("play-button").Click()
def process_browser_log_entry(entry):
response = json.loads(entry['message'])['message']
return response
while True:
browser_log = driver.get_log('performance')
events = [process_browser_log_entry(entry) for entry in browser_log]
events = [event for event in events if 'Network.response' in event['method']]
for e in events:
if e['params']['response']['url'].endswith('.ts'):
url = e['params']['response']['url']
r1 = requests.get(url, stream=True)
if(r1.status_code == 200):
with open('testvod.mpeg', 'ab') as f:
for chunk in r1.iter_content(chunk_size=1024):
f.write(chunk)
else:
print("Received unexpected status code {}".format(r1.status_code))
driver.quit()
As opposed to:
import m3u8
import requests
from bs4 import BeautifulSoup
from tqdm import tqdm_notebook as tqdm
import subprocess
sess = requests.Session()
r = sess.get("https://www.iplt20.com/video/144829/final-csk-vs-srh-fbb-stylish-player-of-the-match-lungi-ngidi")
soup = BeautifulSoup(r.content, 'html5lib')
video_id = soup.find('video', attrs={'id': 'playlistPlayer'})['data-video-id']
account_id = soup.find('video', attrs={'id': 'playlistPlayer'})['data-account']
url = "https://secure.brightcove.com/services/mobile/streaming/index/master.m3u8"
params = {
'videoId': video_id,
'pubId': account_id,
'secure': True
}
r = sess.get(url, params=params)
m3u8_master = m3u8.loads(r.text)
m3u8_playlist_uris = [playlist['uri'] for playlist in m3u8_master.data['playlists']]
playlist_uri = m3u8_playlist_uris[0]
r = sess.get(playlist_uri)
playlist = m3u8.loads(r.text)
m3u8_segment_uris = [segment['uri'] for segment in playlist.data['segments']]
with open("video.ts", 'wb') as f:
for segment_uri in tqdm(m3u8_segment_uris):
r = sess.get(segment_uri)
f.write(r.content)
subprocess.run(['ffmpeg', '-i', 'video.ts', 'video.mp4'])
Few things that will be nice to have is:
- if they ask for captcha I want to pause and manually solve that. If I am not there to solve it (after three tries) it will stop the process.
- It will click here and there, scroll up and down to emulate human behaviour.
This has been hello of a journey. I learned to do scraping using Python because I thought this is how it is going to be from now on. While trying out, I had an idea, I thought that it will be better if I could use selenium to download videos so that it trigger less bot detection. I was trying to automatically use the selenium developer tool (using selenium) to download videos so that it resembles human more closely.
youtube-dl has made the point numerous times they wanna minimize non-stdlib requirements.
But even if Selenium was used, it needs a massive rewrite of the infrastructure. I'd say borderline complete rewrite.
^ And that's not even counting that for consistency all of the extractors would need to be rewritten for Selenium (although that wouldn't be necessary functionality-wise). That's a rather extreme task, as I've seen the count of extractors, let alone the code.
Also, Selenium has features to not need mechanical, horrifyingly unreliable, waiting like you did in the example.
For the future, Selenium would indeed be the way to go, though.
@rautamiekka some sites are creating huge problem now a days. For example, I had a post on quora answering Tutorials: How can I download PluralSight training videos? A lot of people are commenting that they have been banned, their account had been blocked due to high bot scores etc.
If we could come up with a single page script (not talking about rewrite, maybe a template) using selenium, then I could give it to them. I think if someone modifies the sample I have provided then it will be working for those sites.
Just to be clear, we are not looking for a universal solution, but a solution which we can tweak with the trouble making sites. For example: If we could get a working script (with --batch-file, --cookies, --min-sleep-interval, --max-sleep-interval, --abort-on-error, -f, -o ) that download pluralsight courses using selenium then we could modify that script for other sites (and also can try different bot bypass strategies with that script).
^ Hmm, very true.
Never heard of someone getting banned over at PluralSight (pretty much nowhere for that matter, apart from Fb being extremely fidgety, as many reports here has proved; even I've gotten soft-blocked for reasons that couldn't be true: despite my acc having all the 2FA's enabled, I were repeatedly soft-blocked cuz my acc allegedly was abusing things like their graphs API which I had never used, nor never used yt-dl on Fb. Gladly it stopped eventually. I think it was Fb's systems fucking up) for downloaders, but I'd be lying if I said it's a surprise.
This is the best news I heard all day! I'm very happy to see Microsoft/GitHub change course and siding with opensource devs!
Bring out the crabs
π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦π¦
Never heard of someone getting banned over at PluralSight
Please check the quora post link. You will find some evidence there.
So glad to see such an essential tool as youtube-dl reinstated, and GitHub taking a stand for developers. However, even though this battle was won, the war isn't: DMCA still contains language about anti-circumvention measures, and nothing prevents RIAA or anyone else from invoking them again, including against youtube-dl. Also: nothing prevents RIAA from putting pressure on YouTube to tighten the screws w.r.t. DRM, and then we all lose the ability to download video tutorials, conference talks etc. for offline-viewing. Even if we may succeed in publicly shaming RIAA and their members for their anti-social behavior (doubt it, they stopped caring being a good Netizen a long time ago), smaller obscure rights-groups could just as well do damage here.
This is the moment for Big Tech to join the fight to lobby for a better DMCA. This will be pretty difficult, but with GitHub/Microsoft jumping on the bandwagon, maybe Google/Alphabet could make sure YouTube won't cave in and give to the RIAA pressure to default-DRM everything. This is IMHO the most pressing issue to resolve on the political level: to get Google to join us, and to stick with us. This, and of course, to lobby the hell out of Congress to fix the DMCA and push for a modernization of WIPO.
Thanks GitHub for actually listening!
RIAA should really re-evaluate what they're doing... They appear to not know what they stand for anymore...
Long live the king youtube-dl!
Please let us know if/how we can do anything to help fund legal defense--I'm glad you're back up, but it's not clear to me this is over.
Welcome back!
Welcome back.
@dstftw Congratutions for great work you've kept doing so far, we're lucky to have you and your team working so hard on this project !
π
God bless you abundantly you, your team and respective families !
I am so happy! Thanks to everyone supporting the cause in any way!
This is a HUGE win for freedom and against bullshit!
Read EFF's blog here: https://www.eff.org/deeplinks/2020/11/github-reinstates-youtube-dl-after-riaas-abuse-dmca
Thanks @github and @EFForg
welcome back! π
hello youtube dl. welcome back to the arena!
εΌεΏ εεεεεε
π₯³ WOOOO!!!
Yesssssss but... Why still use github?
Good on GitHub, the EFF, and everyone else involved for helping common sense to prevail.
As for the RIAA all I have to say is this:
Is there any chance new "open source" video test cases can be something other than "Big Buck Bunny" :-)
We could use that blender video with the girl and dog, I forgot what its called.
For Blender Reels, Sintinel, Agent 327 and a lot of others.
I read the legal reply to Github. I think the legal help you had was a good one. The wording is just amazing. Kudos to EFForg.
Glad to see developers standing up for each other and fighting RIAA's illegal intimidation tactics
it`s a good news!
I cannot overstate how important youtube-dl has been for me for downloading conferences, MOOCs, for offline viewing.
Alpha Mike Foxtrot, RIAA.
I am so late to the news but I am glad that youtube-dl has returned! :) I am so glad that this project lives on!
It's a great victory against RIAA, well done to Gihub and EFF !
Does anyone know how we can get our forks of youtube-dl back?
Mine is still down https://github.com/pukkandan/youtube-dl
@pukkandan GitHub should eventually get around to opening an internal support ticket and sending you an e-mail containing several links to allow you to choose your desired outcome, restoration of your fork, deletion, &c. Given the original repo is back, I went for the "just nuke it, I'll fork again in the future if I wish to share changes again", which I confirmed they were A-OK with.
Notably, they confirmed it's safe to re-fork. In my case, this means nothing in my original fork was lost; I can re-fork, push my local copy, and my two minor changes will re-appear. The message should have the subject:
[GitHub Support] - youtube-dl fork
And will contain a link to your ticket, to respond to, in the form: https://support.github.com/ticket/{id}
If they aren't timely in getting ahold of you, you may be able to open a ticket (or check "My Tickets" on that support site to see if the e-mail got lost) and make your request proactively. There was, additionally, an e-mail with the subject:
An Important Notice From GitHub Trust & Safety
This more generally covered the "overall situation" and provided many, many links to further reading. For full disclosure, here's their initial "important notice" message, and my response, which they fulfilled. The links here, you'll notice, contain "tags". These may be my tags, or they may be a tag for the situation; I wouldn't assume either way. (Open a ticket yourself, not using my links from this Gist. π) Edit: Gist updated with their response.
@amcgregor thanks for the info
Yay
letβs fucking GOOOOOOO
Thanks All.
Welcome back!
https://github.blog/2020-11-16-standing-up-for-developers-youtube-dl-is-back/
Read this. GitHub is pretty serious about protecting open source projects developers.
Virtue signaling. Just see what GitHub did to the developers of Pale Moon: https://forum.palemoon.org/viewtopic.php?f=70&t=25435&start=40#p202146
eyy welcome back lads
@smnthermes "Wow, you sure got your self someone who can spout the social justice party line VERY well" and I stopped reading. Never even really figured out what that thread was about. Saw something about web components, then that. The non-sequitir is real, and relationship to this situation completely obfuscated.
Diving back inβ¦
I do hope that Github becomes defunctβ¦
(Because wishing for the demise of a useful tool is awesome.)
If I were a judge, I'd take it as physical evidence of antitrust law violation... But that's just me.
(I'm glad this fellow is not. Private property, alternatives are available, absolutely no antitrust involvement, hereβthey in no way have a monopoly within this domain. Was about to write "like they do in X"β¦ butβ¦)
Welp, guess Github is dead.
There isn't a sufficiently π€£ emoji. It fails to depict the floor being rolled upon.
Welcome back! This is great news. I'd like to contribute to the project.
Long Live Open Source !
visca
I'm very sorry that I missed this at the time.
So, like everyone here, Welcome Back youtube-dl & Live Long & Prosper Open Source.
Pretty sure that ain't how it works.
So, if we expand the abbreviations, we get 'Downloader-download'?
Do you also have an idea how users are supposed to find a specific product, when it has pretty much the most generic name on the internet?
It's obvious you haven't put more than 10 seconds of thought into your comment.
@smnthermes wow, reading those threads give me a headache
Github Over.
Communist Authoritarianism = Very Yes.
Wow, you sure got your self someone who can spout the social justice party line VERY well.
I do hope that Github becomes defunct and people migrate elsewhere
I'd take it as physical evidence of antitrust law violation... But that's just me.
WTF? Using a new technology is not browser neutral, and make you commie? Are they crazy? Seems like there's just a some altrights and RMS-like people.
The name ""Youtube-dl" was a big error, because 'Youtube' is a trademark. It's necessary to change that name.
I think the name can be 'Downloader-dl'.
not how it works but okay
The name ""Youtube-dl" was a big error, because 'Youtube' is a trademark. It's necessary to change that name.
I think the name can be 'Downloader-dl'.not how it works but okay
I can't create an application and to name it 'Netflix Downloader' or 'Netflix-dl', this is not correct. Netflix would use all its power to destroy the app. It's better different name.
you can tho, you wouldn't host it on github tho
The name ""Youtube-dl" was a big error, because 'Youtube' is a trademark. It's necessary to change that name.
I think the name can be 'Downloader-dl'.not how it works but okay
I can't create an application and to name it 'Netflix Downloader' or 'Netflix-dl', this is not correct. Netflix would use all its power to destroy the app. It's better different name.
you actually can do that but netflix downloader is piracy but youtube downloader isn't because how would it be
Though youtube-dl is back
The issue has resulted to bifurication of youtube-dl into
Ytdl-org youtube-dl and ppawel youtube-dl https://github.com/ppawel/youtube-dl
While ytdl-org is now live,the other branch and it's forks are still in lockdown mode
That's the repo's owner's problem.
@rautamiekka sure but that has led to fragmentation of youtube-dl into two groups
And almost (10.5 k) 68 percentage of the community is still in DMCA lockdown mode,the rest 32 percent (5.2k)
are in these group
If the repo owners just talked to GitHub it would be fixed, it ain't harder than that.
Which reminds me: GitHub shoulda reopened all the affected forks since they had the full list through the DMCA. That makes this weird.
Congrats youtube-dl! Probably one of the best tools ever tbh. Great job to the many corporations that are defending it, too.
Youtube dl is definitely a project that should be hosted on a self-hosted git server, though. These kinds of projects that are heavily dmca'd for almost no valid reasons (we be encouraging piracy omg woah) stand no chance on git servers that have close relations with copyright trash. Github did a good job though.
I might be late to the party, but that won't stop me from celebrating. π
Why has development stalled as of late? Issues and PRs are piling up each week with no end in sight. Is the project in need of a maintainer with more time to devote to the project?
Why has development stalled as of late? Issues and PRs are piling up each week with no end in sight. Is the project in need of a maintainer with more time to devote to the project?
The lack of answers about this, and ongoing stalled development has made me jump over to yt-dlp, which has an active community.