License request from pywidevine different to the one constructed on the device
Closed this issue · 41 comments
Describe the bug
License request constructed by pywidevine gets rejected by license server while the device itself plays the video without issue. The situation indicates that the license server has a means to distinguish the license request constructed by pywidevine comparing to chrome on the device.
The request to license server is constructed correctly, otp, token, headers. Confirmed by more users. So started to wonder if there is anything that the license server can distinguish other than the key+blob since these are correct.
To Reproduce
Create Android device in emulator, Pixel 6 Pro, Android 11. Extract keys for CDM version 16.0.0. Construct license challenge for vdocipher license server. Challenge has token that contains otp, playback info, href, tech, and license request. Make a post request.
Expected behavior
A correctly constructed license request that will in request to license server return the license.
Additional context
pywidevine test succeeds.
The context may look like out of concern of pywidevine at first sight. However, the case indicates that the license server can distinguish license request from pywidevine from android 11's chrome. Therefore it should be a concern of pywidevine construct license request that is identical to the browser on the device.
More information from new experiments. When the keys are extracted from a physical device then pywidevine creates a license request that works with the license server. I will park this issue here since I have a solution and the issue can be either in pywidevine or the dumper (dumping fake keys?).
What's most likely happening is the license server simply blacklisted that specific provision (or the entire System ID). This is almost confirmed by the fact that you had another provision that worked, with no other code changes. I wouldn't be surprised since it's from the Emulator. If its Android app can be installed in the Emulator I'd try to run it and see if it can play anything licensed.
In cases like this for the future, ensure that the Service Certificate is set, and use the correct one. Some services refuse if it's not set or not the right one.
You should also make sure the wvd was made under the correct device type. E.g., it should be ANDROID for this Emulator provision, not Chrome. It's quite important to use the correct one as there are small differences between the challenge messages between them. Some code that uses pywidevine can also query for the type and filter specific code based on the type as well.
Hey, thanks for getting back.
The service certificate is set when extracting the key. The wvd was made with "ANDROID" argument only.
Not sure where you get that "another provision worked" or what you exactly mean by "provision". Keys extracted from devices made by the emulator don't work at all, license server responds with auth failed. Yet, after the auth failed, chrome plays drm videos inside the emulator where the device runs. So no way for me to get keys in order to make my own requests to the license server.
What works are keys extracted from a physical device. It is true, the process and pywidevine is with no change, the same pywidevine and the same process for both physical device and emulator created device.
First, you are misunderstanding what a 'provision' is. A provision is a private key you have from a device. e.g., device_private_key. This combined with the Client ID blob would be a 'provision', often inaccurately referred to as simply 'keys'.
I mean "another provision worked" as you stated the provision from the Emulator does not work, but from the Android device it does. Hence you're using two provisions with the other one working.
However, you state that even after "auth failed" chrome in the emulator still successfully plays videos from the license server? Have you checked to see if it made any further successful requests thereafter? As that's quite strange.
It's also possible that the device failed (auth failed) and defaulted to a lower resolution stream that has no DRM or the weaker AES Clear Key. I'd check the manifest that the Android emulator last gets to see what DRM it supports.
Cool, then as you say a provision extracted from the physical device works yet provisions from the emulator don't. Tested multiple devices and android versions 9, 10, 11, 12.
I paid an attention to the point of checking if the device has been blocked after the auth failed from the license server. I used the default chrome on the device, I refreshed the bitmovin page and played the video, I load the watermark demo page from vdocipher and play their video. This works after say 20-40 such auth failed attempts and 24+ hours since then. Throughout and after the emulated device is able to play the videos in its browser.
When I find out how to check the manifest I will do it and be back with update. Visually there has been no visible change, large device resolution 3120x1440, so I doubt any change.
That comment about pywidevine was a bit confusing and then quickly deleted. The comment talked about testing of pywidevine vs classic widevine but later talked about "older" pywidevine. Would need more clarification from the author.
I test with the following videos,
- https://bitmovin.com/demos/drm
- https://www.vdocipher.com/blog/2014/12/add-text-to-videos-with-watermark/
If there is a specific older pywidevine that I could try I am happy to do it. Let me know the commit hash or tag to checkout.
I had a similar experience with Vdocipher.
A few months ago I did a comparison test between @rlaphoenix's pywidevine and the "classic" one (the earlier WKS version to be exact) with two similar Python scripts except for the decryption keys function of course.
The test was performed on a set of Vdocipher videos from the same website and the result was that the older pywidevine implementation worked, as opposed to @rlaphoenix's module. I had ran a successful test with the produced .wvd file against the Bitmovin demo video to make sure that it's actually working.
Please note that my necessary files were extracted from a virtual Pixel 5 running an Android Q system.
If I have to guess the reason I will say that the version of pywidevine I used had this .json file which may "spoof" the real origin of the keys files.
client info from emulator device,
{
"application_name": "com.android.chrome",
"company_name": "Google",
"model_name": "sdk_gphone_arm64",
"architecture_name": "arm64-v8a",
"device_name": "emulator_arm64",
"product_name": "sdk_gphone_arm64",
"build_info": "google/sdk_gphone_arm64/emulator_arm64:11/RSR1.210722.002/7602718:userdebug/dev-keys",
"widevine_cdm_version": "16.0.0",
"oem_crypto_security_patch_level": "0",
"oem_crypto_build_information": "OEMCrypto Level3 Code 22590 May 12 2021 23:37:56"
}
Will try the one referenced, if not I will need to get one from another device.
Well from what I'm reading it's as if you only have issues with the Emulator provisions. It seems to be regardless of what version of pywidevine you're using unless I misread it. This leads me to believe what I thought earlier that it's blocked by vdocipher is in fact the case.
How it seems to be working somewhat after 40 calls or so like you said I'm not entirely sure unless it's fallen back to a lower resolution and was allowed. E.g., not allowed 1080p license so it tried to load lower resolutions and bitrates until it got a successful request for 480p or something. That's my best guess.
The only thing from my glance at the 2nd demo link is that the token data for OTP is sourced from an unknown place. I can't find where that data is sourced from. It seems to be timestamped 2016-03-13, so maybe it's not a true OTP and can be re-used forever. I'm not entirely sure. The Playback Info seems easy enough to generate, and so would be the href of course. So it's just the OTP and the Challenge here that are the variables in play to pay attention to.
At the end of the day, I really can't see there being an issue just by using a specific private key or client ID on pywidevine's side of the process. Therefore I really just think it's the case of vdocipher blocking the emulator provisions.
--
Again if you can generate a challenge from one code base that works, that doesn't in this pywidevine codebase, using the same provision, save both the challenges, and I'll compare the differences and see if there's anything wrong. However, I really don't think there is. I think they just block emulator ones, but maybe let's em through for lower resolutions/bitrates.
Their OTP is generated by their backend on each page refresh and placed inside the page source. OTP has TTL. Works a couple of moments and minutes, e.g. the browser's request to the license can be repeated from cmd line. After the TTL, the license server rejects it with License verification failed.
How can I display the manifest? Does this give you idea about the quality? The highest of the three offered qualities.
Well from what I'm reading it's as if you only have issues with the Emulator provisions. It seems to be regardless of what version of pywidevine you're using unless I misread it.
...
However, I really don't think there is. I think they just block emulator ones ...
I would like to confirm again that the the same CDM files which were extracted from the Android emulator were used with both my test scripts, that is once with the .wvd file using @rlaphoenix's module and another time with WKS version.
A listing of the available video qualities mentioned earlier:
ID EXT RESOLUTION │ TBR PROTO │ VCODEC VBR ACODEC MORE INFO
───────────────────────────────────────────────────────────────────────────────────
0 mp4 1366x720 │ 275k https │ avc1.4d4028 275k video only DASH video, mp4_dash
1 mp4 2048x1080 │ 688k https │ avc1.64001f 688k video only DASH video, mp4_dash
2 mp4 2048x1080 │ 1387k https │ avc1.640028 1387k video only DASH video, mp4_dash
Obtained by running this command line:
yt-dlp "https://d2lrwez4x0gs00.cloudfront.net/media/aWYD1JUhUz8sh/725eff64/stream.mpd" --allow-u -F
Well from what I'm reading it's as if you only have issues with the Emulator provisions. It seems to be regardless of what version of pywidevine you're using unless I misread it.
...
However, I really don't think there is. I think they just block emulator ones ...I would like to confirm again that the the same CDM files which were extracted from the Android emulator were used with both my test scripts, that is once with the
.wvdfile using @rlaphoenix's module and another time with WKS version.
Ok, Interesting. Can you generate a challenge from WKS version, save it, test it make sure it worked, then do the same for my pywidevine and send me the two saved challenges making sure you mark which is which? If you can, disable service certs so I can verify the client ID information as well.
Well, I ran my test scripts and both worked fine this time.
The token of a video in VdoCipher's official website was used.
I don't know what service certificates means to be honest.
What device and android version you selected? I will try it too. None of the provisions from emulator worked for me, yet you say you have a working one.
I use diazole's fork of the dumper. The original can do only the older android 9 and 8. The result from both for me is the same, auth failed. Do you use one from a different source?
cdm = Cdm.from_device(device)
session_id = cdm.open()
cdm.set_service_certificate(session_id, cdm.common_privacy_cert)I used this dumper and frida server but with another script.js file from a friend.
What is the emulated device and android version you created?
That sounds like the way the provision is extracted is the only difference in our setups since yours works with emulated devices. I use the original wvdumper's or diazole's.
If that would mean it is about the provision extracted, it is even more mind boggling that wvdumper/diazole's script.js extracts a different provision from android 10 emulated devices versus android 10 physical device. Since one does not work with license server and the other does.
This was my setup:
Please note that my necessary files were extracted from a virtual Pixel 5 running an Android Q system.
+ it's an x86 image.
Indeed, your theory makes sense. Maybe the script.js file is what makes the difference.
I am pissed off. I've got a physical device, Lenovo tablet, got provision dumped, yet keep getting Auth failed from vdocipher using pywidevine constructed wvd, while the device happily plays the videos. Going bonkers.
no magic with the lib. cdm 15.0.0, vendor/lib/libwvhidl.so hooked at 0xf3657000, func kqzqahjq.
I keep seeing the same issue all over again this time with physical device. The device happily plays the videos. However, license requests constructed with the challenge from pywidevine receive auth failed.
I switch back and forth between the chrome on device and pywidevine requests. That shows the extracted provision is valid and not blocked by the auth server. If it was blocked then the chrome on device would be rejected too. It has the same provision, in fact any repeated dump yields the same provision.
I decode, print and compare payloads and headers, compare them with those on the device. I debug chrome's session running on the device. Headers and payloads match, just the challenge is constructed in wvd device,
device = Device.load(wvd_path)
cdm = Cdm.from_device(device)
session_id = cdm.open()
cdm.set_service_certificate(session_id, cdm.common_privacy_cert)
challenge = cdm.get_license_challenge(
session_id,
PSSH(pssh),
privacy_mode=True
)The challenge goes to the request,
token_dict = {}
token_dict['otp'] = otp
token_dict['playbackInfo'] = playback_info
token_dict['href'] = href
token_dict['tech'] = tech
token_dict['licenseRequest'] = challenge
token = json.dumps(
token_dict.to_json(),
cls=ComplexJsonEncoder
)
token_raw = base64.b64encode(token.encode('UTF-8')).decode('UTF-8')
headers = {
'user-agent': 'Mozilla/5.0 (Linux; Android 10; Lenovo TB-8505X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36',
'origin': f'{origin_url.scheme}://{origin_url.hostname}/',
'referer': f'{origin_url.scheme}://{origin_url.hostname}/',
'vdo-ref': video-ref
}
req = requests.post(
'https://license.vdocipher.com/auth',
json={'token': token_raw},
headers=headers
)Can you suggest what to debug further? I can share all of the info including the extracted provision - I will reinstall if it voids. We could move it to a private channel too.
Doesn't the licenseRequest value need to be base64-encoded? You left it as bytes.
Also all the things your doing JSON wise is just funky.
Do this instead:
token_b64 = base64.b64encode(json.dumps(token_dict).encode()).decode()
# ...
req = requests.post(
'https://license.vdocipher.com/auth',
json={'token': token_b64},
headers=headers
)I took the code from a larger generalised code-base and was making it more condensed for the post. I lost the challenge encoding during copying. The "funky" json was doing additional checks in the original code. The point remains the same.
I re-tested the copy below. token_b64 is from the chrome session on the device running the watermark demo. This ends with {"code":2048,"message":"Authentication failed"}. The device itself keeps playing the videos at the highest quality after page reload and device restart.
token_dict = json.loads(base64.b64decode(token_b64))
otp = otp if otp else token_dict['otp']
playback_info = token_dict['playbackInfo']
href = token_dict['href']
tech = token_dict['tech']
device = Device.load(wvd_path)
cdm = Cdm.from_device(device)
session_id = cdm.open()
cdm.set_service_certificate(session_id, cdm.common_privacy_cert)
challenge = cdm.get_license_challenge(
session_id,
PSSH(pssh),
privacy_mode=True
)
challenge_b64 = base64.b64encode(challenge).decode('UTF-8')
token_dict = {}
token_dict['otp'] = otp
token_dict['playbackInfo'] = playback_info
token_dict['href'] = href
token_dict['tech'] = tech
token_dict['licenseRequest'] = challenge_b64
token_json = json.dumps(token_dict)
token_b64 = base64.b64encode(token_json.encode('UTF-8')).decode('UTF-8')
origin_url = urlparse(mpd)
headers = {
'user-agent': 'Mozilla/5.0 (Linux; Android 10; Lenovo TB-8505X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36',
'origin': f'{origin_url.scheme}://{origin_url.hostname}/',
'referer': f'{origin_url.scheme}://{origin_url.hostname}/',
'vdo-ref': video_ref
}
req = requests.post(
'https://license.vdocipher.com/auth',
json={'token': token_b64},
headers=headers
)
print(req.text)Looks like no one has an idea. With the code above that constructs the challenge correctly, I conclude that faulty is either the provision being dumped or the pywidevine constructing the challenge. Both cases look at least suspicious.
The first would mean the CDM has an ability to provide a fake provision, i.e. function kqzqahjq on cdm15 32bit is not the right one and there is another one in the device, since the device constructs correct challenges. The dumped provision however decrypts bitmovin just fine despite the license server does not like it.
The second would mean that some provisions construct good challenges, as reported by others, and some don't, as reported here and others using emulator.
You have yet to supply a challenge that you say definitely works. If you can provide one then I can deconstruct it and see what differs (provide one you make from pywidevine as well via your provision, so I can compare). However, I'm confident it's your provisions. Some ways of dumping even L3 are generally bad. I remember one script going around actually had an error and was dumping bad data as the key. Another script faked the result.
The shared code takes a generated challenge and delivers it to license server. I shared it to show that I send it to the server correctly to clear this part. The problem remains in the challenge generated by pywidevine using the provision. pywidevine test command for this provision/wvd decrypts bitmovin drm with success, the device holding the provision plays videos. That means the CDM on the device can create challenges using the provision that get accepted by license server.
With the bad ways of dumping L3, are you saying that a bad provision decrypts bitmovin? My provision decrypts bitmovin demo. This makes me assume the provision is good. Can you comment if there can be a bad provision that can decrypt bitmovin?
I am happy to send you the provision and challenge. The point with provisions is that I have not got a provision that gets accepted by the license server. So I don't have a case that you can debug for a difference. You have not said you want to look at my current provision.
When I started the thread, I worked with emulated devices, all provisions failed with the license server, yet the emulated devices played the drm videos. The same issue but emulated devices. I shared the experience with another guy, both of us had this issue. When he switched to a physical device, onn tv box, he confirmed that that has fixed the issue with failed auth, so I opened this thread. I got the Lenovo Tab M8 but sadly the issue has remained for me. I can just keep testing new devices.
While you did share code to send to the license server, you can't dismiss it as the cause. For one, you have token_b64 which could be specific to a provision, device, IP, timestamp, etc. I briefly looked at vdocipher when this issue first came up and I too couldn't find any genuine info about the token, or where it came from. Specifically the OTP in it.
Also just because bitmovin accepts it, doesn't mean other places do as well. All services can use a blacklist or whitelist on both the license server SDK and their own API front. Also, you say it plays videos on the device. We don't know if it's even using Widevine, playing the same quality you are trying to license, or even using the same provision. Even an L3 device could have it's own L3 certification, and an L3-haystack certification to fall back on. The haystack one is typically a 4443 or such System ID, and usually are mass blocked or at least limited.
With the bad ways of dumping L3, are you saying that a bad provision decrypts bitmovin?
No, they likely cant. What I was saying is there are scripts/tools out there that either mislead or just fake results. E.g., generate random data, return data parsed/processed incorrectly, or just fake.
I am happy to send you the provision and challenge.
Please do not share the provision. But do share a challenge generated by your device, from your device, as well as a challenge that pywidevine is generating, of which you have tried to send and failed. I will compare the two and see what differs.
The point with provisions is that I have not got a provision that gets accepted by the license server. So I don't have a case that you can debug for a difference.
You can. You say the device itself works and plays. So just use HTTP-Toolkit and sniff the request and save the challenge that was sent to the license server. For Pywidevine just save it to a file, or base64 and print it and copy it. We want to compare challenges, not provisions.
I've given a look at the watermark demo for vdocipher, and could only find one difference with our challenges. The request ID. Now. Here's the thing. It's up to the CDM to decide what to use for the request ID. License Servers aren't meant to dictate what format it should be but it seems like vdocipher does.
Chrome's CDM (chromecdm) just does a random 16-byte request ID, while Android OEMCrypto does a more or less strange approach. It generates 4 random bytes followed by 4 0x00 bytes. It then uses the session number in little-endian to fill up 8 more bytes. The result would usually be <4 rand><4 0x00><7 0x00><1 0x01 or small number>. From there OEMCrypto seems to have a bug as I doubt it was intentional, but they encode it incorrectly. They convert it to upper-case HEX (now 64chars in length), then encode that as base64.
Since you and I are both using Android CDMs, this is the correct request ID formula. It just seems vdocipher is trying to dictate what it should be. However, a reminder that we are comparing a Chrome challenge with an Android-like challenge. I would still need to see a real Android challenge from your physical device to really confirm what's happening.
I'll also just mention now that I too think the android formula that's currently in use is wrong. However, I've had strong pushback behind the scenes with some proof to show that Android challenges (at least some) did it like that. And yes, specifically that encoding.
What I'd be interested is to see what your Lenovo device uses as the request ID.
Now I get that you want the challenge from the device and want to compare it with a challenge from pywidevine. Sure, that's easy, I will send that.
The API is clear about the OTP, input to OTP are client secret and selection of options that control restrictions, ttl, geo, etc. Important for our case is, it does not consider info about the client who will play the video. That leads to the token_b64, nothing special that would affect us, otp, video id, and the challenge. So again, only thing that can go wrong in this part is the challenge.
I tried explicitly using a similar length and encoding request ID (chrome style request ID) and it still failed. I'm also having issues with a code 2013 "timed out" response.
We don't know if it's even using Widevine,
You want to say that the chrome browser is not using widevine?
playing the same quality you are trying to license,
The http request to license server does not specify the quality.
We explored the quality played by the device earlier in the thread, it plays the highest quality.
or even using the same provision.
We rely on the hooked methods in the library being used by chrome. Assuming that since they get called these are the correct functions. Many users have reported that the provision flowing through that hooked function is a good one. So we assume this not a honey pot. But if it was a honeypot, then the extract provision would not decrypt a drm video. However, this one decrypts videos. That gives us all indications that this is the provision used by the device itself.
Do you have theories or indications that the device have multiple provisions that all decrypt the video yet only selected ones get accepted by license servers?
I'm also having issues with a code 2013 "timed out" response.
OTP has time to live. It gets set by the backend when requesting the OTP. Recommended to be at least 5 minutes. Default 6 hours. The OTPs usually last a couple of minutes. The watermark demo lasts a few minutes, looks 5 minutes but I didn't test it specifically. Each HTTP request gives you new OTP.
What you sent is exactly what pywidevine is generating for android devices. So, it's not the challenge at cause.
However. I might have found out something. After a couple of calls under the same-ish code, the key will get blacklisted. But I might have found a solid fix to this entire issue. First of all, the OTP is genuinely one time use. Re-use it and your key is dead. However, how do you get the OTP? Well, it's sitting in the demo page in multiple places, but won't change for some reason. The fix? See that Origin header? Just randomize it lmfao.
import base64
import json
import re
import requests
from urllib.parse import urlparse
from os import urandom
from pywidevine import Cdm, Device, PSSH
DEVICE = Device.load(r"C:\Users\Admin\AppData\Local\devine\WVDs\provision.wvd")
MPD_URL = urlparse("https://d2lrwez4x0gs00.cloudfront.net/media/aWYD1JUhUz8sh/725eff64/stream.mpd")
PSSH_ = "AAAAUnBzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAADIiKnZkb2NpcGhlcjoxYWUzMzFlMzBjZjI0YWQ3OWY4ZTI4ZTk5MjQzNDU3M0jj3JWbBg=="
session = requests.Session()
session.headers.update({
"Origin": f"https://{urandom(8).hex()}.com",
})
def get_otp() -> str:
# this grabs OTP for the vdocipher demo page
# it's important to match headers, e.t.c between the call here and the license call
# so use a requests session to keep hold of the headers, cookies, e.t.c
demo_page = session.get("https://www.vdocipher.com/blog/2014/12/add-text-to-videos-with-watermark/").text
return re.search(r"<div id='vdo(20160313versIND313[^']*)'", demo_page).group(1)
cdm = Cdm.from_device(DEVICE)
session_id = cdm.open()
cdm.set_service_certificate(session_id, cdm.common_privacy_cert)
challenge = cdm.get_license_challenge(
session_id,
PSSH(PSSH_),
privacy_mode=True
)
otp = get_otp()
print("OTP:", otp)
token = {
"otp": otp,
"playbackInfo": "eyJ2aWRlb0lkIjoiYzE0ODBkNmYwNTdiNGFlZjg5YTcwNTc4ZTdmOWQzM2UifQ==",
"href": "https://www.vdocipher.com/blog/2014/12/add-text-to-videos-with-watermark/",
"tech": "wv",
"licenseRequest": base64.b64encode(challenge).decode()
}
print("Token:", token)
token_b64 = base64.b64encode(json.dumps(token).encode()).decode()
req = session.post(
url="https://license.vdocipher.com/auth",
json={
"token": token_b64
}
)
print(req.text)Works like a charm! It seems as though if you keep the origin different each time you request, they won't block your provision.
I'm going to close this now as it's outright confirmed not to be the challenge at cause for the issues. I think I've also given you an ample enough solution to the problem at hand. Feel free to continue the discussion if needed though.
(Also I just deleted two comments that I felt were too unnecessary/spammy, but I see GitHub now makes events, further filling up the discussion log lol)
Many thanks for checking. Sending flowers and wine.
For more discussion, or for someone who will be interested in the future, what remains unclear about auth failed is, that the ban mechanism of the license server looks more sophisticated than just banning the provision.
The device that holds it is not affected by the ban, videos play in chrome. Yet wvd requests from other hosts are all banned, I did requests from remote servers in addition to local computer.
Extracting the provision again, yields the same provision, so the functions keep using the same provision.
I am just saying there is something more behind the scene that led to the confused situation.
Additionally, the emulator devices work too, tested Pixel C, api 29.
To be clear, my randomized origin method worked for you? Did it work on all of your provisions? It seems once blocked, it stays blocked. But yes, you are right, it seems a bit sophisticated on how it is blocking.
Regardless, if you keep using a randomized origin to not get it blocked in the first place, it shouldn't happen again.
You should sniff your device and see how its retrieving the OTP, and what headers/cookies its using as it may be related.
You were so so helpful. I needed someone with the experience to give me the final nudge. The fix was the requesting of OTP inside the script and using the single session between requests. Noticing that getting the page via requests returns the same OTP was a cherry on top.
Before, I was taking OTP from the browser only. Dumped fresh provision, blocked the license server, refreshed page, got OTP, made wvd request. Despite that got auth failed. The sophisticated ban mechanism led me to deadlock, since the provision works on the device but not via wvd therefore I didn't stress dumping new provisions again for further tests. I would try this again to double confirm, but it is not imporant anymore.
In chrome on device, or in chrome/firefox on local computer, you can keep sending the auth request repeatedly to the license server always receiving a success response, no ban. By sending repeatedly, I mean repeating the single request of POST auth, not the page reload.