Pixiv 新 UI 下无法抓取作品
Closed this issue · 2 comments
LenKiMo commented
使用的系统、浏览器、Pxer版本
- Windows 10 1803
- Chrome Dev 71.0.3551.3 64-Bit
- Pxer-Der 7
Bug发生时的地址
https://www.pixiv.net/member_illust.php?id=211515&type=illust
Bug复现的方式
Pixiv 近几日可能正在以 AB 测的方式推送新版用户界面（官方尚未发公告正式宣布或介绍），用户资料页面、作品页面彻底重构。导致 Pxer 无法获取作品数据。
目前尚无法判断有多少用户已切换至新界面，可能开发者一时也无法接触新界面。
新版界面示例：
触发Bug的操作和Bug发生时Pxer的状态
点击 Run 后 Pxer 立即报告错误：PxerApp.executePageTask: taskList is empty
eternal-flame-AD commented
看到了，这个问题会放在一个较高的priority处理的。
P站打算越来越依赖于react了，新版本的dom更加难以优雅的基于view进行抓取
目前如果我们继续坚持纯基于dom inspection的抓取，我想整个代码会有很多magic string和magic number，会很不好看也不可靠
我认为是否可以考虑逐步转移至基于CGI的抓取，现在用户作品页也是类似于探索页，通过后续ajax请求获得数据的，我相信这是P站今后的趋势，也能带来更高的性能和对目标站点更低的负载。我们应该迎合而不是回避
HTTP GET https://www.pixiv.net/ajax/user/211515/profile/all
{
"error":false,
"message":"",
"body":{
"illusts":{
"1098570":null,
...works
},
"manga":{
"8059073":null,
...works
},
"novels":[
],
"mangaSeries":[
],
"novelSeries":[
],
"pickup":[
],
"bookmarkCount":{
"public":{
"illust":1,
"novel":0
},
"private":{
"illust":0,
"novel":0
}
}
}
}
HTTP GET https://www.pixiv.net/ajax/user/211515/profile/illusts?ids[]=65061467&ids[]=65105380&is_manga_top=0
{
"error":false,
"message":"",
"body":{
"works":{
"65061467":{
"id":"65061467",
"title":"角っ子ちゃん",
"illustType":0,
"xRestrict":0,
"url":"https://i.pximg.net/c/250x250_80_a2/img-master/img/2017/09/21/00/13/32/65061467_p0_square1200.jpg",
"tags":[
"オリジナル",
"女の子",
"フォークロア",
"角娘",
"オリジナル1000users入り",
"ボブカット"
],
"userId":"211515",
"width":750,
"height":1125,
"pageCount":1,
"isBookmarkable":true,
"bookmarkData":null
},
"65105380":{
"id":"65105380",
"title":"鼓動の深淵",
"illustType":0,
"xRestrict":0,
"url":"https://i.pximg.net/c/250x250_80_a2/img-master/img/2017/09/24/00/18/10/65105380_p0_square1200.jpg",
"tags":[
"オリジナル",
"女の子",
"猫耳",
"美人",
"オリジナル1000users入り",
"アルビノ"
],
"userId":"211515",
"width":760,
"height":1000,
"pageCount":1,
"isBookmarkable":true,
"bookmarkData":null
}
},
"zoneConfig":{
"header":{
"url":"https://d.pixiv.org/show?zone_id=header&format=js&s=1&up=0&a=20&ng=w&l=zh&K=2a916e5d1e20a&ab_test_digits_first=7&ab_test_digits_second=7&Yuid=gEU0BBA&suid=Pfva4qmjly29jz3z0&num=5b9c8f99651&t=EdHvbaqh4D&t=QoMGZ-ADd1&t=A6lyMOEwVK&t=WFa6D9S0vU&t=Y7S_D7yI9m&t=_FEqdcTJcb&t=-DphRRNG21&t=RTJMXD26Ak&t=U_bD7M6bNc&t=jhcp_KSmm7&t=kMfEqczvrm&t=L5whpJPGGF&t=pFXyQUZWQm&t=g5qKpoRb7G&t=JemYQ6sFs9&t=lBkaym7Y_i&t=x39Z3Y8YmE&t=uusOs0ipBx&t=jhuUT0OJva&t=1Xpow_4E0Y&t=jNY6LUmcR6&t=Bo7J-JbNu0&t=e9EFq9kkOU&t=IDgdYKXRNx&t=0Cw8G7l_R2&t=3W0FvX5WNz&t=JiOF--6n9q&t=ltWGWhi6yl&t=5SPMcXcBSY&t=OPmau8HTOc&t=q3eUobDMJW&t=NXxDJr1D_u&t=X_X0hPGItJ&t=2pZ4K1syEF&t=jH0uD88V6F&t=0Jr43qklGT&t=PwDMGzD6xn&t=eVxus64GZU&t=n7yldlT6Wd&t=H9vtPmr1rZ&t=GsDXLSOMhr&t=p0wL4uRMAe&t=bjvoeQ48k_&t=UxtxZzUi0g&t=5ctTiS0-qW&t=3e4_dJLuSN&t=iHZprwR7oi&t=KxXzuwrjBE&t=XSbCMPF_3_&t=yPNaP3JSNF&t=ofcDCLl35P&t=yRW6jUeLh0&t=-oGijJmC5S&t=cYitwadFW9&t=rTtU3ZTxac&t=WCD7DR8bZ2&t=Jn8UqLm2kA&t=AHxDcFqeqw&t=cz8oHglg-o&t=3deaaPj0p-&t=b2pIuSlV4v&t=_hSAdpN9rx&t=SCKvbXWUzm&t=4pwg6tC91U&t=YJtQxxgwZx&t=svKogfYWcS&t=N7i1r51559&t=-cvUa0HoiX&t=IQ7tgDp_ul&t=4ptuZxmi2Z&t=RrGTPeTw5S&t=_9aPt_C37G&t=vuWaZwFv8Y&t=r_Jjn6Ua2V&t=kP7msdIeEU&t=YX3tU8uRAA&t=OT4SuGenFI&t=jaqkarpwly&t=tUf41RBUn5&t=T4MNOIzhrH&t=qojJ3f98K3&t=ChnYmNJoqJ&t=Zi6jH2w_J0&t=k-EVeilbVq&t=iXASjloBb5&t=m95lKAaPqa&t=6BnLjaN0i_&t=Z1Tr_4YaYU&t=Ru5N44LdSZ&t=SaUUiAESXE&t=kimgK7cJmi&t=kokMsqJMrY&t=h8g1prF0k3&t=Ie2c51_4Sp&t=l9SkxFl1y3&t=BPTuAspoUa&t=1WVq2ZNfjm&t=duOK3fn341&t=j5a5lF_J5h&t=cWT4J9vtRZ"
},
"footer":{
"url":"https://d.pixiv.org/show?zone_id=footer&format=js&s=1&up=0&a=20&ng=w&l=zh&K=2a916e5d1e20a&ab_test_digits_first=7&ab_test_digits_second=7&Yuid=gEU0BBA&suid=Pfva4qmjm6nc48z7r&num=5b9c8f99232&t=EdHvbaqh4D&t=QoMGZ-ADd1&t=A6lyMOEwVK&t=WFa6D9S0vU&t=Y7S_D7yI9m&t=_FEqdcTJcb&t=-DphRRNG21&t=RTJMXD26Ak&t=U_bD7M6bNc&t=jhcp_KSmm7&t=kMfEqczvrm&t=L5whpJPGGF&t=pFXyQUZWQm&t=g5qKpoRb7G&t=JemYQ6sFs9&t=lBkaym7Y_i&t=x39Z3Y8YmE&t=uusOs0ipBx&t=jhuUT0OJva&t=1Xpow_4E0Y&t=jNY6LUmcR6&t=Bo7J-JbNu0&t=e9EFq9kkOU&t=IDgdYKXRNx&t=0Cw8G7l_R2&t=3W0FvX5WNz&t=JiOF--6n9q&t=ltWGWhi6yl&t=5SPMcXcBSY&t=OPmau8HTOc&t=q3eUobDMJW&t=NXxDJr1D_u&t=X_X0hPGItJ&t=2pZ4K1syEF&t=jH0uD88V6F&t=0Jr43qklGT&t=PwDMGzD6xn&t=eVxus64GZU&t=n7yldlT6Wd&t=H9vtPmr1rZ&t=GsDXLSOMhr&t=p0wL4uRMAe&t=bjvoeQ48k_&t=UxtxZzUi0g&t=5ctTiS0-qW&t=3e4_dJLuSN&t=iHZprwR7oi&t=KxXzuwrjBE&t=XSbCMPF_3_&t=yPNaP3JSNF&t=ofcDCLl35P&t=yRW6jUeLh0&t=-oGijJmC5S&t=cYitwadFW9&t=rTtU3ZTxac&t=WCD7DR8bZ2&t=Jn8UqLm2kA&t=AHxDcFqeqw&t=cz8oHglg-o&t=3deaaPj0p-&t=b2pIuSlV4v&t=_hSAdpN9rx&t=SCKvbXWUzm&t=4pwg6tC91U&t=YJtQxxgwZx&t=svKogfYWcS&t=N7i1r51559&t=-cvUa0HoiX&t=IQ7tgDp_ul&t=4ptuZxmi2Z&t=RrGTPeTw5S&t=_9aPt_C37G&t=vuWaZwFv8Y&t=r_Jjn6Ua2V&t=kP7msdIeEU&t=YX3tU8uRAA&t=OT4SuGenFI&t=jaqkarpwly&t=tUf41RBUn5&t=T4MNOIzhrH&t=qojJ3f98K3&t=ChnYmNJoqJ&t=Zi6jH2w_J0&t=k-EVeilbVq&t=iXASjloBb5&t=m95lKAaPqa&t=6BnLjaN0i_&t=Z1Tr_4YaYU&t=Ru5N44LdSZ&t=SaUUiAESXE&t=kimgK7cJmi&t=kokMsqJMrY&t=h8g1prF0k3&t=Ie2c51_4Sp&t=l9SkxFl1y3&t=BPTuAspoUa&t=1WVq2ZNfjm&t=duOK3fn341&t=j5a5lF_J5h&t=cWT4J9vtRZ"
},
"logo":{
"url":"https://d.pixiv.org/show?zone_id=logo_side&format=js&s=1&up=0&a=20&ng=w&l=zh&K=2a916e5d1e20a&ab_test_digits_first=7&ab_test_digits_second=7&Yuid=gEU0BBA&suid=Pfva4qmjmczt0140j&num=5b9c8f9910&t=EdHvbaqh4D&t=QoMGZ-ADd1&t=A6lyMOEwVK&t=WFa6D9S0vU&t=Y7S_D7yI9m&t=_FEqdcTJcb&t=-DphRRNG21&t=RTJMXD26Ak&t=U_bD7M6bNc&t=jhcp_KSmm7&t=kMfEqczvrm&t=L5whpJPGGF&t=pFXyQUZWQm&t=g5qKpoRb7G&t=JemYQ6sFs9&t=lBkaym7Y_i&t=x39Z3Y8YmE&t=uusOs0ipBx&t=jhuUT0OJva&t=1Xpow_4E0Y&t=jNY6LUmcR6&t=Bo7J-JbNu0&t=e9EFq9kkOU&t=IDgdYKXRNx&t=0Cw8G7l_R2&t=3W0FvX5WNz&t=JiOF--6n9q&t=ltWGWhi6yl&t=5SPMcXcBSY&t=OPmau8HTOc&t=q3eUobDMJW&t=NXxDJr1D_u&t=X_X0hPGItJ&t=2pZ4K1syEF&t=jH0uD88V6F&t=0Jr43qklGT&t=PwDMGzD6xn&t=eVxus64GZU&t=n7yldlT6Wd&t=H9vtPmr1rZ&t=GsDXLSOMhr&t=p0wL4uRMAe&t=bjvoeQ48k_&t=UxtxZzUi0g&t=5ctTiS0-qW&t=3e4_dJLuSN&t=iHZprwR7oi&t=KxXzuwrjBE&t=XSbCMPF_3_&t=yPNaP3JSNF&t=ofcDCLl35P&t=yRW6jUeLh0&t=-oGijJmC5S&t=cYitwadFW9&t=rTtU3ZTxac&t=WCD7DR8bZ2&t=Jn8UqLm2kA&t=AHxDcFqeqw&t=cz8oHglg-o&t=3deaaPj0p-&t=b2pIuSlV4v&t=_hSAdpN9rx&t=SCKvbXWUzm&t=4pwg6tC91U&t=YJtQxxgwZx&t=svKogfYWcS&t=N7i1r51559&t=-cvUa0HoiX&t=IQ7tgDp_ul&t=4ptuZxmi2Z&t=RrGTPeTw5S&t=_9aPt_C37G&t=vuWaZwFv8Y&t=r_Jjn6Ua2V&t=kP7msdIeEU&t=YX3tU8uRAA&t=OT4SuGenFI&t=jaqkarpwly&t=tUf41RBUn5&t=T4MNOIzhrH&t=qojJ3f98K3&t=ChnYmNJoqJ&t=Zi6jH2w_J0&t=k-EVeilbVq&t=iXASjloBb5&t=m95lKAaPqa&t=6BnLjaN0i_&t=Z1Tr_4YaYU&t=Ru5N44LdSZ&t=SaUUiAESXE&t=kimgK7cJmi&t=kokMsqJMrY&t=h8g1prF0k3&t=Ie2c51_4Sp&t=l9SkxFl1y3&t=BPTuAspoUa&t=1WVq2ZNfjm&t=duOK3fn341&t=j5a5lF_J5h&t=cWT4J9vtRZ"
}
}
}
}
eternal-flame-AD commented
Resolved in 7.1.1