mirror of
https://github.com/ucmao/media-parser.git
synced 2026-11-20 08:04:56 +08:00
增强好看视频、豆包、小云雀与微博多源解析兼容性与测试用例
This commit is contained in:
@@ -9,9 +9,9 @@
|
||||
<a href="#-核心特性">核心特性</a> •
|
||||
<a href="#-支持的平台矩阵">支持平台</a> •
|
||||
<a href="#-部署指南">部署指南</a> •
|
||||
<a href="#-api-核心接口说明">开发者文档</a> •
|
||||
<a href="#-api-接口">API 接口</a> •
|
||||
<a href="#-自动化测试与健康自检">测试自检</a> •
|
||||
<a href="#-开发者文档与逆向百科">开发文档</a> •
|
||||
<a href="#-开发者文档与逆向百科">架构与逆向</a> •
|
||||
<a href="#-联系作者">联系作者</a>
|
||||
</p>
|
||||
|
||||
@@ -37,58 +37,58 @@ Media-Parser是一款专为短视频创作者与开发者打造的**100%原生
|
||||
|
||||
## 💾 支持的平台矩阵
|
||||
|
||||
| 平台名称 | 作者 | 标题 | 文案 | 封面 | 视频 | 图集 | 多视频 | 实况 | 音频 | 字幕 |
|
||||
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
||||
| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
|
||||
| **视频号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | |
|
||||
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | |
|
||||
| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | |
|
||||
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **可灵AI** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **夸克AI** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | |
|
||||
| **通义千问** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **腾讯元宝** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **闲鱼** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | |
|
||||
| **拼多多** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | |
|
||||
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **汽水音乐** | ✓ | ✓ | | ✓ | ✓ | | | | ✓ | ✓ |
|
||||
| **QQ音乐** | ✓ | ✓ | | ✓ | ✓ | | ✓ | | ✓ | ✓ |
|
||||
| **网易云音乐** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | ✓ |
|
||||
| **酷狗音乐** | ✓ | ✓ | | ✓ | ✓ | | | | ✓ | |
|
||||
| **配音秀** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **松果时刻** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | ✓ | |
|
||||
| **腾讯频道** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **剪映 / CapCut** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | |
|
||||
| **快影** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | ✓ | |
|
||||
| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | |
|
||||
| **微视** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **AcFun** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | |
|
||||
| **今日头条** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | |
|
||||
| **绿洲** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | |
|
||||
| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **全民K歌** | | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **新片场** | ✓ | ✓ | | ✓ | ✓ | | ✓ | | | |
|
||||
| **好看视频** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **梨视频** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **虎牙** | | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **美拍** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **最右** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | |
|
||||
| **番茄小说** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **红果短剧** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **红果漫剧** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **星绘AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **央视** | ✓ | ✓ | | ✓ | ✓ | | | | | |
|
||||
| **央视频** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | |
|
||||
| 平台名称 | 作者 | 标题 | 文案 | 封面 | 视频 | 图集 | 多视频 | 实况 | 音频 | 字幕 | 📖 逆向思路 |
|
||||
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
||||
| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | [查看](docs/parsers/douyin.md) |
|
||||
| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | | [查看](docs/parsers/xiaohongshu.md) |
|
||||
| **视频号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | | [查看](docs/parsers/wechat-channels.md) |
|
||||
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | [查看](docs/parsers/wechat-mp.md) |
|
||||
| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | | [查看](docs/parsers/kuaishou.md) |
|
||||
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | [查看](docs/parsers/bilibili.md) |
|
||||
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | [查看](docs/parsers/doubao.md) |
|
||||
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | [查看](docs/parsers/jimeng.md) |
|
||||
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/xiaoyunque.md) |
|
||||
| **可灵AI** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/kling.md) |
|
||||
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/hailuo.md) |
|
||||
| **夸克AI** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/quark-ai.md) |
|
||||
| **通义千问** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | | | [查看](docs/parsers/qianwen.md) |
|
||||
| **腾讯元宝** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | | | [查看](docs/parsers/yuanbao.md) |
|
||||
| **闲鱼** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/xianyu.md) |
|
||||
| **拼多多** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | | [查看](docs/parsers/pinduoduo.md) |
|
||||
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/soul.md) |
|
||||
| **汽水音乐** | ✓ | ✓ | | ✓ | ✓ | | | | ✓ | ✓ | [查看](docs/parsers/qsmusic.md) |
|
||||
| **QQ音乐** | ✓ | ✓ | | ✓ | ✓ | | ✓ | | ✓ | ✓ | [查看](docs/parsers/qqmusic.md) |
|
||||
| **网易云音乐** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | ✓ | [查看](docs/parsers/netease-music.md) |
|
||||
| **酷狗音乐** | ✓ | ✓ | | ✓ | ✓ | | | | ✓ | | [查看](docs/parsers/kugou-music.md) |
|
||||
| **配音秀** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/peiyinxiu.md) |
|
||||
| **松果时刻** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | ✓ | | [查看](docs/parsers/pinecone-moment.md) |
|
||||
| **腾讯频道** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/tencent-channel.md) |
|
||||
| **剪映 / CapCut** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | | [查看](docs/parsers/jianying.md) |
|
||||
| **快影** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | ✓ | | [查看](docs/parsers/kwaiying.md) |
|
||||
| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | [查看](docs/parsers/pipigaoxiao.md) |
|
||||
| **微视** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/weishi.md) |
|
||||
| **AcFun** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/acfun.md) |
|
||||
| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | [查看](docs/parsers/xigua.md) |
|
||||
| **今日头条** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | [查看](docs/parsers/toutiao.md) |
|
||||
| **绿洲** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/lvzhou.md) |
|
||||
| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/pipixia.md) |
|
||||
| **全民K歌** | | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/quanminkge.md) |
|
||||
| **新片场** | ✓ | ✓ | | ✓ | ✓ | | ✓ | | | | [查看](docs/parsers/xinpianchang.md) |
|
||||
| **好看视频** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/haokan.md) |
|
||||
| **梨视频** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/lishipin.md) |
|
||||
| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/weibo.md) |
|
||||
| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/zhihu.md) |
|
||||
| **虎牙** | | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/huya.md) |
|
||||
| **美拍** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/meipai.md) |
|
||||
| **最右** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | [查看](docs/parsers/zuiyou.md) |
|
||||
| **番茄小说** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/fanqie.md) |
|
||||
| **红果短剧** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/fanqie.md) |
|
||||
| **红果漫剧** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/fanqie.md) |
|
||||
| **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/dewu.md) |
|
||||
| **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/lofter.md) |
|
||||
| **星绘AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/butterflyai.md) |
|
||||
| **央视** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/cctv.md) |
|
||||
| **央视频** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/yangshipin.md) |
|
||||
|
||||
*注:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。
|
||||
|
||||
|
||||
@@ -7,19 +7,23 @@
|
||||
"changdunovel.com": "番茄小说",
|
||||
"kylin.hainanyuyue.com": "红果漫剧",
|
||||
"hainanyuyue.com": "红果漫剧",
|
||||
"xiaohongshu.com": "小红书",
|
||||
"www.xiaohongshu.com": "小红书",
|
||||
"xhslink.com": "小红书",
|
||||
"xhslink.cn": "小红书",
|
||||
"douyin.com": "抖音",
|
||||
"www.douyin.com": "抖音",
|
||||
"v.douyin.com": "抖音",
|
||||
"www.iesdouyin.com": "抖音",
|
||||
"iesdouyin.com": "抖音",
|
||||
"kuaishou.com": "快手",
|
||||
"www.kuaishou.com": "快手",
|
||||
"v.kuaishou.com": "快手",
|
||||
"m.kuaishou.com": "快手",
|
||||
"c.kuaishou.com": "快手",
|
||||
"v.m.chenzhongtech.com": "快手",
|
||||
"m.chenzhongtech.com": "快手",
|
||||
"bilibili.com": "哔哩哔哩",
|
||||
"www.bilibili.com": "哔哩哔哩",
|
||||
"b23.tv": "哔哩哔哩",
|
||||
"m.bilibili.com": "哔哩哔哩",
|
||||
@@ -33,12 +37,16 @@
|
||||
"mbd.baidu.com": "好看视频",
|
||||
"isee.weishi.qq.com": "微视",
|
||||
"video.weishi.qq.com": "微视",
|
||||
"pearvideo.com": "梨视频",
|
||||
"www.pearvideo.com": "梨视频",
|
||||
"h5.pipigx.com": "皮皮搞笑",
|
||||
"h5.ippzone.com": "皮皮搞笑",
|
||||
"acfun.cn": "AcFun",
|
||||
"m.acfun.cn": "AcFun",
|
||||
"www.acfun.cn": "AcFun",
|
||||
"weibo.com": "微博",
|
||||
"www.weibo.com": "微博",
|
||||
"weibo.cn": "微博",
|
||||
"m.weibo.cn": "微博",
|
||||
"video.weibo.com": "微博",
|
||||
"h5.video.weibo.com": "微博",
|
||||
@@ -51,8 +59,10 @@
|
||||
"www.toutiao.com": "今日头条",
|
||||
"toutiao.com": "今日头条",
|
||||
"is.snssdk.com": "今日头条",
|
||||
"zhihu.com": "知乎",
|
||||
"www.zhihu.com": "知乎",
|
||||
"zhuanlan.zhihu.com": "知乎",
|
||||
"huya.com": "虎牙",
|
||||
"www.huya.com": "虎牙",
|
||||
"v.huya.com": "虎牙",
|
||||
"m.huya.com": "虎牙",
|
||||
@@ -69,6 +79,7 @@
|
||||
"xinpianchang.com": "新片场",
|
||||
"izuiyou.com": "最右",
|
||||
"share.xiaochuankeji.cn": "最右",
|
||||
"doubao.com": "豆包",
|
||||
"www.doubao.com": "豆包",
|
||||
"jimeng.jianying.com": "即梦AI",
|
||||
"m.jimeng.jianying.com": "即梦AI",
|
||||
@@ -95,6 +106,7 @@
|
||||
"www.music.163.com": "网易云音乐",
|
||||
"y.music.163.com": "网易云音乐",
|
||||
"fn.music.163.com": "网易云音乐",
|
||||
"kugou.com": "酷狗音乐",
|
||||
"t1.kugou.com": "酷狗音乐",
|
||||
"m.kugou.com": "酷狗音乐",
|
||||
"m3ws.kugou.com": "酷狗音乐",
|
||||
|
||||
+11
-5
@@ -9,31 +9,37 @@
|
||||
* **平台标识**:`豆包`
|
||||
* **支持媒体类型**:
|
||||
* AI 生成 1080P 高清无水印视频 (MP4)
|
||||
* AI 创作音乐原声与音频流 (MP3/M4A/MP4 Audio)
|
||||
* AI 生成图文 / 提示词生图 (PNG/JPEG)
|
||||
* 对话问答正文 (Markdown/纯文本内容)
|
||||
* 对话标题、Prompt 提示词与创作者信息
|
||||
* **常见链接形态**:
|
||||
* 独立视频分享:`https://www.doubao.com/video-sharing?share_id=41356597786354690&video_id=v0d69cg10004d6978e2ljht0i4fdpp00`
|
||||
* 对话历史/线程分享:`https://www.doubao.com/thread/w8293749281`
|
||||
* AI 音乐分享:`https://www.doubao.com/music-sharing?vid=v02292g10006dadu4kq7dld48hgk0dn0&share_id=55311664453668866`
|
||||
* 对话历史/线程分享:`https://www.doubao.com/thread/w8293749281` 或 `https://www.doubao.com/chat/38441154953364226`
|
||||
* **Cookie 依赖**:
|
||||
* **图片与对话正文解析**:**无需 Cookie**(公开元数据直接包含 `image_ori_raw` 高清原图与完整对话文本)。
|
||||
* **视频无水印解析**:**必须配置 Cookie**(1080P 原始流被权限隔离,未登录仅能获取服务端硬压制水印的预览切片)。
|
||||
* **图片、音乐与对话正文解析**:**无需 Cookie**(公开元数据直接包含 `image_ori_raw` 高清原图、音乐音频流与完整对话文本)。
|
||||
* **视频无水印解析**:**建议配置 Cookie**(1080P 原始流被权限隔离,未登录仅能获取服务端硬压制水印的预览切片)。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心逆向流程
|
||||
|
||||
[DoubaoParser](file:///Users/leo/Projects/media-parser/src/parsers/doubao_parser.py) 内部实现了双分支解析引擎与优雅降级策略:
|
||||
[DoubaoParser](file:///Users/leo/Projects/media-parser/src/parsers/doubao_parser.py) 内部实现了多路由分发解析引擎与优雅降级策略:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start["输入豆包分享 URL"] --> Route{"判断 URL 路径"}
|
||||
Route -->|"路径包含 /thread/"| ThreadFlow["1. 会话线程解析"]
|
||||
Route -->|"路径包含 /thread/ 或 /chat/"| ThreadFlow["1. 会话线程解析"]
|
||||
Route -->|"路径为 /video-sharing"| VideoFlow["2. 独立视频解析"]
|
||||
Route -->|"路径为 /music-sharing"| MusicFlow["3. AI 音乐分享解析"]
|
||||
|
||||
ThreadFlow --> ScriptPayload["提取 HTML 中 script 注入的 JSON 负载"]
|
||||
ScriptPayload --> Creations["递归遍历提取 creation 实体 (视频/图像)"]
|
||||
|
||||
MusicFlow --> VidDecrypt["通过 vid 调用 alice/resource/get_video_model 或 get_play_info"]
|
||||
VidDecrypt --> AudioStream["提取原始无损音频直链与高清视频流"]
|
||||
|
||||
VideoFlow --> AuthCheck{"是否配置了 DOUBAO_COOKIE?"}
|
||||
AuthCheck -->|"已配置 Cookie"| SamanthaAPI["调用 samantha/media/get_play_info"]
|
||||
SamanthaAPI --> AES["执行 FPLAY KDF 盐值派生与 AES-128-CBC 解密"]
|
||||
|
||||
+14
-5
@@ -7,22 +7,31 @@
|
||||
## 1. 平台特征与支持能力
|
||||
|
||||
* **平台标识**:`好看视频`
|
||||
* **支持媒体类型**:无水印高清视频 (MP4) / 视频封面 / 标题与作者
|
||||
* **支持媒体类型**:无水印 1080P/超清/高清视频 (MP4) / 视频封面 / 标题与作者
|
||||
* **常见链接形态**:
|
||||
* 网页链接:`https://haokan.baidu.com/v?vid=17831460188721240800`
|
||||
* 百度分享短链:`https://mr.baidu.com/r/22HM1zborMA?f=cp&u=c88bf4dbb5418758`
|
||||
* 百度落地页:`https://mbd.baidu.com/newspage/data/videoshare?nid=sv_10423411367049244882`
|
||||
* 百度搜索视频页:`https://m.baidu.com/video/page?pd=video_page&nid=14766924042149789226&sign=...`
|
||||
* **Cookie 依赖**:无需 Cookie。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心逆向流程
|
||||
|
||||
1. **提取 VID**:从 Query 中提取 `vid` 参数。
|
||||
1. **短链跳转与参数提取**:
|
||||
- 跟踪 `mr.baidu.com` 重定向并保留核心白名单参数(`nid`, `vid`, `id`, `context`, `pd`, `sign`, `word` 等)。
|
||||
2. **提取 HTML 内嵌视频源**:
|
||||
* 请求好看视频播放落地页;
|
||||
* 正则匹配 `playurl` 或 HTML 内的 `<video>` 播放源链接。
|
||||
- **现代版本**:匹配提取 `window.jsonData` 中的 `data.videoInfo`,从 `clarityArr` 按清晰度 rank 降序提取最高画质(如 1080P/超清)视频直链;
|
||||
- **旧版版本**:提取 `window.__PRELOADED_STATE__` 中的 `curVideoMeta.clarityUrl`;
|
||||
- **移动搜索落地页**:从 script 标签内嵌 JSON 中提取协议相对(`//vd...`)播放源并补齐 HTTPS 协议。
|
||||
|
||||
---
|
||||
|
||||
## 3. 测试与验证
|
||||
|
||||
* **样本验证**:`python tests/manual_verify_parsers.py --platform 好看视频`
|
||||
* **单元测试**:[tests/test_haokan_parser.py](file:///Users/leo/Projects/media-parser/tests/test_haokan_parser.py)
|
||||
* **样本验证**:
|
||||
```bash
|
||||
python -m unittest tests/test_haokan_parser.py
|
||||
```
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
* 视频页:`https://video.weibo.com/show?fid=1034:5336219874426938`
|
||||
* 直播/回放页:`https://weibo.com/l/wblive/p/show/1022:2321325311149536575703`
|
||||
* 网页长链:`https://weibo.com/1234567890/Mabcdef`
|
||||
* 移动端带渠道后缀:`https://m.weibo.cn/7753941940/5332536008119716/qq?wm=3333_2001`
|
||||
* **Cookie 依赖**:无需登录 Cookie(内置自动生成临时 Visitor 访客会话)。
|
||||
|
||||
---
|
||||
@@ -24,6 +25,7 @@
|
||||
|
||||
### 2.1 微博 Base62 转换算法 (`mid_to_id`)
|
||||
微博长链中的字符串 ID(如 `Mabcdef`)为 Base62 编码。在请求数据前,解析器通过 `base62_decode` 将其还原为数据库中的真实纯数字 `id`。
|
||||
同时支持识别并提取 `m.weibo.cn/<uid>/<mid>/<channel_subpath>?...` 移动端嵌套路径中的纯数字 mid。
|
||||
|
||||
### 2.2 视频流、直播回放与图文分支提取
|
||||
* **分支 1 (视频专页 `1034:xxx` 与直播回放 `1022:xxx`)**:
|
||||
@@ -39,4 +41,4 @@
|
||||
## 3. 测试与验证
|
||||
|
||||
* **单元测试**:[tests/test_weibo_parser.py](file:///Users/leo/Projects/media-parser/tests/test_weibo_parser.py)
|
||||
* **执行命令**:`pytest tests/test_weibo_parser.py`
|
||||
* **执行命令**:`python -m unittest tests/test_weibo_parser.py`
|
||||
|
||||
@@ -9,9 +9,11 @@
|
||||
* **平台标识**:`小云雀AI`
|
||||
* **支持媒体类型**:
|
||||
* AI 创作图集 (PNG/JPEG)
|
||||
* AI 创作高清视频 (MP4)
|
||||
* 提示词 Prompt、封面与作者信息
|
||||
* **常见链接形态**:
|
||||
* 分享短链:`https://xiaoyunque.jianying.com/s/z_7nWGLGruM/`
|
||||
* 灵感分享长链:`https://xiaoyunque.jianying.com/activities/pippit_share?inspiration_id=...`
|
||||
* **Cookie 依赖**:公开短链**无需配置 Cookie**。
|
||||
|
||||
---
|
||||
@@ -23,12 +25,14 @@
|
||||
|
||||
### 2.2 核心数据接口
|
||||
* **接口地址**:
|
||||
`GET/POST https://xiaoyunque.jianying.com/luckycat/cn/jianying/campaign/v1/pippit/share/landing_page`
|
||||
`POST https://xiaoyunque.jianying.com/luckycat/cn/jianying/campaign/v1/pippit/share/landing_page`
|
||||
* **请求头**:
|
||||
* `User-Agent`: 现代浏览器 UA
|
||||
* `Accept`: `application/json, text/plain, */*`
|
||||
* `Content-Type`: `application/json`
|
||||
* **提取数据**:
|
||||
* 从返回的 `data.item_info` 或 `data.share_info` 中提取高清图集 URL 与封面。
|
||||
* 解析返回的 `data.page_info`,兼容 `generate_page`(生成页)、`inspiration_page`(灵感创作/壁纸)与 `template_page`(模板页);
|
||||
* 从 `item_info` 中提取列表型 `video_info` 视频直链、`image_info` 高清图集及创作者(`nick_name`, `avatar_url`)元数据。
|
||||
|
||||
---
|
||||
|
||||
@@ -37,6 +41,5 @@
|
||||
* **单元测试**:[tests/test_xiaoyunque_parser.py](file:///Users/leo/Projects/media-parser/tests/test_xiaoyunque_parser.py)
|
||||
* **执行测试**:
|
||||
```bash
|
||||
pytest tests/test_xiaoyunque_parser.py
|
||||
python tests/manual_verify_parsers.py --platform 小云雀AI
|
||||
python -m unittest tests/test_xiaoyunque_parser.py
|
||||
```
|
||||
|
||||
@@ -51,16 +51,19 @@ class DoubaoParser(BaseParser):
|
||||
"cover_url": None,
|
||||
"author": None,
|
||||
"image_list": [],
|
||||
"audio_url": None,
|
||||
}
|
||||
self._parse_once()
|
||||
|
||||
def _parse_once(self):
|
||||
path = urlparse(self.real_url).path.rstrip("/")
|
||||
try:
|
||||
if path.startswith("/thread/"):
|
||||
if path.startswith("/thread/") or path.startswith("/chat/"):
|
||||
self.data.update(self._parse_thread())
|
||||
elif path == "/video-sharing":
|
||||
self.data.update(self._parse_video_sharing())
|
||||
elif path == "/music-sharing":
|
||||
self.data.update(self._parse_music_sharing())
|
||||
else:
|
||||
logger.warning(f"Unsupported Doubao share URL: {self.real_url}")
|
||||
except Exception as exc:
|
||||
@@ -241,6 +244,80 @@ class DoubaoParser(BaseParser):
|
||||
"image_list": [],
|
||||
}
|
||||
|
||||
def _parse_music_sharing(self):
|
||||
query = parse_qs(urlparse(self.real_url).query)
|
||||
vid = self._first(query.get("vid")) or self._first(query.get("video_id"))
|
||||
|
||||
headers = dict(self.headers)
|
||||
headers.update({
|
||||
"Content-Type": "application/json",
|
||||
"Origin": "https://www.doubao.com",
|
||||
"Referer": self.real_url,
|
||||
})
|
||||
|
||||
params = {
|
||||
"version_code": "20800",
|
||||
"language": "zh-CN",
|
||||
"device_platform": "web",
|
||||
"aid": "497858",
|
||||
"real_aid": "497858",
|
||||
"pkg_type": "release_version",
|
||||
"samantha_web": "1",
|
||||
"use-olympus-account": "1",
|
||||
}
|
||||
|
||||
audio_urls = []
|
||||
video_urls = []
|
||||
cover_url = None
|
||||
|
||||
if vid:
|
||||
# 1. 优先通过 alice/resource/get_video_model + FPLAY 解密获取无水印流
|
||||
unwatermarked_urls, unwatermarked_poster = self._fetch_unwatermarked_video_by_vid(vid)
|
||||
if unwatermarked_urls:
|
||||
video_urls.extend(unwatermarked_urls)
|
||||
if unwatermarked_poster:
|
||||
cover_url = unwatermarked_poster
|
||||
|
||||
# 2. 次选 samantha/media/get_play_info
|
||||
try:
|
||||
play_resp = self.session.post(
|
||||
self.PLAY_INFO_API,
|
||||
params=params,
|
||||
headers=headers,
|
||||
json={"key": vid},
|
||||
timeout=10,
|
||||
)
|
||||
if play_resp.status_code == 200:
|
||||
play_json = play_resp.json()
|
||||
if play_json.get("code") == 0 and play_json.get("data"):
|
||||
pdata = play_json["data"]
|
||||
orig_info = pdata.get("original_media_info") or {}
|
||||
raw_orig = orig_info.get("main_url")
|
||||
if raw_orig:
|
||||
clean_url = self._sanitize_video_url(raw_orig)
|
||||
if pdata.get("media_type") == "audio":
|
||||
audio_urls.append(clean_url)
|
||||
else:
|
||||
video_urls.append(clean_url)
|
||||
if not cover_url and pdata.get("poster_url"):
|
||||
cover_url = pdata["poster_url"]
|
||||
except Exception as err:
|
||||
logger.debug(f"Doubao samantha get_play_info request failed: {err}")
|
||||
|
||||
final_video_url = video_urls[0] if video_urls else None
|
||||
final_audio_url = audio_urls[0] if audio_urls else final_video_url
|
||||
|
||||
return {
|
||||
"title": "豆包AI音乐分享",
|
||||
"desc": None,
|
||||
"video_url": final_video_url,
|
||||
"video_list": video_urls,
|
||||
"audio_url": final_audio_url,
|
||||
"cover_url": cover_url,
|
||||
"author": None,
|
||||
"image_list": [],
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def _decipher_fplay_url(cls, url_raw: str, key_seed: str) -> str:
|
||||
if not url_raw or not key_seed:
|
||||
@@ -655,3 +732,6 @@ class DoubaoParser(BaseParser):
|
||||
|
||||
def get_image_list(self):
|
||||
return self.data.get("image_list") or []
|
||||
|
||||
def get_audio_url(self):
|
||||
return self.data.get("audio_url")
|
||||
|
||||
+155
-29
@@ -21,55 +21,181 @@ class HaokanParser(BaseParser):
|
||||
self.data = self.fetch_html_data()
|
||||
|
||||
def fetch_html_data(self):
|
||||
self.html_content = self.fetch_html_content()
|
||||
pattern = re.compile(r'window\.__PRELOADED_STATE__\s*=\s*(\{.*\};)', re.DOTALL)
|
||||
json_data = BaseParser.parse_html_data(self.html_content, pattern)
|
||||
return json_data
|
||||
try:
|
||||
resp = self.session.get(self.real_url, headers=self.headers, timeout=10)
|
||||
if resp.status_code == 200:
|
||||
self.html_content = resp.content.decode('utf-8', errors='replace')
|
||||
else:
|
||||
self.html_content = ""
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to fetch HTML content for {self.real_url}: {e}")
|
||||
self.html_content = ""
|
||||
|
||||
if not self.html_content:
|
||||
return None
|
||||
|
||||
# 1. 优先尝试提取现代百度/好看视频使用的 window.jsonData
|
||||
pattern_json_data = re.compile(r'window\.jsonData\s*=\s*(\{.*?\});\s*(?:window\.|\n|<)', re.DOTALL)
|
||||
match = pattern_json_data.search(self.html_content)
|
||||
if not match:
|
||||
pattern_json_data = re.compile(r'window\.jsonData\s*=\s*(\{.*)', re.DOTALL)
|
||||
match = pattern_json_data.search(self.html_content)
|
||||
if match:
|
||||
raw_json = match.group(1).strip()
|
||||
if raw_json.endswith(';'):
|
||||
raw_json = raw_json[:-1].strip()
|
||||
if '</script>' in raw_json:
|
||||
raw_json = raw_json.split('</script>')[0].strip().rstrip(';')
|
||||
try:
|
||||
data_dict = json.loads(raw_json)
|
||||
return json.dumps(data_dict)
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to decode window.jsonData: {e}")
|
||||
|
||||
# 2. 兜底提取旧版 window.__PRELOADED_STATE__
|
||||
pattern_preloaded = re.compile(r'window\.__PRELOADED_STATE__\s*=\s*(\{.*\};)', re.DOTALL)
|
||||
json_data = BaseParser.parse_html_data(self.html_content, pattern_preloaded)
|
||||
if json_data:
|
||||
return json_data
|
||||
|
||||
# 3. 兜底提取百度移动端视频搜索落地页中的 script JSON 数据
|
||||
for s in re.findall(r'<script[^>]*>(.*?)</script>', self.html_content, re.DOTALL):
|
||||
st = s.strip()
|
||||
if st.startswith('{') and st.endswith('}') and ('"url"' in st or '"play_url"' in st or '"vid"' in st):
|
||||
try:
|
||||
parsed_obj = json.loads(st)
|
||||
if isinstance(parsed_obj, dict):
|
||||
inner = parsed_obj.get('data') if isinstance(parsed_obj.get('data'), dict) else parsed_obj
|
||||
if inner.get('url') or inner.get('play_url') or inner.get('videoInfo'):
|
||||
return json.dumps(parsed_obj)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return None
|
||||
|
||||
def _get_data_dict(self):
|
||||
if not self.data:
|
||||
return {}
|
||||
if isinstance(self.data, dict):
|
||||
return self.data
|
||||
try:
|
||||
return json.loads(self.data)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
def get_real_video_url(self):
|
||||
try:
|
||||
data_dict = json.loads(self.data)
|
||||
clarity_url = data_dict.get('curVideoMeta', {}).get('clarityUrl', [])
|
||||
if clarity_url:
|
||||
data_dict = self._get_data_dict()
|
||||
if not data_dict:
|
||||
return None
|
||||
|
||||
# 结构 A: window.jsonData (现代版本)
|
||||
inner_data = data_dict.get('data') if isinstance(data_dict.get('data'), dict) else data_dict
|
||||
video_info = inner_data.get('videoInfo') or inner_data.get('response') or {}
|
||||
if isinstance(video_info, dict):
|
||||
clarity_arr = video_info.get('clarityArr') or video_info.get('clarity_arr') or []
|
||||
if clarity_arr and isinstance(clarity_arr, list):
|
||||
# 按 rank 降序排列选取最高画质流
|
||||
sorted_clarity = sorted(
|
||||
[item for item in clarity_arr if isinstance(item, dict) and item.get('url')],
|
||||
key=lambda x: int(x.get('rank', 0)),
|
||||
reverse=True
|
||||
)
|
||||
if sorted_clarity:
|
||||
url = unquote(sorted_clarity[0]['url']).replace("\\/", "/")
|
||||
return f"https:{url}" if url.startswith("//") else url
|
||||
play_url = video_info.get('play_url') or video_info.get('playurl') or video_info.get('url')
|
||||
if play_url:
|
||||
url = unquote(play_url).replace("\\/", "/")
|
||||
return f"https:{url}" if url.startswith("//") else url
|
||||
|
||||
# 结构 B: window.__PRELOADED_STATE__ (旧版版本)
|
||||
cur_meta = data_dict.get('curVideoMeta') or inner_data.get('curVideoMeta') or {}
|
||||
clarity_url = cur_meta.get('clarityUrl', [])
|
||||
if clarity_url and isinstance(clarity_url, list):
|
||||
video_url = clarity_url[-1].get('url', '')
|
||||
return unquote(video_url).replace("\/", "/")
|
||||
if video_url:
|
||||
url = unquote(video_url).replace("\\/", "/")
|
||||
return f"https:{url}" if url.startswith("//") else url
|
||||
if cur_meta.get('playurl'):
|
||||
url = unquote(cur_meta['playurl']).replace("\\/", "/")
|
||||
return f"https:{url}" if url.startswith("//") else url
|
||||
|
||||
# 结构 C: 搜索落地页直接包含的 url
|
||||
if inner_data.get('url') and isinstance(inner_data['url'], str) and ('.mp4' in inner_data['url'] or 'bdstatic' in inner_data['url']):
|
||||
url = unquote(inner_data['url']).replace("\\/", "/")
|
||||
return f"https:{url}" if url.startswith("//") else url
|
||||
|
||||
return None
|
||||
except (KeyError, json.JSONDecodeError, TypeError) as e:
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to parse Haokan video URL: {e}")
|
||||
return None
|
||||
|
||||
def get_title_content(self):
|
||||
try:
|
||||
data_dict = json.loads(self.data)
|
||||
return data_dict.get('curVideoMeta', {}).get('title', '')
|
||||
except (KeyError, json.JSONDecodeError, TypeError) as e:
|
||||
data_dict = self._get_data_dict()
|
||||
if not data_dict:
|
||||
return ""
|
||||
|
||||
inner_data = data_dict.get('data') if isinstance(data_dict.get('data'), dict) else data_dict
|
||||
video_info = inner_data.get('videoInfo') or inner_data.get('response') or {}
|
||||
if isinstance(video_info, dict) and video_info.get('title'):
|
||||
return video_info['title']
|
||||
if inner_data.get('title'):
|
||||
return inner_data['title']
|
||||
|
||||
cur_meta = data_dict.get('curVideoMeta') or inner_data.get('curVideoMeta') or {}
|
||||
return cur_meta.get('title', '')
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to parse Haokan title: {e}")
|
||||
return ""
|
||||
|
||||
def get_cover_photo_url(self):
|
||||
try:
|
||||
data_dict = json.loads(self.data)
|
||||
cover_url = data_dict.get('curVideoMeta', {}).get('poster', '')
|
||||
return cover_url.replace("\/", "/") if cover_url else ""
|
||||
except (KeyError, json.JSONDecodeError, TypeError) as e:
|
||||
data_dict = self._get_data_dict()
|
||||
if not data_dict:
|
||||
return ""
|
||||
|
||||
inner_data = data_dict.get('data') if isinstance(data_dict.get('data'), dict) else data_dict
|
||||
video_info = inner_data.get('videoInfo') or inner_data.get('response') or {}
|
||||
if isinstance(video_info, dict):
|
||||
cover = video_info.get('posterImage') or video_info.get('poster') or video_info.get('cover')
|
||||
if cover:
|
||||
return cover.replace("\\/", "/")
|
||||
if inner_data.get('poster'):
|
||||
return inner_data['poster'].replace("\\/", "/")
|
||||
|
||||
cur_meta = data_dict.get('curVideoMeta') or inner_data.get('curVideoMeta') or {}
|
||||
cover_url = cur_meta.get('poster', '')
|
||||
return cover_url.replace("\\/", "/") if cover_url else ""
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to parse Haokan cover URL: {e}")
|
||||
return ""
|
||||
|
||||
def get_author_info(self):
|
||||
try:
|
||||
data_dict = json.loads(self.data)
|
||||
# 定位到作者信息节点
|
||||
author_node = data_dict.get('curVideoMeta', {}).get('mth', {})
|
||||
data_dict = self._get_data_dict()
|
||||
if not data_dict:
|
||||
return {}
|
||||
|
||||
# 提取并格式化为统一的字典结构
|
||||
author_info = {
|
||||
'nickname': author_node.get('author_name', ''),
|
||||
# 好看视频的作者唯一ID是 mthid
|
||||
'author_id': str(author_node.get('mthid', '')),
|
||||
# URL中可能会有转义字符,做一个安全替换
|
||||
'avatar': author_node.get('author_photo', '').replace('\\/', '/')
|
||||
}
|
||||
return author_info
|
||||
except (KeyError, json.JSONDecodeError, AttributeError) as e:
|
||||
inner_data = data_dict.get('data') if isinstance(data_dict.get('data'), dict) else data_dict
|
||||
author_node = inner_data.get('author') or {}
|
||||
if isinstance(author_node, dict) and (author_node.get('name') or author_node.get('author_name')):
|
||||
return {
|
||||
'nickname': author_node.get('name') or author_node.get('author_name', ''),
|
||||
'author_id': str(author_node.get('uk') or author_node.get('mthid') or author_node.get('id', '')),
|
||||
'avatar': (author_node.get('icon') or author_node.get('author_photo') or author_node.get('avatar', '')).replace('\\/', '/')
|
||||
}
|
||||
|
||||
cur_meta = data_dict.get('curVideoMeta') or inner_data.get('curVideoMeta') or {}
|
||||
mth_node = cur_meta.get('mth', {})
|
||||
if isinstance(mth_node, dict):
|
||||
return {
|
||||
'nickname': mth_node.get('author_name', ''),
|
||||
'author_id': str(mth_node.get('mthid', '')),
|
||||
'avatar': mth_node.get('author_photo', '').replace('\\/', '/')
|
||||
}
|
||||
return {}
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to parse author info: {e}")
|
||||
return {}
|
||||
|
||||
@@ -71,8 +71,8 @@ class WeiboParser(BaseParser):
|
||||
if match := re.search(r"/show/\d+:(\d+)", self.real_url):
|
||||
return match.group(1)
|
||||
|
||||
# PC URL, like: weibo.com/123456789/O8yqz0I8Q or weibo.com/7928442102/5331959570240710
|
||||
match = re.search(r'weibo\.com/\d+/([a-zA-Z0-9]+)', self.real_url)
|
||||
# PC / Mobile URL, like: weibo.com/123456789/O8yqz0I8Q or m.weibo.cn/7753941940/5332536008119716/qq
|
||||
match = re.search(r'weibo\.(?:com|cn)/\d+/([a-zA-Z0-9]+)(?:/|\?|$)', self.real_url)
|
||||
if match:
|
||||
segment = match.group(1)
|
||||
return segment if segment.isdigit() else mid_to_id(segment)
|
||||
|
||||
@@ -46,8 +46,9 @@ class XiaoyunqueParser(BaseParser):
|
||||
qdict = {k: v[0] for k, v in parse_qs(parsed.query).items()}
|
||||
|
||||
if not qdict and "/s/" in parsed.path:
|
||||
target_fetch_url = url if url.endswith('/') else f"{url}/"
|
||||
response = self.session.get(
|
||||
url,
|
||||
target_fetch_url,
|
||||
headers={"User-Agent": self.USER_AGENT},
|
||||
allow_redirects=True,
|
||||
timeout=15,
|
||||
@@ -78,42 +79,75 @@ class XiaoyunqueParser(BaseParser):
|
||||
@classmethod
|
||||
def _format_data(cls, data):
|
||||
page_info = data.get("page_info") or {}
|
||||
generate_page = page_info.get("generate_page") or {}
|
||||
user_info = generate_page.get("user_info") or {}
|
||||
item_info = generate_page.get("item_info") or {}
|
||||
|
||||
# 寻找命中的有效页面节点(兼容 generate_page, inspiration_page, template_page 等)
|
||||
target_page = {}
|
||||
for key in ("generate_page", "inspiration_page", "template_page", "share_page"):
|
||||
if isinstance(page_info.get(key), dict):
|
||||
target_page = page_info[key]
|
||||
break
|
||||
if not target_page:
|
||||
for val in page_info.values():
|
||||
if isinstance(val, dict) and ("item_info" in val or "user_info" in val):
|
||||
target_page = val
|
||||
break
|
||||
|
||||
user_info = target_page.get("user_info") or {}
|
||||
item_info = target_page.get("item_info") or {}
|
||||
|
||||
title = item_info.get("title") or None
|
||||
desc = item_info.get("desc") or None
|
||||
|
||||
# 图片列表
|
||||
image_info = item_info.get("image_info") or []
|
||||
image_info = item_info.get("image_info") or item_info.get("images") or []
|
||||
image_list = []
|
||||
for img in image_info:
|
||||
if isinstance(img, dict) and img.get("image_url"):
|
||||
image_list.append(img["image_url"])
|
||||
elif isinstance(img, str):
|
||||
image_list.append(img)
|
||||
if isinstance(image_info, list):
|
||||
for img in image_info:
|
||||
if isinstance(img, dict) and (img.get("image_url") or img.get("url")):
|
||||
image_list.append(img.get("image_url") or img.get("url"))
|
||||
elif isinstance(img, str):
|
||||
image_list.append(img)
|
||||
elif isinstance(image_info, dict):
|
||||
img_url = image_info.get("image_url") or image_info.get("url")
|
||||
if img_url:
|
||||
image_list.append(img_url)
|
||||
|
||||
# 视频列表
|
||||
video_url = item_info.get("video_url") or item_info.get("video_play_url")
|
||||
video_info = item_info.get("video_info") or {}
|
||||
if not video_url and isinstance(video_info, dict):
|
||||
video_url = video_info.get("main_url") or video_info.get("video_url")
|
||||
# 视频列表与主视频
|
||||
video_list = []
|
||||
raw_video_url = item_info.get("video_url") or item_info.get("video_play_url")
|
||||
video_info = item_info.get("video_info") or item_info.get("video") or []
|
||||
|
||||
if isinstance(video_info, list):
|
||||
for v in video_info:
|
||||
if isinstance(v, dict):
|
||||
v_url = v.get("video_url") or v.get("main_url") or v.get("url")
|
||||
if v_url and v_url not in video_list:
|
||||
video_list.append(v_url)
|
||||
elif isinstance(v, str) and v not in video_list:
|
||||
video_list.append(v)
|
||||
elif isinstance(video_info, dict):
|
||||
v_url = video_info.get("main_url") or video_info.get("video_url") or video_info.get("url")
|
||||
if v_url and v_url not in video_list:
|
||||
video_list.append(v_url)
|
||||
|
||||
video_list = [video_url] if video_url else []
|
||||
if raw_video_url and raw_video_url not in video_list:
|
||||
video_list.insert(0, raw_video_url)
|
||||
|
||||
primary_video_url = video_list[0] if video_list else None
|
||||
|
||||
# 封面图
|
||||
cover_url = item_info.get("cover_url")
|
||||
cover_url = item_info.get("cover_url") or item_info.get("poster") or item_info.get("cover")
|
||||
|
||||
author = {
|
||||
"nickname": user_info.get("nick_name") or "",
|
||||
"nickname": user_info.get("nick_name") or user_info.get("nickname") or "",
|
||||
"author_id": str(user_info.get("user_id") or user_info.get("sec_uid") or ""),
|
||||
"avatar": user_info.get("avatar_url") or "",
|
||||
"avatar": user_info.get("avatar_url") or user_info.get("avatar") or "",
|
||||
}
|
||||
|
||||
return {
|
||||
"title": title,
|
||||
"desc": desc,
|
||||
"video_url": video_url,
|
||||
"video_url": primary_video_url,
|
||||
"video_list": video_list,
|
||||
"cover_url": cover_url,
|
||||
"author": author,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"schema_version": 2,
|
||||
"description": "50 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)",
|
||||
"total_cases": 310,
|
||||
"total_cases": 315,
|
||||
"cases": [
|
||||
{
|
||||
"platform": "网易云音乐",
|
||||
@@ -4473,7 +4473,9 @@
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: 星绘AI 视频生成作品",
|
||||
"pattern": "AI 视频生成作品"
|
||||
"pattern": "AI 视频生成作品",
|
||||
"status": "expired",
|
||||
"expired_reason": "星绘AI分享Token已失效"
|
||||
},
|
||||
{
|
||||
"platform": "央视",
|
||||
@@ -4518,6 +4520,77 @@
|
||||
],
|
||||
"note": "2026-09-06 已验证: 央视频短链与微视频详情",
|
||||
"pattern": "微视频及短剧详情"
|
||||
},
|
||||
{
|
||||
"platform": "豆包",
|
||||
"url": "https://www.doubao.com/music-sharing?vid=v02292g10006dadu4kq7dld48hgk0dn0&share_id=55311664453668866",
|
||||
"media_types": [
|
||||
"audio",
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"audio",
|
||||
"video",
|
||||
"title"
|
||||
],
|
||||
"note": "2026-09-14 已验证: 豆包 AI 音乐生成分享,提取音频与视频流",
|
||||
"pattern": "AI 音乐分享"
|
||||
},
|
||||
{
|
||||
"platform": "好看视频",
|
||||
"url": "https://mr.baidu.com/r/22HM1zborMA?f=cp&u=c88bf4dbb5418758",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"cover",
|
||||
"title"
|
||||
],
|
||||
"note": "2026-09-14 已验证: 百度分享短链与多清晰度落地页",
|
||||
"pattern": "百度短链与清晰度矩阵"
|
||||
},
|
||||
{
|
||||
"platform": "好看视频",
|
||||
"url": "https://m.baidu.com/video/page?pd=video_page&nid=14766924042149789226&sign=2495427256712467844&word=%E6%9A%97%E5%8C%BA%E7%AA%81%E5%9B%B4S1%20PV&oword=%E6%9A%97%E5%8C%BA%E7%AA%81%E5%9B%B4S1%20PV&atn=index&frsrcid=4185",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"cover",
|
||||
"title"
|
||||
],
|
||||
"note": "2026-09-14 已验证: 百度移动搜索视频落地页",
|
||||
"pattern": "移动端搜索落地页"
|
||||
},
|
||||
{
|
||||
"platform": "小云雀AI",
|
||||
"url": "https://xiaoyunque.jianying.com/s/eguwTcLQIxw/",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-14 已验证: 小云雀灵感创作与壁纸生成分享",
|
||||
"pattern": "灵感创作与壁纸生成"
|
||||
},
|
||||
{
|
||||
"platform": "微博",
|
||||
"url": "https://m.weibo.cn/7753941940/5332536008119716/qq?wm=3333_2001",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"cover",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-14 已验证: 微博移动端带渠道后缀视频博文",
|
||||
"pattern": "移动端渠道后缀博文"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -47,10 +47,15 @@ def collect_media(parser):
|
||||
}
|
||||
|
||||
|
||||
def verify_case(case):
|
||||
def verify_case(case, include_expired=False):
|
||||
platform = case["platform"]
|
||||
pattern = case.get("pattern", "标准链接")
|
||||
url = case.get("url", "")
|
||||
is_marked_expired = case.get("status") == "expired" or case.get("is_expired") is True
|
||||
if is_marked_expired and not include_expired:
|
||||
reason = case.get("expired_reason") or case.get("note") or "用例已标记为失效/下架"
|
||||
return "SKIPPED", platform, pattern, f"跳过已失效用例 ({reason})", url
|
||||
|
||||
if not url:
|
||||
return "MISSING", platform, pattern, "未配置真实链接", url
|
||||
try:
|
||||
@@ -65,18 +70,37 @@ def verify_case(case):
|
||||
expected = case.get("expected_fields", ["title"])
|
||||
missing = [field for field in expected if not found.get(field)]
|
||||
if missing:
|
||||
terminal_detail = getattr(parser, "terminal_error", None) or getattr(parser, "_terminal_filter_detail", None)
|
||||
if isinstance(terminal_detail, dict):
|
||||
detail_msg = terminal_detail.get("detail_msg") or terminal_detail.get("notice") or "作品已被作者删除或设为私密"
|
||||
return "EXPIRED", platform, pattern, f"链接已失效: {detail_msg}", url
|
||||
if getattr(parser, "no_media_in_content", False):
|
||||
return "EXPIRED", platform, pattern, "分享内容仅为纯文本,未包含媒体资源", url
|
||||
return "FAILED", platform, pattern, f"缺少字段:{', '.join(missing)}", url
|
||||
present = ", ".join(name for name, value in found.items() if value)
|
||||
return "PASSED", platform, pattern, f"已取得:{present}", url
|
||||
except Exception as exc:
|
||||
err_msg = str(exc)
|
||||
if "MEDIA_DELETED" in err_msg or "作品权限或已被删除" in err_msg:
|
||||
return "EXPIRED", platform, pattern, f"链接已失效: {type(exc).__name__}: {exc}", url
|
||||
return "FAILED", platform, pattern, f"{type(exc).__name__}: {exc}", url
|
||||
|
||||
|
||||
def save_cases(cases_data):
|
||||
"""回写更新后的用例到 live_parser_samples.json。"""
|
||||
raw_content = json.loads(SAMPLES_PATH.read_text(encoding="utf-8"))
|
||||
raw_content["cases"] = cases_data
|
||||
raw_content["total_cases"] = len(cases_data)
|
||||
SAMPLES_PATH.write_text(json.dumps(raw_content, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def main():
|
||||
ParserFactory._discover()
|
||||
parser = argparse.ArgumentParser(description="一键多形态验证真实分享链接")
|
||||
parser.add_argument("--platform", action="append", help="仅验证指定平台;可重复传入")
|
||||
parser.add_argument("--list-missing", action="store_true", help="仅列出缺少链接的平台")
|
||||
parser.add_argument("--include-expired", action="store_true", help="强制包含标记为失效的用例一起运行")
|
||||
parser.add_argument("--auto-mark-expired", action="store_true", help="自动将检测到已失效的用例标记为 status=expired")
|
||||
parser.add_argument("--limit", type=int, default=0, help="限制每个平台运行的最大用例数 (默认全部)")
|
||||
parser.add_argument("--workers", type=int, default=20, help="并发线程数 (默认 20,0 表示单线程顺序执行)")
|
||||
parser.add_argument("--summary-only", action="store_true", help="仅输出最终汇总,适合全量回归")
|
||||
@@ -84,7 +108,8 @@ def main():
|
||||
if args.summary_only:
|
||||
logging.disable(logging.CRITICAL)
|
||||
|
||||
cases = load_cases()
|
||||
all_cases = load_cases()
|
||||
cases = all_cases
|
||||
if args.platform:
|
||||
requested = set(args.platform)
|
||||
cases = [case for case in cases if case["platform"] in requested]
|
||||
@@ -111,9 +136,17 @@ def main():
|
||||
print(f"🧪 开始执行 {platform_count} 平台自动化多形态链接验证 (共 {len(cases)} 条用例, 线程数: {args.workers if args.workers > 0 else 1})...\n", flush=True)
|
||||
|
||||
results = []
|
||||
status_icons = {
|
||||
"PASSED": "✅ PASSED ",
|
||||
"FAILED": "❌ FAILED ",
|
||||
"EXPIRED": "⚠️ EXPIRED ",
|
||||
"SKIPPED": "⚪ SKIPPED ",
|
||||
"MISSING": "⚪ MISSING ",
|
||||
}
|
||||
|
||||
if args.workers > 1:
|
||||
with ThreadPoolExecutor(max_workers=args.workers) as executor:
|
||||
future_to_case = {executor.submit(verify_case, case): case for case in cases}
|
||||
future_to_case = {executor.submit(verify_case, case, args.include_expired): case for case in cases}
|
||||
for future in as_completed(future_to_case):
|
||||
try:
|
||||
res = future.result()
|
||||
@@ -122,31 +155,49 @@ def main():
|
||||
res = ("FAILED", c["platform"], c.get("pattern", ""), f"{type(exc).__name__}: {exc}", c.get("url", ""))
|
||||
results.append(res)
|
||||
status, platform, pattern, detail, url = res
|
||||
status_tag = f"✅ {status}" if status == "PASSED" else (f"❌ {status}" if status == "FAILED" else f"⚪ {status}")
|
||||
status_tag = status_icons.get(status, f"[{status}]")
|
||||
if not args.summary_only:
|
||||
print(f"{status_tag:<10} [{platform:<6}] ({pattern[:25]}): {detail}", flush=True)
|
||||
print(f"{status_tag} [{platform:<6}] ({pattern[:25]}): {detail}", flush=True)
|
||||
else:
|
||||
for case in cases:
|
||||
res = verify_case(case)
|
||||
res = verify_case(case, args.include_expired)
|
||||
results.append(res)
|
||||
status, platform, pattern, detail, url = res
|
||||
status_tag = f"✅ {status}" if status == "PASSED" else (f"❌ {status}" if status == "FAILED" else f"⚪ {status}")
|
||||
status_tag = status_icons.get(status, f"[{status}]")
|
||||
if not args.summary_only:
|
||||
print(f"{status_tag:<10} [{platform:<6}] ({pattern[:25]}): {detail}", flush=True)
|
||||
print(f"{status_tag} [{platform:<6}] ({pattern[:25]}): {detail}", flush=True)
|
||||
|
||||
summary = {status: sum(1 for result in results if result[0] == status) for status in ("PASSED", "FAILED", "MISSING")}
|
||||
summary = {status: sum(1 for result in results if result[0] == status) for status in ("PASSED", "FAILED", "EXPIRED", "SKIPPED", "MISSING")}
|
||||
print("\n" + "=" * 60, flush=True)
|
||||
print("📊 验证汇总:" + ", ".join(f"{key}={value}" for key, value in summary.items()), flush=True)
|
||||
total = len(results)
|
||||
if total > 0:
|
||||
pass_rate = (summary['PASSED'] / total) * 100
|
||||
print(f"🎯 整体通过率:{pass_rate:.1f}% ({summary['PASSED']}/{total})", flush=True)
|
||||
if args.summary_only:
|
||||
failures = [result for result in results if result[0] != "PASSED"]
|
||||
print("📊 验证汇总:" + ", ".join(f"{key}={value}" for key, value in summary.items() if value > 0 or key in ("PASSED", "FAILED")), flush=True)
|
||||
|
||||
active_total = summary["PASSED"] + summary["FAILED"] + summary["EXPIRED"] + summary["MISSING"]
|
||||
if active_total > 0:
|
||||
pass_rate = (summary["PASSED"] / active_total) * 100
|
||||
print(f"🎯 有效用例通过率:{pass_rate:.1f}% ({summary['PASSED']}/{active_total})", flush=True)
|
||||
if summary.get("SKIPPED", 0) > 0:
|
||||
print(f"💡 已跳过失效/下架用例: {summary['SKIPPED']} 条 (传入 --include-expired 可强制测试)", flush=True)
|
||||
|
||||
if args.summary_only or summary.get("FAILED", 0) > 0 or summary.get("EXPIRED", 0) > 0:
|
||||
failures = [result for result in results if result[0] in ("FAILED", "EXPIRED")]
|
||||
if failures:
|
||||
print("失败用例:", flush=True)
|
||||
for status, platform, pattern, detail, _url in failures:
|
||||
print(f"- [{platform}] {pattern}: {detail}", flush=True)
|
||||
print("\n🚨 需关注/异常用例:", flush=True)
|
||||
for status, platform, pattern, detail, url in failures:
|
||||
print(f"- [{status}] [{platform}] {pattern}: {detail} -> {url}", flush=True)
|
||||
|
||||
if args.auto_mark_expired and summary.get("EXPIRED", 0) > 0:
|
||||
expired_urls = {res[4]: res[3] for res in results if res[0] == "EXPIRED" and res[4]}
|
||||
marked_count = 0
|
||||
for case in all_cases:
|
||||
case_url = case.get("url")
|
||||
if case_url in expired_urls and case.get("status") != "expired":
|
||||
case["status"] = "expired"
|
||||
case["expired_reason"] = expired_urls[case_url]
|
||||
marked_count += 1
|
||||
if marked_count > 0:
|
||||
save_cases(all_cases)
|
||||
print(f"\n💾 已自动将 {marked_count} 条失效链接标记为 status='expired' 并持久化回用例库。", flush=True)
|
||||
|
||||
print("=" * 60, flush=True)
|
||||
|
||||
|
||||
|
||||
@@ -177,11 +177,39 @@ class DoubaoParserTest(unittest.TestCase):
|
||||
with patch("requests.Session.get", return_value=response):
|
||||
parser = DoubaoParser("https://www.doubao.com/thread/pure-text")
|
||||
|
||||
self.assertTrue(parser.no_media_in_content)
|
||||
self.assertIn("请写一首关于秋天的古诗", parser.get_description())
|
||||
self.assertIn("空山新雨后,天气晚来秋", parser.get_description())
|
||||
self.assertIsNone(parser.get_real_video_url())
|
||||
self.assertEqual(parser.get_image_list(), [])
|
||||
def test_music_sharing_extracts_audio_and_video(self):
|
||||
play_info_response = Mock(
|
||||
status_code=200,
|
||||
json=Mock(return_value={
|
||||
"code": 0,
|
||||
"data": {
|
||||
"media_type": "audio",
|
||||
"original_media_info": {
|
||||
"main_url": "https://v9-videoweb.doubao.com/test_audio.mp4"
|
||||
},
|
||||
"poster_url": "https://image.example.com/music_poster.jpg"
|
||||
}
|
||||
})
|
||||
)
|
||||
url = "https://www.doubao.com/music-sharing?vid=v02292g10006dadu4kq7dld48hgk0dn0&share_id=55311664453668866"
|
||||
with patch.object(DoubaoParser, "_fetch_unwatermarked_video_by_vid", return_value=(["https://clean.example.com/audio.mp4"], "https://clean.example.com/poster.jpg")):
|
||||
with patch("requests.Session.post", return_value=play_info_response):
|
||||
parser = DoubaoParser(url)
|
||||
|
||||
self.assertEqual(parser.get_title_content(), "豆包AI音乐分享")
|
||||
self.assertEqual(parser.get_real_video_url(), "https://clean.example.com/audio.mp4")
|
||||
self.assertEqual(parser.get_audio_url(), "https://v9-videoweb.doubao.com/test_audio.mp4")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://clean.example.com/poster.jpg")
|
||||
|
||||
def test_chat_route_supported(self):
|
||||
page = "<html><head></head><body></body></html>"
|
||||
response = Mock(text=page)
|
||||
response.raise_for_status.return_value = None
|
||||
|
||||
with patch("requests.Session.get", return_value=response):
|
||||
parser = DoubaoParser("https://www.doubao.com/chat/38441154953364226")
|
||||
|
||||
self.assertEqual(parser.get_title_content(), "豆包对话分享")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
import json
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from src.parsers.haokan_parser import HaokanParser
|
||||
from utils.web_fetcher import UrlParser
|
||||
|
||||
|
||||
class HaokanParserTest(unittest.TestCase):
|
||||
def test_parses_window_json_data(self):
|
||||
json_data_payload = {
|
||||
"data": {
|
||||
"title": "测试好看视频标题",
|
||||
"videoInfo": {
|
||||
"title": "测试好看视频标题",
|
||||
"posterImage": "https://image.example.com/poster.jpg",
|
||||
"clarityArr": [
|
||||
{"rank": 1, "url": "https://video.example.com/480p.mp4"},
|
||||
{"rank": 3, "url": "https://video.example.com/1080p.mp4"},
|
||||
{"rank": 2, "url": "https://video.example.com/720p.mp4"}
|
||||
]
|
||||
},
|
||||
"author": {
|
||||
"name": "测试作者",
|
||||
"uk": "123456",
|
||||
"icon": "https://image.example.com/avatar.jpg"
|
||||
}
|
||||
}
|
||||
}
|
||||
html_page = (
|
||||
'<html><head>'
|
||||
f'<script>window.jsonData = {json.dumps(json_data_payload)};</script>'
|
||||
'</head><body></body></html>'
|
||||
)
|
||||
response = Mock(status_code=200, content=html_page.encode("utf-8"))
|
||||
|
||||
with patch("requests.Session.get", return_value=response):
|
||||
parser = HaokanParser("https://mbd.baidu.com/newspage/data/videoshare?nid=sv_123456")
|
||||
|
||||
self.assertEqual(parser.get_title_content(), "测试好看视频标题")
|
||||
self.assertEqual(parser.get_real_video_url(), "https://video.example.com/1080p.mp4")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://image.example.com/poster.jpg")
|
||||
self.assertEqual(parser.get_author_info()["nickname"], "测试作者")
|
||||
self.assertEqual(parser.get_author_info()["author_id"], "123456")
|
||||
|
||||
def test_url_parser_extracts_nid_and_sign(self):
|
||||
url = "https://mbd.baidu.com/newspage/data/videoshare?nid=sv_10423411367049244882&noise=123"
|
||||
extracted = UrlParser.extract_video_address(url)
|
||||
self.assertEqual(extracted, "https://mbd.baidu.com/newspage/data/videoshare?nid=sv_10423411367049244882")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -39,6 +39,11 @@ class WeiboParserTest(unittest.TestCase):
|
||||
parser.real_url = "https://m.weibo.cn/detail/O8yqz0I8Q"
|
||||
self.assertEqual(parser._extract_id(), "5020389670169684")
|
||||
|
||||
def test_extracts_id_from_mobile_channel_url(self):
|
||||
parser = WeiboParser.__new__(WeiboParser)
|
||||
parser.real_url = "https://m.weibo.cn/7753941940/5332536008119716/qq?wm=3333_2001"
|
||||
self.assertEqual(parser._extract_id(), "5332536008119716")
|
||||
|
||||
def test_image_list_extraction(self):
|
||||
parser = WeiboParser.__new__(WeiboParser)
|
||||
parser.post_data = {
|
||||
|
||||
@@ -44,6 +44,40 @@ class XiaoyunqueParserTest(unittest.TestCase):
|
||||
"12345",
|
||||
)
|
||||
|
||||
def test_inspiration_page_response(self):
|
||||
response = Mock()
|
||||
response.raise_for_status.return_value = None
|
||||
response.json.return_value = {
|
||||
"err_no": 0,
|
||||
"err_tips": "success",
|
||||
"data": {
|
||||
"page_info": {
|
||||
"inspiration_page": {
|
||||
"user_info": {
|
||||
"nick_name": "茉莱",
|
||||
"avatar_url": "https://p26-passport.byteacctimg.com/avatar.jpg",
|
||||
},
|
||||
"item_info": {
|
||||
"title": "三角洲原图壁纸",
|
||||
"desc": "壁纸生成",
|
||||
"video_info": [
|
||||
{"video_url": "https://v11-xyq-video.jianying.com/video.mp4"}
|
||||
],
|
||||
},
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
url = "https://xiaoyunque.jianying.com/activities/pippit_share?inspiration_id=7684286783865146686"
|
||||
|
||||
with patch("requests.Session.post", return_value=response):
|
||||
parser = XiaoyunqueParser(url)
|
||||
|
||||
self.assertEqual(parser.get_title_content(), "三角洲原图壁纸")
|
||||
self.assertEqual(parser.get_description(), "壁纸生成")
|
||||
self.assertEqual(parser.get_real_video_url(), "https://v11-xyq-video.jianying.com/video.mp4")
|
||||
self.assertEqual(parser.get_author_info()["nickname"], "茉莱")
|
||||
|
||||
def test_url_parser_recognizes_xiaoyunque(self):
|
||||
url = "https://xiaoyunque.jianying.com/s/z_7nWGLGruM/"
|
||||
self.assertEqual(UrlParser.get_platform(url), "小云雀AI")
|
||||
|
||||
@@ -159,12 +159,13 @@ class UrlParser:
|
||||
if platform == '好看视频':
|
||||
query_params = parse_qs(parsed_url.query)
|
||||
preserved_params = []
|
||||
for key in ('vid', 'id', 'context', 'pd', 'src'):
|
||||
for key in ('vid', 'id', 'context', 'pd', 'src', 'nid', 'sign', 'word', 'oword', 'atn', 'frsrcid'):
|
||||
value = query_params.get(key, [None])[0]
|
||||
if value is not None:
|
||||
preserved_params.append((key, value))
|
||||
if preserved_params:
|
||||
address = f"{address}?{urlencode(preserved_params)}"
|
||||
from urllib.parse import quote
|
||||
address = f"{address}?{urlencode(preserved_params, quote_via=quote)}"
|
||||
elif platform == "微视":
|
||||
query_params = parse_qs(parsed_url.query)
|
||||
vid = query_params.get('id', [None])[0] # 使用 get 方法避免 KeyError
|
||||
@@ -203,7 +204,7 @@ class UrlParser:
|
||||
elif platform == "豆包":
|
||||
query_params = parse_qs(parsed_url.query)
|
||||
preserved_params = []
|
||||
for key in ('share_id', 'source_type', 'video_id', 'share_scene'):
|
||||
for key in ('share_id', 'source_type', 'video_id', 'share_scene', 'vid'):
|
||||
value = query_params.get(key, [None])[0]
|
||||
if value is not None:
|
||||
preserved_params.append((key, value))
|
||||
|
||||
Reference in New Issue
Block a user