增强好看视频、豆包、小云雀与微博多源解析兼容性与测试用例

This commit is contained in:
ucmao
2026-09-14 13:41:32 +08:00
parent 532c118211
commit 3bdb3731a9
17 changed files with 667 additions and 150 deletions
+54 -54
View File
@@ -9,9 +9,9 @@
<a href="#-核心特性">核心特性</a> •
<a href="#-支持的平台矩阵">支持平台</a> •
<a href="#-部署指南">部署指南</a> •
<a href="#-api-核心接口说明">开发者文档</a> •
<a href="#-api-接口">API 接口</a> •
<a href="#-自动化测试与健康自检">测试自检</a> •
<a href="#-开发者文档与逆向百科">开发文档</a> •
<a href="#-开发者文档与逆向百科">架构与逆向</a> •
<a href="#-联系作者">联系作者</a>
</p>
@@ -37,58 +37,58 @@ Media-Parser是一款专为短视频创作者与开发者打造的**100%原生
## 💾 支持的平台矩阵
| 平台名称 | 作者 | 标题 | 文案 | 封面 | 视频 | 图集 | 多视频 | 实况 | 音频 | 字幕 |
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
| **视频号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | |
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | |
| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | |
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **可灵AI** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **夸克AI** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | |
| **通义千问** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | | |
| **腾讯元宝** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | | |
| **闲鱼** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | |
| **拼多多** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | |
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **汽水音乐** | ✓ | ✓ | | ✓ | ✓ | | | | ✓ | ✓ |
| **QQ音乐** | ✓ | ✓ | | ✓ | ✓ | | ✓ | | ✓ | ✓ |
| **网易云音乐** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | ✓ |
| **酷狗音乐** | ✓ | ✓ | | ✓ | ✓ | | | | ✓ | |
| **配音秀** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **松果时刻** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | ✓ | |
| **腾讯频道** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **剪映 / CapCut** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | |
| **快影** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | ✓ | |
| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | |
| **微视** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **AcFun** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | |
| **今日头条** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | |
| **绿洲** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | |
| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **全民K歌** | | ✓ | | ✓ | ✓ | | | | | |
| **新片场** | ✓ | ✓ | | ✓ | ✓ | | ✓ | | | |
| **好看视频** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **梨视频** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **虎牙** | | ✓ | | ✓ | ✓ | | | | | |
| **美拍** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **最右** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | |
| **番茄小说** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **红果短剧** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **红果漫剧** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **星绘AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | |
| **央视** | ✓ | ✓ | | ✓ | ✓ | | | | | |
| **央视频** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | |
| 平台名称 | 作者 | 标题 | 文案 | 封面 | 视频 | 图集 | 多视频 | 实况 | 音频 | 字幕 | 📖 逆向思路 |
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | [查看](docs/parsers/douyin.md) |
| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | | [查看](docs/parsers/xiaohongshu.md) |
| **视频号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | | [查看](docs/parsers/wechat-channels.md) |
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | [查看](docs/parsers/wechat-mp.md) |
| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | | [查看](docs/parsers/kuaishou.md) |
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | [查看](docs/parsers/bilibili.md) |
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | [查看](docs/parsers/doubao.md) |
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | [查看](docs/parsers/jimeng.md) |
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/xiaoyunque.md) |
| **可灵AI** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/kling.md) |
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/hailuo.md) |
| **夸克AI** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/quark-ai.md) |
| **通义千问** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | | | [查看](docs/parsers/qianwen.md) |
| **腾讯元宝** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | | | [查看](docs/parsers/yuanbao.md) |
| **闲鱼** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/xianyu.md) |
| **拼多多** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | | [查看](docs/parsers/pinduoduo.md) |
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/soul.md) |
| **汽水音乐** | ✓ | ✓ | | ✓ | ✓ | | | | ✓ | ✓ | [查看](docs/parsers/qsmusic.md) |
| **QQ音乐** | ✓ | ✓ | | ✓ | ✓ | | ✓ | | ✓ | ✓ | [查看](docs/parsers/qqmusic.md) |
| **网易云音乐** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | ✓ | [查看](docs/parsers/netease-music.md) |
| **酷狗音乐** | ✓ | ✓ | | ✓ | ✓ | | | | ✓ | | [查看](docs/parsers/kugou-music.md) |
| **配音秀** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/peiyinxiu.md) |
| **松果时刻** | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | | ✓ | | [查看](docs/parsers/pinecone-moment.md) |
| **腾讯频道** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/tencent-channel.md) |
| **剪映 / CapCut** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | ✓ | | [查看](docs/parsers/jianying.md) |
| **快影** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | ✓ | | [查看](docs/parsers/kwaiying.md) |
| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | [查看](docs/parsers/pipigaoxiao.md) |
| **微视** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/weishi.md) |
| **AcFun** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/acfun.md) |
| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | [查看](docs/parsers/xigua.md) |
| **今日头条** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | [查看](docs/parsers/toutiao.md) |
| **绿洲** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/lvzhou.md) |
| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/pipixia.md) |
| **全民K歌** | | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/quanminkge.md) |
| **新片场** | ✓ | ✓ | | ✓ | ✓ | | ✓ | | | | [查看](docs/parsers/xinpianchang.md) |
| **好看视频** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/haokan.md) |
| **梨视频** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/lishipin.md) |
| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/weibo.md) |
| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/zhihu.md) |
| **虎牙** | | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/huya.md) |
| **美拍** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/meipai.md) |
| **最右** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | [查看](docs/parsers/zuiyou.md) |
| **番茄小说** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/fanqie.md) |
| **红果短剧** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/fanqie.md) |
| **红果漫剧** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/fanqie.md) |
| **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/dewu.md) |
| **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/lofter.md) |
| **星绘AI** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/butterflyai.md) |
| **央视** | ✓ | ✓ | | ✓ | ✓ | | | | | | [查看](docs/parsers/cctv.md) |
| **央视频** | ✓ | ✓ | | ✓ | ✓ | ✓ | | | | | [查看](docs/parsers/yangshipin.md) |
*注:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。
+12
View File
@@ -7,19 +7,23 @@
"changdunovel.com": "番茄小说",
"kylin.hainanyuyue.com": "红果漫剧",
"hainanyuyue.com": "红果漫剧",
"xiaohongshu.com": "小红书",
"www.xiaohongshu.com": "小红书",
"xhslink.com": "小红书",
"xhslink.cn": "小红书",
"douyin.com": "抖音",
"www.douyin.com": "抖音",
"v.douyin.com": "抖音",
"www.iesdouyin.com": "抖音",
"iesdouyin.com": "抖音",
"kuaishou.com": "快手",
"www.kuaishou.com": "快手",
"v.kuaishou.com": "快手",
"m.kuaishou.com": "快手",
"c.kuaishou.com": "快手",
"v.m.chenzhongtech.com": "快手",
"m.chenzhongtech.com": "快手",
"bilibili.com": "哔哩哔哩",
"www.bilibili.com": "哔哩哔哩",
"b23.tv": "哔哩哔哩",
"m.bilibili.com": "哔哩哔哩",
@@ -33,12 +37,16 @@
"mbd.baidu.com": "好看视频",
"isee.weishi.qq.com": "微视",
"video.weishi.qq.com": "微视",
"pearvideo.com": "梨视频",
"www.pearvideo.com": "梨视频",
"h5.pipigx.com": "皮皮搞笑",
"h5.ippzone.com": "皮皮搞笑",
"acfun.cn": "AcFun",
"m.acfun.cn": "AcFun",
"www.acfun.cn": "AcFun",
"weibo.com": "微博",
"www.weibo.com": "微博",
"weibo.cn": "微博",
"m.weibo.cn": "微博",
"video.weibo.com": "微博",
"h5.video.weibo.com": "微博",
@@ -51,8 +59,10 @@
"www.toutiao.com": "今日头条",
"toutiao.com": "今日头条",
"is.snssdk.com": "今日头条",
"zhihu.com": "知乎",
"www.zhihu.com": "知乎",
"zhuanlan.zhihu.com": "知乎",
"huya.com": "虎牙",
"www.huya.com": "虎牙",
"v.huya.com": "虎牙",
"m.huya.com": "虎牙",
@@ -69,6 +79,7 @@
"xinpianchang.com": "新片场",
"izuiyou.com": "最右",
"share.xiaochuankeji.cn": "最右",
"doubao.com": "豆包",
"www.doubao.com": "豆包",
"jimeng.jianying.com": "即梦AI",
"m.jimeng.jianying.com": "即梦AI",
@@ -95,6 +106,7 @@
"www.music.163.com": "网易云音乐",
"y.music.163.com": "网易云音乐",
"fn.music.163.com": "网易云音乐",
"kugou.com": "酷狗音乐",
"t1.kugou.com": "酷狗音乐",
"m.kugou.com": "酷狗音乐",
"m3ws.kugou.com": "酷狗音乐",
+11 -5
View File
@@ -9,31 +9,37 @@
* **平台标识**:`豆包`
* **支持媒体类型**:
* AI 生成 1080P 高清无水印视频 (MP4)
* AI 创作音乐原声与音频流 (MP3/M4A/MP4 Audio)
* AI 生成图文 / 提示词生图 (PNG/JPEG)
* 对话问答正文 (Markdown/纯文本内容)
* 对话标题、Prompt 提示词与创作者信息
* **常见链接形态**:
* 独立视频分享:`https://www.doubao.com/video-sharing?share_id=41356597786354690&video_id=v0d69cg10004d6978e2ljht0i4fdpp00`
* 对话历史/线程分享:`https://www.doubao.com/thread/w8293749281`
* AI 音乐分享:`https://www.doubao.com/music-sharing?vid=v02292g10006dadu4kq7dld48hgk0dn0&share_id=55311664453668866`
* 对话历史/线程分享:`https://www.doubao.com/thread/w8293749281` 或 `https://www.doubao.com/chat/38441154953364226`
* **Cookie 依赖**:
* **图片与对话正文解析**:**无需 Cookie**(公开元数据直接包含 `image_ori_raw` 高清原图与完整对话文本)。
* **视频无水印解析**:**必须配置 Cookie**(1080P 原始流被权限隔离,未登录仅能获取服务端硬压制水印的预览切片)。
* **图片、音乐与对话正文解析**:**无需 Cookie**(公开元数据直接包含 `image_ori_raw` 高清原图、音乐音频流与完整对话文本)。
* **视频无水印解析**:**建议配置 Cookie**(1080P 原始流被权限隔离,未登录仅能获取服务端硬压制水印的预览切片)。
---
## 2. 核心逆向流程
[DoubaoParser](file:///Users/leo/Projects/media-parser/src/parsers/doubao_parser.py) 内部实现了双分支解析引擎与优雅降级策略:
[DoubaoParser](file:///Users/leo/Projects/media-parser/src/parsers/doubao_parser.py) 内部实现了多路由分发解析引擎与优雅降级策略:
```mermaid
flowchart TD
Start["输入豆包分享 URL"] --> Route{"判断 URL 路径"}
Route -->|"路径包含 /thread/"| ThreadFlow["1. 会话线程解析"]
Route -->|"路径包含 /thread/ 或 /chat/"| ThreadFlow["1. 会话线程解析"]
Route -->|"路径为 /video-sharing"| VideoFlow["2. 独立视频解析"]
Route -->|"路径为 /music-sharing"| MusicFlow["3. AI 音乐分享解析"]
ThreadFlow --> ScriptPayload["提取 HTML 中 script 注入的 JSON 负载"]
ScriptPayload --> Creations["递归遍历提取 creation 实体 (视频/图像)"]
MusicFlow --> VidDecrypt["通过 vid 调用 alice/resource/get_video_model 或 get_play_info"]
VidDecrypt --> AudioStream["提取原始无损音频直链与高清视频流"]
VideoFlow --> AuthCheck{"是否配置了 DOUBAO_COOKIE?"}
AuthCheck -->|"已配置 Cookie"| SamanthaAPI["调用 samantha/media/get_play_info"]
SamanthaAPI --> AES["执行 FPLAY KDF 盐值派生与 AES-128-CBC 解密"]
+14 -5
View File
@@ -7,22 +7,31 @@
## 1. 平台特征与支持能力
* **平台标识**:`好看视频`
* **支持媒体类型**:无水印高清视频 (MP4) / 视频封面 / 标题与作者
* **支持媒体类型**:无水印 1080P/超清/高清视频 (MP4) / 视频封面 / 标题与作者
* **常见链接形态**:
* 网页链接:`https://haokan.baidu.com/v?vid=17831460188721240800`
* 百度分享短链:`https://mr.baidu.com/r/22HM1zborMA?f=cp&u=c88bf4dbb5418758`
* 百度落地页:`https://mbd.baidu.com/newspage/data/videoshare?nid=sv_10423411367049244882`
* 百度搜索视频页:`https://m.baidu.com/video/page?pd=video_page&nid=14766924042149789226&sign=...`
* **Cookie 依赖**:无需 Cookie。
---
## 2. 核心逆向流程
1. **提取 VID**:从 Query 中提取 `vid` 参数。
1. **短链跳转与参数提取**:
- 跟踪 `mr.baidu.com` 重定向并保留核心白名单参数(`nid`, `vid`, `id`, `context`, `pd`, `sign`, `word` 等)。
2. **提取 HTML 内嵌视频源**:
* 请求好看视频播放落地页;
* 正则匹配 `playurl` 或 HTML 内的 `<video>` 播放源链接。
- **现代版本**:匹配提取 `window.jsonData` 中的 `data.videoInfo`,从 `clarityArr` 按清晰度 rank 降序提取最高画质(如 1080P/超清)视频直链;
- **旧版版本**:提取 `window.__PRELOADED_STATE__` 中的 `curVideoMeta.clarityUrl`;
- **移动搜索落地页**:从 script 标签内嵌 JSON 中提取协议相对(`//vd...`)播放源并补齐 HTTPS 协议。
---
## 3. 测试与验证
* **样本验证**:`python tests/manual_verify_parsers.py --platform 好看视频`
* **单元测试**:[tests/test_haokan_parser.py](file:///Users/leo/Projects/media-parser/tests/test_haokan_parser.py)
* **样本验证**:
```bash
python -m unittest tests/test_haokan_parser.py
```
+3 -1
View File
@@ -16,6 +16,7 @@
* 视频页:`https://video.weibo.com/show?fid=1034:5336219874426938`
* 直播/回放页:`https://weibo.com/l/wblive/p/show/1022:2321325311149536575703`
* 网页长链:`https://weibo.com/1234567890/Mabcdef`
* 移动端带渠道后缀:`https://m.weibo.cn/7753941940/5332536008119716/qq?wm=3333_2001`
* **Cookie 依赖**:无需登录 Cookie(内置自动生成临时 Visitor 访客会话)。
---
@@ -24,6 +25,7 @@
### 2.1 微博 Base62 转换算法 (`mid_to_id`)
微博长链中的字符串 ID(如 `Mabcdef`)为 Base62 编码。在请求数据前,解析器通过 `base62_decode` 将其还原为数据库中的真实纯数字 `id`。
同时支持识别并提取 `m.weibo.cn/<uid>/<mid>/<channel_subpath>?...` 移动端嵌套路径中的纯数字 mid。
### 2.2 视频流、直播回放与图文分支提取
* **分支 1 (视频专页 `1034:xxx` 与直播回放 `1022:xxx`)**:
@@ -39,4 +41,4 @@
## 3. 测试与验证
* **单元测试**:[tests/test_weibo_parser.py](file:///Users/leo/Projects/media-parser/tests/test_weibo_parser.py)
* **执行命令**:`pytest tests/test_weibo_parser.py`
* **执行命令**:`python -m unittest tests/test_weibo_parser.py`
+7 -4
View File
@@ -9,9 +9,11 @@
* **平台标识**:`小云雀AI`
* **支持媒体类型**:
* AI 创作图集 (PNG/JPEG)
* AI 创作高清视频 (MP4)
* 提示词 Prompt、封面与作者信息
* **常见链接形态**:
* 分享短链:`https://xiaoyunque.jianying.com/s/z_7nWGLGruM/`
* 灵感分享长链:`https://xiaoyunque.jianying.com/activities/pippit_share?inspiration_id=...`
* **Cookie 依赖**:公开短链**无需配置 Cookie**。
---
@@ -23,12 +25,14 @@
### 2.2 核心数据接口
* **接口地址**:
`GET/POST https://xiaoyunque.jianying.com/luckycat/cn/jianying/campaign/v1/pippit/share/landing_page`
`POST https://xiaoyunque.jianying.com/luckycat/cn/jianying/campaign/v1/pippit/share/landing_page`
* **请求头**:
* `User-Agent`: 现代浏览器 UA
* `Accept`: `application/json, text/plain, */*`
* `Content-Type`: `application/json`
* **提取数据**:
* 从返回的 `data.item_info` 或 `data.share_info` 中提取高清图集 URL 与封面。
* 解析返回的 `data.page_info`,兼容 `generate_page`(生成页)、`inspiration_page`(灵感创作/壁纸)与 `template_page`(模板页);
* 从 `item_info` 中提取列表型 `video_info` 视频直链、`image_info` 高清图集及创作者(`nick_name`, `avatar_url`)元数据。
---
@@ -37,6 +41,5 @@
* **单元测试**:[tests/test_xiaoyunque_parser.py](file:///Users/leo/Projects/media-parser/tests/test_xiaoyunque_parser.py)
* **执行测试**:
```bash
pytest tests/test_xiaoyunque_parser.py
python tests/manual_verify_parsers.py --platform 小云雀AI
python -m unittest tests/test_xiaoyunque_parser.py
```
+81 -1
View File
@@ -51,16 +51,19 @@ class DoubaoParser(BaseParser):
"cover_url": None,
"author": None,
"image_list": [],
"audio_url": None,
}
self._parse_once()
def _parse_once(self):
path = urlparse(self.real_url).path.rstrip("/")
try:
if path.startswith("/thread/"):
if path.startswith("/thread/") or path.startswith("/chat/"):
self.data.update(self._parse_thread())
elif path == "/video-sharing":
self.data.update(self._parse_video_sharing())
elif path == "/music-sharing":
self.data.update(self._parse_music_sharing())
else:
logger.warning(f"Unsupported Doubao share URL: {self.real_url}")
except Exception as exc:
@@ -241,6 +244,80 @@ class DoubaoParser(BaseParser):
"image_list": [],
}
def _parse_music_sharing(self):
query = parse_qs(urlparse(self.real_url).query)
vid = self._first(query.get("vid")) or self._first(query.get("video_id"))
headers = dict(self.headers)
headers.update({
"Content-Type": "application/json",
"Origin": "https://www.doubao.com",
"Referer": self.real_url,
})
params = {
"version_code": "20800",
"language": "zh-CN",
"device_platform": "web",
"aid": "497858",
"real_aid": "497858",
"pkg_type": "release_version",
"samantha_web": "1",
"use-olympus-account": "1",
}
audio_urls = []
video_urls = []
cover_url = None
if vid:
# 1. 优先通过 alice/resource/get_video_model + FPLAY 解密获取无水印流
unwatermarked_urls, unwatermarked_poster = self._fetch_unwatermarked_video_by_vid(vid)
if unwatermarked_urls:
video_urls.extend(unwatermarked_urls)
if unwatermarked_poster:
cover_url = unwatermarked_poster
# 2. 次选 samantha/media/get_play_info
try:
play_resp = self.session.post(
self.PLAY_INFO_API,
params=params,
headers=headers,
json={"key": vid},
timeout=10,
)
if play_resp.status_code == 200:
play_json = play_resp.json()
if play_json.get("code") == 0 and play_json.get("data"):
pdata = play_json["data"]
orig_info = pdata.get("original_media_info") or {}
raw_orig = orig_info.get("main_url")
if raw_orig:
clean_url = self._sanitize_video_url(raw_orig)
if pdata.get("media_type") == "audio":
audio_urls.append(clean_url)
else:
video_urls.append(clean_url)
if not cover_url and pdata.get("poster_url"):
cover_url = pdata["poster_url"]
except Exception as err:
logger.debug(f"Doubao samantha get_play_info request failed: {err}")
final_video_url = video_urls[0] if video_urls else None
final_audio_url = audio_urls[0] if audio_urls else final_video_url
return {
"title": "豆包AI音乐分享",
"desc": None,
"video_url": final_video_url,
"video_list": video_urls,
"audio_url": final_audio_url,
"cover_url": cover_url,
"author": None,
"image_list": [],
}
@classmethod
def _decipher_fplay_url(cls, url_raw: str, key_seed: str) -> str:
if not url_raw or not key_seed:
@@ -655,3 +732,6 @@ class DoubaoParser(BaseParser):
def get_image_list(self):
return self.data.get("image_list") or []
def get_audio_url(self):
return self.data.get("audio_url")
+155 -29
View File
@@ -21,55 +21,181 @@ class HaokanParser(BaseParser):
self.data = self.fetch_html_data()
def fetch_html_data(self):
self.html_content = self.fetch_html_content()
pattern = re.compile(r'window\.__PRELOADED_STATE__\s*=\s*(\{.*\};)', re.DOTALL)
json_data = BaseParser.parse_html_data(self.html_content, pattern)
return json_data
try:
resp = self.session.get(self.real_url, headers=self.headers, timeout=10)
if resp.status_code == 200:
self.html_content = resp.content.decode('utf-8', errors='replace')
else:
self.html_content = ""
except Exception as e:
logger.warning(f"Failed to fetch HTML content for {self.real_url}: {e}")
self.html_content = ""
if not self.html_content:
return None
# 1. 优先尝试提取现代百度/好看视频使用的 window.jsonData
pattern_json_data = re.compile(r'window\.jsonData\s*=\s*(\{.*?\});\s*(?:window\.|\n|<)', re.DOTALL)
match = pattern_json_data.search(self.html_content)
if not match:
pattern_json_data = re.compile(r'window\.jsonData\s*=\s*(\{.*)', re.DOTALL)
match = pattern_json_data.search(self.html_content)
if match:
raw_json = match.group(1).strip()
if raw_json.endswith(';'):
raw_json = raw_json[:-1].strip()
if '</script>' in raw_json:
raw_json = raw_json.split('</script>')[0].strip().rstrip(';')
try:
data_dict = json.loads(raw_json)
return json.dumps(data_dict)
except Exception as e:
logger.debug(f"Failed to decode window.jsonData: {e}")
# 2. 兜底提取旧版 window.__PRELOADED_STATE__
pattern_preloaded = re.compile(r'window\.__PRELOADED_STATE__\s*=\s*(\{.*\};)', re.DOTALL)
json_data = BaseParser.parse_html_data(self.html_content, pattern_preloaded)
if json_data:
return json_data
# 3. 兜底提取百度移动端视频搜索落地页中的 script JSON 数据
for s in re.findall(r'<script[^>]*>(.*?)</script>', self.html_content, re.DOTALL):
st = s.strip()
if st.startswith('{') and st.endswith('}') and ('"url"' in st or '"play_url"' in st or '"vid"' in st):
try:
parsed_obj = json.loads(st)
if isinstance(parsed_obj, dict):
inner = parsed_obj.get('data') if isinstance(parsed_obj.get('data'), dict) else parsed_obj
if inner.get('url') or inner.get('play_url') or inner.get('videoInfo'):
return json.dumps(parsed_obj)
except Exception:
pass
return None
def _get_data_dict(self):
if not self.data:
return {}
if isinstance(self.data, dict):
return self.data
try:
return json.loads(self.data)
except Exception:
return {}
def get_real_video_url(self):
try:
data_dict = json.loads(self.data)
clarity_url = data_dict.get('curVideoMeta', {}).get('clarityUrl', [])
if clarity_url:
data_dict = self._get_data_dict()
if not data_dict:
return None
# 结构 A: window.jsonData (现代版本)
inner_data = data_dict.get('data') if isinstance(data_dict.get('data'), dict) else data_dict
video_info = inner_data.get('videoInfo') or inner_data.get('response') or {}
if isinstance(video_info, dict):
clarity_arr = video_info.get('clarityArr') or video_info.get('clarity_arr') or []
if clarity_arr and isinstance(clarity_arr, list):
# 按 rank 降序排列选取最高画质流
sorted_clarity = sorted(
[item for item in clarity_arr if isinstance(item, dict) and item.get('url')],
key=lambda x: int(x.get('rank', 0)),
reverse=True
)
if sorted_clarity:
url = unquote(sorted_clarity[0]['url']).replace("\\/", "/")
return f"https:{url}" if url.startswith("//") else url
play_url = video_info.get('play_url') or video_info.get('playurl') or video_info.get('url')
if play_url:
url = unquote(play_url).replace("\\/", "/")
return f"https:{url}" if url.startswith("//") else url
# 结构 B: window.__PRELOADED_STATE__ (旧版版本)
cur_meta = data_dict.get('curVideoMeta') or inner_data.get('curVideoMeta') or {}
clarity_url = cur_meta.get('clarityUrl', [])
if clarity_url and isinstance(clarity_url, list):
video_url = clarity_url[-1].get('url', '')
return unquote(video_url).replace("\/", "/")
if video_url:
url = unquote(video_url).replace("\\/", "/")
return f"https:{url}" if url.startswith("//") else url
if cur_meta.get('playurl'):
url = unquote(cur_meta['playurl']).replace("\\/", "/")
return f"https:{url}" if url.startswith("//") else url
# 结构 C: 搜索落地页直接包含的 url
if inner_data.get('url') and isinstance(inner_data['url'], str) and ('.mp4' in inner_data['url'] or 'bdstatic' in inner_data['url']):
url = unquote(inner_data['url']).replace("\\/", "/")
return f"https:{url}" if url.startswith("//") else url
return None
except (KeyError, json.JSONDecodeError, TypeError) as e:
except Exception as e:
logger.warning(f"Failed to parse Haokan video URL: {e}")
return None
def get_title_content(self):
try:
data_dict = json.loads(self.data)
return data_dict.get('curVideoMeta', {}).get('title', '')
except (KeyError, json.JSONDecodeError, TypeError) as e:
data_dict = self._get_data_dict()
if not data_dict:
return ""
inner_data = data_dict.get('data') if isinstance(data_dict.get('data'), dict) else data_dict
video_info = inner_data.get('videoInfo') or inner_data.get('response') or {}
if isinstance(video_info, dict) and video_info.get('title'):
return video_info['title']
if inner_data.get('title'):
return inner_data['title']
cur_meta = data_dict.get('curVideoMeta') or inner_data.get('curVideoMeta') or {}
return cur_meta.get('title', '')
except Exception as e:
logger.warning(f"Failed to parse Haokan title: {e}")
return ""
def get_cover_photo_url(self):
try:
data_dict = json.loads(self.data)
cover_url = data_dict.get('curVideoMeta', {}).get('poster', '')
return cover_url.replace("\/", "/") if cover_url else ""
except (KeyError, json.JSONDecodeError, TypeError) as e:
data_dict = self._get_data_dict()
if not data_dict:
return ""
inner_data = data_dict.get('data') if isinstance(data_dict.get('data'), dict) else data_dict
video_info = inner_data.get('videoInfo') or inner_data.get('response') or {}
if isinstance(video_info, dict):
cover = video_info.get('posterImage') or video_info.get('poster') or video_info.get('cover')
if cover:
return cover.replace("\\/", "/")
if inner_data.get('poster'):
return inner_data['poster'].replace("\\/", "/")
cur_meta = data_dict.get('curVideoMeta') or inner_data.get('curVideoMeta') or {}
cover_url = cur_meta.get('poster', '')
return cover_url.replace("\\/", "/") if cover_url else ""
except Exception as e:
logger.warning(f"Failed to parse Haokan cover URL: {e}")
return ""
def get_author_info(self):
try:
data_dict = json.loads(self.data)
# 定位到作者信息节点
author_node = data_dict.get('curVideoMeta', {}).get('mth', {})
data_dict = self._get_data_dict()
if not data_dict:
return {}
# 提取并格式化为统一的字典结构
author_info = {
'nickname': author_node.get('author_name', ''),
# 好看视频的作者唯一ID是 mthid
'author_id': str(author_node.get('mthid', '')),
# URL中可能会有转义字符,做一个安全替换
'avatar': author_node.get('author_photo', '').replace('\\/', '/')
}
return author_info
except (KeyError, json.JSONDecodeError, AttributeError) as e:
inner_data = data_dict.get('data') if isinstance(data_dict.get('data'), dict) else data_dict
author_node = inner_data.get('author') or {}
if isinstance(author_node, dict) and (author_node.get('name') or author_node.get('author_name')):
return {
'nickname': author_node.get('name') or author_node.get('author_name', ''),
'author_id': str(author_node.get('uk') or author_node.get('mthid') or author_node.get('id', '')),
'avatar': (author_node.get('icon') or author_node.get('author_photo') or author_node.get('avatar', '')).replace('\\/', '/')
}
cur_meta = data_dict.get('curVideoMeta') or inner_data.get('curVideoMeta') or {}
mth_node = cur_meta.get('mth', {})
if isinstance(mth_node, dict):
return {
'nickname': mth_node.get('author_name', ''),
'author_id': str(mth_node.get('mthid', '')),
'avatar': mth_node.get('author_photo', '').replace('\\/', '/')
}
return {}
except Exception as e:
logger.warning(f"Failed to parse author info: {e}")
return {}
+2 -2
View File
@@ -71,8 +71,8 @@ class WeiboParser(BaseParser):
if match := re.search(r"/show/\d+:(\d+)", self.real_url):
return match.group(1)
# PC URL, like: weibo.com/123456789/O8yqz0I8Q or weibo.com/7928442102/5331959570240710
match = re.search(r'weibo\.com/\d+/([a-zA-Z0-9]+)', self.real_url)
# PC / Mobile URL, like: weibo.com/123456789/O8yqz0I8Q or m.weibo.cn/7753941940/5332536008119716/qq
match = re.search(r'weibo\.(?:com|cn)/\d+/([a-zA-Z0-9]+)(?:/|\?|$)', self.real_url)
if match:
segment = match.group(1)
return segment if segment.isdigit() else mid_to_id(segment)
+54 -20
View File
@@ -46,8 +46,9 @@ class XiaoyunqueParser(BaseParser):
qdict = {k: v[0] for k, v in parse_qs(parsed.query).items()}
if not qdict and "/s/" in parsed.path:
target_fetch_url = url if url.endswith('/') else f"{url}/"
response = self.session.get(
url,
target_fetch_url,
headers={"User-Agent": self.USER_AGENT},
allow_redirects=True,
timeout=15,
@@ -78,42 +79,75 @@ class XiaoyunqueParser(BaseParser):
@classmethod
def _format_data(cls, data):
page_info = data.get("page_info") or {}
generate_page = page_info.get("generate_page") or {}
user_info = generate_page.get("user_info") or {}
item_info = generate_page.get("item_info") or {}
# 寻找命中的有效页面节点(兼容 generate_page, inspiration_page, template_page 等)
target_page = {}
for key in ("generate_page", "inspiration_page", "template_page", "share_page"):
if isinstance(page_info.get(key), dict):
target_page = page_info[key]
break
if not target_page:
for val in page_info.values():
if isinstance(val, dict) and ("item_info" in val or "user_info" in val):
target_page = val
break
user_info = target_page.get("user_info") or {}
item_info = target_page.get("item_info") or {}
title = item_info.get("title") or None
desc = item_info.get("desc") or None
# 图片列表
image_info = item_info.get("image_info") or []
image_info = item_info.get("image_info") or item_info.get("images") or []
image_list = []
for img in image_info:
if isinstance(img, dict) and img.get("image_url"):
image_list.append(img["image_url"])
elif isinstance(img, str):
image_list.append(img)
if isinstance(image_info, list):
for img in image_info:
if isinstance(img, dict) and (img.get("image_url") or img.get("url")):
image_list.append(img.get("image_url") or img.get("url"))
elif isinstance(img, str):
image_list.append(img)
elif isinstance(image_info, dict):
img_url = image_info.get("image_url") or image_info.get("url")
if img_url:
image_list.append(img_url)
# 视频列表
video_url = item_info.get("video_url") or item_info.get("video_play_url")
video_info = item_info.get("video_info") or {}
if not video_url and isinstance(video_info, dict):
video_url = video_info.get("main_url") or video_info.get("video_url")
# 视频列表与主视频
video_list = []
raw_video_url = item_info.get("video_url") or item_info.get("video_play_url")
video_info = item_info.get("video_info") or item_info.get("video") or []
if isinstance(video_info, list):
for v in video_info:
if isinstance(v, dict):
v_url = v.get("video_url") or v.get("main_url") or v.get("url")
if v_url and v_url not in video_list:
video_list.append(v_url)
elif isinstance(v, str) and v not in video_list:
video_list.append(v)
elif isinstance(video_info, dict):
v_url = video_info.get("main_url") or video_info.get("video_url") or video_info.get("url")
if v_url and v_url not in video_list:
video_list.append(v_url)
video_list = [video_url] if video_url else []
if raw_video_url and raw_video_url not in video_list:
video_list.insert(0, raw_video_url)
primary_video_url = video_list[0] if video_list else None
# 封面图
cover_url = item_info.get("cover_url")
cover_url = item_info.get("cover_url") or item_info.get("poster") or item_info.get("cover")
author = {
"nickname": user_info.get("nick_name") or "",
"nickname": user_info.get("nick_name") or user_info.get("nickname") or "",
"author_id": str(user_info.get("user_id") or user_info.get("sec_uid") or ""),
"avatar": user_info.get("avatar_url") or "",
"avatar": user_info.get("avatar_url") or user_info.get("avatar") or "",
}
return {
"title": title,
"desc": desc,
"video_url": video_url,
"video_url": primary_video_url,
"video_list": video_list,
"cover_url": cover_url,
"author": author,
+75 -2
View File
@@ -1,7 +1,7 @@
{
"schema_version": 2,
"description": "50 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)",
"total_cases": 310,
"total_cases": 315,
"cases": [
{
"platform": "网易云音乐",
@@ -4473,7 +4473,9 @@
"author"
],
"note": "2026-09-06 已验证: 星绘AI 视频生成作品",
"pattern": "AI 视频生成作品"
"pattern": "AI 视频生成作品",
"status": "expired",
"expired_reason": "星绘AI分享Token已失效"
},
{
"platform": "央视",
@@ -4518,6 +4520,77 @@
],
"note": "2026-09-06 已验证: 央视频短链与微视频详情",
"pattern": "微视频及短剧详情"
},
{
"platform": "豆包",
"url": "https://www.doubao.com/music-sharing?vid=v02292g10006dadu4kq7dld48hgk0dn0&share_id=55311664453668866",
"media_types": [
"audio",
"video"
],
"expected_fields": [
"audio",
"video",
"title"
],
"note": "2026-09-14 已验证: 豆包 AI 音乐生成分享,提取音频与视频流",
"pattern": "AI 音乐分享"
},
{
"platform": "好看视频",
"url": "https://mr.baidu.com/r/22HM1zborMA?f=cp&u=c88bf4dbb5418758",
"media_types": [
"video"
],
"expected_fields": [
"video",
"cover",
"title"
],
"note": "2026-09-14 已验证: 百度分享短链与多清晰度落地页",
"pattern": "百度短链与清晰度矩阵"
},
{
"platform": "好看视频",
"url": "https://m.baidu.com/video/page?pd=video_page&nid=14766924042149789226&sign=2495427256712467844&word=%E6%9A%97%E5%8C%BA%E7%AA%81%E5%9B%B4S1%20PV&oword=%E6%9A%97%E5%8C%BA%E7%AA%81%E5%9B%B4S1%20PV&atn=index&frsrcid=4185",
"media_types": [
"video"
],
"expected_fields": [
"video",
"cover",
"title"
],
"note": "2026-09-14 已验证: 百度移动搜索视频落地页",
"pattern": "移动端搜索落地页"
},
{
"platform": "小云雀AI",
"url": "https://xiaoyunque.jianying.com/s/eguwTcLQIxw/",
"media_types": [
"video"
],
"expected_fields": [
"video",
"title",
"author"
],
"note": "2026-09-14 已验证: 小云雀灵感创作与壁纸生成分享",
"pattern": "灵感创作与壁纸生成"
},
{
"platform": "微博",
"url": "https://m.weibo.cn/7753941940/5332536008119716/qq?wm=3333_2001",
"media_types": [
"video"
],
"expected_fields": [
"video",
"cover",
"author"
],
"note": "2026-09-14 已验证: 微博移动端带渠道后缀视频博文",
"pattern": "移动端渠道后缀博文"
}
]
}
+70 -19
View File
@@ -47,10 +47,15 @@ def collect_media(parser):
}
def verify_case(case):
def verify_case(case, include_expired=False):
platform = case["platform"]
pattern = case.get("pattern", "标准链接")
url = case.get("url", "")
is_marked_expired = case.get("status") == "expired" or case.get("is_expired") is True
if is_marked_expired and not include_expired:
reason = case.get("expired_reason") or case.get("note") or "用例已标记为失效/下架"
return "SKIPPED", platform, pattern, f"跳过已失效用例 ({reason})", url
if not url:
return "MISSING", platform, pattern, "未配置真实链接", url
try:
@@ -65,18 +70,37 @@ def verify_case(case):
expected = case.get("expected_fields", ["title"])
missing = [field for field in expected if not found.get(field)]
if missing:
terminal_detail = getattr(parser, "terminal_error", None) or getattr(parser, "_terminal_filter_detail", None)
if isinstance(terminal_detail, dict):
detail_msg = terminal_detail.get("detail_msg") or terminal_detail.get("notice") or "作品已被作者删除或设为私密"
return "EXPIRED", platform, pattern, f"链接已失效: {detail_msg}", url
if getattr(parser, "no_media_in_content", False):
return "EXPIRED", platform, pattern, "分享内容仅为纯文本,未包含媒体资源", url
return "FAILED", platform, pattern, f"缺少字段:{', '.join(missing)}", url
present = ", ".join(name for name, value in found.items() if value)
return "PASSED", platform, pattern, f"已取得:{present}", url
except Exception as exc:
err_msg = str(exc)
if "MEDIA_DELETED" in err_msg or "作品权限或已被删除" in err_msg:
return "EXPIRED", platform, pattern, f"链接已失效: {type(exc).__name__}: {exc}", url
return "FAILED", platform, pattern, f"{type(exc).__name__}: {exc}", url
def save_cases(cases_data):
"""回写更新后的用例到 live_parser_samples.json。"""
raw_content = json.loads(SAMPLES_PATH.read_text(encoding="utf-8"))
raw_content["cases"] = cases_data
raw_content["total_cases"] = len(cases_data)
SAMPLES_PATH.write_text(json.dumps(raw_content, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def main():
ParserFactory._discover()
parser = argparse.ArgumentParser(description="一键多形态验证真实分享链接")
parser.add_argument("--platform", action="append", help="仅验证指定平台;可重复传入")
parser.add_argument("--list-missing", action="store_true", help="仅列出缺少链接的平台")
parser.add_argument("--include-expired", action="store_true", help="强制包含标记为失效的用例一起运行")
parser.add_argument("--auto-mark-expired", action="store_true", help="自动将检测到已失效的用例标记为 status=expired")
parser.add_argument("--limit", type=int, default=0, help="限制每个平台运行的最大用例数 (默认全部)")
parser.add_argument("--workers", type=int, default=20, help="并发线程数 (默认 20,0 表示单线程顺序执行)")
parser.add_argument("--summary-only", action="store_true", help="仅输出最终汇总,适合全量回归")
@@ -84,7 +108,8 @@ def main():
if args.summary_only:
logging.disable(logging.CRITICAL)
cases = load_cases()
all_cases = load_cases()
cases = all_cases
if args.platform:
requested = set(args.platform)
cases = [case for case in cases if case["platform"] in requested]
@@ -111,9 +136,17 @@ def main():
print(f"🧪 开始执行 {platform_count} 平台自动化多形态链接验证 (共 {len(cases)} 条用例, 线程数: {args.workers if args.workers > 0 else 1})...\n", flush=True)
results = []
status_icons = {
"PASSED": "✅ PASSED ",
"FAILED": "❌ FAILED ",
"EXPIRED": "⚠️ EXPIRED ",
"SKIPPED": "⚪ SKIPPED ",
"MISSING": "⚪ MISSING ",
}
if args.workers > 1:
with ThreadPoolExecutor(max_workers=args.workers) as executor:
future_to_case = {executor.submit(verify_case, case): case for case in cases}
future_to_case = {executor.submit(verify_case, case, args.include_expired): case for case in cases}
for future in as_completed(future_to_case):
try:
res = future.result()
@@ -122,31 +155,49 @@ def main():
res = ("FAILED", c["platform"], c.get("pattern", ""), f"{type(exc).__name__}: {exc}", c.get("url", ""))
results.append(res)
status, platform, pattern, detail, url = res
status_tag = f"✅ {status}" if status == "PASSED" else (f"❌ {status}" if status == "FAILED" else f"⚪ {status}")
status_tag = status_icons.get(status, f"[{status}]")
if not args.summary_only:
print(f"{status_tag:<10} [{platform:<6}] ({pattern[:25]}): {detail}", flush=True)
print(f"{status_tag} [{platform:<6}] ({pattern[:25]}): {detail}", flush=True)
else:
for case in cases:
res = verify_case(case)
res = verify_case(case, args.include_expired)
results.append(res)
status, platform, pattern, detail, url = res
status_tag = f"✅ {status}" if status == "PASSED" else (f"❌ {status}" if status == "FAILED" else f"⚪ {status}")
status_tag = status_icons.get(status, f"[{status}]")
if not args.summary_only:
print(f"{status_tag:<10} [{platform:<6}] ({pattern[:25]}): {detail}", flush=True)
print(f"{status_tag} [{platform:<6}] ({pattern[:25]}): {detail}", flush=True)
summary = {status: sum(1 for result in results if result[0] == status) for status in ("PASSED", "FAILED", "MISSING")}
summary = {status: sum(1 for result in results if result[0] == status) for status in ("PASSED", "FAILED", "EXPIRED", "SKIPPED", "MISSING")}
print("\n" + "=" * 60, flush=True)
print("📊 验证汇总:" + ", ".join(f"{key}={value}" for key, value in summary.items()), flush=True)
total = len(results)
if total > 0:
pass_rate = (summary['PASSED'] / total) * 100
print(f"🎯 整体通过率:{pass_rate:.1f}% ({summary['PASSED']}/{total})", flush=True)
if args.summary_only:
failures = [result for result in results if result[0] != "PASSED"]
print("📊 验证汇总:" + ", ".join(f"{key}={value}" for key, value in summary.items() if value > 0 or key in ("PASSED", "FAILED")), flush=True)
active_total = summary["PASSED"] + summary["FAILED"] + summary["EXPIRED"] + summary["MISSING"]
if active_total > 0:
pass_rate = (summary["PASSED"] / active_total) * 100
print(f"🎯 有效用例通过率:{pass_rate:.1f}% ({summary['PASSED']}/{active_total})", flush=True)
if summary.get("SKIPPED", 0) > 0:
print(f"💡 已跳过失效/下架用例: {summary['SKIPPED']} 条 (传入 --include-expired 可强制测试)", flush=True)
if args.summary_only or summary.get("FAILED", 0) > 0 or summary.get("EXPIRED", 0) > 0:
failures = [result for result in results if result[0] in ("FAILED", "EXPIRED")]
if failures:
print("失败用例:", flush=True)
for status, platform, pattern, detail, _url in failures:
print(f"- [{platform}] {pattern}: {detail}", flush=True)
print("\n🚨 需关注/异常用例:", flush=True)
for status, platform, pattern, detail, url in failures:
print(f"- [{status}] [{platform}] {pattern}: {detail} -> {url}", flush=True)
if args.auto_mark_expired and summary.get("EXPIRED", 0) > 0:
expired_urls = {res[4]: res[3] for res in results if res[0] == "EXPIRED" and res[4]}
marked_count = 0
for case in all_cases:
case_url = case.get("url")
if case_url in expired_urls and case.get("status") != "expired":
case["status"] = "expired"
case["expired_reason"] = expired_urls[case_url]
marked_count += 1
if marked_count > 0:
save_cases(all_cases)
print(f"\n💾 已自动将 {marked_count} 条失效链接标记为 status='expired' 并持久化回用例库。", flush=True)
print("=" * 60, flush=True)
+33 -5
View File
@@ -177,11 +177,39 @@ class DoubaoParserTest(unittest.TestCase):
with patch("requests.Session.get", return_value=response):
parser = DoubaoParser("https://www.doubao.com/thread/pure-text")
self.assertTrue(parser.no_media_in_content)
self.assertIn("请写一首关于秋天的古诗", parser.get_description())
self.assertIn("空山新雨后,天气晚来秋", parser.get_description())
self.assertIsNone(parser.get_real_video_url())
self.assertEqual(parser.get_image_list(), [])
def test_music_sharing_extracts_audio_and_video(self):
play_info_response = Mock(
status_code=200,
json=Mock(return_value={
"code": 0,
"data": {
"media_type": "audio",
"original_media_info": {
"main_url": "https://v9-videoweb.doubao.com/test_audio.mp4"
},
"poster_url": "https://image.example.com/music_poster.jpg"
}
})
)
url = "https://www.doubao.com/music-sharing?vid=v02292g10006dadu4kq7dld48hgk0dn0&share_id=55311664453668866"
with patch.object(DoubaoParser, "_fetch_unwatermarked_video_by_vid", return_value=(["https://clean.example.com/audio.mp4"], "https://clean.example.com/poster.jpg")):
with patch("requests.Session.post", return_value=play_info_response):
parser = DoubaoParser(url)
self.assertEqual(parser.get_title_content(), "豆包AI音乐分享")
self.assertEqual(parser.get_real_video_url(), "https://clean.example.com/audio.mp4")
self.assertEqual(parser.get_audio_url(), "https://v9-videoweb.doubao.com/test_audio.mp4")
self.assertEqual(parser.get_cover_photo_url(), "https://clean.example.com/poster.jpg")
def test_chat_route_supported(self):
page = "<html><head></head><body></body></html>"
response = Mock(text=page)
response.raise_for_status.return_value = None
with patch("requests.Session.get", return_value=response):
parser = DoubaoParser("https://www.doubao.com/chat/38441154953364226")
self.assertEqual(parser.get_title_content(), "豆包对话分享")
if __name__ == "__main__":
+53
View File
@@ -0,0 +1,53 @@
import json
import unittest
from unittest.mock import Mock, patch
from src.parsers.haokan_parser import HaokanParser
from utils.web_fetcher import UrlParser
class HaokanParserTest(unittest.TestCase):
def test_parses_window_json_data(self):
json_data_payload = {
"data": {
"title": "测试好看视频标题",
"videoInfo": {
"title": "测试好看视频标题",
"posterImage": "https://image.example.com/poster.jpg",
"clarityArr": [
{"rank": 1, "url": "https://video.example.com/480p.mp4"},
{"rank": 3, "url": "https://video.example.com/1080p.mp4"},
{"rank": 2, "url": "https://video.example.com/720p.mp4"}
]
},
"author": {
"name": "测试作者",
"uk": "123456",
"icon": "https://image.example.com/avatar.jpg"
}
}
}
html_page = (
'<html><head>'
f'<script>window.jsonData = {json.dumps(json_data_payload)};</script>'
'</head><body></body></html>'
)
response = Mock(status_code=200, content=html_page.encode("utf-8"))
with patch("requests.Session.get", return_value=response):
parser = HaokanParser("https://mbd.baidu.com/newspage/data/videoshare?nid=sv_123456")
self.assertEqual(parser.get_title_content(), "测试好看视频标题")
self.assertEqual(parser.get_real_video_url(), "https://video.example.com/1080p.mp4")
self.assertEqual(parser.get_cover_photo_url(), "https://image.example.com/poster.jpg")
self.assertEqual(parser.get_author_info()["nickname"], "测试作者")
self.assertEqual(parser.get_author_info()["author_id"], "123456")
def test_url_parser_extracts_nid_and_sign(self):
url = "https://mbd.baidu.com/newspage/data/videoshare?nid=sv_10423411367049244882&noise=123"
extracted = UrlParser.extract_video_address(url)
self.assertEqual(extracted, "https://mbd.baidu.com/newspage/data/videoshare?nid=sv_10423411367049244882")
if __name__ == "__main__":
unittest.main()
+5
View File
@@ -39,6 +39,11 @@ class WeiboParserTest(unittest.TestCase):
parser.real_url = "https://m.weibo.cn/detail/O8yqz0I8Q"
self.assertEqual(parser._extract_id(), "5020389670169684")
def test_extracts_id_from_mobile_channel_url(self):
parser = WeiboParser.__new__(WeiboParser)
parser.real_url = "https://m.weibo.cn/7753941940/5332536008119716/qq?wm=3333_2001"
self.assertEqual(parser._extract_id(), "5332536008119716")
def test_image_list_extraction(self):
parser = WeiboParser.__new__(WeiboParser)
parser.post_data = {
+34
View File
@@ -44,6 +44,40 @@ class XiaoyunqueParserTest(unittest.TestCase):
"12345",
)
def test_inspiration_page_response(self):
response = Mock()
response.raise_for_status.return_value = None
response.json.return_value = {
"err_no": 0,
"err_tips": "success",
"data": {
"page_info": {
"inspiration_page": {
"user_info": {
"nick_name": "茉莱",
"avatar_url": "https://p26-passport.byteacctimg.com/avatar.jpg",
},
"item_info": {
"title": "三角洲原图壁纸",
"desc": "壁纸生成",
"video_info": [
{"video_url": "https://v11-xyq-video.jianying.com/video.mp4"}
],
},
}
}
},
}
url = "https://xiaoyunque.jianying.com/activities/pippit_share?inspiration_id=7684286783865146686"
with patch("requests.Session.post", return_value=response):
parser = XiaoyunqueParser(url)
self.assertEqual(parser.get_title_content(), "三角洲原图壁纸")
self.assertEqual(parser.get_description(), "壁纸生成")
self.assertEqual(parser.get_real_video_url(), "https://v11-xyq-video.jianying.com/video.mp4")
self.assertEqual(parser.get_author_info()["nickname"], "茉莱")
def test_url_parser_recognizes_xiaoyunque(self):
url = "https://xiaoyunque.jianying.com/s/z_7nWGLGruM/"
self.assertEqual(UrlParser.get_platform(url), "小云雀AI")
+4 -3
View File
@@ -159,12 +159,13 @@ class UrlParser:
if platform == '好看视频':
query_params = parse_qs(parsed_url.query)
preserved_params = []
for key in ('vid', 'id', 'context', 'pd', 'src'):
for key in ('vid', 'id', 'context', 'pd', 'src', 'nid', 'sign', 'word', 'oword', 'atn', 'frsrcid'):
value = query_params.get(key, [None])[0]
if value is not None:
preserved_params.append((key, value))
if preserved_params:
address = f"{address}?{urlencode(preserved_params)}"
from urllib.parse import quote
address = f"{address}?{urlencode(preserved_params, quote_via=quote)}"
elif platform == "微视":
query_params = parse_qs(parsed_url.query)
vid = query_params.get('id', [None])[0] # 使用 get 方法避免 KeyError
@@ -203,7 +204,7 @@ class UrlParser:
elif platform == "豆包":
query_params = parse_qs(parsed_url.query)
preserved_params = []
for key in ('share_id', 'source_type', 'video_id', 'share_scene'):
for key in ('share_id', 'source_type', 'video_id', 'share_scene', 'vid'):
value = query_params.get(key, [None])[0]
if value is not None:
preserved_params.append((key, value))