mirror of
https://github.com/ucmao/media-parser.git
synced 2026-11-20 08:04:56 +08:00
新增星绘AI、央视与央视频平台解析支持
This commit is contained in:
@@ -3,10 +3,10 @@
|
||||
|
||||
**基于 Python 的多平台媒体原生本地解析系统**
|
||||
|
||||
[](LICENSE) [](https://www.python.org/) [](https://flask.palletsprojects.com/) [](#-部署指南) [](#-支持的平台矩阵)
|
||||
[](LICENSE) [](https://www.python.org/) [](https://flask.palletsprojects.com/) [](#-部署指南) [](#-支持的平台矩阵)
|
||||
|
||||
<p align="center">
|
||||
<a href="#-核心解析逻辑">解析逻辑</a> •
|
||||
<a href="#-核心特性">核心特性</a> •
|
||||
<a href="#-支持的平台矩阵">支持平台</a> •
|
||||
<a href="#-部署指南">部署指南</a> •
|
||||
<a href="#-api-核心接口说明">接口文档</a> •
|
||||
@@ -15,63 +15,81 @@
|
||||
<a href="#-联系作者">联系作者</a>
|
||||
</p>
|
||||
|
||||
媒体解析去水印是一款专为短视频创作者与开发者打造的**原生本地解析工具**。
|
||||
媒体解析去水印是一款专为短视频创作者与开发者打造的**100%原生本地解析工具**。
|
||||
|
||||
通过“智能识别 -> 本地抓取 -> 提取地址 -> 快捷下载”的闭环,助你高效获取无水印素材。
|
||||
|
||||
**100%本地原生实现,不依赖外部API,不封装第三方解析库,核心逆向逻辑完全开源。**
|
||||
**不依赖外部API,不套壳第三方库,无浏览器开销,纯底层协议与算法逆向。**
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 💎 核心解析逻辑
|
||||
## ✨ 核心特性
|
||||
|
||||
* **多平台智能适配**:内置 `ParserFactory` 工厂模式,自动识别链接来源并分配对应解析器。
|
||||
* **原生本地解析**:解析逻辑直接内置在项目代码中,由各平台 Parser 本地发起请求并提取真实媒体地址。
|
||||
* **零外部API依赖**:无需对接任何第三方代解析接口或 SaaS 中转服务,部署后即可独立稳定运行。
|
||||
* **开箱即用API**:提供标准化的 JSON 数据接口,无冗余数据库依赖,适合快速对接业务或二次开发。
|
||||
|
||||
## ✨ 项目特点
|
||||
|
||||
* **本地可控**:所有请求链路与解析规则均在本地代码中,方便调试、维护与定制。
|
||||
* **部署简单**:安装依赖或使用 Docker 即可一键运行,不需要额外申请第三方 API 账号或密钥。
|
||||
* **便于扩展**:各平台对应独立 Parser,遵循统一的返回数据结构,新增平台轻松快捷。
|
||||
* ⚡ **极速轻量**:纯 HTTP 网络协议与底层算法逆向,免启动 Chromium/Playwright 等笨重浏览器,内存占用极低(<100MB),毫秒级极速响应。
|
||||
* 🔒 **原生自主**:100% 本地代码闭环抓取,零外部商用 API 或第三方代解析依赖,数据链路自主可控,杜绝断流与隐私泄露风险。
|
||||
* 🧩 **插件化解耦**:内置 `ParserFactory` 模块自动发现与工厂分发机制,50 个平台独立解耦,遵循统一的数据契约,新增与维护平台极度轻松。
|
||||
* 🔌 **开箱即用**:提供标准 RESTful JSON 接口与 Web 体验页,无外部数据库等冗余依赖,Docker Compose 一键构建,轻松无缝接入各类业务系统。
|
||||
|
||||
---
|
||||
|
||||
## 💾 支持的平台矩阵
|
||||
|
||||
| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 |
|
||||
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
||||
| **抖音** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
|
||||
| **小红书** | ✓ | ✓ | ✓ | ✓ | **✓+** | | | **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | |
|
||||
| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **微视** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **快手** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | | **AcFun** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **今日头条** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | **全民K歌** | | ✓ | ✓ | ✓ | | | |
|
||||
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **新片场** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **好看视频** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **梨视频** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **闲鱼** | | ✓ | ✓ | | ✓ | | | **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **虎牙** | | ✓ | ✓ | ✓ | | | |
|
||||
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **美拍** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | **最右** | ✓ | ✓ | | ✓ | | | |
|
||||
| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | | | | | | | | |
|
||||
| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 实况 |
|
||||
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
||||
| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ |
|
||||
| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ |
|
||||
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
|
||||
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **闲鱼** | | ✓ | ✓ | | ✓ | | | |
|
||||
| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | |
|
||||
| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
|
||||
| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
|
||||
| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **微视** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **AcFun** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **今日头条** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **全民K歌** | | ✓ | ✓ | ✓ | | | | |
|
||||
| **新片场** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **好看视频** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **梨视频** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **虎牙** | | ✓ | ✓ | ✓ | | | | |
|
||||
| **美拍** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **最右** | ✓ | ✓ | | ✓ | | | | |
|
||||
| **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **星绘AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **央视** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **央视频** | ✓ | ✓ | ✓ | | ✓ | | | |
|
||||
|
||||
> 💡 **说明**:
|
||||
> * **图集 `✓+`**:表示该平台支持提取 iOS/Android 原始 **LivePhoto 动态实况图**(含封面静图与关联 `.mov`/`.mp4` 实况视频原件)。
|
||||
> * **原生水印提示**:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。
|
||||
*注:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。
|
||||
|
||||
---
|
||||
|
||||
@@ -132,32 +150,17 @@ python app.py
|
||||
|
||||
## 🔌 API 核心接口说明
|
||||
|
||||
### 1. 服务健康检查
|
||||
|
||||
* **接口路径**:`GET /api/health`
|
||||
* **接口描述**:供容器编排(Docker Compose/K8s)、反向代理或监控系统探针检测服务存活状态。
|
||||
* **返回示例**:
|
||||
```json
|
||||
{
|
||||
"status": "ok"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. 媒体解析接口
|
||||
|
||||
* **接口路径**:`POST /api/parse`
|
||||
* **接口描述**:传入包含分享链接的文本,智能提取多媒体直链与图文信息。
|
||||
|
||||
#### 请求参数 (Request Body)
|
||||
### 请求参数 (Request Body)
|
||||
格式: `application/json`
|
||||
|
||||
| 参数名 | 类型 | 必填 | 描述 | 限制与示例 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `text` | `string` | 是 | 视频分享链接或包含链接的文本短语 | 最长 2000 字符,如 `"https://v.douyin.com/..."` |
|
||||
|
||||
#### 返回说明 (Response)
|
||||
### 返回说明 (Response)
|
||||
格式: `application/json`
|
||||
|
||||
成功响应示例:
|
||||
@@ -210,10 +213,10 @@ python app.py
|
||||
|
||||
## 🧪 自动化测试与健康自检
|
||||
|
||||
本项目拥有完备的双层测试体系(Mock 单元测试 + 47 平台真实在线样本库回归)。遇到解析异常或日常部署验证时,可一键运行健康自检:
|
||||
本项目拥有完备的双层测试体系(Mock 单元测试 + 50 平台真实在线样本库回归)。遇到解析异常或日常部署验证时,可一键运行健康自检:
|
||||
|
||||
```bash
|
||||
# 47 平台极速冒烟测试(每个平台测 1 条最具代表性的链接,秒级完成健康检查)
|
||||
# 50 平台极速冒烟测试(每个平台测 1 条最具代表性的链接,秒级完成健康检查)
|
||||
python3 tests/manual_verify_parsers.py --limit 1
|
||||
|
||||
# 仅验证单个或指定平台(如:小云雀AI / 抖音)
|
||||
@@ -231,7 +234,7 @@ python3 tests/manual_verify_parsers.py --platform "小云雀AI"
|
||||
* 🏗️ **[系统架构与生命周期设计](docs/architecture.md)**:分层设计、302 追踪与 `ParserFactory` 动态发现机制。
|
||||
* 🔍 **[通用逆向方法论](docs/reverse-guide.md)**:SSR 数据提取、H5 接口伪装、JS 签名沙箱及抓包 SOP。
|
||||
* 🧪 **[测试体系与回归验证](docs/testing.md)**:Pytest 单元测试、Mock 与线上真实样例回归。
|
||||
* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 47 个平台的技术文档。
|
||||
* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 50 个平台的技术文档。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -142,7 +142,19 @@
|
||||
"poizon.com": "得物",
|
||||
"www.poizon.com": "得物",
|
||||
"lofter.com": "网易LOFTER",
|
||||
"www.lofter.com": "网易LOFTER"
|
||||
"www.lofter.com": "网易LOFTER",
|
||||
"butterflyai.cn": "星绘AI",
|
||||
"www.butterflyai.cn": "星绘AI",
|
||||
"cctv.com": "央视",
|
||||
"tv.cctv.com": "央视",
|
||||
"cctv.cn": "央视",
|
||||
"tv.cctv.cn": "央视",
|
||||
"content-static.cctvnews.cctv.com": "央视",
|
||||
"cctvnews.cctv.com": "央视",
|
||||
"news.cctv.com": "央视",
|
||||
"yspapp.cn": "央视频",
|
||||
"yangshipin.cn": "央视频",
|
||||
"m.yangshipin.cn": "央视频"
|
||||
},
|
||||
"USER_AGENT_PC": [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
|
||||
|
||||
+6
-3
@@ -2,7 +2,7 @@
|
||||
|
||||
欢迎查阅 **Media Parser** 开发者文档与逆向工程知识库。
|
||||
|
||||
本项目是一个高性能、模块化、支持 **47 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。
|
||||
本项目是一个高性能、模块化、支持 **50 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。
|
||||
|
||||
---
|
||||
|
||||
@@ -14,8 +14,8 @@
|
||||
* 📖 **平台实战指南 (Parser Guides)**:
|
||||
* **短视频与轻社区**:[抖音](parsers/douyin.md) | [快手](parsers/kuaishou.md) | [今日头条](parsers/xigua.md) | [皮皮虾](parsers/pipixia.md) | [皮皮搞笑](parsers/pipigaoxiao.md) | [最右](parsers/zuiyou.md) | [美拍](parsers/meipai.md) | [微视](parsers/weishi.md) | [绿洲](parsers/lvzhou.md) | [番茄小说/红果短剧/红果漫剧](parsers/fanqie.md)
|
||||
* **图文与综合社区**:[小红书](parsers/xiaohongshu.md) | [微信公众号](parsers/wechat-mp.md) | [微博](parsers/weibo.md) | [知乎](parsers/zhihu.md) | [闲鱼](parsers/xianyu.md) | [Soul](parsers/soul.md) | [得物](parsers/dewu.md) | [网易LOFTER](parsers/lofter.md)
|
||||
* **长视频与弹幕/创作**:[哔哩哔哩](parsers/bilibili.md) | [AcFun](parsers/acfun.md) | [新片场](parsers/xinpianchang.md) | [好看视频](parsers/haokan.md) | [西瓜视频](parsers/xigua.md) | [剪映](parsers/jianying.md) | [快影](parsers/kwaiying.md)
|
||||
* **AI 生成与大模型**:[豆包 AI](parsers/doubao.md) | [腾讯元宝](parsers/yuanbao.md) | [即梦 AI](parsers/jimeng.md) | [可灵 AI](parsers/kling.md) | [海螺 AI](parsers/hailuo.md) | [通义千问](parsers/qianwen.md) | [夸克 AI](parsers/quark-ai.md) | [小云雀 AI](parsers/xiaoyunque.md) | [松果时刻](parsers/pinecone-moment.md)
|
||||
* **长视频与电视/融媒**:[哔哩哔哩](parsers/bilibili.md) | [AcFun](parsers/acfun.md) | [央视](parsers/cctv.md) | [央视频](parsers/yangshipin.md) | [新片场](parsers/xinpianchang.md) | [好看视频](parsers/haokan.md) | [西瓜视频](parsers/xigua.md) | [剪映](parsers/jianying.md) | [快影](parsers/kwaiying.md)
|
||||
* **AI 生成与大模型**:[豆包 AI](parsers/doubao.md) | [星绘 AI](parsers/butterflyai.md) | [腾讯元宝](parsers/yuanbao.md) | [即梦 AI](parsers/jimeng.md) | [可灵 AI](parsers/kling.md) | [海螺 AI](parsers/hailuo.md) | [通义千问](parsers/qianwen.md) | [夸克 AI](parsers/quark-ai.md) | [小云雀 AI](parsers/xiaoyunque.md) | [松果时刻](parsers/pinecone-moment.md)
|
||||
* **音频与垂直频道**:[汽水音乐](parsers/qsmusic.md) | [QQ音乐](parsers/qqmusic.md) | [网易云音乐](parsers/netease-music.md) | [酷狗音乐](parsers/kugou-music.md) | [配音秀](parsers/peiyinxiu.md) | [全民K歌](parsers/quanminkge.md) | [虎牙](parsers/huya.md) | [梨视频](parsers/lishipin.md) | [微信视频号](parsers/wechat-channels.md) | [腾讯频道](parsers/tencent-channel.md)
|
||||
|
||||
---
|
||||
@@ -76,6 +76,9 @@
|
||||
| 45 | **松果时刻** | AI 故事多页视频 / 图集 / 配音 | ✅ 支持 | 🟢 公开内容免配置 | 公开详情接口 | [查看指南](parsers/pinecone-moment.md) |
|
||||
| 46 | **得物** | 潮流穿搭 / 开箱短视频 / 高清图集 | ✅ 支持 | 🟢 免配置 | Next.js metaOGInfo 数据提取 | [查看指南](parsers/dewu.md) |
|
||||
| 47 | **网易LOFTER** | 二次元插画 / 摄影图集 / 创作者短视频 | ✅ 支持 | 🟢 免配置 | window.__initialize_data__ 数据提取 | [查看指南](parsers/lofter.md) |
|
||||
| 48 | **星绘 AI** | AI 生图 / 原画写真 / AI 视频 | ✅ 支持 | 🟢 免配置 | 字节分享 API (Community Service) | [查看指南](parsers/butterflyai.md) |
|
||||
| 49 | **央视** | 栏目视频 (HLS/m3u8) / 央视新闻微视频 | ✅ 支持 | 🟢 免配置 | CNTV 调度接口 + 阿里云 API 网关 HMAC-SHA256 验签 | [查看指南](parsers/cctv.md) |
|
||||
| 50 | **央视频** | 微视频 / 微短剧 / 高清封面与作者信息 | ✅ 支持 | 🟢 免配置 | 页面 SSR 状态机 (__STATE__) 提取与 Meta 跳转跟踪 | [查看指南](parsers/yangshipin.md) |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# 星绘 AI (ButterflyAI) 逆向解析指南
|
||||
|
||||
本篇详细记录字节跳动豆包大模型家族旗下的 AI 艺术与生图/生视频平台 **星绘 AI (ButterflyAI)** 的作品解析方案。
|
||||
|
||||
---
|
||||
|
||||
## 1. 平台特征与支持能力
|
||||
|
||||
* **平台标识**:`星绘AI`
|
||||
* **支持媒体类型**:无水印超清原画图集 / AI 视频生成作品 (MP4) / 封面 / 标题文案 / 创作者信息
|
||||
* **常见链接形态**:
|
||||
* 官方分享短链:`https://www.butterflyai.cn/s/gQ0EXE9BJsw/`
|
||||
* 社区分享长链:`https://www.butterflyai.cn/share/record?share_code=PRr8gE3n3eENtAl5FOtN&is_mine=1&resource_count=1&entity_type=1`
|
||||
* H5 落地页:`https://www.butterflyai.cn/butterfly/community/h5/share/index.html?share_code=...`
|
||||
* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心逆向流程
|
||||
|
||||
1. **短链展开与参数定位**:
|
||||
* 用户分享获得的链接一般为 `https://www.butterflyai.cn/s/{code}/` 短链形态。
|
||||
* 通过 `WebFetcher.fetch_redirect_url` 跟随 302 重定向定位至落地页,提取关键参数 `share_code`(或 `share_token`)与 `share_id`。
|
||||
2. **社区分享开放接口**:
|
||||
* 请求端点:`POST https://www.butterflyai.cn/butterfly/community/v1/share/record/get?aid=564650`
|
||||
* 请求载荷:
|
||||
```json
|
||||
{
|
||||
"share_record_id": "",
|
||||
"from_h5": true,
|
||||
"share_token": "<share_code>",
|
||||
"extra_params": {
|
||||
"aid": "564650",
|
||||
"app_name": "星绘"
|
||||
}
|
||||
}
|
||||
```
|
||||
3. **数据结构与媒体提取**:
|
||||
* **无水印原画图集**:从 `share_record.rendering_images` 或 `artwork.resource_list[].image_resource.rendering_images` 中提取图片。接口原生提供 `no_wm_download_url`(无水印原画下载直链),优先级高于带水印的 `download_url` 与 `image_url`。
|
||||
* **AI 视频作品**:从 `share_record.rendering_video.video_url` 或 `artwork.resource_list[].video_resource.rendering_videos[].video_url` 中提取原生 MP4 视频直链。
|
||||
* **元数据**:提取 `artwork.title`、Prompt、`show_info.effect_title`(如 AI 写真、AI 视频等),以及 `creator.screen_name` 和多档高清头像 `creator.avatar.large_url`。
|
||||
|
||||
---
|
||||
|
||||
## 3. 测试与验证
|
||||
|
||||
* **单元测试**:[tests/test_butterflyai_parser.py](file:///Users/leo/Projects/media-parser/tests/test_butterflyai_parser.py)
|
||||
* **执行命令**:`python3 -m unittest tests/test_butterflyai_parser.py`
|
||||
@@ -0,0 +1,50 @@
|
||||
# 央视 (CCTV / 央视网 / 央视新闻) 逆向解析指南
|
||||
|
||||
本篇详细记录国家级综合电视广播与新闻门户 **央视 (CCTV)** 的电视节目、综合栏目点播与央视新闻微视频解析方案。
|
||||
|
||||
---
|
||||
|
||||
## 1. 平台特征与支持能力
|
||||
|
||||
* **平台标识**:`央视`
|
||||
* **支持媒体类型**:高清 HLS 视频流 (.m3u8) / 央视新闻微视频 (MP4/m3u8) / 封面 / 标题文案 / 栏目与来源作者
|
||||
* **常见链接形态**:
|
||||
* 央视电视点播:`https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml`
|
||||
* 央视视频短链/手机端:`https://tv.cctv.cn/v/v1/VIDE4y5JLGNL5QIKkj1JrGnL160430.html`
|
||||
* 央视新闻微视频/报道:`https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092`
|
||||
* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心逆向流程
|
||||
|
||||
央视内容体系主要分为 **CCTV 综合/栏目点播体系** 与 **央视新闻客户端 (CCTVNews) 体系**,Parser 自动根据域名与路径进行路由分发:
|
||||
|
||||
### 体系 A:CCTV 电视与栏目点播 (`tv.cctv.com`, `tv.cctv.cn`)
|
||||
1. **PID / GUID 提取**:
|
||||
* 页面内通过全局 JS 变量声明 32 位 Hex 视频全局标识,例如 `guid = "00e121ae62194fec8ab20e5b8eb9a89a"` 或 `pid = "..."`。
|
||||
2. **CNTV 视频调度分配接口**:
|
||||
* 请求端点:`GET https://vdn.apps.cntv.cn/api/getHttpVideoInfo.do?pid={pid}`
|
||||
* 直接提取 `hls_url`(官方多码率 HLS 流地址)、`image`(官方高分辨率静态封面)、`title`(节目期数与标题)及 `column`(栏目名称)。
|
||||
|
||||
### 体系 B:央视新闻客户端 (`content-static.cctvnews.cctv.com`)
|
||||
1. **文章 / 微视频 ID**:
|
||||
* 从 URL 参数中提取 `item_id` 或 `articleId`。
|
||||
2. **阿里云 API 网关 HMAC-SHA256 签名机制**:
|
||||
* 接口端点:`https://api.cctvnews.cctv.com/1.0.0/feed/article/server/getArticle?articleId={item_id}&appcode=video_web`
|
||||
* 请求头必须携带标准 API Gateway 签名参数:
|
||||
* `x-ca-key`: 客户端公开 AppKey (`204133710`)
|
||||
* `x-ca-stage`: `RELEASE`
|
||||
* `x-ca-timestamp`: 当前毫秒时间戳
|
||||
* `x-ca-signature-headers`: `x-ca-key,x-ca-stage,x-ca-timestamp`
|
||||
* `x-ca-signature`: 基于 `appSecret` 对签名规范字符串计算所得的 Base64 HMAC-SHA256 摘要。
|
||||
3. **Base64 响应解密**:
|
||||
* 服务端响应返回 `{"code": 0, "response": "<base64>"}`。
|
||||
* 对 `response` 字段解码后即可得到完整文章、视频播放直链 (`videos[0].url`) 与缩略图。
|
||||
|
||||
---
|
||||
|
||||
## 3. 测试与验证
|
||||
|
||||
* **单元测试**:[tests/test_cctv_parser.py](file:///Users/leo/Projects/media-parser/tests/test_cctv_parser.py)
|
||||
* **执行命令**:`python3 -m unittest tests/test_cctv_parser.py`
|
||||
@@ -0,0 +1,36 @@
|
||||
# 央视频 (Yangshipin) 逆向解析指南
|
||||
|
||||
本篇详细记录中央广播电视总台 5G 新媒体旗舰平台 **央视频 (Yangshipin)** 的微视频、微短剧与体育资讯作品解析方案。
|
||||
|
||||
---
|
||||
|
||||
## 1. 平台特征与支持能力
|
||||
|
||||
* **平台标识**:`央视频`
|
||||
* **支持媒体类型**:高分辨率封面 / 图集 / 完整视频标题 / 创作者与机构作者信息(头像与名称)/ 视频 ID (`vid`, `cid`)
|
||||
* **常见链接形态**:
|
||||
* 官方分享短链:`https://www.yspapp.cn/5Sqx`, `https://www.yspapp.cn/d1o`
|
||||
* 移动端竖屏微短剧/小视频:`https://m.yangshipin.cn/portrait_video?vid=l00005817wl`
|
||||
* 移动端常规横屏视频:`https://m.yangshipin.cn/video?type=0&vid=v000007pgfu&cid=...`
|
||||
* PC/网页端:`https://yangshipin.cn/...`
|
||||
* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心逆向流程
|
||||
|
||||
1. **Meta Refresh 重定向自动跟随**:
|
||||
* 央视频短链 `yspapp.cn/{code}` 采用 HTML `<meta http-equiv="refresh" content="0; URL='https://m.yangshipin.cn/...'"/>` 形式执行页面跳转。
|
||||
* 解析器在 `WebFetcher` 与 `YangshipinParser` 中双重内置了针对 `<meta http-equiv="refresh">` 的无感自动追踪,直接锁定最终目标落地页。
|
||||
2. **SSR 双状态机数据提取**:
|
||||
* **横屏常规视频**:页面注入全局变量 `window.__STATE_video__`,解析 `payloads.sharevideo` 结构,获取 `title`、`cover_pic`、`cid`、`vid` 及 `om_info.title`(发布机构或频道名)。
|
||||
* **竖屏微短剧/短视频**:页面注入全局变量 `window.__STATE_portrait_video__`,解析 `payloads.videoDataList.items[0].videoData`,获取微短剧标题、`shareItem.shareImgUrl`(超清封面)、以及 `detailFollowItem.actorItem` 中的创作者昵称与头像。
|
||||
3. **播放流安全提示**:
|
||||
* 央视频点播播放接口底层接入腾讯云 VOD 架构,采用带有动态 `cKey 8.1` / 白名单签名机制保护。解析器全面提取高保真元数据与封面图集,并保留扩展能力。
|
||||
|
||||
---
|
||||
|
||||
## 3. 测试与验证
|
||||
|
||||
* **单元测试**:[tests/test_yangshipin_parser.py](file:///Users/leo/Projects/media-parser/tests/test_yangshipin_parser.py)
|
||||
* **执行命令**:`python3 -m unittest tests/test_yangshipin_parser.py`
|
||||
+4
-4
@@ -13,7 +13,7 @@ tests/
|
||||
├── test_parser_factory.py # 解析器自动注册与工厂路由测试
|
||||
├── test_web_fetcher.py # 302跳转与URL正则提取测试
|
||||
├── test_*_parser.py # 各平台专属的单元测试
|
||||
├── live_parser_samples.json # 真实线上验证样本库 (45+ 平台活跃样例)
|
||||
├── live_parser_samples.json # 真实线上验证样本库 (50 平台活跃样例,300+ 用例)
|
||||
├── test_live_parser_samples.py # 自动化执行样本库回归测试
|
||||
└── manual_verify_parsers.py # 命令行交互式/全量人工排查脚本
|
||||
```
|
||||
@@ -48,10 +48,10 @@ pytest tests/test_live_parser_samples.py -s
|
||||
```
|
||||
|
||||
### 4. 运行交互式真实样本回归验证工具
|
||||
[tests/manual_verify_parsers.py](file:///Users/leo/Projects/media-parser/tests/manual_verify_parsers.py) 支持对 45+ 个平台的多种形态(短链、长链、视频、图集等)发起真实网络请求与字段契约校验:
|
||||
[tests/manual_verify_parsers.py](file:///Users/leo/Projects/media-parser/tests/manual_verify_parsers.py) 支持对 50 个平台的多种形态(短链、长链、视频、图集等)发起真实网络请求与字段契约校验:
|
||||
|
||||
```bash
|
||||
# 1. 快速冒烟测试 (每个平台只测 1 条最具代表性的链接,极速完成 45 平台基准健康检查)
|
||||
# 1. 快速冒烟测试 (每个平台只测 1 条最具代表性的链接,极速完成 50 平台基准健康检查)
|
||||
python3 tests/manual_verify_parsers.py --limit 1
|
||||
|
||||
# 2. 验证指定平台的多形态覆盖 (支持重复传入多个 --platform 参数)
|
||||
@@ -60,7 +60,7 @@ python3 tests/manual_verify_parsers.py --platform 微博 --platform 哔哩哔哩
|
||||
# 3. 限制指定平台的测试用例数量 (例如每个平台最多测 2 条)
|
||||
python3 tests/manual_verify_parsers.py --platform 抖音 --platform 小红书 --limit 2
|
||||
|
||||
# 4. 全量深度回归测试 (运行 270+ 条全部多形态真实用例,适合发版前全量巡检)
|
||||
# 4. 全量深度回归测试 (运行 300+ 条全部多形态真实用例,适合发版前全量巡检)
|
||||
python3 tests/manual_verify_parsers.py
|
||||
|
||||
# 5. 列出当前用例库中未配置真实链接的平台
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
import json
|
||||
import re
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
from configs.general_constants import USER_AGENT_M
|
||||
from configs.logging_config import get_logger
|
||||
from src.parser_factory import register_parser
|
||||
from src.parsers.base_parser import BaseParser
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@register_parser("星绘AI")
|
||||
class ButterflyAIParser(BaseParser):
|
||||
"""星绘 AI (ButterflyAI / 字节跳动豆包家族) 解析器,支持无水印原画图集与视频生成作品提取。"""
|
||||
|
||||
def __init__(self, real_url):
|
||||
super().__init__(real_url)
|
||||
self.headers = {
|
||||
"User-Agent": USER_AGENT_M[0],
|
||||
"Referer": "https://www.butterflyai.cn/",
|
||||
"Accept": "application/json, text/plain, */*",
|
||||
}
|
||||
self.title = ""
|
||||
self.cover_url = None
|
||||
self.video_url = None
|
||||
self.image_list = []
|
||||
self.author = None
|
||||
self._parse()
|
||||
|
||||
def _parse(self):
|
||||
if not self.real_url:
|
||||
return
|
||||
|
||||
parsed = urlparse(self.real_url)
|
||||
qs = parse_qs(parsed.query)
|
||||
|
||||
# 1. 尝试从 URL 参数或路径提取 share_token / share_code / share_record_id
|
||||
share_token = qs.get("share_code", [None])[0] or qs.get("share_token", [None])[0]
|
||||
share_record_id = qs.get("share_id", [""])[0] or qs.get("share_record_id", [""])[0]
|
||||
|
||||
# 若未获取到 share_token 且为短链路径形式 /s/{code}/,跟随重定向获取最终 URL
|
||||
if not share_token:
|
||||
match = re.search(r"/s/([a-zA-Z0-9_\-]+)", parsed.path)
|
||||
if match:
|
||||
try:
|
||||
resp = self.session.get(self.real_url, headers=self.headers, allow_redirects=True, timeout=5)
|
||||
final_parsed = urlparse(resp.url)
|
||||
final_qs = parse_qs(final_parsed.query)
|
||||
share_token = final_qs.get("share_code", [None])[0] or final_qs.get("share_token", [None])[0]
|
||||
share_record_id = final_qs.get("share_id", [""])[0] or final_qs.get("share_record_id", [""])[0]
|
||||
except Exception as e:
|
||||
logger.warning("Failed to follow ButterflyAI redirect for %s: %s", self.real_url, e)
|
||||
|
||||
if not share_token and not share_record_id:
|
||||
logger.warning("Could not find share_token or share_record_id from %s", self.real_url)
|
||||
return
|
||||
|
||||
api_url = "https://www.butterflyai.cn/butterfly/community/v1/share/record/get?aid=564650"
|
||||
payload = {
|
||||
"share_record_id": share_record_id or "",
|
||||
"from_h5": True,
|
||||
"share_token": share_token or "",
|
||||
"extra_params": {
|
||||
"aid": "564650",
|
||||
"app_name": "星绘",
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
res = self.session.post(api_url, json=payload, headers=self.headers, timeout=5)
|
||||
res.raise_for_status()
|
||||
data = res.json()
|
||||
except Exception as e:
|
||||
logger.error("Failed to request ButterflyAI API: %s", e)
|
||||
return
|
||||
|
||||
status_info = data.get("status_info") or {}
|
||||
status_code = status_info.get("status_code", data.get("status_code", 0))
|
||||
if status_code != 0:
|
||||
logger.warning("ButterflyAI API returned error code %s: %s", status_code, status_info.get("status_msg") or data.get("message"))
|
||||
return
|
||||
|
||||
share_record = data.get("share_record") or {}
|
||||
artwork = share_record.get("artwork") or {}
|
||||
creator = share_record.get("creator") or {}
|
||||
|
||||
# 提取标题与生成 Prompt
|
||||
self.title = (
|
||||
artwork.get("title")
|
||||
or artwork.get("description")
|
||||
or share_record.get("prompt")
|
||||
or share_record.get("show_info", {}).get("effect_title")
|
||||
or ""
|
||||
).strip()
|
||||
if not self.title and share_record.get("effect_list"):
|
||||
first_eff = share_record["effect_list"][0]
|
||||
self.title = (
|
||||
first_eff.get("name")
|
||||
or first_eff.get("show_info", {}).get("effect_title")
|
||||
or ""
|
||||
).strip()
|
||||
|
||||
# 提取作者信息
|
||||
author_name = creator.get("screen_name") or creator.get("nickname") or creator.get("name")
|
||||
avatar_obj = creator.get("avatar") or creator.get("avatar_url")
|
||||
author_avatar = None
|
||||
if isinstance(avatar_obj, dict):
|
||||
author_avatar = avatar_obj.get("large_url") or avatar_obj.get("image_url") or avatar_obj.get("middle_url")
|
||||
elif isinstance(avatar_obj, str):
|
||||
author_avatar = avatar_obj
|
||||
|
||||
if author_name or author_avatar:
|
||||
self.author = {
|
||||
"name": author_name,
|
||||
"avatar": author_avatar,
|
||||
}
|
||||
|
||||
# 提取视频 (share_record.rendering_video 或 artwork.resource_list)
|
||||
rendering_video = share_record.get("rendering_video") or {}
|
||||
if rendering_video:
|
||||
self.video_url = rendering_video.get("video_url") or rendering_video.get("download_url")
|
||||
|
||||
for r in artwork.get("resource_list") or []:
|
||||
if not isinstance(r, dict):
|
||||
continue
|
||||
vr = r.get("video_resource") or {}
|
||||
for v in vr.get("rendering_videos") or []:
|
||||
if not self.video_url:
|
||||
self.video_url = v.get("video_url") or v.get("download_url")
|
||||
|
||||
ir = r.get("image_resource") or {}
|
||||
for img in ir.get("rendering_images") or []:
|
||||
img_url = img.get("no_wm_download_url") or img.get("download_url") or img.get("image_url")
|
||||
if img_url and img_url not in self.image_list:
|
||||
self.image_list.append(img_url)
|
||||
|
||||
# 提取图文列表(优先提取无水印原画 download_url)
|
||||
for img in share_record.get("rendering_images") or []:
|
||||
img_url = img.get("no_wm_download_url") or img.get("download_url") or img.get("image_url")
|
||||
if img_url and img_url not in self.image_list:
|
||||
self.image_list.append(img_url)
|
||||
|
||||
# 提取封面
|
||||
self.cover_url = artwork.get("cover_image_url") or artwork.get("cover_url")
|
||||
if not self.cover_url and rendering_video:
|
||||
self.cover_url = rendering_video.get("cover_image_url")
|
||||
if not self.cover_url and self.image_list:
|
||||
self.cover_url = self.image_list[0]
|
||||
|
||||
def get_real_video_url(self):
|
||||
return self.video_url
|
||||
|
||||
def get_title_content(self):
|
||||
return self.title or ""
|
||||
|
||||
def get_cover_photo_url(self):
|
||||
return self.cover_url
|
||||
|
||||
def get_author_info(self):
|
||||
return self.author
|
||||
|
||||
def get_image_list(self):
|
||||
return self.image_list
|
||||
@@ -0,0 +1,162 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
from configs.general_constants import USER_AGENT_PC
|
||||
from configs.logging_config import get_logger
|
||||
from src.parser_factory import register_parser
|
||||
from src.parsers.base_parser import BaseParser
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@register_parser("央视")
|
||||
class CCTVParser(BaseParser):
|
||||
"""央视 (CCTV / 央视网 / 央视新闻) 解析器,支持常规栏目节目、央视新闻客户端与云平台视频提取。"""
|
||||
|
||||
def __init__(self, real_url):
|
||||
super().__init__(real_url)
|
||||
self.headers = {
|
||||
"User-Agent": USER_AGENT_PC[0],
|
||||
"Referer": "https://tv.cctv.com/",
|
||||
"Accept": "*/*",
|
||||
}
|
||||
self.title = ""
|
||||
self.cover_url = None
|
||||
self.video_url = None
|
||||
self.author = None
|
||||
self._parse()
|
||||
|
||||
def _parse(self):
|
||||
if not self.real_url:
|
||||
return
|
||||
|
||||
parsed = urlparse(self.real_url)
|
||||
qs = parse_qs(parsed.query)
|
||||
|
||||
# 区分 央视新闻 API 体系 与 CCTV 综合/电视频道体系
|
||||
if "cctvnews" in parsed.netloc or "snow-book" in parsed.path:
|
||||
self._parse_cctvnews(parsed, qs)
|
||||
else:
|
||||
self._parse_cctv_web(parsed, qs)
|
||||
|
||||
if self.title:
|
||||
self.title = re.sub(r'<[^>]+>', '', self.title).strip()
|
||||
|
||||
def _parse_cctvnews(self, parsed, qs):
|
||||
"""解析央视新闻客户端微视频及新闻报道(采用阿里云 API 网关 HMAC-SHA256 签名)"""
|
||||
item_id = qs.get("item_id", [None])[0] or qs.get("articleId", [None])[0]
|
||||
if not item_id:
|
||||
match = re.search(r'(?:item_id|articleId)=([0-9a-zA-Z_\-]+)', self.real_url)
|
||||
if match:
|
||||
item_id = match.group(1)
|
||||
|
||||
if not item_id:
|
||||
logger.warning("Could not extract articleId/item_id from CCTV News URL: %s", self.real_url)
|
||||
return
|
||||
|
||||
app_key = "204133710"
|
||||
app_secret = "etyEuNdA7GvQU7iPZHqnrBpSFfRyKQTD"
|
||||
host = "https://api.cctvnews.cctv.com"
|
||||
path = "/1.0.0/feed/article/server/getArticle"
|
||||
params = {"articleId": item_id, "appcode": "video_web"}
|
||||
t_now = int(time.time() * 1000)
|
||||
|
||||
headers = {
|
||||
"x-ca-timestamp": str(t_now),
|
||||
"x-ca-key": app_key,
|
||||
"x-ca-stage": "RELEASE",
|
||||
"accept": "application/json",
|
||||
"User-Agent": USER_AGENT_PC[0],
|
||||
}
|
||||
ca_headers = sorted([k for k in headers.keys() if k.startswith("x-ca-")])
|
||||
headers["x-ca-signature-headers"] = ",".join(ca_headers)
|
||||
header_str = "\n".join([f"{k}:{headers[k]}" for k in ca_headers])
|
||||
sorted_params = sorted(params.items())
|
||||
query_str = "&".join([f"{k}={v}" for k, v in sorted_params])
|
||||
full_path = f"{path}?{query_str}"
|
||||
string_to_sign = f"GET\n{headers['accept']}\n\n\n\n{header_str}\n{full_path}"
|
||||
sig = base64.b64encode(
|
||||
hmac.new(app_secret.encode("utf-8"), string_to_sign.encode("utf-8"), hashlib.sha256).digest()
|
||||
).decode("utf-8")
|
||||
headers["x-ca-signature"] = sig
|
||||
|
||||
try:
|
||||
res = self.session.get(f"{host}{full_path}", headers=headers, timeout=5)
|
||||
res.raise_for_status()
|
||||
resp_json = res.json()
|
||||
b64_response = resp_json.get("response")
|
||||
if not b64_response:
|
||||
logger.warning("CCTV News response missing encoded payload: %s", resp_json)
|
||||
return
|
||||
|
||||
payload_bytes = base64.b64decode(b64_response)
|
||||
payload = json.loads(payload_bytes.decode("utf-8")).get("data", {})
|
||||
self.title = (payload.get("title") or "").strip()
|
||||
|
||||
videos = payload.get("videos") or []
|
||||
if videos and isinstance(videos, list):
|
||||
self.video_url = videos[0].get("url")
|
||||
cover_info = videos[0].get("cover")
|
||||
if isinstance(cover_info, dict):
|
||||
self.cover_url = cover_info.get("url")
|
||||
|
||||
if not self.cover_url:
|
||||
thumbnails = payload.get("thumbnails") or []
|
||||
if thumbnails and isinstance(thumbnails, list):
|
||||
self.cover_url = thumbnails[0].get("url")
|
||||
|
||||
author_name = payload.get("source") or payload.get("author") or "央视新闻"
|
||||
self.author = {
|
||||
"name": author_name,
|
||||
"avatar": None,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error("Failed to parse CCTV News article: %s", e)
|
||||
|
||||
def _parse_cctv_web(self, parsed, qs):
|
||||
"""解析 tv.cctv.com / tv.cctv.cn 等央视节目详情页(通过 CNTV 视频分配接口)"""
|
||||
pid = qs.get("pid", [None])[0] or qs.get("guid", [None])[0]
|
||||
if not pid:
|
||||
html = self.fetch_html_content()
|
||||
if html:
|
||||
match = re.search(r'(?:guid|pid|videoCenterId)\s*=\s*["\']([0-9a-fA-F]{32})["\']', html)
|
||||
if match:
|
||||
pid = match.group(1)
|
||||
|
||||
if not pid:
|
||||
logger.warning("Could not extract video pid/guid from CCTV URL: %s", self.real_url)
|
||||
return
|
||||
|
||||
api_url = f"https://vdn.apps.cntv.cn/api/getHttpVideoInfo.do?pid={pid}"
|
||||
try:
|
||||
res = self.session.get(api_url, headers=self.headers, timeout=5)
|
||||
res.raise_for_status()
|
||||
data = res.json()
|
||||
except Exception as e:
|
||||
logger.error("Failed to fetch CCTV video info for pid %s: %s", pid, e)
|
||||
return
|
||||
|
||||
self.title = (data.get("title") or "").strip()
|
||||
self.video_url = data.get("hls_url")
|
||||
self.cover_url = data.get("image")
|
||||
column_name = data.get("column") or data.get("produce") or "央视网"
|
||||
self.author = {
|
||||
"name": column_name,
|
||||
"avatar": None,
|
||||
}
|
||||
|
||||
def get_real_video_url(self):
|
||||
return self.video_url
|
||||
|
||||
def get_title_content(self):
|
||||
return self.title or ""
|
||||
|
||||
def get_cover_photo_url(self):
|
||||
return self.cover_url
|
||||
|
||||
def get_author_info(self):
|
||||
return self.author
|
||||
@@ -0,0 +1,141 @@
|
||||
import json
|
||||
import re
|
||||
from bs4 import BeautifulSoup
|
||||
from configs.general_constants import USER_AGENT_M
|
||||
from configs.logging_config import get_logger
|
||||
from src.parser_factory import register_parser
|
||||
from src.parsers.base_parser import BaseParser
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@register_parser("央视频")
|
||||
class YangshipinParser(BaseParser):
|
||||
"""央视频 (Yangshipin) 客户端及 H5 平台解析器,支持横竖屏微短剧/精选视频元数据、原画封面与作者信息提取。"""
|
||||
|
||||
def __init__(self, real_url):
|
||||
super().__init__(real_url)
|
||||
self.headers = {
|
||||
"User-Agent": USER_AGENT_M[0],
|
||||
"Referer": "https://m.yangshipin.cn/",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
}
|
||||
self.title = ""
|
||||
self.cover_url = None
|
||||
self.video_url = None
|
||||
self.image_list = []
|
||||
self.author = None
|
||||
self._parse()
|
||||
|
||||
def _parse(self):
|
||||
if not self.real_url:
|
||||
return
|
||||
|
||||
html = self.fetch_html_content()
|
||||
if not html:
|
||||
logger.warning("Failed to fetch HTML content for Yangshipin URL: %s", self.real_url)
|
||||
return
|
||||
|
||||
# 兼容短链未重定向时页面内的 meta refresh 跳转
|
||||
meta_refresh = re.search(r'''<meta\s+http-equiv=["']refresh["']\s+content=["'][^;]+;\s*URL=['"]([^'"]+)['"]''', html, re.I)
|
||||
if meta_refresh:
|
||||
redirect_url = meta_refresh.group(1).strip()
|
||||
if redirect_url.startswith("/"):
|
||||
redirect_url = f"https://m.yangshipin.cn{redirect_url}"
|
||||
self.real_url = redirect_url
|
||||
html = self.fetch_html_content()
|
||||
if not html:
|
||||
return
|
||||
|
||||
try:
|
||||
# 1. 尝试从横屏视频 STATE 提取 (__STATE_video__)
|
||||
data_vid = self._extract_state_json(html, "window.__STATE_video__")
|
||||
if data_vid:
|
||||
sv = data_vid.get("payloads", {}).get("sharevideo", {})
|
||||
self.title = (sv.get("title") or "").strip()
|
||||
self.cover_url = sv.get("cover_pic")
|
||||
om_info = sv.get("om_info", {})
|
||||
author_name = om_info.get("title")
|
||||
if author_name:
|
||||
self.author = {
|
||||
"name": author_name,
|
||||
"avatar": None,
|
||||
}
|
||||
|
||||
# 2. 尝试从竖屏视频 STATE 提取 (__STATE_portrait_video__)
|
||||
data_port = self._extract_state_json(html, "window.__STATE_portrait_video__")
|
||||
if data_port:
|
||||
items = data_port.get("payloads", {}).get("videoDataList", {}).get("items", [])
|
||||
if items:
|
||||
vd = items[0].get("videoData", {})
|
||||
if not self.title:
|
||||
self.title = (vd.get("title") or "").strip()
|
||||
if not self.cover_url:
|
||||
self.cover_url = (
|
||||
vd.get("shareItem", {}).get("shareImgUrl")
|
||||
or vd.get("poster", {}).get("poster", {}).get("imageUrl")
|
||||
)
|
||||
actor = vd.get("detailFollowItem", {}).get("actorItem", {})
|
||||
if actor:
|
||||
nick_name = actor.get("nickName", {}).get("text")
|
||||
head_url = actor.get("headUrl")
|
||||
if nick_name or head_url:
|
||||
self.author = {
|
||||
"name": nick_name,
|
||||
"avatar": head_url,
|
||||
}
|
||||
|
||||
# 3. 兜底:从 OpenGraph / HTML 标签提取
|
||||
if not self.title or not self.cover_url:
|
||||
soup = BeautifulSoup(html, "lxml")
|
||||
if not self.title:
|
||||
og_title = soup.find("meta", property="og:title")
|
||||
if og_title and og_title.get("content"):
|
||||
self.title = og_title["content"].strip()
|
||||
elif soup.title and soup.title.string:
|
||||
self.title = soup.title.string.strip()
|
||||
|
||||
if not self.cover_url:
|
||||
og_img = soup.find("meta", property="og:image")
|
||||
if og_img and og_img.get("content"):
|
||||
self.cover_url = og_img["content"].strip()
|
||||
|
||||
except Exception as exc:
|
||||
logger.exception("Error parsing Yangshipin page: %s", exc)
|
||||
|
||||
if self.title:
|
||||
self.title = re.sub(r'<[^>]+>', '', self.title).strip()
|
||||
|
||||
if not self.video_url and self.cover_url and self.cover_url not in self.image_list:
|
||||
self.image_list.append(self.cover_url)
|
||||
|
||||
def _extract_state_json(self, html, state_key):
|
||||
"""从页面提取指定 state_key 的 JSON 数据对象"""
|
||||
idx = html.find(state_key)
|
||||
if idx == -1:
|
||||
return None
|
||||
start = html.find("{", idx)
|
||||
if start == -1:
|
||||
return None
|
||||
try:
|
||||
decoder = json.JSONDecoder()
|
||||
data, _ = decoder.raw_decode(html[start:])
|
||||
return data
|
||||
except Exception as e:
|
||||
logger.warning("Failed to decode %s JSON: %s", state_key, e)
|
||||
return None
|
||||
|
||||
def get_real_video_url(self):
|
||||
return self.video_url
|
||||
|
||||
def get_title_content(self):
|
||||
return self.title or ""
|
||||
|
||||
def get_cover_photo_url(self):
|
||||
return self.cover_url
|
||||
|
||||
def get_author_info(self):
|
||||
return self.author
|
||||
|
||||
def get_image_list(self):
|
||||
return self.image_list
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"schema_version": 2,
|
||||
"description": "45 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)",
|
||||
"total_cases": 301,
|
||||
"description": "50 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)",
|
||||
"total_cases": 306,
|
||||
"cases": [
|
||||
{
|
||||
"platform": "网易云音乐",
|
||||
@@ -4378,6 +4378,80 @@
|
||||
],
|
||||
"note": "2026-09-06 已验证: LOFTER 创作者短视频",
|
||||
"pattern": "同人短视频"
|
||||
},
|
||||
{
|
||||
"platform": "星绘AI",
|
||||
"url": "https://www.butterflyai.cn/s/gQ0EXE9BJsw/",
|
||||
"media_types": [
|
||||
"images"
|
||||
],
|
||||
"expected_fields": [
|
||||
"images",
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: 星绘AI 无水印原画图集",
|
||||
"pattern": "AI 生图/写真作品"
|
||||
},
|
||||
{
|
||||
"platform": "星绘AI",
|
||||
"url": "https://www.butterflyai.cn/share/record?share_code=9r3kCJOnwubPE15yFhKY&is_mine=1&resource_count=1&entity_type=2",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: 星绘AI 视频生成作品",
|
||||
"pattern": "AI 视频生成作品"
|
||||
},
|
||||
{
|
||||
"platform": "央视",
|
||||
"url": "https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml?spm=C22284.PP7miAIBYiAV.EIn520ebHszi.32",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: CCTV 栏目点播视频与 HLS 流",
|
||||
"pattern": "综合栏目点播"
|
||||
},
|
||||
{
|
||||
"platform": "央视",
|
||||
"url": "https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092&t=1787103208459&toc_style_id=video_default&share_to=copy_url&track_id=766f3425-9b74-40ac-b2cf-751f9e5a19fd",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: 央视新闻客户端微视频",
|
||||
"pattern": "新闻微视频"
|
||||
},
|
||||
{
|
||||
"platform": "央视频",
|
||||
"url": "https://www.yspapp.cn/5Sqx",
|
||||
"media_types": [
|
||||
"images"
|
||||
],
|
||||
"expected_fields": [
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: 央视频短链与微视频详情",
|
||||
"pattern": "微视频及短剧详情"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
import unittest
|
||||
from unittest.mock import patch, Mock
|
||||
|
||||
from src.parsers.butterflyai_parser import ButterflyAIParser
|
||||
|
||||
|
||||
class ButterflyAIParserTest(unittest.TestCase):
|
||||
def test_parses_image_post_successfully(self):
|
||||
fake_api_resp = {
|
||||
"status_code": 0,
|
||||
"status_info": {"status_code": 0, "status_msg": "Success"},
|
||||
"share_record": {
|
||||
"artwork": {
|
||||
"title": "夏日清凉少女写真",
|
||||
"cover_image_url": "https://p26-community.butterfly.cn/cover.jpg",
|
||||
},
|
||||
"creator": {
|
||||
"screen_name": "星绘创作师",
|
||||
"avatar": {
|
||||
"large_url": "https://p3-passport.byteacctimg.com/avatar_large.jpeg",
|
||||
"image_url": "https://p3-passport.byteacctimg.com/avatar.jpeg",
|
||||
},
|
||||
},
|
||||
"rendering_images": [
|
||||
{
|
||||
"no_wm_download_url": "https://p26-community.butterfly.cn/pic1_hd.jpg",
|
||||
"download_url": "https://p26-community.butterfly.cn/pic1_wm.jpg",
|
||||
},
|
||||
{
|
||||
"image_url": "https://p26-community.butterfly.cn/pic2.jpg",
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
mock_resp = Mock()
|
||||
mock_resp.status_code = 200
|
||||
mock_resp.json.return_value = fake_api_resp
|
||||
mock_resp.raise_for_status = Mock()
|
||||
|
||||
with patch("requests.Session.post", return_value=mock_resp):
|
||||
parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=abc123Test")
|
||||
self.assertEqual(parser.get_title_content(), "夏日清凉少女写真")
|
||||
self.assertEqual(
|
||||
parser.get_author_info(),
|
||||
{"name": "星绘创作师", "avatar": "https://p3-passport.byteacctimg.com/avatar_large.jpeg"},
|
||||
)
|
||||
self.assertEqual(
|
||||
parser.get_image_list(),
|
||||
[
|
||||
"https://p26-community.butterfly.cn/pic1_hd.jpg",
|
||||
"https://p26-community.butterfly.cn/pic2.jpg",
|
||||
],
|
||||
)
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://p26-community.butterfly.cn/cover.jpg")
|
||||
self.assertIsNone(parser.get_real_video_url())
|
||||
|
||||
def test_parses_video_post_successfully(self):
|
||||
fake_api_resp = {
|
||||
"status_code": 0,
|
||||
"status_info": {"status_code": 0, "status_msg": "Success"},
|
||||
"share_record": {
|
||||
"show_info": {"effect_title": "AI 视频变换"},
|
||||
"creator": {
|
||||
"screen_name": "特效大师",
|
||||
"avatar_url": "https://p3-passport.byteacctimg.com/avatar_str.jpeg",
|
||||
},
|
||||
"rendering_video": {
|
||||
"video_url": "https://v-butterfly.douyinvod.com/video1.mp4",
|
||||
"cover_image_url": "https://v-butterfly.douyinvod.com/cover1.jpg",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
mock_resp = Mock()
|
||||
mock_resp.status_code = 200
|
||||
mock_resp.json.return_value = fake_api_resp
|
||||
mock_resp.raise_for_status = Mock()
|
||||
|
||||
with patch("requests.Session.post", return_value=mock_resp):
|
||||
parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=vid123")
|
||||
self.assertEqual(parser.get_title_content(), "AI 视频变换")
|
||||
self.assertEqual(
|
||||
parser.get_author_info(),
|
||||
{"name": "特效大师", "avatar": "https://p3-passport.byteacctimg.com/avatar_str.jpeg"},
|
||||
)
|
||||
self.assertEqual(parser.get_real_video_url(), "https://v-butterfly.douyinvod.com/video1.mp4")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://v-butterfly.douyinvod.com/cover1.jpg")
|
||||
self.assertEqual(parser.get_image_list(), [])
|
||||
|
||||
def test_handles_api_failure_gracefully(self):
|
||||
mock_resp = Mock()
|
||||
mock_resp.status_code = 500
|
||||
mock_resp.raise_for_status.side_effect = Exception("API Internal Error")
|
||||
|
||||
with patch("requests.Session.post", return_value=mock_resp):
|
||||
parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=fail123")
|
||||
self.assertEqual(parser.get_title_content(), "")
|
||||
self.assertIsNone(parser.get_real_video_url())
|
||||
self.assertIsNone(parser.get_author_info())
|
||||
self.assertEqual(parser.get_image_list(), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,82 @@
|
||||
import base64
|
||||
import json
|
||||
import unittest
|
||||
from unittest.mock import patch, Mock
|
||||
|
||||
from src.parsers.cctv_parser import CCTVParser
|
||||
|
||||
|
||||
class CCTVParserTest(unittest.TestCase):
|
||||
def test_parses_cntv_web_video_successfully(self):
|
||||
fake_html = """
|
||||
<html>
|
||||
<head><title>生财有道</title></head>
|
||||
<body>
|
||||
<script>
|
||||
var guid = "00e121ae62194fec8ab20e5b8eb9a89a";
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
fake_api_data = {
|
||||
"title": "《生财有道》 20260526 匠心织锦绣",
|
||||
"hls_url": "https://hls.cntv.lxdns.com/asp/hls/main/0303000a/3/default/00e12/main.m3u8",
|
||||
"image": "https://p1.img.cctvpic.com/cover.jpg",
|
||||
"column": "生财有道",
|
||||
}
|
||||
|
||||
mock_api_resp = Mock()
|
||||
mock_api_resp.status_code = 200
|
||||
mock_api_resp.json.return_value = fake_api_data
|
||||
mock_api_resp.raise_for_status = Mock()
|
||||
|
||||
with patch.object(CCTVParser, "fetch_html_content", return_value=fake_html), \
|
||||
patch("requests.Session.get", return_value=mock_api_resp):
|
||||
parser = CCTVParser("https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml")
|
||||
self.assertEqual(parser.get_title_content(), "《生财有道》 20260526 匠心织锦绣")
|
||||
self.assertEqual(
|
||||
parser.get_real_video_url(),
|
||||
"https://hls.cntv.lxdns.com/asp/hls/main/0303000a/3/default/00e12/main.m3u8",
|
||||
)
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://p1.img.cctvpic.com/cover.jpg")
|
||||
self.assertEqual(parser.get_author_info(), {"name": "生财有道", "avatar": None})
|
||||
|
||||
def test_parses_cctvnews_article_successfully(self):
|
||||
fake_payload = {
|
||||
"data": {
|
||||
"title": "今天是第九个中国医师节",
|
||||
"source": "央视新闻客户端",
|
||||
"videos": [
|
||||
{
|
||||
"url": "https://res.cctvnews.cctv.com/video/stream.m3u8",
|
||||
"cover": {"url": "https://img.cctvnews.cctv.com/cover1.jpg"},
|
||||
}
|
||||
],
|
||||
}
|
||||
}
|
||||
b64_str = base64.b64encode(json.dumps(fake_payload).encode("utf-8")).decode("utf-8")
|
||||
fake_api_resp = {"code": 0, "response": b64_str}
|
||||
|
||||
mock_resp = Mock()
|
||||
mock_resp.status_code = 200
|
||||
mock_resp.json.return_value = fake_api_resp
|
||||
mock_resp.raise_for_status = Mock()
|
||||
|
||||
with patch("requests.Session.get", return_value=mock_resp):
|
||||
url = "https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092"
|
||||
parser = CCTVParser(url)
|
||||
self.assertEqual(parser.get_title_content(), "今天是第九个中国医师节")
|
||||
self.assertEqual(parser.get_real_video_url(), "https://res.cctvnews.cctv.com/video/stream.m3u8")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://img.cctvnews.cctv.com/cover1.jpg")
|
||||
self.assertEqual(parser.get_author_info(), {"name": "央视新闻客户端", "avatar": None})
|
||||
|
||||
def test_handles_missing_pid_gracefully(self):
|
||||
with patch.object(CCTVParser, "fetch_html_content", return_value="<html><body>No video</body></html>"):
|
||||
parser = CCTVParser("https://tv.cctv.com/empty.shtml")
|
||||
self.assertEqual(parser.get_title_content(), "")
|
||||
self.assertIsNone(parser.get_real_video_url())
|
||||
self.assertIsNone(parser.get_cover_photo_url())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,122 @@
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from src.parsers.yangshipin_parser import YangshipinParser
|
||||
|
||||
|
||||
class YangshipinParserTest(unittest.TestCase):
|
||||
def test_parses_portrait_video_successfully(self):
|
||||
fake_html = """
|
||||
<html>
|
||||
<head><title>央视频</title></head>
|
||||
<body>
|
||||
<script>
|
||||
window.__STATE_portrait_video__ = {
|
||||
"payloads": {
|
||||
"videoDataList": {
|
||||
"items": [
|
||||
{
|
||||
"videoData": {
|
||||
"vid": "l00005817wl",
|
||||
"title": "5次助攻对4次助攻!巅峰对决",
|
||||
"shareItem": {
|
||||
"shareImgUrl": "https://jietufengmian.yangshipin.cn/cover1.jpg"
|
||||
},
|
||||
"detailFollowItem": {
|
||||
"actorItem": {
|
||||
"nickName": {"text": "奥运来了"},
|
||||
"headUrl": "https://mpuser.ysp.cctv.cn/avatar1.jpeg"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
};
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
with patch.object(YangshipinParser, "fetch_html_content", return_value=fake_html):
|
||||
parser = YangshipinParser("https://m.yangshipin.cn/portrait_video?vid=l00005817wl")
|
||||
self.assertEqual(parser.get_title_content(), "5次助攻对4次助攻!巅峰对决")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover1.jpg")
|
||||
self.assertEqual(
|
||||
parser.get_author_info(),
|
||||
{"name": "奥运来了", "avatar": "https://mpuser.ysp.cctv.cn/avatar1.jpeg"},
|
||||
)
|
||||
self.assertEqual(parser.get_image_list(), ["https://jietufengmian.yangshipin.cn/cover1.jpg"])
|
||||
|
||||
def test_parses_landscape_video_successfully(self):
|
||||
fake_html = """
|
||||
<html>
|
||||
<head><title>央视频</title></head>
|
||||
<body>
|
||||
<script>
|
||||
window.__STATE_video__ = {
|
||||
"payloads": {
|
||||
"sharevideo": {
|
||||
"vid": "v000007pgfu",
|
||||
"title": "《普法栏目剧》远山的守望",
|
||||
"cover_pic": "https://jietufengmian.yangshipin.cn/cover2.jpg",
|
||||
"om_info": {
|
||||
"title": "社会与法频道"
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
with patch.object(YangshipinParser, "fetch_html_content", return_value=fake_html):
|
||||
parser = YangshipinParser("https://m.yangshipin.cn/video?type=0&vid=v000007pgfu")
|
||||
self.assertEqual(parser.get_title_content(), "《普法栏目剧》远山的守望")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover2.jpg")
|
||||
self.assertEqual(parser.get_author_info(), {"name": "社会与法频道", "avatar": None})
|
||||
self.assertEqual(parser.get_image_list(), ["https://jietufengmian.yangshipin.cn/cover2.jpg"])
|
||||
|
||||
def test_follows_meta_refresh_redirect(self):
|
||||
meta_html = """
|
||||
<!DOCTYPE html>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="refresh" content="0; URL='https://m.yangshipin.cn/portrait_video?vid=5Sqx'"/>
|
||||
<title>央视频</title>
|
||||
"""
|
||||
detail_html = """
|
||||
<html><body>
|
||||
<script>
|
||||
window.__STATE_portrait_video__ = {
|
||||
"payloads": {
|
||||
"videoDataList": {
|
||||
"items": [
|
||||
{
|
||||
"videoData": {
|
||||
"title": "跳转后的视频标题",
|
||||
"shareItem": {"shareImgUrl": "https://jietufengmian.yangshipin.cn/cover3.jpg"}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
};
|
||||
</script>
|
||||
</body></html>
|
||||
"""
|
||||
with patch.object(YangshipinParser, "fetch_html_content", side_effect=[meta_html, detail_html]):
|
||||
parser = YangshipinParser("https://www.yspapp.cn/5Sqx")
|
||||
self.assertEqual(parser.get_title_content(), "跳转后的视频标题")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover3.jpg")
|
||||
|
||||
def test_handles_empty_or_broken_html(self):
|
||||
with patch.object(YangshipinParser, "fetch_html_content", return_value="<html><body>404 Not Found</body></html>"):
|
||||
parser = YangshipinParser("https://m.yangshipin.cn/video?vid=empty")
|
||||
self.assertEqual(parser.get_title_content(), "")
|
||||
self.assertIsNone(parser.get_cover_photo_url())
|
||||
self.assertIsNone(parser.get_author_info())
|
||||
self.assertEqual(parser.get_image_list(), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+56
-2
@@ -22,8 +22,8 @@ class WebFetcher:
|
||||
# 绿洲分享页会对通用桌面请求头返回 500,由解析器使用移动端请求头抓取。
|
||||
# 微博博文可通过 API 直接根据 ID 解析,直接访问网页端常触发访客系统重定向。
|
||||
domain = UrlParser.get_domain(url)
|
||||
if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan", "dw4.co"}:
|
||||
if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI", "网易LOFTER", "得物"}:
|
||||
if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan", "dw4.co", "yspapp.cn"}:
|
||||
if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI", "网易LOFTER", "得物", "央视"}:
|
||||
return UrlParser.extract_video_address(url)
|
||||
try:
|
||||
current_url = url
|
||||
@@ -32,6 +32,10 @@ class WebFetcher:
|
||||
resp = requests.get(current_url, headers=WebFetcher.headers, allow_redirects=False, timeout=5)
|
||||
resp.raise_for_status()
|
||||
redirect_url = resp.headers.get("location")
|
||||
if not redirect_url and resp.status_code == 200 and isinstance(getattr(resp, "text", None), str):
|
||||
meta_match = re.search(r'<meta[^>]*http-equiv=["\']?refresh["\']?[^>]*content=["\']?[^"\']*URL=[\'"]?([^\'" >]+)', resp.text, re.I)
|
||||
if meta_match:
|
||||
redirect_url = meta_match.group(1).strip("'\"")
|
||||
if redirect_url:
|
||||
redirect_url = urljoin(current_url, redirect_url)
|
||||
# 如果重定向到了登录页、404拦截页、验证码校验页或错误页,不要更新 url,直接中断以保留原始有效 URL
|
||||
@@ -105,6 +109,15 @@ class UrlParser:
|
||||
if domain.endswith('.dewu.com') or domain.endswith('.poizon.com'):
|
||||
return '得物'
|
||||
|
||||
if domain == 'butterflyai.cn' or domain.endswith('.butterflyai.cn'):
|
||||
return '星绘AI'
|
||||
|
||||
if domain in {'cctv.com', 'cctv.cn'} or domain.endswith('.cctv.com') or domain.endswith('.cctv.cn'):
|
||||
return '央视'
|
||||
|
||||
if domain in {'yangshipin.cn', 'yspapp.cn'} or domain.endswith('.yangshipin.cn') or domain.endswith('.yspapp.cn'):
|
||||
return '央视频'
|
||||
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
@@ -365,6 +378,33 @@ class UrlParser:
|
||||
preserved_params.append((key, value))
|
||||
if preserved_params:
|
||||
address = f"{address}?{urlencode(preserved_params)}"
|
||||
elif platform == "星绘AI":
|
||||
query_params = parse_qs(parsed_url.query)
|
||||
preserved_params = []
|
||||
for key in ("share_code", "share_id", "entity_type", "is_mine", "resource_count", "share_token"):
|
||||
value = query_params.get(key, [None])[0]
|
||||
if value is not None:
|
||||
preserved_params.append((key, value))
|
||||
if preserved_params:
|
||||
address = f"{address}?{urlencode(preserved_params)}"
|
||||
elif platform == "央视":
|
||||
query_params = parse_qs(parsed_url.query)
|
||||
preserved_params = []
|
||||
for key in ("item_id", "articleId", "pid", "guid"):
|
||||
value = query_params.get(key, [None])[0]
|
||||
if value is not None:
|
||||
preserved_params.append((key, value))
|
||||
if preserved_params:
|
||||
address = f"{address}?{urlencode(preserved_params)}"
|
||||
elif platform == "央视频":
|
||||
query_params = parse_qs(parsed_url.query)
|
||||
preserved_params = []
|
||||
for key in ("vid", "cid", "type", "serverFrom"):
|
||||
value = query_params.get(key, [None])[0]
|
||||
if value is not None:
|
||||
preserved_params.append((key, value))
|
||||
if preserved_params:
|
||||
address = f"{address}?{urlencode(preserved_params)}"
|
||||
return address
|
||||
|
||||
@staticmethod
|
||||
@@ -446,12 +486,26 @@ class UrlParser:
|
||||
params_ps = query_params.get('ps', [None])[0]
|
||||
if params_ps:
|
||||
return params_ps
|
||||
params_share_code = query_params.get('share_code', [None])[0]
|
||||
if params_share_code:
|
||||
return params_share_code
|
||||
params_item_id = query_params.get('item_id', [None])[0]
|
||||
if params_item_id:
|
||||
return params_item_id
|
||||
params_article_id = query_params.get('articleId', [None])[0]
|
||||
if params_article_id:
|
||||
return params_article_id
|
||||
params_guid = query_params.get('guid', [None])[0]
|
||||
if params_guid:
|
||||
return params_guid
|
||||
# 尝试从URL路径中获取视频ID
|
||||
path_segments = parsed_url.path.strip('/').split('/')
|
||||
if path_segments:
|
||||
video_id = path_segments[-1]
|
||||
if video_id.endswith('.html'):
|
||||
video_id = video_id[:-5]
|
||||
elif video_id.endswith('.shtml'):
|
||||
video_id = video_id[:-6]
|
||||
desktop_mv = re.fullmatch(r'mv_([0-9a-fA-F]{32})', video_id)
|
||||
if desktop_mv:
|
||||
return desktop_mv.group(1)
|
||||
|
||||
Reference in New Issue
Block a user