新增星绘AI、央视与央视频平台解析支持

This commit is contained in:
ucmao
2026-09-06 10:38:13 +08:00
parent e2e6634dbf
commit 30d6a0648b
15 changed files with 1131 additions and 76 deletions
+67 -64
View File
@@ -3,10 +3,10 @@
**基于 Python 的多平台媒体原生本地解析系统**
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) [![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/) [![Flask](https://img.shields.io/badge/Framework-Flask-000000?logo=flask&logoColor=white)](https://flask.palletsprojects.com/) [![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?logo=docker&logoColor=white)](#-部署指南) [![Support](https://img.shields.io/badge/support-47%20Platforms-brightgreen.svg)](#-支持的平台矩阵)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) [![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/) [![Flask](https://img.shields.io/badge/Framework-Flask-000000?logo=flask&logoColor=white)](https://flask.palletsprojects.com/) [![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?logo=docker&logoColor=white)](#-部署指南) [![Support](https://img.shields.io/badge/support-50%20Platforms-brightgreen.svg)](#-支持的平台矩阵)
<p align="center">
<a href="#-核心解析逻辑">解析逻辑</a> •
<a href="#-核心特性">核心特性</a> •
<a href="#-支持的平台矩阵">支持平台</a> •
<a href="#-部署指南">部署指南</a> •
<a href="#-api-核心接口说明">接口文档</a> •
@@ -15,63 +15,81 @@
<a href="#-联系作者">联系作者</a>
</p>
媒体解析去水印是一款专为短视频创作者与开发者打造的**原生本地解析工具**。
媒体解析去水印是一款专为短视频创作者与开发者打造的**100%原生本地解析工具**。
通过“智能识别 -> 本地抓取 -> 提取地址 -> 快捷下载”的闭环,助你高效获取无水印素材。
**100%本地原生实现,不依赖外部API,不封装第三方解析库,核心逆向逻辑完全开源。**
**不依赖外部API,不套壳第三方库,无浏览器开销,纯底层协议与算法逆向。**
</div>
---
## 💎 核心解析逻辑
## ✨ 核心特性
* **多平台智能适配**:内置 `ParserFactory` 工厂模式,自动识别链接来源并分配对应解析器。
* **原生本地解析**:解析逻辑直接内置在项目代码中,由各平台 Parser 本地发起请求并提取真实媒体地址。
* **零外部API依赖**:无需对接任何第三方代解析接口或 SaaS 中转服务,部署后即可独立稳定运行。
* **开箱即用API**:提供标准化的 JSON 数据接口,无冗余数据库依赖,适合快速对接业务或二次开发。
## ✨ 项目特点
* **本地可控**:所有请求链路与解析规则均在本地代码中,方便调试、维护与定制。
* **部署简单**:安装依赖或使用 Docker 即可一键运行,不需要额外申请第三方 API 账号或密钥。
* **便于扩展**:各平台对应独立 Parser,遵循统一的返回数据结构,新增平台轻松快捷。
* ⚡ **极速轻量**:纯 HTTP 网络协议与底层算法逆向,免启动 Chromium/Playwright 等笨重浏览器,内存占用极低(<100MB),毫秒级极速响应。
* 🔒 **原生自主**:100% 本地代码闭环抓取,零外部商用 API 或第三方代解析依赖,数据链路自主可控,杜绝断流与隐私泄露风险。
* 🧩 **插件化解耦**:内置 `ParserFactory` 模块自动发现与工厂分发机制,50 个平台独立解耦,遵循统一的数据契约,新增与维护平台极度轻松。
* 🔌 **开箱即用**:提供标准 RESTful JSON 接口与 Web 体验页,无外部数据库等冗余依赖,Docker Compose 一键构建,轻松无缝接入各类业务系统。
---
## 💾 支持的平台矩阵
| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **抖音** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| **小红书** | ✓ | ✓ | ✓ | ✓ | **✓+** | | | **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | |
| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | |
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **微视** | ✓ | ✓ | ✓ | ✓ | | | |
| **快手** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | | **AcFun** | ✓ | ✓ | ✓ | ✓ | | | |
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | |
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **今日头条** | ✓ | ✓ | ✓ | ✓ | | | |
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | |
| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | **全民K歌** | | ✓ | ✓ | ✓ | | | |
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **新片场** | ✓ | ✓ | ✓ | ✓ | | | |
| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **好看视频** | ✓ | ✓ | ✓ | ✓ | | | |
| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **梨视频** | ✓ | ✓ | ✓ | ✓ | | | |
| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
| **闲鱼** | | ✓ | ✓ | | ✓ | | | **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **虎牙** | | ✓ | ✓ | ✓ | | | |
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **美拍** | ✓ | ✓ | ✓ | ✓ | | | |
| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | **最右** | ✓ | ✓ | | ✓ | | | |
| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | |
| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | |
| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | |
| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | | | | | | | | |
| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 实况 |
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ |
| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | |
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ |
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | |
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | |
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **闲鱼** | | ✓ | ✓ | | ✓ | | | |
| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | |
| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | |
| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | |
| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | |
| **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
| **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | | |
| **微视** | ✓ | ✓ | ✓ | ✓ | | | | |
| **AcFun** | ✓ | ✓ | ✓ | ✓ | | | | |
| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | | |
| **今日头条** | ✓ | ✓ | ✓ | ✓ | | | | |
| **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | | |
| **全民K歌** | | ✓ | ✓ | ✓ | | | | |
| **新片场** | ✓ | ✓ | ✓ | ✓ | | | | |
| **好看视频** | ✓ | ✓ | ✓ | ✓ | | | | |
| **梨视频** | ✓ | ✓ | ✓ | ✓ | | | | |
| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **虎牙** | | ✓ | ✓ | ✓ | | | | |
| **美拍** | ✓ | ✓ | ✓ | ✓ | | | | |
| **最右** | ✓ | ✓ | | ✓ | | | | |
| **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | | |
| **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | | |
| **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | | |
| **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **星绘AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
| **央视** | ✓ | ✓ | ✓ | ✓ | | | | |
| **央视频** | ✓ | ✓ | ✓ | | ✓ | | | |
> 💡 **说明**:
> * **图集 `✓+`**:表示该平台支持提取 iOS/Android 原始 **LivePhoto 动态实况图**(含封面静图与关联 `.mov`/`.mp4` 实况视频原件)。
> * **原生水印提示**:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。
*注:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。
---
@@ -132,32 +150,17 @@ python app.py
## 🔌 API 核心接口说明
### 1. 服务健康检查
* **接口路径**:`GET /api/health`
* **接口描述**:供容器编排(Docker Compose/K8s)、反向代理或监控系统探针检测服务存活状态。
* **返回示例**:
```json
{
"status": "ok"
}
```
---
### 2. 媒体解析接口
* **接口路径**:`POST /api/parse`
* **接口描述**:传入包含分享链接的文本,智能提取多媒体直链与图文信息。
#### 请求参数 (Request Body)
### 请求参数 (Request Body)
格式: `application/json`
| 参数名 | 类型 | 必填 | 描述 | 限制与示例 |
| --- | --- | --- | --- | --- |
| `text` | `string` | 是 | 视频分享链接或包含链接的文本短语 | 最长 2000 字符,如 `"https://v.douyin.com/..."` |
#### 返回说明 (Response)
### 返回说明 (Response)
格式: `application/json`
成功响应示例:
@@ -210,10 +213,10 @@ python app.py
## 🧪 自动化测试与健康自检
本项目拥有完备的双层测试体系(Mock 单元测试 + 47 平台真实在线样本库回归)。遇到解析异常或日常部署验证时,可一键运行健康自检:
本项目拥有完备的双层测试体系(Mock 单元测试 + 50 平台真实在线样本库回归)。遇到解析异常或日常部署验证时,可一键运行健康自检:
```bash
# 47 平台极速冒烟测试(每个平台测 1 条最具代表性的链接,秒级完成健康检查)
# 50 平台极速冒烟测试(每个平台测 1 条最具代表性的链接,秒级完成健康检查)
python3 tests/manual_verify_parsers.py --limit 1
# 仅验证单个或指定平台(如:小云雀AI / 抖音)
@@ -231,7 +234,7 @@ python3 tests/manual_verify_parsers.py --platform "小云雀AI"
* 🏗️ **[系统架构与生命周期设计](docs/architecture.md)**:分层设计、302 追踪与 `ParserFactory` 动态发现机制。
* 🔍 **[通用逆向方法论](docs/reverse-guide.md)**:SSR 数据提取、H5 接口伪装、JS 签名沙箱及抓包 SOP。
* 🧪 **[测试体系与回归验证](docs/testing.md)**:Pytest 单元测试、Mock 与线上真实样例回归。
* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 47 个平台的技术文档。
* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 50 个平台的技术文档。
---
+13 -1
View File
@@ -142,7 +142,19 @@
"poizon.com": "得物",
"www.poizon.com": "得物",
"lofter.com": "网易LOFTER",
"www.lofter.com": "网易LOFTER"
"www.lofter.com": "网易LOFTER",
"butterflyai.cn": "星绘AI",
"www.butterflyai.cn": "星绘AI",
"cctv.com": "央视",
"tv.cctv.com": "央视",
"cctv.cn": "央视",
"tv.cctv.cn": "央视",
"content-static.cctvnews.cctv.com": "央视",
"cctvnews.cctv.com": "央视",
"news.cctv.com": "央视",
"yspapp.cn": "央视频",
"yangshipin.cn": "央视频",
"m.yangshipin.cn": "央视频"
},
"USER_AGENT_PC": [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
+6 -3
View File
@@ -2,7 +2,7 @@
欢迎查阅 **Media Parser** 开发者文档与逆向工程知识库。
本项目是一个高性能、模块化、支持 **47 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。
本项目是一个高性能、模块化、支持 **50 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。
---
@@ -14,8 +14,8 @@
* 📖 **平台实战指南 (Parser Guides)**:
* **短视频与轻社区**:[抖音](parsers/douyin.md) | [快手](parsers/kuaishou.md) | [今日头条](parsers/xigua.md) | [皮皮虾](parsers/pipixia.md) | [皮皮搞笑](parsers/pipigaoxiao.md) | [最右](parsers/zuiyou.md) | [美拍](parsers/meipai.md) | [微视](parsers/weishi.md) | [绿洲](parsers/lvzhou.md) | [番茄小说/红果短剧/红果漫剧](parsers/fanqie.md)
* **图文与综合社区**:[小红书](parsers/xiaohongshu.md) | [微信公众号](parsers/wechat-mp.md) | [微博](parsers/weibo.md) | [知乎](parsers/zhihu.md) | [闲鱼](parsers/xianyu.md) | [Soul](parsers/soul.md) | [得物](parsers/dewu.md) | [网易LOFTER](parsers/lofter.md)
* **长视频与弹幕/创作**:[哔哩哔哩](parsers/bilibili.md) | [AcFun](parsers/acfun.md) | [新片场](parsers/xinpianchang.md) | [好看视频](parsers/haokan.md) | [西瓜视频](parsers/xigua.md) | [剪映](parsers/jianying.md) | [快影](parsers/kwaiying.md)
* **AI 生成与大模型**:[豆包 AI](parsers/doubao.md) | [腾讯元宝](parsers/yuanbao.md) | [即梦 AI](parsers/jimeng.md) | [可灵 AI](parsers/kling.md) | [海螺 AI](parsers/hailuo.md) | [通义千问](parsers/qianwen.md) | [夸克 AI](parsers/quark-ai.md) | [小云雀 AI](parsers/xiaoyunque.md) | [松果时刻](parsers/pinecone-moment.md)
* **长视频与电视/融媒**:[哔哩哔哩](parsers/bilibili.md) | [AcFun](parsers/acfun.md) | [央视](parsers/cctv.md) | [央视频](parsers/yangshipin.md) | [新片场](parsers/xinpianchang.md) | [好看视频](parsers/haokan.md) | [西瓜视频](parsers/xigua.md) | [剪映](parsers/jianying.md) | [快影](parsers/kwaiying.md)
* **AI 生成与大模型**:[豆包 AI](parsers/doubao.md) | [星绘 AI](parsers/butterflyai.md) | [腾讯元宝](parsers/yuanbao.md) | [即梦 AI](parsers/jimeng.md) | [可灵 AI](parsers/kling.md) | [海螺 AI](parsers/hailuo.md) | [通义千问](parsers/qianwen.md) | [夸克 AI](parsers/quark-ai.md) | [小云雀 AI](parsers/xiaoyunque.md) | [松果时刻](parsers/pinecone-moment.md)
* **音频与垂直频道**:[汽水音乐](parsers/qsmusic.md) | [QQ音乐](parsers/qqmusic.md) | [网易云音乐](parsers/netease-music.md) | [酷狗音乐](parsers/kugou-music.md) | [配音秀](parsers/peiyinxiu.md) | [全民K歌](parsers/quanminkge.md) | [虎牙](parsers/huya.md) | [梨视频](parsers/lishipin.md) | [微信视频号](parsers/wechat-channels.md) | [腾讯频道](parsers/tencent-channel.md)
---
@@ -76,6 +76,9 @@
| 45 | **松果时刻** | AI 故事多页视频 / 图集 / 配音 | ✅ 支持 | 🟢 公开内容免配置 | 公开详情接口 | [查看指南](parsers/pinecone-moment.md) |
| 46 | **得物** | 潮流穿搭 / 开箱短视频 / 高清图集 | ✅ 支持 | 🟢 免配置 | Next.js metaOGInfo 数据提取 | [查看指南](parsers/dewu.md) |
| 47 | **网易LOFTER** | 二次元插画 / 摄影图集 / 创作者短视频 | ✅ 支持 | 🟢 免配置 | window.__initialize_data__ 数据提取 | [查看指南](parsers/lofter.md) |
| 48 | **星绘 AI** | AI 生图 / 原画写真 / AI 视频 | ✅ 支持 | 🟢 免配置 | 字节分享 API (Community Service) | [查看指南](parsers/butterflyai.md) |
| 49 | **央视** | 栏目视频 (HLS/m3u8) / 央视新闻微视频 | ✅ 支持 | 🟢 免配置 | CNTV 调度接口 + 阿里云 API 网关 HMAC-SHA256 验签 | [查看指南](parsers/cctv.md) |
| 50 | **央视频** | 微视频 / 微短剧 / 高清封面与作者信息 | ✅ 支持 | 🟢 免配置 | 页面 SSR 状态机 (__STATE__) 提取与 Meta 跳转跟踪 | [查看指南](parsers/yangshipin.md) |
---
+48
View File
@@ -0,0 +1,48 @@
# 星绘 AI (ButterflyAI) 逆向解析指南
本篇详细记录字节跳动豆包大模型家族旗下的 AI 艺术与生图/生视频平台 **星绘 AI (ButterflyAI)** 的作品解析方案。
---
## 1. 平台特征与支持能力
* **平台标识**:`星绘AI`
* **支持媒体类型**:无水印超清原画图集 / AI 视频生成作品 (MP4) / 封面 / 标题文案 / 创作者信息
* **常见链接形态**:
* 官方分享短链:`https://www.butterflyai.cn/s/gQ0EXE9BJsw/`
* 社区分享长链:`https://www.butterflyai.cn/share/record?share_code=PRr8gE3n3eENtAl5FOtN&is_mine=1&resource_count=1&entity_type=1`
* H5 落地页:`https://www.butterflyai.cn/butterfly/community/h5/share/index.html?share_code=...`
* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。
---
## 2. 核心逆向流程
1. **短链展开与参数定位**:
* 用户分享获得的链接一般为 `https://www.butterflyai.cn/s/{code}/` 短链形态。
* 通过 `WebFetcher.fetch_redirect_url` 跟随 302 重定向定位至落地页,提取关键参数 `share_code`(或 `share_token`)与 `share_id`。
2. **社区分享开放接口**:
* 请求端点:`POST https://www.butterflyai.cn/butterfly/community/v1/share/record/get?aid=564650`
* 请求载荷:
```json
{
"share_record_id": "",
"from_h5": true,
"share_token": "<share_code>",
"extra_params": {
"aid": "564650",
"app_name": "星绘"
}
}
```
3. **数据结构与媒体提取**:
* **无水印原画图集**:从 `share_record.rendering_images` 或 `artwork.resource_list[].image_resource.rendering_images` 中提取图片。接口原生提供 `no_wm_download_url`(无水印原画下载直链),优先级高于带水印的 `download_url` 与 `image_url`。
* **AI 视频作品**:从 `share_record.rendering_video.video_url` 或 `artwork.resource_list[].video_resource.rendering_videos[].video_url` 中提取原生 MP4 视频直链。
* **元数据**:提取 `artwork.title`、Prompt、`show_info.effect_title`(如 AI 写真、AI 视频等),以及 `creator.screen_name` 和多档高清头像 `creator.avatar.large_url`。
---
## 3. 测试与验证
* **单元测试**:[tests/test_butterflyai_parser.py](file:///Users/leo/Projects/media-parser/tests/test_butterflyai_parser.py)
* **执行命令**:`python3 -m unittest tests/test_butterflyai_parser.py`
+50
View File
@@ -0,0 +1,50 @@
# 央视 (CCTV / 央视网 / 央视新闻) 逆向解析指南
本篇详细记录国家级综合电视广播与新闻门户 **央视 (CCTV)** 的电视节目、综合栏目点播与央视新闻微视频解析方案。
---
## 1. 平台特征与支持能力
* **平台标识**:`央视`
* **支持媒体类型**:高清 HLS 视频流 (.m3u8) / 央视新闻微视频 (MP4/m3u8) / 封面 / 标题文案 / 栏目与来源作者
* **常见链接形态**:
* 央视电视点播:`https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml`
* 央视视频短链/手机端:`https://tv.cctv.cn/v/v1/VIDE4y5JLGNL5QIKkj1JrGnL160430.html`
* 央视新闻微视频/报道:`https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092`
* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。
---
## 2. 核心逆向流程
央视内容体系主要分为 **CCTV 综合/栏目点播体系** 与 **央视新闻客户端 (CCTVNews) 体系**,Parser 自动根据域名与路径进行路由分发:
### 体系 A:CCTV 电视与栏目点播 (`tv.cctv.com`, `tv.cctv.cn`)
1. **PID / GUID 提取**:
* 页面内通过全局 JS 变量声明 32 位 Hex 视频全局标识,例如 `guid = "00e121ae62194fec8ab20e5b8eb9a89a"` 或 `pid = "..."`。
2. **CNTV 视频调度分配接口**:
* 请求端点:`GET https://vdn.apps.cntv.cn/api/getHttpVideoInfo.do?pid={pid}`
* 直接提取 `hls_url`(官方多码率 HLS 流地址)、`image`(官方高分辨率静态封面)、`title`(节目期数与标题)及 `column`(栏目名称)。
### 体系 B:央视新闻客户端 (`content-static.cctvnews.cctv.com`)
1. **文章 / 微视频 ID**:
* 从 URL 参数中提取 `item_id` 或 `articleId`。
2. **阿里云 API 网关 HMAC-SHA256 签名机制**:
* 接口端点:`https://api.cctvnews.cctv.com/1.0.0/feed/article/server/getArticle?articleId={item_id}&appcode=video_web`
* 请求头必须携带标准 API Gateway 签名参数:
* `x-ca-key`: 客户端公开 AppKey (`204133710`)
* `x-ca-stage`: `RELEASE`
* `x-ca-timestamp`: 当前毫秒时间戳
* `x-ca-signature-headers`: `x-ca-key,x-ca-stage,x-ca-timestamp`
* `x-ca-signature`: 基于 `appSecret` 对签名规范字符串计算所得的 Base64 HMAC-SHA256 摘要。
3. **Base64 响应解密**:
* 服务端响应返回 `{"code": 0, "response": "<base64>"}`。
* 对 `response` 字段解码后即可得到完整文章、视频播放直链 (`videos[0].url`) 与缩略图。
---
## 3. 测试与验证
* **单元测试**:[tests/test_cctv_parser.py](file:///Users/leo/Projects/media-parser/tests/test_cctv_parser.py)
* **执行命令**:`python3 -m unittest tests/test_cctv_parser.py`
+36
View File
@@ -0,0 +1,36 @@
# 央视频 (Yangshipin) 逆向解析指南
本篇详细记录中央广播电视总台 5G 新媒体旗舰平台 **央视频 (Yangshipin)** 的微视频、微短剧与体育资讯作品解析方案。
---
## 1. 平台特征与支持能力
* **平台标识**:`央视频`
* **支持媒体类型**:高分辨率封面 / 图集 / 完整视频标题 / 创作者与机构作者信息(头像与名称)/ 视频 ID (`vid`, `cid`)
* **常见链接形态**:
* 官方分享短链:`https://www.yspapp.cn/5Sqx`, `https://www.yspapp.cn/d1o`
* 移动端竖屏微短剧/小视频:`https://m.yangshipin.cn/portrait_video?vid=l00005817wl`
* 移动端常规横屏视频:`https://m.yangshipin.cn/video?type=0&vid=v000007pgfu&cid=...`
* PC/网页端:`https://yangshipin.cn/...`
* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。
---
## 2. 核心逆向流程
1. **Meta Refresh 重定向自动跟随**:
* 央视频短链 `yspapp.cn/{code}` 采用 HTML `<meta http-equiv="refresh" content="0; URL='https://m.yangshipin.cn/...'"/>` 形式执行页面跳转。
* 解析器在 `WebFetcher` 与 `YangshipinParser` 中双重内置了针对 `<meta http-equiv="refresh">` 的无感自动追踪,直接锁定最终目标落地页。
2. **SSR 双状态机数据提取**:
* **横屏常规视频**:页面注入全局变量 `window.__STATE_video__`,解析 `payloads.sharevideo` 结构,获取 `title`、`cover_pic`、`cid`、`vid` 及 `om_info.title`(发布机构或频道名)。
* **竖屏微短剧/短视频**:页面注入全局变量 `window.__STATE_portrait_video__`,解析 `payloads.videoDataList.items[0].videoData`,获取微短剧标题、`shareItem.shareImgUrl`(超清封面)、以及 `detailFollowItem.actorItem` 中的创作者昵称与头像。
3. **播放流安全提示**:
* 央视频点播播放接口底层接入腾讯云 VOD 架构,采用带有动态 `cKey 8.1` / 白名单签名机制保护。解析器全面提取高保真元数据与封面图集,并保留扩展能力。
---
## 3. 测试与验证
* **单元测试**:[tests/test_yangshipin_parser.py](file:///Users/leo/Projects/media-parser/tests/test_yangshipin_parser.py)
* **执行命令**:`python3 -m unittest tests/test_yangshipin_parser.py`
+4 -4
View File
@@ -13,7 +13,7 @@ tests/
├── test_parser_factory.py # 解析器自动注册与工厂路由测试
├── test_web_fetcher.py # 302跳转与URL正则提取测试
├── test_*_parser.py # 各平台专属的单元测试
├── live_parser_samples.json # 真实线上验证样本库 (45+ 平台活跃样例)
├── live_parser_samples.json # 真实线上验证样本库 (50 平台活跃样例,300+ 用例)
├── test_live_parser_samples.py # 自动化执行样本库回归测试
└── manual_verify_parsers.py # 命令行交互式/全量人工排查脚本
```
@@ -48,10 +48,10 @@ pytest tests/test_live_parser_samples.py -s
```
### 4. 运行交互式真实样本回归验证工具
[tests/manual_verify_parsers.py](file:///Users/leo/Projects/media-parser/tests/manual_verify_parsers.py) 支持对 45+ 个平台的多种形态(短链、长链、视频、图集等)发起真实网络请求与字段契约校验:
[tests/manual_verify_parsers.py](file:///Users/leo/Projects/media-parser/tests/manual_verify_parsers.py) 支持对 50 个平台的多种形态(短链、长链、视频、图集等)发起真实网络请求与字段契约校验:
```bash
# 1. 快速冒烟测试 (每个平台只测 1 条最具代表性的链接,极速完成 45 平台基准健康检查)
# 1. 快速冒烟测试 (每个平台只测 1 条最具代表性的链接,极速完成 50 平台基准健康检查)
python3 tests/manual_verify_parsers.py --limit 1
# 2. 验证指定平台的多形态覆盖 (支持重复传入多个 --platform 参数)
@@ -60,7 +60,7 @@ python3 tests/manual_verify_parsers.py --platform 微博 --platform 哔哩哔哩
# 3. 限制指定平台的测试用例数量 (例如每个平台最多测 2 条)
python3 tests/manual_verify_parsers.py --platform 抖音 --platform 小红书 --limit 2
# 4. 全量深度回归测试 (运行 270+ 条全部多形态真实用例,适合发版前全量巡检)
# 4. 全量深度回归测试 (运行 300+ 条全部多形态真实用例,适合发版前全量巡检)
python3 tests/manual_verify_parsers.py
# 5. 列出当前用例库中未配置真实链接的平台
+163
View File
@@ -0,0 +1,163 @@
import json
import re
from urllib.parse import parse_qs, urlparse
from configs.general_constants import USER_AGENT_M
from configs.logging_config import get_logger
from src.parser_factory import register_parser
from src.parsers.base_parser import BaseParser
logger = get_logger(__name__)
@register_parser("星绘AI")
class ButterflyAIParser(BaseParser):
"""星绘 AI (ButterflyAI / 字节跳动豆包家族) 解析器,支持无水印原画图集与视频生成作品提取。"""
def __init__(self, real_url):
super().__init__(real_url)
self.headers = {
"User-Agent": USER_AGENT_M[0],
"Referer": "https://www.butterflyai.cn/",
"Accept": "application/json, text/plain, */*",
}
self.title = ""
self.cover_url = None
self.video_url = None
self.image_list = []
self.author = None
self._parse()
def _parse(self):
if not self.real_url:
return
parsed = urlparse(self.real_url)
qs = parse_qs(parsed.query)
# 1. 尝试从 URL 参数或路径提取 share_token / share_code / share_record_id
share_token = qs.get("share_code", [None])[0] or qs.get("share_token", [None])[0]
share_record_id = qs.get("share_id", [""])[0] or qs.get("share_record_id", [""])[0]
# 若未获取到 share_token 且为短链路径形式 /s/{code}/,跟随重定向获取最终 URL
if not share_token:
match = re.search(r"/s/([a-zA-Z0-9_\-]+)", parsed.path)
if match:
try:
resp = self.session.get(self.real_url, headers=self.headers, allow_redirects=True, timeout=5)
final_parsed = urlparse(resp.url)
final_qs = parse_qs(final_parsed.query)
share_token = final_qs.get("share_code", [None])[0] or final_qs.get("share_token", [None])[0]
share_record_id = final_qs.get("share_id", [""])[0] or final_qs.get("share_record_id", [""])[0]
except Exception as e:
logger.warning("Failed to follow ButterflyAI redirect for %s: %s", self.real_url, e)
if not share_token and not share_record_id:
logger.warning("Could not find share_token or share_record_id from %s", self.real_url)
return
api_url = "https://www.butterflyai.cn/butterfly/community/v1/share/record/get?aid=564650"
payload = {
"share_record_id": share_record_id or "",
"from_h5": True,
"share_token": share_token or "",
"extra_params": {
"aid": "564650",
"app_name": "星绘",
},
}
try:
res = self.session.post(api_url, json=payload, headers=self.headers, timeout=5)
res.raise_for_status()
data = res.json()
except Exception as e:
logger.error("Failed to request ButterflyAI API: %s", e)
return
status_info = data.get("status_info") or {}
status_code = status_info.get("status_code", data.get("status_code", 0))
if status_code != 0:
logger.warning("ButterflyAI API returned error code %s: %s", status_code, status_info.get("status_msg") or data.get("message"))
return
share_record = data.get("share_record") or {}
artwork = share_record.get("artwork") or {}
creator = share_record.get("creator") or {}
# 提取标题与生成 Prompt
self.title = (
artwork.get("title")
or artwork.get("description")
or share_record.get("prompt")
or share_record.get("show_info", {}).get("effect_title")
or ""
).strip()
if not self.title and share_record.get("effect_list"):
first_eff = share_record["effect_list"][0]
self.title = (
first_eff.get("name")
or first_eff.get("show_info", {}).get("effect_title")
or ""
).strip()
# 提取作者信息
author_name = creator.get("screen_name") or creator.get("nickname") or creator.get("name")
avatar_obj = creator.get("avatar") or creator.get("avatar_url")
author_avatar = None
if isinstance(avatar_obj, dict):
author_avatar = avatar_obj.get("large_url") or avatar_obj.get("image_url") or avatar_obj.get("middle_url")
elif isinstance(avatar_obj, str):
author_avatar = avatar_obj
if author_name or author_avatar:
self.author = {
"name": author_name,
"avatar": author_avatar,
}
# 提取视频 (share_record.rendering_video 或 artwork.resource_list)
rendering_video = share_record.get("rendering_video") or {}
if rendering_video:
self.video_url = rendering_video.get("video_url") or rendering_video.get("download_url")
for r in artwork.get("resource_list") or []:
if not isinstance(r, dict):
continue
vr = r.get("video_resource") or {}
for v in vr.get("rendering_videos") or []:
if not self.video_url:
self.video_url = v.get("video_url") or v.get("download_url")
ir = r.get("image_resource") or {}
for img in ir.get("rendering_images") or []:
img_url = img.get("no_wm_download_url") or img.get("download_url") or img.get("image_url")
if img_url and img_url not in self.image_list:
self.image_list.append(img_url)
# 提取图文列表(优先提取无水印原画 download_url)
for img in share_record.get("rendering_images") or []:
img_url = img.get("no_wm_download_url") or img.get("download_url") or img.get("image_url")
if img_url and img_url not in self.image_list:
self.image_list.append(img_url)
# 提取封面
self.cover_url = artwork.get("cover_image_url") or artwork.get("cover_url")
if not self.cover_url and rendering_video:
self.cover_url = rendering_video.get("cover_image_url")
if not self.cover_url and self.image_list:
self.cover_url = self.image_list[0]
def get_real_video_url(self):
return self.video_url
def get_title_content(self):
return self.title or ""
def get_cover_photo_url(self):
return self.cover_url
def get_author_info(self):
return self.author
def get_image_list(self):
return self.image_list
+162
View File
@@ -0,0 +1,162 @@
import base64
import hashlib
import hmac
import json
import re
import time
from urllib.parse import parse_qs, urlparse
from configs.general_constants import USER_AGENT_PC
from configs.logging_config import get_logger
from src.parser_factory import register_parser
from src.parsers.base_parser import BaseParser
logger = get_logger(__name__)
@register_parser("央视")
class CCTVParser(BaseParser):
"""央视 (CCTV / 央视网 / 央视新闻) 解析器,支持常规栏目节目、央视新闻客户端与云平台视频提取。"""
def __init__(self, real_url):
super().__init__(real_url)
self.headers = {
"User-Agent": USER_AGENT_PC[0],
"Referer": "https://tv.cctv.com/",
"Accept": "*/*",
}
self.title = ""
self.cover_url = None
self.video_url = None
self.author = None
self._parse()
def _parse(self):
if not self.real_url:
return
parsed = urlparse(self.real_url)
qs = parse_qs(parsed.query)
# 区分 央视新闻 API 体系 与 CCTV 综合/电视频道体系
if "cctvnews" in parsed.netloc or "snow-book" in parsed.path:
self._parse_cctvnews(parsed, qs)
else:
self._parse_cctv_web(parsed, qs)
if self.title:
self.title = re.sub(r'<[^>]+>', '', self.title).strip()
def _parse_cctvnews(self, parsed, qs):
"""解析央视新闻客户端微视频及新闻报道(采用阿里云 API 网关 HMAC-SHA256 签名)"""
item_id = qs.get("item_id", [None])[0] or qs.get("articleId", [None])[0]
if not item_id:
match = re.search(r'(?:item_id|articleId)=([0-9a-zA-Z_\-]+)', self.real_url)
if match:
item_id = match.group(1)
if not item_id:
logger.warning("Could not extract articleId/item_id from CCTV News URL: %s", self.real_url)
return
app_key = "204133710"
app_secret = "etyEuNdA7GvQU7iPZHqnrBpSFfRyKQTD"
host = "https://api.cctvnews.cctv.com"
path = "/1.0.0/feed/article/server/getArticle"
params = {"articleId": item_id, "appcode": "video_web"}
t_now = int(time.time() * 1000)
headers = {
"x-ca-timestamp": str(t_now),
"x-ca-key": app_key,
"x-ca-stage": "RELEASE",
"accept": "application/json",
"User-Agent": USER_AGENT_PC[0],
}
ca_headers = sorted([k for k in headers.keys() if k.startswith("x-ca-")])
headers["x-ca-signature-headers"] = ",".join(ca_headers)
header_str = "\n".join([f"{k}:{headers[k]}" for k in ca_headers])
sorted_params = sorted(params.items())
query_str = "&".join([f"{k}={v}" for k, v in sorted_params])
full_path = f"{path}?{query_str}"
string_to_sign = f"GET\n{headers['accept']}\n\n\n\n{header_str}\n{full_path}"
sig = base64.b64encode(
hmac.new(app_secret.encode("utf-8"), string_to_sign.encode("utf-8"), hashlib.sha256).digest()
).decode("utf-8")
headers["x-ca-signature"] = sig
try:
res = self.session.get(f"{host}{full_path}", headers=headers, timeout=5)
res.raise_for_status()
resp_json = res.json()
b64_response = resp_json.get("response")
if not b64_response:
logger.warning("CCTV News response missing encoded payload: %s", resp_json)
return
payload_bytes = base64.b64decode(b64_response)
payload = json.loads(payload_bytes.decode("utf-8")).get("data", {})
self.title = (payload.get("title") or "").strip()
videos = payload.get("videos") or []
if videos and isinstance(videos, list):
self.video_url = videos[0].get("url")
cover_info = videos[0].get("cover")
if isinstance(cover_info, dict):
self.cover_url = cover_info.get("url")
if not self.cover_url:
thumbnails = payload.get("thumbnails") or []
if thumbnails and isinstance(thumbnails, list):
self.cover_url = thumbnails[0].get("url")
author_name = payload.get("source") or payload.get("author") or "央视新闻"
self.author = {
"name": author_name,
"avatar": None,
}
except Exception as e:
logger.error("Failed to parse CCTV News article: %s", e)
def _parse_cctv_web(self, parsed, qs):
"""解析 tv.cctv.com / tv.cctv.cn 等央视节目详情页(通过 CNTV 视频分配接口)"""
pid = qs.get("pid", [None])[0] or qs.get("guid", [None])[0]
if not pid:
html = self.fetch_html_content()
if html:
match = re.search(r'(?:guid|pid|videoCenterId)\s*=\s*["\']([0-9a-fA-F]{32})["\']', html)
if match:
pid = match.group(1)
if not pid:
logger.warning("Could not extract video pid/guid from CCTV URL: %s", self.real_url)
return
api_url = f"https://vdn.apps.cntv.cn/api/getHttpVideoInfo.do?pid={pid}"
try:
res = self.session.get(api_url, headers=self.headers, timeout=5)
res.raise_for_status()
data = res.json()
except Exception as e:
logger.error("Failed to fetch CCTV video info for pid %s: %s", pid, e)
return
self.title = (data.get("title") or "").strip()
self.video_url = data.get("hls_url")
self.cover_url = data.get("image")
column_name = data.get("column") or data.get("produce") or "央视网"
self.author = {
"name": column_name,
"avatar": None,
}
def get_real_video_url(self):
return self.video_url
def get_title_content(self):
return self.title or ""
def get_cover_photo_url(self):
return self.cover_url
def get_author_info(self):
return self.author
+141
View File
@@ -0,0 +1,141 @@
import json
import re
from bs4 import BeautifulSoup
from configs.general_constants import USER_AGENT_M
from configs.logging_config import get_logger
from src.parser_factory import register_parser
from src.parsers.base_parser import BaseParser
logger = get_logger(__name__)
@register_parser("央视频")
class YangshipinParser(BaseParser):
"""央视频 (Yangshipin) 客户端及 H5 平台解析器,支持横竖屏微短剧/精选视频元数据、原画封面与作者信息提取。"""
def __init__(self, real_url):
super().__init__(real_url)
self.headers = {
"User-Agent": USER_AGENT_M[0],
"Referer": "https://m.yangshipin.cn/",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}
self.title = ""
self.cover_url = None
self.video_url = None
self.image_list = []
self.author = None
self._parse()
def _parse(self):
if not self.real_url:
return
html = self.fetch_html_content()
if not html:
logger.warning("Failed to fetch HTML content for Yangshipin URL: %s", self.real_url)
return
# 兼容短链未重定向时页面内的 meta refresh 跳转
meta_refresh = re.search(r'''<meta\s+http-equiv=["']refresh["']\s+content=["'][^;]+;\s*URL=['"]([^'"]+)['"]''', html, re.I)
if meta_refresh:
redirect_url = meta_refresh.group(1).strip()
if redirect_url.startswith("/"):
redirect_url = f"https://m.yangshipin.cn{redirect_url}"
self.real_url = redirect_url
html = self.fetch_html_content()
if not html:
return
try:
# 1. 尝试从横屏视频 STATE 提取 (__STATE_video__)
data_vid = self._extract_state_json(html, "window.__STATE_video__")
if data_vid:
sv = data_vid.get("payloads", {}).get("sharevideo", {})
self.title = (sv.get("title") or "").strip()
self.cover_url = sv.get("cover_pic")
om_info = sv.get("om_info", {})
author_name = om_info.get("title")
if author_name:
self.author = {
"name": author_name,
"avatar": None,
}
# 2. 尝试从竖屏视频 STATE 提取 (__STATE_portrait_video__)
data_port = self._extract_state_json(html, "window.__STATE_portrait_video__")
if data_port:
items = data_port.get("payloads", {}).get("videoDataList", {}).get("items", [])
if items:
vd = items[0].get("videoData", {})
if not self.title:
self.title = (vd.get("title") or "").strip()
if not self.cover_url:
self.cover_url = (
vd.get("shareItem", {}).get("shareImgUrl")
or vd.get("poster", {}).get("poster", {}).get("imageUrl")
)
actor = vd.get("detailFollowItem", {}).get("actorItem", {})
if actor:
nick_name = actor.get("nickName", {}).get("text")
head_url = actor.get("headUrl")
if nick_name or head_url:
self.author = {
"name": nick_name,
"avatar": head_url,
}
# 3. 兜底:从 OpenGraph / HTML 标签提取
if not self.title or not self.cover_url:
soup = BeautifulSoup(html, "lxml")
if not self.title:
og_title = soup.find("meta", property="og:title")
if og_title and og_title.get("content"):
self.title = og_title["content"].strip()
elif soup.title and soup.title.string:
self.title = soup.title.string.strip()
if not self.cover_url:
og_img = soup.find("meta", property="og:image")
if og_img and og_img.get("content"):
self.cover_url = og_img["content"].strip()
except Exception as exc:
logger.exception("Error parsing Yangshipin page: %s", exc)
if self.title:
self.title = re.sub(r'<[^>]+>', '', self.title).strip()
if not self.video_url and self.cover_url and self.cover_url not in self.image_list:
self.image_list.append(self.cover_url)
def _extract_state_json(self, html, state_key):
"""从页面提取指定 state_key 的 JSON 数据对象"""
idx = html.find(state_key)
if idx == -1:
return None
start = html.find("{", idx)
if start == -1:
return None
try:
decoder = json.JSONDecoder()
data, _ = decoder.raw_decode(html[start:])
return data
except Exception as e:
logger.warning("Failed to decode %s JSON: %s", state_key, e)
return None
def get_real_video_url(self):
return self.video_url
def get_title_content(self):
return self.title or ""
def get_cover_photo_url(self):
return self.cover_url
def get_author_info(self):
return self.author
def get_image_list(self):
return self.image_list
+76 -2
View File
@@ -1,7 +1,7 @@
{
"schema_version": 2,
"description": "45 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)",
"total_cases": 301,
"description": "50 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)",
"total_cases": 306,
"cases": [
{
"platform": "网易云音乐",
@@ -4378,6 +4378,80 @@
],
"note": "2026-09-06 已验证: LOFTER 创作者短视频",
"pattern": "同人短视频"
},
{
"platform": "星绘AI",
"url": "https://www.butterflyai.cn/s/gQ0EXE9BJsw/",
"media_types": [
"images"
],
"expected_fields": [
"images",
"cover",
"title",
"author"
],
"note": "2026-09-06 已验证: 星绘AI 无水印原画图集",
"pattern": "AI 生图/写真作品"
},
{
"platform": "星绘AI",
"url": "https://www.butterflyai.cn/share/record?share_code=9r3kCJOnwubPE15yFhKY&is_mine=1&resource_count=1&entity_type=2",
"media_types": [
"video"
],
"expected_fields": [
"video",
"cover",
"title",
"author"
],
"note": "2026-09-06 已验证: 星绘AI 视频生成作品",
"pattern": "AI 视频生成作品"
},
{
"platform": "央视",
"url": "https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml?spm=C22284.PP7miAIBYiAV.EIn520ebHszi.32",
"media_types": [
"video"
],
"expected_fields": [
"video",
"cover",
"title",
"author"
],
"note": "2026-09-06 已验证: CCTV 栏目点播视频与 HLS 流",
"pattern": "综合栏目点播"
},
{
"platform": "央视",
"url": "https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092&t=1787103208459&toc_style_id=video_default&share_to=copy_url&track_id=766f3425-9b74-40ac-b2cf-751f9e5a19fd",
"media_types": [
"video"
],
"expected_fields": [
"video",
"cover",
"title",
"author"
],
"note": "2026-09-06 已验证: 央视新闻客户端微视频",
"pattern": "新闻微视频"
},
{
"platform": "央视频",
"url": "https://www.yspapp.cn/5Sqx",
"media_types": [
"images"
],
"expected_fields": [
"cover",
"title",
"author"
],
"note": "2026-09-06 已验证: 央视频短链与微视频详情",
"pattern": "微视频及短剧详情"
}
]
}
+105
View File
@@ -0,0 +1,105 @@
import unittest
from unittest.mock import patch, Mock
from src.parsers.butterflyai_parser import ButterflyAIParser
class ButterflyAIParserTest(unittest.TestCase):
def test_parses_image_post_successfully(self):
fake_api_resp = {
"status_code": 0,
"status_info": {"status_code": 0, "status_msg": "Success"},
"share_record": {
"artwork": {
"title": "夏日清凉少女写真",
"cover_image_url": "https://p26-community.butterfly.cn/cover.jpg",
},
"creator": {
"screen_name": "星绘创作师",
"avatar": {
"large_url": "https://p3-passport.byteacctimg.com/avatar_large.jpeg",
"image_url": "https://p3-passport.byteacctimg.com/avatar.jpeg",
},
},
"rendering_images": [
{
"no_wm_download_url": "https://p26-community.butterfly.cn/pic1_hd.jpg",
"download_url": "https://p26-community.butterfly.cn/pic1_wm.jpg",
},
{
"image_url": "https://p26-community.butterfly.cn/pic2.jpg",
}
],
},
}
mock_resp = Mock()
mock_resp.status_code = 200
mock_resp.json.return_value = fake_api_resp
mock_resp.raise_for_status = Mock()
with patch("requests.Session.post", return_value=mock_resp):
parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=abc123Test")
self.assertEqual(parser.get_title_content(), "夏日清凉少女写真")
self.assertEqual(
parser.get_author_info(),
{"name": "星绘创作师", "avatar": "https://p3-passport.byteacctimg.com/avatar_large.jpeg"},
)
self.assertEqual(
parser.get_image_list(),
[
"https://p26-community.butterfly.cn/pic1_hd.jpg",
"https://p26-community.butterfly.cn/pic2.jpg",
],
)
self.assertEqual(parser.get_cover_photo_url(), "https://p26-community.butterfly.cn/cover.jpg")
self.assertIsNone(parser.get_real_video_url())
def test_parses_video_post_successfully(self):
fake_api_resp = {
"status_code": 0,
"status_info": {"status_code": 0, "status_msg": "Success"},
"share_record": {
"show_info": {"effect_title": "AI 视频变换"},
"creator": {
"screen_name": "特效大师",
"avatar_url": "https://p3-passport.byteacctimg.com/avatar_str.jpeg",
},
"rendering_video": {
"video_url": "https://v-butterfly.douyinvod.com/video1.mp4",
"cover_image_url": "https://v-butterfly.douyinvod.com/cover1.jpg",
},
},
}
mock_resp = Mock()
mock_resp.status_code = 200
mock_resp.json.return_value = fake_api_resp
mock_resp.raise_for_status = Mock()
with patch("requests.Session.post", return_value=mock_resp):
parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=vid123")
self.assertEqual(parser.get_title_content(), "AI 视频变换")
self.assertEqual(
parser.get_author_info(),
{"name": "特效大师", "avatar": "https://p3-passport.byteacctimg.com/avatar_str.jpeg"},
)
self.assertEqual(parser.get_real_video_url(), "https://v-butterfly.douyinvod.com/video1.mp4")
self.assertEqual(parser.get_cover_photo_url(), "https://v-butterfly.douyinvod.com/cover1.jpg")
self.assertEqual(parser.get_image_list(), [])
def test_handles_api_failure_gracefully(self):
mock_resp = Mock()
mock_resp.status_code = 500
mock_resp.raise_for_status.side_effect = Exception("API Internal Error")
with patch("requests.Session.post", return_value=mock_resp):
parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=fail123")
self.assertEqual(parser.get_title_content(), "")
self.assertIsNone(parser.get_real_video_url())
self.assertIsNone(parser.get_author_info())
self.assertEqual(parser.get_image_list(), [])
if __name__ == "__main__":
unittest.main()
+82
View File
@@ -0,0 +1,82 @@
import base64
import json
import unittest
from unittest.mock import patch, Mock
from src.parsers.cctv_parser import CCTVParser
class CCTVParserTest(unittest.TestCase):
def test_parses_cntv_web_video_successfully(self):
fake_html = """
<html>
<head><title>生财有道</title></head>
<body>
<script>
var guid = "00e121ae62194fec8ab20e5b8eb9a89a";
</script>
</body>
</html>
"""
fake_api_data = {
"title": "《生财有道》 20260526 匠心织锦绣",
"hls_url": "https://hls.cntv.lxdns.com/asp/hls/main/0303000a/3/default/00e12/main.m3u8",
"image": "https://p1.img.cctvpic.com/cover.jpg",
"column": "生财有道",
}
mock_api_resp = Mock()
mock_api_resp.status_code = 200
mock_api_resp.json.return_value = fake_api_data
mock_api_resp.raise_for_status = Mock()
with patch.object(CCTVParser, "fetch_html_content", return_value=fake_html), \
patch("requests.Session.get", return_value=mock_api_resp):
parser = CCTVParser("https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml")
self.assertEqual(parser.get_title_content(), "《生财有道》 20260526 匠心织锦绣")
self.assertEqual(
parser.get_real_video_url(),
"https://hls.cntv.lxdns.com/asp/hls/main/0303000a/3/default/00e12/main.m3u8",
)
self.assertEqual(parser.get_cover_photo_url(), "https://p1.img.cctvpic.com/cover.jpg")
self.assertEqual(parser.get_author_info(), {"name": "生财有道", "avatar": None})
def test_parses_cctvnews_article_successfully(self):
fake_payload = {
"data": {
"title": "今天是第九个中国医师节",
"source": "央视新闻客户端",
"videos": [
{
"url": "https://res.cctvnews.cctv.com/video/stream.m3u8",
"cover": {"url": "https://img.cctvnews.cctv.com/cover1.jpg"},
}
],
}
}
b64_str = base64.b64encode(json.dumps(fake_payload).encode("utf-8")).decode("utf-8")
fake_api_resp = {"code": 0, "response": b64_str}
mock_resp = Mock()
mock_resp.status_code = 200
mock_resp.json.return_value = fake_api_resp
mock_resp.raise_for_status = Mock()
with patch("requests.Session.get", return_value=mock_resp):
url = "https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092"
parser = CCTVParser(url)
self.assertEqual(parser.get_title_content(), "今天是第九个中国医师节")
self.assertEqual(parser.get_real_video_url(), "https://res.cctvnews.cctv.com/video/stream.m3u8")
self.assertEqual(parser.get_cover_photo_url(), "https://img.cctvnews.cctv.com/cover1.jpg")
self.assertEqual(parser.get_author_info(), {"name": "央视新闻客户端", "avatar": None})
def test_handles_missing_pid_gracefully(self):
with patch.object(CCTVParser, "fetch_html_content", return_value="<html><body>No video</body></html>"):
parser = CCTVParser("https://tv.cctv.com/empty.shtml")
self.assertEqual(parser.get_title_content(), "")
self.assertIsNone(parser.get_real_video_url())
self.assertIsNone(parser.get_cover_photo_url())
if __name__ == "__main__":
unittest.main()
+122
View File
@@ -0,0 +1,122 @@
import unittest
from unittest.mock import patch
from src.parsers.yangshipin_parser import YangshipinParser
class YangshipinParserTest(unittest.TestCase):
def test_parses_portrait_video_successfully(self):
fake_html = """
<html>
<head><title>央视频</title></head>
<body>
<script>
window.__STATE_portrait_video__ = {
"payloads": {
"videoDataList": {
"items": [
{
"videoData": {
"vid": "l00005817wl",
"title": "5次助攻对4次助攻!巅峰对决",
"shareItem": {
"shareImgUrl": "https://jietufengmian.yangshipin.cn/cover1.jpg"
},
"detailFollowItem": {
"actorItem": {
"nickName": {"text": "奥运来了"},
"headUrl": "https://mpuser.ysp.cctv.cn/avatar1.jpeg"
}
}
}
}
]
}
}
};
</script>
</body>
</html>
"""
with patch.object(YangshipinParser, "fetch_html_content", return_value=fake_html):
parser = YangshipinParser("https://m.yangshipin.cn/portrait_video?vid=l00005817wl")
self.assertEqual(parser.get_title_content(), "5次助攻对4次助攻!巅峰对决")
self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover1.jpg")
self.assertEqual(
parser.get_author_info(),
{"name": "奥运来了", "avatar": "https://mpuser.ysp.cctv.cn/avatar1.jpeg"},
)
self.assertEqual(parser.get_image_list(), ["https://jietufengmian.yangshipin.cn/cover1.jpg"])
def test_parses_landscape_video_successfully(self):
fake_html = """
<html>
<head><title>央视频</title></head>
<body>
<script>
window.__STATE_video__ = {
"payloads": {
"sharevideo": {
"vid": "v000007pgfu",
"title": "《普法栏目剧》远山的守望",
"cover_pic": "https://jietufengmian.yangshipin.cn/cover2.jpg",
"om_info": {
"title": "社会与法频道"
}
}
}
};
</script>
</body>
</html>
"""
with patch.object(YangshipinParser, "fetch_html_content", return_value=fake_html):
parser = YangshipinParser("https://m.yangshipin.cn/video?type=0&vid=v000007pgfu")
self.assertEqual(parser.get_title_content(), "《普法栏目剧》远山的守望")
self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover2.jpg")
self.assertEqual(parser.get_author_info(), {"name": "社会与法频道", "avatar": None})
self.assertEqual(parser.get_image_list(), ["https://jietufengmian.yangshipin.cn/cover2.jpg"])
def test_follows_meta_refresh_redirect(self):
meta_html = """
<!DOCTYPE html>
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; URL='https://m.yangshipin.cn/portrait_video?vid=5Sqx'"/>
<title>央视频</title>
"""
detail_html = """
<html><body>
<script>
window.__STATE_portrait_video__ = {
"payloads": {
"videoDataList": {
"items": [
{
"videoData": {
"title": "跳转后的视频标题",
"shareItem": {"shareImgUrl": "https://jietufengmian.yangshipin.cn/cover3.jpg"}
}
}
]
}
}
};
</script>
</body></html>
"""
with patch.object(YangshipinParser, "fetch_html_content", side_effect=[meta_html, detail_html]):
parser = YangshipinParser("https://www.yspapp.cn/5Sqx")
self.assertEqual(parser.get_title_content(), "跳转后的视频标题")
self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover3.jpg")
def test_handles_empty_or_broken_html(self):
with patch.object(YangshipinParser, "fetch_html_content", return_value="<html><body>404 Not Found</body></html>"):
parser = YangshipinParser("https://m.yangshipin.cn/video?vid=empty")
self.assertEqual(parser.get_title_content(), "")
self.assertIsNone(parser.get_cover_photo_url())
self.assertIsNone(parser.get_author_info())
self.assertEqual(parser.get_image_list(), [])
if __name__ == "__main__":
unittest.main()
+56 -2
View File
@@ -22,8 +22,8 @@ class WebFetcher:
# 绿洲分享页会对通用桌面请求头返回 500,由解析器使用移动端请求头抓取。
# 微博博文可通过 API 直接根据 ID 解析,直接访问网页端常触发访客系统重定向。
domain = UrlParser.get_domain(url)
if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan", "dw4.co"}:
if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI", "网易LOFTER", "得物"}:
if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan", "dw4.co", "yspapp.cn"}:
if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI", "网易LOFTER", "得物", "央视"}:
return UrlParser.extract_video_address(url)
try:
current_url = url
@@ -32,6 +32,10 @@ class WebFetcher:
resp = requests.get(current_url, headers=WebFetcher.headers, allow_redirects=False, timeout=5)
resp.raise_for_status()
redirect_url = resp.headers.get("location")
if not redirect_url and resp.status_code == 200 and isinstance(getattr(resp, "text", None), str):
meta_match = re.search(r'<meta[^>]*http-equiv=["\']?refresh["\']?[^>]*content=["\']?[^"\']*URL=[\'"]?([^\'" >]+)', resp.text, re.I)
if meta_match:
redirect_url = meta_match.group(1).strip("'\"")
if redirect_url:
redirect_url = urljoin(current_url, redirect_url)
# 如果重定向到了登录页、404拦截页、验证码校验页或错误页,不要更新 url,直接中断以保留原始有效 URL
@@ -105,6 +109,15 @@ class UrlParser:
if domain.endswith('.dewu.com') or domain.endswith('.poizon.com'):
return '得物'
if domain == 'butterflyai.cn' or domain.endswith('.butterflyai.cn'):
return '星绘AI'
if domain in {'cctv.com', 'cctv.cn'} or domain.endswith('.cctv.com') or domain.endswith('.cctv.cn'):
return '央视'
if domain in {'yangshipin.cn', 'yspapp.cn'} or domain.endswith('.yangshipin.cn') or domain.endswith('.yspapp.cn'):
return '央视频'
return None
@staticmethod
@@ -365,6 +378,33 @@ class UrlParser:
preserved_params.append((key, value))
if preserved_params:
address = f"{address}?{urlencode(preserved_params)}"
elif platform == "星绘AI":
query_params = parse_qs(parsed_url.query)
preserved_params = []
for key in ("share_code", "share_id", "entity_type", "is_mine", "resource_count", "share_token"):
value = query_params.get(key, [None])[0]
if value is not None:
preserved_params.append((key, value))
if preserved_params:
address = f"{address}?{urlencode(preserved_params)}"
elif platform == "央视":
query_params = parse_qs(parsed_url.query)
preserved_params = []
for key in ("item_id", "articleId", "pid", "guid"):
value = query_params.get(key, [None])[0]
if value is not None:
preserved_params.append((key, value))
if preserved_params:
address = f"{address}?{urlencode(preserved_params)}"
elif platform == "央视频":
query_params = parse_qs(parsed_url.query)
preserved_params = []
for key in ("vid", "cid", "type", "serverFrom"):
value = query_params.get(key, [None])[0]
if value is not None:
preserved_params.append((key, value))
if preserved_params:
address = f"{address}?{urlencode(preserved_params)}"
return address
@staticmethod
@@ -446,12 +486,26 @@ class UrlParser:
params_ps = query_params.get('ps', [None])[0]
if params_ps:
return params_ps
params_share_code = query_params.get('share_code', [None])[0]
if params_share_code:
return params_share_code
params_item_id = query_params.get('item_id', [None])[0]
if params_item_id:
return params_item_id
params_article_id = query_params.get('articleId', [None])[0]
if params_article_id:
return params_article_id
params_guid = query_params.get('guid', [None])[0]
if params_guid:
return params_guid
# 尝试从URL路径中获取视频ID
path_segments = parsed_url.path.strip('/').split('/')
if path_segments:
video_id = path_segments[-1]
if video_id.endswith('.html'):
video_id = video_id[:-5]
elif video_id.endswith('.shtml'):
video_id = video_id[:-6]
desktop_mv = re.fullmatch(r'mv_([0-9a-fA-F]{32})', video_id)
if desktop_mv:
return desktop_mv.group(1)