diff --git a/README.md b/README.md index d1479a8..99c5947 100644 --- a/README.md +++ b/README.md @@ -3,10 +3,10 @@ **基于 Python 的多平台媒体原生本地解析系统** -[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) [![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/) [![Flask](https://img.shields.io/badge/Framework-Flask-000000?logo=flask&logoColor=white)](https://flask.palletsprojects.com/) [![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?logo=docker&logoColor=white)](#-部署指南) [![Support](https://img.shields.io/badge/support-47%20Platforms-brightgreen.svg)](#-支持的平台矩阵) +[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) [![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/) [![Flask](https://img.shields.io/badge/Framework-Flask-000000?logo=flask&logoColor=white)](https://flask.palletsprojects.com/) [![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?logo=docker&logoColor=white)](#-部署指南) [![Support](https://img.shields.io/badge/support-50%20Platforms-brightgreen.svg)](#-支持的平台矩阵)

-解析逻辑 • +核心特性 • 支持平台 • 部署指南 • 接口文档 • @@ -15,63 +15,81 @@ 联系作者

-媒体解析去水印是一款专为短视频创作者与开发者打造的**原生本地解析工具**。 +媒体解析去水印是一款专为短视频创作者与开发者打造的**100%原生本地解析工具**。 通过“智能识别 -> 本地抓取 -> 提取地址 -> 快捷下载”的闭环,助你高效获取无水印素材。 -**100%本地原生实现,不依赖外部API,不封装第三方解析库,核心逆向逻辑完全开源。** +**不依赖外部API,不套壳第三方库,无浏览器开销,纯底层协议与算法逆向。** --- -## 💎 核心解析逻辑 +## ✨ 核心特性 -* **多平台智能适配**:内置 `ParserFactory` 工厂模式,自动识别链接来源并分配对应解析器。 -* **原生本地解析**:解析逻辑直接内置在项目代码中,由各平台 Parser 本地发起请求并提取真实媒体地址。 -* **零外部API依赖**:无需对接任何第三方代解析接口或 SaaS 中转服务,部署后即可独立稳定运行。 -* **开箱即用API**:提供标准化的 JSON 数据接口,无冗余数据库依赖,适合快速对接业务或二次开发。 - -## ✨ 项目特点 - -* **本地可控**:所有请求链路与解析规则均在本地代码中,方便调试、维护与定制。 -* **部署简单**:安装依赖或使用 Docker 即可一键运行,不需要额外申请第三方 API 账号或密钥。 -* **便于扩展**:各平台对应独立 Parser,遵循统一的返回数据结构,新增平台轻松快捷。 +* ⚡ **极速轻量**:纯 HTTP 网络协议与底层算法逆向,免启动 Chromium/Playwright 等笨重浏览器,内存占用极低(<100MB),毫秒级极速响应。 +* 🔒 **原生自主**:100% 本地代码闭环抓取,零外部商用 API 或第三方代解析依赖,数据链路自主可控,杜绝断流与隐私泄露风险。 +* 🧩 **插件化解耦**:内置 `ParserFactory` 模块自动发现与工厂分发机制,50 个平台独立解耦,遵循统一的数据契约,新增与维护平台极度轻松。 +* 🔌 **开箱即用**:提供标准 RESTful JSON 接口与 Web 体验页,无外部数据库等冗余依赖,Docker Compose 一键构建,轻松无缝接入各类业务系统。 --- ## 💾 支持的平台矩阵 -| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | -| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | -| **抖音** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | -| **小红书** | ✓ | ✓ | ✓ | ✓ | **✓+** | | | **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | | -| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | | -| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **微视** | ✓ | ✓ | ✓ | ✓ | | | | -| **快手** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | | **AcFun** | ✓ | ✓ | ✓ | ✓ | | | | -| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | | -| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **今日头条** | ✓ | ✓ | ✓ | ✓ | | | | -| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | | -| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | | -| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | **全民K歌** | | ✓ | ✓ | ✓ | | | | -| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **新片场** | ✓ | ✓ | ✓ | ✓ | | | | -| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **好看视频** | ✓ | ✓ | ✓ | ✓ | | | | -| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **梨视频** | ✓ | ✓ | ✓ | ✓ | | | | -| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | | -| **闲鱼** | | ✓ | ✓ | | ✓ | | | **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | | -| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **虎牙** | | ✓ | ✓ | ✓ | | | | -| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **美拍** | ✓ | ✓ | ✓ | ✓ | | | | -| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | **最右** | ✓ | ✓ | | ✓ | | | | -| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | | -| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | | -| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | | -| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | | | -| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | | | -| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | | | | | | | | | +| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 实况 | +| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | +| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | +| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | | +| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | +| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | +| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | | +| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | | +| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | | +| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **闲鱼** | | ✓ | ✓ | | ✓ | | | | +| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | | +| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | | +| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | | +| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | | +| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | +| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | | +| **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | +| **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | | | +| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | | | +| **微视** | ✓ | ✓ | ✓ | ✓ | | | | | +| **AcFun** | ✓ | ✓ | ✓ | ✓ | | | | | +| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | | | +| **今日头条** | ✓ | ✓ | ✓ | ✓ | | | | | +| **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | | | +| **全民K歌** | | ✓ | ✓ | ✓ | | | | | +| **新片场** | ✓ | ✓ | ✓ | ✓ | | | | | +| **好看视频** | ✓ | ✓ | ✓ | ✓ | | | | | +| **梨视频** | ✓ | ✓ | ✓ | ✓ | | | | | +| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **虎牙** | | ✓ | ✓ | ✓ | | | | | +| **美拍** | ✓ | ✓ | ✓ | ✓ | | | | | +| **最右** | ✓ | ✓ | | ✓ | | | | | +| **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | | | +| **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | | | +| **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | | | +| **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **星绘AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | +| **央视** | ✓ | ✓ | ✓ | ✓ | | | | | +| **央视频** | ✓ | ✓ | ✓ | | ✓ | | | | -> 💡 **说明**: -> * **图集 `✓+`**:表示该平台支持提取 iOS/Android 原始 **LivePhoto 动态实况图**(含封面静图与关联 `.mov`/`.mp4` 实况视频原件)。 -> * **原生水印提示**:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。 +*注:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。 --- @@ -132,32 +150,17 @@ python app.py ## 🔌 API 核心接口说明 -### 1. 服务健康检查 - -* **接口路径**:`GET /api/health` -* **接口描述**:供容器编排(Docker Compose/K8s)、反向代理或监控系统探针检测服务存活状态。 -* **返回示例**: - ```json - { - "status": "ok" - } - ``` - ---- - -### 2. 媒体解析接口 - * **接口路径**:`POST /api/parse` * **接口描述**:传入包含分享链接的文本,智能提取多媒体直链与图文信息。 -#### 请求参数 (Request Body) +### 请求参数 (Request Body) 格式: `application/json` | 参数名 | 类型 | 必填 | 描述 | 限制与示例 | | --- | --- | --- | --- | --- | | `text` | `string` | 是 | 视频分享链接或包含链接的文本短语 | 最长 2000 字符,如 `"https://v.douyin.com/..."` | -#### 返回说明 (Response) +### 返回说明 (Response) 格式: `application/json` 成功响应示例: @@ -210,10 +213,10 @@ python app.py ## 🧪 自动化测试与健康自检 -本项目拥有完备的双层测试体系(Mock 单元测试 + 47 平台真实在线样本库回归)。遇到解析异常或日常部署验证时,可一键运行健康自检: +本项目拥有完备的双层测试体系(Mock 单元测试 + 50 平台真实在线样本库回归)。遇到解析异常或日常部署验证时,可一键运行健康自检: ```bash -# 47 平台极速冒烟测试(每个平台测 1 条最具代表性的链接,秒级完成健康检查) +# 50 平台极速冒烟测试(每个平台测 1 条最具代表性的链接,秒级完成健康检查) python3 tests/manual_verify_parsers.py --limit 1 # 仅验证单个或指定平台(如:小云雀AI / 抖音) @@ -231,7 +234,7 @@ python3 tests/manual_verify_parsers.py --platform "小云雀AI" * 🏗️ **[系统架构与生命周期设计](docs/architecture.md)**:分层设计、302 追踪与 `ParserFactory` 动态发现机制。 * 🔍 **[通用逆向方法论](docs/reverse-guide.md)**:SSR 数据提取、H5 接口伪装、JS 签名沙箱及抓包 SOP。 * 🧪 **[测试体系与回归验证](docs/testing.md)**:Pytest 单元测试、Mock 与线上真实样例回归。 -* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 47 个平台的技术文档。 +* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 50 个平台的技术文档。 --- diff --git a/configs/business_config.json b/configs/business_config.json index f2595f2..6e5d515 100755 --- a/configs/business_config.json +++ b/configs/business_config.json @@ -142,7 +142,19 @@ "poizon.com": "得物", "www.poizon.com": "得物", "lofter.com": "网易LOFTER", - "www.lofter.com": "网易LOFTER" + "www.lofter.com": "网易LOFTER", + "butterflyai.cn": "星绘AI", + "www.butterflyai.cn": "星绘AI", + "cctv.com": "央视", + "tv.cctv.com": "央视", + "cctv.cn": "央视", + "tv.cctv.cn": "央视", + "content-static.cctvnews.cctv.com": "央视", + "cctvnews.cctv.com": "央视", + "news.cctv.com": "央视", + "yspapp.cn": "央视频", + "yangshipin.cn": "央视频", + "m.yangshipin.cn": "央视频" }, "USER_AGENT_PC": [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36", diff --git a/docs/index.md b/docs/index.md index b831179..cdd30b2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,7 +2,7 @@ 欢迎查阅 **Media Parser** 开发者文档与逆向工程知识库。 -本项目是一个高性能、模块化、支持 **47 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。 +本项目是一个高性能、模块化、支持 **50 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。 --- @@ -14,8 +14,8 @@ * 📖 **平台实战指南 (Parser Guides)**: * **短视频与轻社区**:[抖音](parsers/douyin.md) | [快手](parsers/kuaishou.md) | [今日头条](parsers/xigua.md) | [皮皮虾](parsers/pipixia.md) | [皮皮搞笑](parsers/pipigaoxiao.md) | [最右](parsers/zuiyou.md) | [美拍](parsers/meipai.md) | [微视](parsers/weishi.md) | [绿洲](parsers/lvzhou.md) | [番茄小说/红果短剧/红果漫剧](parsers/fanqie.md) * **图文与综合社区**:[小红书](parsers/xiaohongshu.md) | [微信公众号](parsers/wechat-mp.md) | [微博](parsers/weibo.md) | [知乎](parsers/zhihu.md) | [闲鱼](parsers/xianyu.md) | [Soul](parsers/soul.md) | [得物](parsers/dewu.md) | [网易LOFTER](parsers/lofter.md) - * **长视频与弹幕/创作**:[哔哩哔哩](parsers/bilibili.md) | [AcFun](parsers/acfun.md) | [新片场](parsers/xinpianchang.md) | [好看视频](parsers/haokan.md) | [西瓜视频](parsers/xigua.md) | [剪映](parsers/jianying.md) | [快影](parsers/kwaiying.md) - * **AI 生成与大模型**:[豆包 AI](parsers/doubao.md) | [腾讯元宝](parsers/yuanbao.md) | [即梦 AI](parsers/jimeng.md) | [可灵 AI](parsers/kling.md) | [海螺 AI](parsers/hailuo.md) | [通义千问](parsers/qianwen.md) | [夸克 AI](parsers/quark-ai.md) | [小云雀 AI](parsers/xiaoyunque.md) | [松果时刻](parsers/pinecone-moment.md) + * **长视频与电视/融媒**:[哔哩哔哩](parsers/bilibili.md) | [AcFun](parsers/acfun.md) | [央视](parsers/cctv.md) | [央视频](parsers/yangshipin.md) | [新片场](parsers/xinpianchang.md) | [好看视频](parsers/haokan.md) | [西瓜视频](parsers/xigua.md) | [剪映](parsers/jianying.md) | [快影](parsers/kwaiying.md) + * **AI 生成与大模型**:[豆包 AI](parsers/doubao.md) | [星绘 AI](parsers/butterflyai.md) | [腾讯元宝](parsers/yuanbao.md) | [即梦 AI](parsers/jimeng.md) | [可灵 AI](parsers/kling.md) | [海螺 AI](parsers/hailuo.md) | [通义千问](parsers/qianwen.md) | [夸克 AI](parsers/quark-ai.md) | [小云雀 AI](parsers/xiaoyunque.md) | [松果时刻](parsers/pinecone-moment.md) * **音频与垂直频道**:[汽水音乐](parsers/qsmusic.md) | [QQ音乐](parsers/qqmusic.md) | [网易云音乐](parsers/netease-music.md) | [酷狗音乐](parsers/kugou-music.md) | [配音秀](parsers/peiyinxiu.md) | [全民K歌](parsers/quanminkge.md) | [虎牙](parsers/huya.md) | [梨视频](parsers/lishipin.md) | [微信视频号](parsers/wechat-channels.md) | [腾讯频道](parsers/tencent-channel.md) --- @@ -76,6 +76,9 @@ | 45 | **松果时刻** | AI 故事多页视频 / 图集 / 配音 | ✅ 支持 | 🟢 公开内容免配置 | 公开详情接口 | [查看指南](parsers/pinecone-moment.md) | | 46 | **得物** | 潮流穿搭 / 开箱短视频 / 高清图集 | ✅ 支持 | 🟢 免配置 | Next.js metaOGInfo 数据提取 | [查看指南](parsers/dewu.md) | | 47 | **网易LOFTER** | 二次元插画 / 摄影图集 / 创作者短视频 | ✅ 支持 | 🟢 免配置 | window.__initialize_data__ 数据提取 | [查看指南](parsers/lofter.md) | +| 48 | **星绘 AI** | AI 生图 / 原画写真 / AI 视频 | ✅ 支持 | 🟢 免配置 | 字节分享 API (Community Service) | [查看指南](parsers/butterflyai.md) | +| 49 | **央视** | 栏目视频 (HLS/m3u8) / 央视新闻微视频 | ✅ 支持 | 🟢 免配置 | CNTV 调度接口 + 阿里云 API 网关 HMAC-SHA256 验签 | [查看指南](parsers/cctv.md) | +| 50 | **央视频** | 微视频 / 微短剧 / 高清封面与作者信息 | ✅ 支持 | 🟢 免配置 | 页面 SSR 状态机 (__STATE__) 提取与 Meta 跳转跟踪 | [查看指南](parsers/yangshipin.md) | --- diff --git a/docs/parsers/butterflyai.md b/docs/parsers/butterflyai.md new file mode 100644 index 0000000..fe6bf3b --- /dev/null +++ b/docs/parsers/butterflyai.md @@ -0,0 +1,48 @@ +# 星绘 AI (ButterflyAI) 逆向解析指南 + +本篇详细记录字节跳动豆包大模型家族旗下的 AI 艺术与生图/生视频平台 **星绘 AI (ButterflyAI)** 的作品解析方案。 + +--- + +## 1. 平台特征与支持能力 + +* **平台标识**:`星绘AI` +* **支持媒体类型**:无水印超清原画图集 / AI 视频生成作品 (MP4) / 封面 / 标题文案 / 创作者信息 +* **常见链接形态**: + * 官方分享短链:`https://www.butterflyai.cn/s/gQ0EXE9BJsw/` + * 社区分享长链:`https://www.butterflyai.cn/share/record?share_code=PRr8gE3n3eENtAl5FOtN&is_mine=1&resource_count=1&entity_type=1` + * H5 落地页:`https://www.butterflyai.cn/butterfly/community/h5/share/index.html?share_code=...` +* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。 + +--- + +## 2. 核心逆向流程 + +1. **短链展开与参数定位**: + * 用户分享获得的链接一般为 `https://www.butterflyai.cn/s/{code}/` 短链形态。 + * 通过 `WebFetcher.fetch_redirect_url` 跟随 302 重定向定位至落地页,提取关键参数 `share_code`(或 `share_token`)与 `share_id`。 +2. **社区分享开放接口**: + * 请求端点:`POST https://www.butterflyai.cn/butterfly/community/v1/share/record/get?aid=564650` + * 请求载荷: + ```json + { + "share_record_id": "", + "from_h5": true, + "share_token": "", + "extra_params": { + "aid": "564650", + "app_name": "星绘" + } + } + ``` +3. **数据结构与媒体提取**: + * **无水印原画图集**:从 `share_record.rendering_images` 或 `artwork.resource_list[].image_resource.rendering_images` 中提取图片。接口原生提供 `no_wm_download_url`(无水印原画下载直链),优先级高于带水印的 `download_url` 与 `image_url`。 + * **AI 视频作品**:从 `share_record.rendering_video.video_url` 或 `artwork.resource_list[].video_resource.rendering_videos[].video_url` 中提取原生 MP4 视频直链。 + * **元数据**:提取 `artwork.title`、Prompt、`show_info.effect_title`(如 AI 写真、AI 视频等),以及 `creator.screen_name` 和多档高清头像 `creator.avatar.large_url`。 + +--- + +## 3. 测试与验证 + +* **单元测试**:[tests/test_butterflyai_parser.py](file:///Users/leo/Projects/media-parser/tests/test_butterflyai_parser.py) +* **执行命令**:`python3 -m unittest tests/test_butterflyai_parser.py` diff --git a/docs/parsers/cctv.md b/docs/parsers/cctv.md new file mode 100644 index 0000000..049d5e3 --- /dev/null +++ b/docs/parsers/cctv.md @@ -0,0 +1,50 @@ +# 央视 (CCTV / 央视网 / 央视新闻) 逆向解析指南 + +本篇详细记录国家级综合电视广播与新闻门户 **央视 (CCTV)** 的电视节目、综合栏目点播与央视新闻微视频解析方案。 + +--- + +## 1. 平台特征与支持能力 + +* **平台标识**:`央视` +* **支持媒体类型**:高清 HLS 视频流 (.m3u8) / 央视新闻微视频 (MP4/m3u8) / 封面 / 标题文案 / 栏目与来源作者 +* **常见链接形态**: + * 央视电视点播:`https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml` + * 央视视频短链/手机端:`https://tv.cctv.cn/v/v1/VIDE4y5JLGNL5QIKkj1JrGnL160430.html` + * 央视新闻微视频/报道:`https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092` +* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。 + +--- + +## 2. 核心逆向流程 + +央视内容体系主要分为 **CCTV 综合/栏目点播体系** 与 **央视新闻客户端 (CCTVNews) 体系**,Parser 自动根据域名与路径进行路由分发: + +### 体系 A:CCTV 电视与栏目点播 (`tv.cctv.com`, `tv.cctv.cn`) +1. **PID / GUID 提取**: + * 页面内通过全局 JS 变量声明 32 位 Hex 视频全局标识,例如 `guid = "00e121ae62194fec8ab20e5b8eb9a89a"` 或 `pid = "..."`。 +2. **CNTV 视频调度分配接口**: + * 请求端点:`GET https://vdn.apps.cntv.cn/api/getHttpVideoInfo.do?pid={pid}` + * 直接提取 `hls_url`(官方多码率 HLS 流地址)、`image`(官方高分辨率静态封面)、`title`(节目期数与标题)及 `column`(栏目名称)。 + +### 体系 B:央视新闻客户端 (`content-static.cctvnews.cctv.com`) +1. **文章 / 微视频 ID**: + * 从 URL 参数中提取 `item_id` 或 `articleId`。 +2. **阿里云 API 网关 HMAC-SHA256 签名机制**: + * 接口端点:`https://api.cctvnews.cctv.com/1.0.0/feed/article/server/getArticle?articleId={item_id}&appcode=video_web` + * 请求头必须携带标准 API Gateway 签名参数: + * `x-ca-key`: 客户端公开 AppKey (`204133710`) + * `x-ca-stage`: `RELEASE` + * `x-ca-timestamp`: 当前毫秒时间戳 + * `x-ca-signature-headers`: `x-ca-key,x-ca-stage,x-ca-timestamp` + * `x-ca-signature`: 基于 `appSecret` 对签名规范字符串计算所得的 Base64 HMAC-SHA256 摘要。 +3. **Base64 响应解密**: + * 服务端响应返回 `{"code": 0, "response": ""}`。 + * 对 `response` 字段解码后即可得到完整文章、视频播放直链 (`videos[0].url`) 与缩略图。 + +--- + +## 3. 测试与验证 + +* **单元测试**:[tests/test_cctv_parser.py](file:///Users/leo/Projects/media-parser/tests/test_cctv_parser.py) +* **执行命令**:`python3 -m unittest tests/test_cctv_parser.py` diff --git a/docs/parsers/yangshipin.md b/docs/parsers/yangshipin.md new file mode 100644 index 0000000..9a8715f --- /dev/null +++ b/docs/parsers/yangshipin.md @@ -0,0 +1,36 @@ +# 央视频 (Yangshipin) 逆向解析指南 + +本篇详细记录中央广播电视总台 5G 新媒体旗舰平台 **央视频 (Yangshipin)** 的微视频、微短剧与体育资讯作品解析方案。 + +--- + +## 1. 平台特征与支持能力 + +* **平台标识**:`央视频` +* **支持媒体类型**:高分辨率封面 / 图集 / 完整视频标题 / 创作者与机构作者信息(头像与名称)/ 视频 ID (`vid`, `cid`) +* **常见链接形态**: + * 官方分享短链:`https://www.yspapp.cn/5Sqx`, `https://www.yspapp.cn/d1o` + * 移动端竖屏微短剧/小视频:`https://m.yangshipin.cn/portrait_video?vid=l00005817wl` + * 移动端常规横屏视频:`https://m.yangshipin.cn/video?type=0&vid=v000007pgfu&cid=...` + * PC/网页端:`https://yangshipin.cn/...` +* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。 + +--- + +## 2. 核心逆向流程 + +1. **Meta Refresh 重定向自动跟随**: + * 央视频短链 `yspapp.cn/{code}` 采用 HTML `` 形式执行页面跳转。 + * 解析器在 `WebFetcher` 与 `YangshipinParser` 中双重内置了针对 `` 的无感自动追踪,直接锁定最终目标落地页。 +2. **SSR 双状态机数据提取**: + * **横屏常规视频**:页面注入全局变量 `window.__STATE_video__`,解析 `payloads.sharevideo` 结构,获取 `title`、`cover_pic`、`cid`、`vid` 及 `om_info.title`(发布机构或频道名)。 + * **竖屏微短剧/短视频**:页面注入全局变量 `window.__STATE_portrait_video__`,解析 `payloads.videoDataList.items[0].videoData`,获取微短剧标题、`shareItem.shareImgUrl`(超清封面)、以及 `detailFollowItem.actorItem` 中的创作者昵称与头像。 +3. **播放流安全提示**: + * 央视频点播播放接口底层接入腾讯云 VOD 架构,采用带有动态 `cKey 8.1` / 白名单签名机制保护。解析器全面提取高保真元数据与封面图集,并保留扩展能力。 + +--- + +## 3. 测试与验证 + +* **单元测试**:[tests/test_yangshipin_parser.py](file:///Users/leo/Projects/media-parser/tests/test_yangshipin_parser.py) +* **执行命令**:`python3 -m unittest tests/test_yangshipin_parser.py` diff --git a/docs/testing.md b/docs/testing.md index 34733ec..0eb12c2 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -13,7 +13,7 @@ tests/ ├── test_parser_factory.py # 解析器自动注册与工厂路由测试 ├── test_web_fetcher.py # 302跳转与URL正则提取测试 ├── test_*_parser.py # 各平台专属的单元测试 -├── live_parser_samples.json # 真实线上验证样本库 (45+ 平台活跃样例) +├── live_parser_samples.json # 真实线上验证样本库 (50 平台活跃样例,300+ 用例) ├── test_live_parser_samples.py # 自动化执行样本库回归测试 └── manual_verify_parsers.py # 命令行交互式/全量人工排查脚本 ``` @@ -48,10 +48,10 @@ pytest tests/test_live_parser_samples.py -s ``` ### 4. 运行交互式真实样本回归验证工具 -[tests/manual_verify_parsers.py](file:///Users/leo/Projects/media-parser/tests/manual_verify_parsers.py) 支持对 45+ 个平台的多种形态(短链、长链、视频、图集等)发起真实网络请求与字段契约校验: +[tests/manual_verify_parsers.py](file:///Users/leo/Projects/media-parser/tests/manual_verify_parsers.py) 支持对 50 个平台的多种形态(短链、长链、视频、图集等)发起真实网络请求与字段契约校验: ```bash -# 1. 快速冒烟测试 (每个平台只测 1 条最具代表性的链接,极速完成 45 平台基准健康检查) +# 1. 快速冒烟测试 (每个平台只测 1 条最具代表性的链接,极速完成 50 平台基准健康检查) python3 tests/manual_verify_parsers.py --limit 1 # 2. 验证指定平台的多形态覆盖 (支持重复传入多个 --platform 参数) @@ -60,7 +60,7 @@ python3 tests/manual_verify_parsers.py --platform 微博 --platform 哔哩哔哩 # 3. 限制指定平台的测试用例数量 (例如每个平台最多测 2 条) python3 tests/manual_verify_parsers.py --platform 抖音 --platform 小红书 --limit 2 -# 4. 全量深度回归测试 (运行 270+ 条全部多形态真实用例,适合发版前全量巡检) +# 4. 全量深度回归测试 (运行 300+ 条全部多形态真实用例,适合发版前全量巡检) python3 tests/manual_verify_parsers.py # 5. 列出当前用例库中未配置真实链接的平台 diff --git a/src/parsers/butterflyai_parser.py b/src/parsers/butterflyai_parser.py new file mode 100644 index 0000000..d8647bb --- /dev/null +++ b/src/parsers/butterflyai_parser.py @@ -0,0 +1,163 @@ +import json +import re +from urllib.parse import parse_qs, urlparse +from configs.general_constants import USER_AGENT_M +from configs.logging_config import get_logger +from src.parser_factory import register_parser +from src.parsers.base_parser import BaseParser + +logger = get_logger(__name__) + + +@register_parser("星绘AI") +class ButterflyAIParser(BaseParser): + """星绘 AI (ButterflyAI / 字节跳动豆包家族) 解析器,支持无水印原画图集与视频生成作品提取。""" + + def __init__(self, real_url): + super().__init__(real_url) + self.headers = { + "User-Agent": USER_AGENT_M[0], + "Referer": "https://www.butterflyai.cn/", + "Accept": "application/json, text/plain, */*", + } + self.title = "" + self.cover_url = None + self.video_url = None + self.image_list = [] + self.author = None + self._parse() + + def _parse(self): + if not self.real_url: + return + + parsed = urlparse(self.real_url) + qs = parse_qs(parsed.query) + + # 1. 尝试从 URL 参数或路径提取 share_token / share_code / share_record_id + share_token = qs.get("share_code", [None])[0] or qs.get("share_token", [None])[0] + share_record_id = qs.get("share_id", [""])[0] or qs.get("share_record_id", [""])[0] + + # 若未获取到 share_token 且为短链路径形式 /s/{code}/,跟随重定向获取最终 URL + if not share_token: + match = re.search(r"/s/([a-zA-Z0-9_\-]+)", parsed.path) + if match: + try: + resp = self.session.get(self.real_url, headers=self.headers, allow_redirects=True, timeout=5) + final_parsed = urlparse(resp.url) + final_qs = parse_qs(final_parsed.query) + share_token = final_qs.get("share_code", [None])[0] or final_qs.get("share_token", [None])[0] + share_record_id = final_qs.get("share_id", [""])[0] or final_qs.get("share_record_id", [""])[0] + except Exception as e: + logger.warning("Failed to follow ButterflyAI redirect for %s: %s", self.real_url, e) + + if not share_token and not share_record_id: + logger.warning("Could not find share_token or share_record_id from %s", self.real_url) + return + + api_url = "https://www.butterflyai.cn/butterfly/community/v1/share/record/get?aid=564650" + payload = { + "share_record_id": share_record_id or "", + "from_h5": True, + "share_token": share_token or "", + "extra_params": { + "aid": "564650", + "app_name": "星绘", + }, + } + + try: + res = self.session.post(api_url, json=payload, headers=self.headers, timeout=5) + res.raise_for_status() + data = res.json() + except Exception as e: + logger.error("Failed to request ButterflyAI API: %s", e) + return + + status_info = data.get("status_info") or {} + status_code = status_info.get("status_code", data.get("status_code", 0)) + if status_code != 0: + logger.warning("ButterflyAI API returned error code %s: %s", status_code, status_info.get("status_msg") or data.get("message")) + return + + share_record = data.get("share_record") or {} + artwork = share_record.get("artwork") or {} + creator = share_record.get("creator") or {} + + # 提取标题与生成 Prompt + self.title = ( + artwork.get("title") + or artwork.get("description") + or share_record.get("prompt") + or share_record.get("show_info", {}).get("effect_title") + or "" + ).strip() + if not self.title and share_record.get("effect_list"): + first_eff = share_record["effect_list"][0] + self.title = ( + first_eff.get("name") + or first_eff.get("show_info", {}).get("effect_title") + or "" + ).strip() + + # 提取作者信息 + author_name = creator.get("screen_name") or creator.get("nickname") or creator.get("name") + avatar_obj = creator.get("avatar") or creator.get("avatar_url") + author_avatar = None + if isinstance(avatar_obj, dict): + author_avatar = avatar_obj.get("large_url") or avatar_obj.get("image_url") or avatar_obj.get("middle_url") + elif isinstance(avatar_obj, str): + author_avatar = avatar_obj + + if author_name or author_avatar: + self.author = { + "name": author_name, + "avatar": author_avatar, + } + + # 提取视频 (share_record.rendering_video 或 artwork.resource_list) + rendering_video = share_record.get("rendering_video") or {} + if rendering_video: + self.video_url = rendering_video.get("video_url") or rendering_video.get("download_url") + + for r in artwork.get("resource_list") or []: + if not isinstance(r, dict): + continue + vr = r.get("video_resource") or {} + for v in vr.get("rendering_videos") or []: + if not self.video_url: + self.video_url = v.get("video_url") or v.get("download_url") + + ir = r.get("image_resource") or {} + for img in ir.get("rendering_images") or []: + img_url = img.get("no_wm_download_url") or img.get("download_url") or img.get("image_url") + if img_url and img_url not in self.image_list: + self.image_list.append(img_url) + + # 提取图文列表(优先提取无水印原画 download_url) + for img in share_record.get("rendering_images") or []: + img_url = img.get("no_wm_download_url") or img.get("download_url") or img.get("image_url") + if img_url and img_url not in self.image_list: + self.image_list.append(img_url) + + # 提取封面 + self.cover_url = artwork.get("cover_image_url") or artwork.get("cover_url") + if not self.cover_url and rendering_video: + self.cover_url = rendering_video.get("cover_image_url") + if not self.cover_url and self.image_list: + self.cover_url = self.image_list[0] + + def get_real_video_url(self): + return self.video_url + + def get_title_content(self): + return self.title or "" + + def get_cover_photo_url(self): + return self.cover_url + + def get_author_info(self): + return self.author + + def get_image_list(self): + return self.image_list diff --git a/src/parsers/cctv_parser.py b/src/parsers/cctv_parser.py new file mode 100644 index 0000000..01f0bb9 --- /dev/null +++ b/src/parsers/cctv_parser.py @@ -0,0 +1,162 @@ +import base64 +import hashlib +import hmac +import json +import re +import time +from urllib.parse import parse_qs, urlparse +from configs.general_constants import USER_AGENT_PC +from configs.logging_config import get_logger +from src.parser_factory import register_parser +from src.parsers.base_parser import BaseParser + +logger = get_logger(__name__) + + +@register_parser("央视") +class CCTVParser(BaseParser): + """央视 (CCTV / 央视网 / 央视新闻) 解析器,支持常规栏目节目、央视新闻客户端与云平台视频提取。""" + + def __init__(self, real_url): + super().__init__(real_url) + self.headers = { + "User-Agent": USER_AGENT_PC[0], + "Referer": "https://tv.cctv.com/", + "Accept": "*/*", + } + self.title = "" + self.cover_url = None + self.video_url = None + self.author = None + self._parse() + + def _parse(self): + if not self.real_url: + return + + parsed = urlparse(self.real_url) + qs = parse_qs(parsed.query) + + # 区分 央视新闻 API 体系 与 CCTV 综合/电视频道体系 + if "cctvnews" in parsed.netloc or "snow-book" in parsed.path: + self._parse_cctvnews(parsed, qs) + else: + self._parse_cctv_web(parsed, qs) + + if self.title: + self.title = re.sub(r'<[^>]+>', '', self.title).strip() + + def _parse_cctvnews(self, parsed, qs): + """解析央视新闻客户端微视频及新闻报道(采用阿里云 API 网关 HMAC-SHA256 签名)""" + item_id = qs.get("item_id", [None])[0] or qs.get("articleId", [None])[0] + if not item_id: + match = re.search(r'(?:item_id|articleId)=([0-9a-zA-Z_\-]+)', self.real_url) + if match: + item_id = match.group(1) + + if not item_id: + logger.warning("Could not extract articleId/item_id from CCTV News URL: %s", self.real_url) + return + + app_key = "204133710" + app_secret = "etyEuNdA7GvQU7iPZHqnrBpSFfRyKQTD" + host = "https://api.cctvnews.cctv.com" + path = "/1.0.0/feed/article/server/getArticle" + params = {"articleId": item_id, "appcode": "video_web"} + t_now = int(time.time() * 1000) + + headers = { + "x-ca-timestamp": str(t_now), + "x-ca-key": app_key, + "x-ca-stage": "RELEASE", + "accept": "application/json", + "User-Agent": USER_AGENT_PC[0], + } + ca_headers = sorted([k for k in headers.keys() if k.startswith("x-ca-")]) + headers["x-ca-signature-headers"] = ",".join(ca_headers) + header_str = "\n".join([f"{k}:{headers[k]}" for k in ca_headers]) + sorted_params = sorted(params.items()) + query_str = "&".join([f"{k}={v}" for k, v in sorted_params]) + full_path = f"{path}?{query_str}" + string_to_sign = f"GET\n{headers['accept']}\n\n\n\n{header_str}\n{full_path}" + sig = base64.b64encode( + hmac.new(app_secret.encode("utf-8"), string_to_sign.encode("utf-8"), hashlib.sha256).digest() + ).decode("utf-8") + headers["x-ca-signature"] = sig + + try: + res = self.session.get(f"{host}{full_path}", headers=headers, timeout=5) + res.raise_for_status() + resp_json = res.json() + b64_response = resp_json.get("response") + if not b64_response: + logger.warning("CCTV News response missing encoded payload: %s", resp_json) + return + + payload_bytes = base64.b64decode(b64_response) + payload = json.loads(payload_bytes.decode("utf-8")).get("data", {}) + self.title = (payload.get("title") or "").strip() + + videos = payload.get("videos") or [] + if videos and isinstance(videos, list): + self.video_url = videos[0].get("url") + cover_info = videos[0].get("cover") + if isinstance(cover_info, dict): + self.cover_url = cover_info.get("url") + + if not self.cover_url: + thumbnails = payload.get("thumbnails") or [] + if thumbnails and isinstance(thumbnails, list): + self.cover_url = thumbnails[0].get("url") + + author_name = payload.get("source") or payload.get("author") or "央视新闻" + self.author = { + "name": author_name, + "avatar": None, + } + except Exception as e: + logger.error("Failed to parse CCTV News article: %s", e) + + def _parse_cctv_web(self, parsed, qs): + """解析 tv.cctv.com / tv.cctv.cn 等央视节目详情页(通过 CNTV 视频分配接口)""" + pid = qs.get("pid", [None])[0] or qs.get("guid", [None])[0] + if not pid: + html = self.fetch_html_content() + if html: + match = re.search(r'(?:guid|pid|videoCenterId)\s*=\s*["\']([0-9a-fA-F]{32})["\']', html) + if match: + pid = match.group(1) + + if not pid: + logger.warning("Could not extract video pid/guid from CCTV URL: %s", self.real_url) + return + + api_url = f"https://vdn.apps.cntv.cn/api/getHttpVideoInfo.do?pid={pid}" + try: + res = self.session.get(api_url, headers=self.headers, timeout=5) + res.raise_for_status() + data = res.json() + except Exception as e: + logger.error("Failed to fetch CCTV video info for pid %s: %s", pid, e) + return + + self.title = (data.get("title") or "").strip() + self.video_url = data.get("hls_url") + self.cover_url = data.get("image") + column_name = data.get("column") or data.get("produce") or "央视网" + self.author = { + "name": column_name, + "avatar": None, + } + + def get_real_video_url(self): + return self.video_url + + def get_title_content(self): + return self.title or "" + + def get_cover_photo_url(self): + return self.cover_url + + def get_author_info(self): + return self.author diff --git a/src/parsers/yangshipin_parser.py b/src/parsers/yangshipin_parser.py new file mode 100644 index 0000000..56bf05d --- /dev/null +++ b/src/parsers/yangshipin_parser.py @@ -0,0 +1,141 @@ +import json +import re +from bs4 import BeautifulSoup +from configs.general_constants import USER_AGENT_M +from configs.logging_config import get_logger +from src.parser_factory import register_parser +from src.parsers.base_parser import BaseParser + +logger = get_logger(__name__) + + +@register_parser("央视频") +class YangshipinParser(BaseParser): + """央视频 (Yangshipin) 客户端及 H5 平台解析器,支持横竖屏微短剧/精选视频元数据、原画封面与作者信息提取。""" + + def __init__(self, real_url): + super().__init__(real_url) + self.headers = { + "User-Agent": USER_AGENT_M[0], + "Referer": "https://m.yangshipin.cn/", + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", + } + self.title = "" + self.cover_url = None + self.video_url = None + self.image_list = [] + self.author = None + self._parse() + + def _parse(self): + if not self.real_url: + return + + html = self.fetch_html_content() + if not html: + logger.warning("Failed to fetch HTML content for Yangshipin URL: %s", self.real_url) + return + + # 兼容短链未重定向时页面内的 meta refresh 跳转 + meta_refresh = re.search(r''']+>', '', self.title).strip() + + if not self.video_url and self.cover_url and self.cover_url not in self.image_list: + self.image_list.append(self.cover_url) + + def _extract_state_json(self, html, state_key): + """从页面提取指定 state_key 的 JSON 数据对象""" + idx = html.find(state_key) + if idx == -1: + return None + start = html.find("{", idx) + if start == -1: + return None + try: + decoder = json.JSONDecoder() + data, _ = decoder.raw_decode(html[start:]) + return data + except Exception as e: + logger.warning("Failed to decode %s JSON: %s", state_key, e) + return None + + def get_real_video_url(self): + return self.video_url + + def get_title_content(self): + return self.title or "" + + def get_cover_photo_url(self): + return self.cover_url + + def get_author_info(self): + return self.author + + def get_image_list(self): + return self.image_list diff --git a/tests/live_parser_samples.json b/tests/live_parser_samples.json index eaeed0d..4ec478c 100644 --- a/tests/live_parser_samples.json +++ b/tests/live_parser_samples.json @@ -1,7 +1,7 @@ { "schema_version": 2, - "description": "45 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)", - "total_cases": 301, + "description": "50 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)", + "total_cases": 306, "cases": [ { "platform": "网易云音乐", @@ -4378,6 +4378,80 @@ ], "note": "2026-09-06 已验证: LOFTER 创作者短视频", "pattern": "同人短视频" + }, + { + "platform": "星绘AI", + "url": "https://www.butterflyai.cn/s/gQ0EXE9BJsw/", + "media_types": [ + "images" + ], + "expected_fields": [ + "images", + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: 星绘AI 无水印原画图集", + "pattern": "AI 生图/写真作品" + }, + { + "platform": "星绘AI", + "url": "https://www.butterflyai.cn/share/record?share_code=9r3kCJOnwubPE15yFhKY&is_mine=1&resource_count=1&entity_type=2", + "media_types": [ + "video" + ], + "expected_fields": [ + "video", + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: 星绘AI 视频生成作品", + "pattern": "AI 视频生成作品" + }, + { + "platform": "央视", + "url": "https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml?spm=C22284.PP7miAIBYiAV.EIn520ebHszi.32", + "media_types": [ + "video" + ], + "expected_fields": [ + "video", + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: CCTV 栏目点播视频与 HLS 流", + "pattern": "综合栏目点播" + }, + { + "platform": "央视", + "url": "https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092&t=1787103208459&toc_style_id=video_default&share_to=copy_url&track_id=766f3425-9b74-40ac-b2cf-751f9e5a19fd", + "media_types": [ + "video" + ], + "expected_fields": [ + "video", + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: 央视新闻客户端微视频", + "pattern": "新闻微视频" + }, + { + "platform": "央视频", + "url": "https://www.yspapp.cn/5Sqx", + "media_types": [ + "images" + ], + "expected_fields": [ + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: 央视频短链与微视频详情", + "pattern": "微视频及短剧详情" } ] } diff --git a/tests/test_butterflyai_parser.py b/tests/test_butterflyai_parser.py new file mode 100644 index 0000000..d411253 --- /dev/null +++ b/tests/test_butterflyai_parser.py @@ -0,0 +1,105 @@ +import unittest +from unittest.mock import patch, Mock + +from src.parsers.butterflyai_parser import ButterflyAIParser + + +class ButterflyAIParserTest(unittest.TestCase): + def test_parses_image_post_successfully(self): + fake_api_resp = { + "status_code": 0, + "status_info": {"status_code": 0, "status_msg": "Success"}, + "share_record": { + "artwork": { + "title": "夏日清凉少女写真", + "cover_image_url": "https://p26-community.butterfly.cn/cover.jpg", + }, + "creator": { + "screen_name": "星绘创作师", + "avatar": { + "large_url": "https://p3-passport.byteacctimg.com/avatar_large.jpeg", + "image_url": "https://p3-passport.byteacctimg.com/avatar.jpeg", + }, + }, + "rendering_images": [ + { + "no_wm_download_url": "https://p26-community.butterfly.cn/pic1_hd.jpg", + "download_url": "https://p26-community.butterfly.cn/pic1_wm.jpg", + }, + { + "image_url": "https://p26-community.butterfly.cn/pic2.jpg", + } + ], + }, + } + + mock_resp = Mock() + mock_resp.status_code = 200 + mock_resp.json.return_value = fake_api_resp + mock_resp.raise_for_status = Mock() + + with patch("requests.Session.post", return_value=mock_resp): + parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=abc123Test") + self.assertEqual(parser.get_title_content(), "夏日清凉少女写真") + self.assertEqual( + parser.get_author_info(), + {"name": "星绘创作师", "avatar": "https://p3-passport.byteacctimg.com/avatar_large.jpeg"}, + ) + self.assertEqual( + parser.get_image_list(), + [ + "https://p26-community.butterfly.cn/pic1_hd.jpg", + "https://p26-community.butterfly.cn/pic2.jpg", + ], + ) + self.assertEqual(parser.get_cover_photo_url(), "https://p26-community.butterfly.cn/cover.jpg") + self.assertIsNone(parser.get_real_video_url()) + + def test_parses_video_post_successfully(self): + fake_api_resp = { + "status_code": 0, + "status_info": {"status_code": 0, "status_msg": "Success"}, + "share_record": { + "show_info": {"effect_title": "AI 视频变换"}, + "creator": { + "screen_name": "特效大师", + "avatar_url": "https://p3-passport.byteacctimg.com/avatar_str.jpeg", + }, + "rendering_video": { + "video_url": "https://v-butterfly.douyinvod.com/video1.mp4", + "cover_image_url": "https://v-butterfly.douyinvod.com/cover1.jpg", + }, + }, + } + + mock_resp = Mock() + mock_resp.status_code = 200 + mock_resp.json.return_value = fake_api_resp + mock_resp.raise_for_status = Mock() + + with patch("requests.Session.post", return_value=mock_resp): + parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=vid123") + self.assertEqual(parser.get_title_content(), "AI 视频变换") + self.assertEqual( + parser.get_author_info(), + {"name": "特效大师", "avatar": "https://p3-passport.byteacctimg.com/avatar_str.jpeg"}, + ) + self.assertEqual(parser.get_real_video_url(), "https://v-butterfly.douyinvod.com/video1.mp4") + self.assertEqual(parser.get_cover_photo_url(), "https://v-butterfly.douyinvod.com/cover1.jpg") + self.assertEqual(parser.get_image_list(), []) + + def test_handles_api_failure_gracefully(self): + mock_resp = Mock() + mock_resp.status_code = 500 + mock_resp.raise_for_status.side_effect = Exception("API Internal Error") + + with patch("requests.Session.post", return_value=mock_resp): + parser = ButterflyAIParser("https://www.butterflyai.cn/share/record?share_code=fail123") + self.assertEqual(parser.get_title_content(), "") + self.assertIsNone(parser.get_real_video_url()) + self.assertIsNone(parser.get_author_info()) + self.assertEqual(parser.get_image_list(), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_cctv_parser.py b/tests/test_cctv_parser.py new file mode 100644 index 0000000..73c0bd7 --- /dev/null +++ b/tests/test_cctv_parser.py @@ -0,0 +1,82 @@ +import base64 +import json +import unittest +from unittest.mock import patch, Mock + +from src.parsers.cctv_parser import CCTVParser + + +class CCTVParserTest(unittest.TestCase): + def test_parses_cntv_web_video_successfully(self): + fake_html = """ + + 生财有道 + + + + + """ + fake_api_data = { + "title": "《生财有道》 20260526 匠心织锦绣", + "hls_url": "https://hls.cntv.lxdns.com/asp/hls/main/0303000a/3/default/00e12/main.m3u8", + "image": "https://p1.img.cctvpic.com/cover.jpg", + "column": "生财有道", + } + + mock_api_resp = Mock() + mock_api_resp.status_code = 200 + mock_api_resp.json.return_value = fake_api_data + mock_api_resp.raise_for_status = Mock() + + with patch.object(CCTVParser, "fetch_html_content", return_value=fake_html), \ + patch("requests.Session.get", return_value=mock_api_resp): + parser = CCTVParser("https://tv.cctv.com/2026/05/26/VIDElJFgf7P8XnEqtQr04Lf7260526.shtml") + self.assertEqual(parser.get_title_content(), "《生财有道》 20260526 匠心织锦绣") + self.assertEqual( + parser.get_real_video_url(), + "https://hls.cntv.lxdns.com/asp/hls/main/0303000a/3/default/00e12/main.m3u8", + ) + self.assertEqual(parser.get_cover_photo_url(), "https://p1.img.cctvpic.com/cover.jpg") + self.assertEqual(parser.get_author_info(), {"name": "生财有道", "avatar": None}) + + def test_parses_cctvnews_article_successfully(self): + fake_payload = { + "data": { + "title": "今天是第九个中国医师节", + "source": "央视新闻客户端", + "videos": [ + { + "url": "https://res.cctvnews.cctv.com/video/stream.m3u8", + "cover": {"url": "https://img.cctvnews.cctv.com/cover1.jpg"}, + } + ], + } + } + b64_str = base64.b64encode(json.dumps(fake_payload).encode("utf-8")).decode("utf-8") + fake_api_resp = {"code": 0, "response": b64_str} + + mock_resp = Mock() + mock_resp.status_code = 200 + mock_resp.json.return_value = fake_api_resp + mock_resp.raise_for_status = Mock() + + with patch("requests.Session.get", return_value=mock_resp): + url = "https://content-static.cctvnews.cctv.com/snow-book/video.html?item_id=12063254061111721092" + parser = CCTVParser(url) + self.assertEqual(parser.get_title_content(), "今天是第九个中国医师节") + self.assertEqual(parser.get_real_video_url(), "https://res.cctvnews.cctv.com/video/stream.m3u8") + self.assertEqual(parser.get_cover_photo_url(), "https://img.cctvnews.cctv.com/cover1.jpg") + self.assertEqual(parser.get_author_info(), {"name": "央视新闻客户端", "avatar": None}) + + def test_handles_missing_pid_gracefully(self): + with patch.object(CCTVParser, "fetch_html_content", return_value="No video"): + parser = CCTVParser("https://tv.cctv.com/empty.shtml") + self.assertEqual(parser.get_title_content(), "") + self.assertIsNone(parser.get_real_video_url()) + self.assertIsNone(parser.get_cover_photo_url()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_yangshipin_parser.py b/tests/test_yangshipin_parser.py new file mode 100644 index 0000000..4d57e89 --- /dev/null +++ b/tests/test_yangshipin_parser.py @@ -0,0 +1,122 @@ +import unittest +from unittest.mock import patch + +from src.parsers.yangshipin_parser import YangshipinParser + + +class YangshipinParserTest(unittest.TestCase): + def test_parses_portrait_video_successfully(self): + fake_html = """ + + 央视频 + + + + + """ + with patch.object(YangshipinParser, "fetch_html_content", return_value=fake_html): + parser = YangshipinParser("https://m.yangshipin.cn/portrait_video?vid=l00005817wl") + self.assertEqual(parser.get_title_content(), "5次助攻对4次助攻!巅峰对决") + self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover1.jpg") + self.assertEqual( + parser.get_author_info(), + {"name": "奥运来了", "avatar": "https://mpuser.ysp.cctv.cn/avatar1.jpeg"}, + ) + self.assertEqual(parser.get_image_list(), ["https://jietufengmian.yangshipin.cn/cover1.jpg"]) + + def test_parses_landscape_video_successfully(self): + fake_html = """ + + 央视频 + + + + + """ + with patch.object(YangshipinParser, "fetch_html_content", return_value=fake_html): + parser = YangshipinParser("https://m.yangshipin.cn/video?type=0&vid=v000007pgfu") + self.assertEqual(parser.get_title_content(), "《普法栏目剧》远山的守望") + self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover2.jpg") + self.assertEqual(parser.get_author_info(), {"name": "社会与法频道", "avatar": None}) + self.assertEqual(parser.get_image_list(), ["https://jietufengmian.yangshipin.cn/cover2.jpg"]) + + def test_follows_meta_refresh_redirect(self): + meta_html = """ + + + + 央视频 + """ + detail_html = """ + + + + """ + with patch.object(YangshipinParser, "fetch_html_content", side_effect=[meta_html, detail_html]): + parser = YangshipinParser("https://www.yspapp.cn/5Sqx") + self.assertEqual(parser.get_title_content(), "跳转后的视频标题") + self.assertEqual(parser.get_cover_photo_url(), "https://jietufengmian.yangshipin.cn/cover3.jpg") + + def test_handles_empty_or_broken_html(self): + with patch.object(YangshipinParser, "fetch_html_content", return_value="404 Not Found"): + parser = YangshipinParser("https://m.yangshipin.cn/video?vid=empty") + self.assertEqual(parser.get_title_content(), "") + self.assertIsNone(parser.get_cover_photo_url()) + self.assertIsNone(parser.get_author_info()) + self.assertEqual(parser.get_image_list(), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/utils/web_fetcher.py b/utils/web_fetcher.py index 9f5ab75..7ce1004 100755 --- a/utils/web_fetcher.py +++ b/utils/web_fetcher.py @@ -22,8 +22,8 @@ class WebFetcher: # 绿洲分享页会对通用桌面请求头返回 500,由解析器使用移动端请求头抓取。 # 微博博文可通过 API 直接根据 ID 解析,直接访问网页端常触发访客系统重定向。 domain = UrlParser.get_domain(url) - if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan", "dw4.co"}: - if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI", "网易LOFTER", "得物"}: + if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan", "dw4.co", "yspapp.cn"}: + if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI", "网易LOFTER", "得物", "央视"}: return UrlParser.extract_video_address(url) try: current_url = url @@ -32,6 +32,10 @@ class WebFetcher: resp = requests.get(current_url, headers=WebFetcher.headers, allow_redirects=False, timeout=5) resp.raise_for_status() redirect_url = resp.headers.get("location") + if not redirect_url and resp.status_code == 200 and isinstance(getattr(resp, "text", None), str): + meta_match = re.search(r']*http-equiv=["\']?refresh["\']?[^>]*content=["\']?[^"\']*URL=[\'"]?([^\'" >]+)', resp.text, re.I) + if meta_match: + redirect_url = meta_match.group(1).strip("'\"") if redirect_url: redirect_url = urljoin(current_url, redirect_url) # 如果重定向到了登录页、404拦截页、验证码校验页或错误页,不要更新 url,直接中断以保留原始有效 URL @@ -105,6 +109,15 @@ class UrlParser: if domain.endswith('.dewu.com') or domain.endswith('.poizon.com'): return '得物' + if domain == 'butterflyai.cn' or domain.endswith('.butterflyai.cn'): + return '星绘AI' + + if domain in {'cctv.com', 'cctv.cn'} or domain.endswith('.cctv.com') or domain.endswith('.cctv.cn'): + return '央视' + + if domain in {'yangshipin.cn', 'yspapp.cn'} or domain.endswith('.yangshipin.cn') or domain.endswith('.yspapp.cn'): + return '央视频' + return None @staticmethod @@ -365,6 +378,33 @@ class UrlParser: preserved_params.append((key, value)) if preserved_params: address = f"{address}?{urlencode(preserved_params)}" + elif platform == "星绘AI": + query_params = parse_qs(parsed_url.query) + preserved_params = [] + for key in ("share_code", "share_id", "entity_type", "is_mine", "resource_count", "share_token"): + value = query_params.get(key, [None])[0] + if value is not None: + preserved_params.append((key, value)) + if preserved_params: + address = f"{address}?{urlencode(preserved_params)}" + elif platform == "央视": + query_params = parse_qs(parsed_url.query) + preserved_params = [] + for key in ("item_id", "articleId", "pid", "guid"): + value = query_params.get(key, [None])[0] + if value is not None: + preserved_params.append((key, value)) + if preserved_params: + address = f"{address}?{urlencode(preserved_params)}" + elif platform == "央视频": + query_params = parse_qs(parsed_url.query) + preserved_params = [] + for key in ("vid", "cid", "type", "serverFrom"): + value = query_params.get(key, [None])[0] + if value is not None: + preserved_params.append((key, value)) + if preserved_params: + address = f"{address}?{urlencode(preserved_params)}" return address @staticmethod @@ -446,12 +486,26 @@ class UrlParser: params_ps = query_params.get('ps', [None])[0] if params_ps: return params_ps + params_share_code = query_params.get('share_code', [None])[0] + if params_share_code: + return params_share_code + params_item_id = query_params.get('item_id', [None])[0] + if params_item_id: + return params_item_id + params_article_id = query_params.get('articleId', [None])[0] + if params_article_id: + return params_article_id + params_guid = query_params.get('guid', [None])[0] + if params_guid: + return params_guid # 尝试从URL路径中获取视频ID path_segments = parsed_url.path.strip('/').split('/') if path_segments: video_id = path_segments[-1] if video_id.endswith('.html'): video_id = video_id[:-5] + elif video_id.endswith('.shtml'): + video_id = video_id[:-6] desktop_mv = re.fullmatch(r'mv_([0-9a-fA-F]{32})', video_id) if desktop_mv: return desktop_mv.group(1)