mirror of
https://github.com/ucmao/media-parser.git
synced 2026-11-20 08:04:56 +08:00
feat: 添加得物与网易LOFTER平台解析支持
This commit is contained in:
@@ -3,15 +3,15 @@
|
||||
|
||||
**基于 Python 的多平台媒体原生本地解析系统**
|
||||
|
||||
[](LICENSE) [](https://www.python.org/) [](https://flask.palletsprojects.com/) [](#-部署指南) [](#-支持的平台矩阵)
|
||||
[](LICENSE) [](https://www.python.org/) [](https://flask.palletsprojects.com/) [](#-部署指南) [](#-支持的平台矩阵)
|
||||
|
||||
<p align="center">
|
||||
<a href="#-核心解析逻辑">解析逻辑</a> •
|
||||
<a href="#-支持的平台矩阵">支持平台</a> •
|
||||
<a href="#-部署指南">部署指南</a> •
|
||||
<a href="#-项目结构">项目结构</a> •
|
||||
<a href="#-开发者文档与逆向百科">开发文档</a> •
|
||||
<a href="#-api-核心接口说明">接口文档</a> •
|
||||
<a href="#-自动化测试与健康自检">测试自检</a> •
|
||||
<a href="#-开发者文档与逆向百科">开发文档</a> •
|
||||
<a href="#-联系作者">联系作者</a>
|
||||
</p>
|
||||
|
||||
@@ -42,55 +42,36 @@
|
||||
|
||||
## 💾 支持的平台矩阵
|
||||
|
||||
| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 实况 |
|
||||
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
||||
| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ |
|
||||
| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ |
|
||||
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
|
||||
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **闲鱼** | | ✓ | ✓ | | ✓ | | | |
|
||||
| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | |
|
||||
| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
|
||||
| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | | |
|
||||
| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **微视** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **AcFun** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **今日头条** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **全民K歌** | | ✓ | ✓ | ✓ | | | | |
|
||||
| **新片场** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **好看视频** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **梨视频** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **虎牙** | | ✓ | ✓ | ✓ | | | | |
|
||||
| **美拍** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **最右** | ✓ | ✓ | | ✓ | | | | |
|
||||
| **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | | |
|
||||
| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 |
|
||||
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
||||
| **抖音** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
|
||||
| **小红书** | ✓ | ✓ | ✓ | ✓ | **✓+** | | | **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | |
|
||||
| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **微视** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **快手** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | | **AcFun** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **今日头条** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | **全民K歌** | | ✓ | ✓ | ✓ | | | |
|
||||
| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **新片场** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **好看视频** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **梨视频** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **闲鱼** | | ✓ | ✓ | | ✓ | | | **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **虎牙** | | ✓ | ✓ | ✓ | | | |
|
||||
| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **美拍** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | **最右** | ✓ | ✓ | | ✓ | | | |
|
||||
| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | |
|
||||
| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | | |
|
||||
| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | | | | | | | | |
|
||||
|
||||
*注:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。
|
||||
> 💡 **说明**:
|
||||
> * **图集 `✓+`**:表示该平台支持提取 iOS/Android 原始 **LivePhoto 动态实况图**(含封面静图与关联 `.mov`/`.mp4` 实况视频原件)。
|
||||
> * **原生水印提示**:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。
|
||||
|
||||
---
|
||||
|
||||
@@ -147,71 +128,6 @@ pip install -r requirements.txt
|
||||
python app.py
|
||||
```
|
||||
|
||||
## 📂 项目结构
|
||||
|
||||
```text
|
||||
media-parser/
|
||||
├── app.py # 应用入口 (Flask Web 服务与 API 启动)
|
||||
├── configs/ # 核心配置 (域名平台映射、日志配置等)
|
||||
├── docs/ # 逆向百科与开发文档
|
||||
│ ├── architecture.md # 系统分层架构与生命周期设计
|
||||
│ ├── reverse-guide.md # 通用逆向方法论 (抓包/SSR/JS签名提取)
|
||||
│ ├── testing.md # 完整测试规范与回归手册
|
||||
│ └── parsers/ # 42 份逆向分析文档,覆盖 45 个平台
|
||||
├── src/ # 核心业务逻辑
|
||||
│ ├── api/ # RESTful API 路由 (/api/parse, /api/health)
|
||||
│ ├── web/ # Demo 体验页与交互蓝图
|
||||
│ ├── parsers/ # 42 个解析器模块,支持 45 个平台 (核心解析逻辑)
|
||||
│ └── parser_factory.py # 工厂分发器 (解析器动态发现与自动注册)
|
||||
├── utils/ # 底层工具库与逆向支持
|
||||
│ ├── signer/ # JS 签名沙箱引擎 (a_bogus, x_bogus 等算法执行)
|
||||
│ └── web_fetcher.py # 智能 URL 识别、重定向追踪与请求封装
|
||||
├── tests/ # 完备的双层测试体系
|
||||
│ ├── live_parser_samples.json # 45 平台真实多形态在线样本库
|
||||
│ ├── manual_verify_parsers.py # 命令行交互式冒烟与健康检查工具
|
||||
│ └── test_*_parser.py # 各平台 Mock 自动化单元测试
|
||||
├── static/ & templates/ # Web 演示页面前端静态资源
|
||||
└── docker-compose.yml # 容器化一键部署编排
|
||||
```
|
||||
|
||||
## 📖 开发者文档与逆向百科
|
||||
|
||||
本项目提供了详尽的技术架构与全平台逆向分析手册,详细内容请查阅 **[`docs/`](docs/)** 目录:
|
||||
|
||||
* 🏗️ **[系统架构与生命周期设计](docs/architecture.md)**:分层设计、302 追踪与 `ParserFactory` 动态发现机制。
|
||||
* 🔍 **[通用逆向方法论](docs/reverse-guide.md)**:SSR 数据提取、H5 接口伪装、JS 签名沙箱及抓包 SOP。
|
||||
* 🧪 **[测试体系与回归验证](docs/testing.md)**:Pytest 单元测试、Mock 与线上真实样例回归。
|
||||
* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 45 个平台的技术文档。
|
||||
|
||||
---
|
||||
|
||||
## 🧪 解析有效性测试
|
||||
|
||||
本项目拥有完备的测试体系,包括基础单元测试与基于真实样本库的在线回归测试:
|
||||
|
||||
### 1. 真实样本交互式验证
|
||||
样例测试库见 [`tests/live_parser_samples.json`](tests/live_parser_samples.json),用于实时验证 45 个平台的解析有效性与多形态覆盖:
|
||||
|
||||
```bash
|
||||
# 快速冒烟测试(每个平台测 1 条最具代表性的链接,极速完成健康检查)
|
||||
python3 tests/manual_verify_parsers.py --limit 1
|
||||
|
||||
# 全量回归验证(覆盖多形态真实用例)
|
||||
python3 tests/manual_verify_parsers.py
|
||||
|
||||
# 仅验证单个或指定平台(如:小云雀AI)
|
||||
python3 tests/manual_verify_parsers.py --platform "小云雀AI"
|
||||
```
|
||||
|
||||
### 2. 自动化单元测试
|
||||
```bash
|
||||
# 运行全部单元测试
|
||||
pytest
|
||||
|
||||
# 运行真实线上样例自动化回归
|
||||
pytest tests/test_live_parser_samples.py -s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔌 API 核心接口说明
|
||||
@@ -292,6 +208,33 @@ pytest tests/test_live_parser_samples.py -s
|
||||
|
||||
---
|
||||
|
||||
## 🧪 自动化测试与健康自检
|
||||
|
||||
本项目拥有完备的双层测试体系(Mock 单元测试 + 47 平台真实在线样本库回归)。遇到解析异常或日常部署验证时,可一键运行健康自检:
|
||||
|
||||
```bash
|
||||
# 47 平台极速冒烟测试(每个平台测 1 条最具代表性的链接,秒级完成健康检查)
|
||||
python3 tests/manual_verify_parsers.py --limit 1
|
||||
|
||||
# 仅验证单个或指定平台(如:小云雀AI / 抖音)
|
||||
python3 tests/manual_verify_parsers.py --platform "小云雀AI"
|
||||
```
|
||||
|
||||
> 💡 更多多形态全量回归测试命令与 Pytest 自动化规范,请参阅 **[测试体系与回归验证指南 (docs/testing.md)](docs/testing.md)**。
|
||||
|
||||
---
|
||||
|
||||
## 📖 开发者文档与逆向百科
|
||||
|
||||
本项目提供了详尽的技术架构与全平台逆向分析手册,详细内容请查阅 **[`docs/`](docs/)** 目录:
|
||||
|
||||
* 🏗️ **[系统架构与生命周期设计](docs/architecture.md)**:分层设计、302 追踪与 `ParserFactory` 动态发现机制。
|
||||
* 🔍 **[通用逆向方法论](docs/reverse-guide.md)**:SSR 数据提取、H5 接口伪装、JS 签名沙箱及抓包 SOP。
|
||||
* 🧪 **[测试体系与回归验证](docs/testing.md)**:Pytest 单元测试、Mock 与线上真实样例回归。
|
||||
* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 47 个平台的技术文档。
|
||||
|
||||
---
|
||||
|
||||
## 📩 联系作者
|
||||
|
||||
如果您在安装、使用过程中遇到问题,或有定制需求,请通过以下方式联系:
|
||||
|
||||
@@ -134,7 +134,15 @@
|
||||
"yangkeduo.com": "拼多多",
|
||||
"pinduoduo.com": "拼多多",
|
||||
"www.pinduoduo.com": "拼多多",
|
||||
"mobile.pinduoduo.com": "拼多多"
|
||||
"mobile.pinduoduo.com": "拼多多",
|
||||
"dw4.co": "得物",
|
||||
"dewu.com": "得物",
|
||||
"www.dewu.com": "得物",
|
||||
"m.dewu.com": "得物",
|
||||
"poizon.com": "得物",
|
||||
"www.poizon.com": "得物",
|
||||
"lofter.com": "网易LOFTER",
|
||||
"www.lofter.com": "网易LOFTER"
|
||||
},
|
||||
"USER_AGENT_PC": [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
|
||||
|
||||
@@ -54,6 +54,33 @@ flowchart TD
|
||||
ParseAPI --> CommonUtils
|
||||
```
|
||||
|
||||
### 📂 源码目录组织与模块映射
|
||||
|
||||
```text
|
||||
media-parser/
|
||||
├── app.py # 应用入口 (Flask Web 服务与 API 启动)
|
||||
├── configs/ # 核心配置 (域名平台映射、日志配置等)
|
||||
├── docs/ # 逆向百科与开发文档体系
|
||||
│ ├── architecture.md # 系统分层架构与生命周期设计
|
||||
│ ├── reverse-guide.md # 通用逆向方法论 (抓包/SSR/JS签名提取)
|
||||
│ ├── testing.md # 完整测试规范与回归手册
|
||||
│ └── parsers/ # 平台逆向分析手册
|
||||
├── src/ # 核心业务逻辑
|
||||
│ ├── api/ # RESTful API 路由 (/api/parse, /api/health)
|
||||
│ ├── web/ # Demo 体验页与交互蓝图
|
||||
│ ├── parsers/ # 各平台解析器模块 (核心解析逻辑)
|
||||
│ └── parser_factory.py # 工厂分发器 (解析器动态发现与自动注册)
|
||||
├── utils/ # 底层工具库与逆向支持
|
||||
│ ├── signer/ # JS 签名沙箱引擎 (a_bogus 等算法执行)
|
||||
│ └── web_fetcher.py # 智能 URL 识别、重定向追踪与请求封装
|
||||
├── tests/ # 完备的双层测试体系
|
||||
│ ├── live_parser_samples.json # 平台真实多形态在线样本库
|
||||
│ ├── manual_verify_parsers.py # 命令行交互式冒烟与健康检查工具
|
||||
│ └── test_*_parser.py # 各平台 Mock 自动化单元测试
|
||||
├── static/ & templates/ # Web 演示页面前端静态资源
|
||||
└── docker-compose.yml # 容器化一键部署编排
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 请求处理生命周期 (Lifecycle)
|
||||
|
||||
+4
-2
@@ -2,7 +2,7 @@
|
||||
|
||||
欢迎查阅 **Media Parser** 开发者文档与逆向工程知识库。
|
||||
|
||||
本项目是一个高性能、模块化、支持 **45 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。
|
||||
本项目是一个高性能、模块化、支持 **47 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。
|
||||
|
||||
---
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
* 🧪 **[测试与回归验证 (Testing Guide)](testing.md)**:学习 Pytest 单元测试、Mock 构造与真实样本(Live Samples)测试。
|
||||
* 📖 **平台实战指南 (Parser Guides)**:
|
||||
* **短视频与轻社区**:[抖音](parsers/douyin.md) | [快手](parsers/kuaishou.md) | [今日头条](parsers/xigua.md) | [皮皮虾](parsers/pipixia.md) | [皮皮搞笑](parsers/pipigaoxiao.md) | [最右](parsers/zuiyou.md) | [美拍](parsers/meipai.md) | [微视](parsers/weishi.md) | [绿洲](parsers/lvzhou.md) | [番茄小说/红果短剧/红果漫剧](parsers/fanqie.md)
|
||||
* **图文与综合社区**:[小红书](parsers/xiaohongshu.md) | [微信公众号](parsers/wechat-mp.md) | [微博](parsers/weibo.md) | [知乎](parsers/zhihu.md) | [闲鱼](parsers/xianyu.md) | [Soul](parsers/soul.md)
|
||||
* **图文与综合社区**:[小红书](parsers/xiaohongshu.md) | [微信公众号](parsers/wechat-mp.md) | [微博](parsers/weibo.md) | [知乎](parsers/zhihu.md) | [闲鱼](parsers/xianyu.md) | [Soul](parsers/soul.md) | [得物](parsers/dewu.md) | [网易LOFTER](parsers/lofter.md)
|
||||
* **长视频与弹幕/创作**:[哔哩哔哩](parsers/bilibili.md) | [AcFun](parsers/acfun.md) | [新片场](parsers/xinpianchang.md) | [好看视频](parsers/haokan.md) | [西瓜视频](parsers/xigua.md) | [剪映](parsers/jianying.md) | [快影](parsers/kwaiying.md)
|
||||
* **AI 生成与大模型**:[豆包 AI](parsers/doubao.md) | [腾讯元宝](parsers/yuanbao.md) | [即梦 AI](parsers/jimeng.md) | [可灵 AI](parsers/kling.md) | [海螺 AI](parsers/hailuo.md) | [通义千问](parsers/qianwen.md) | [夸克 AI](parsers/quark-ai.md) | [小云雀 AI](parsers/xiaoyunque.md) | [松果时刻](parsers/pinecone-moment.md)
|
||||
* **音频与垂直频道**:[汽水音乐](parsers/qsmusic.md) | [QQ音乐](parsers/qqmusic.md) | [网易云音乐](parsers/netease-music.md) | [酷狗音乐](parsers/kugou-music.md) | [配音秀](parsers/peiyinxiu.md) | [全民K歌](parsers/quanminkge.md) | [虎牙](parsers/huya.md) | [梨视频](parsers/lishipin.md) | [微信视频号](parsers/wechat-channels.md) | [腾讯频道](parsers/tencent-channel.md)
|
||||
@@ -74,6 +74,8 @@
|
||||
| 43 | **酷狗音乐** | MV / 免费歌曲 / 多档 MP4 | ✅ 支持 | 🟢 公开内容免配置 | H5 公开接口签名 + 分享页数据 | [查看指南](parsers/kugou-music.md) |
|
||||
| 44 | **配音秀** | 配音作品视频 (MP4) | ✅ 支持 | 🟢 公开内容免配置 | 作品页初始化数据提取 | [查看指南](parsers/peiyinxiu.md) |
|
||||
| 45 | **松果时刻** | AI 故事多页视频 / 图集 / 配音 | ✅ 支持 | 🟢 公开内容免配置 | 公开详情接口 | [查看指南](parsers/pinecone-moment.md) |
|
||||
| 46 | **得物** | 潮流穿搭 / 开箱短视频 / 高清图集 | ✅ 支持 | 🟢 免配置 | Next.js metaOGInfo 数据提取 | [查看指南](parsers/dewu.md) |
|
||||
| 47 | **网易LOFTER** | 二次元插画 / 摄影图集 / 创作者短视频 | ✅ 支持 | 🟢 免配置 | window.__initialize_data__ 数据提取 | [查看指南](parsers/lofter.md) |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# 得物 (Dewu) 逆向解析指南
|
||||
|
||||
本篇详细记录潮流网购与 UGC 社区 **得物 (Poizon)** 的短视频与图集解析方案。
|
||||
|
||||
---
|
||||
|
||||
## 1. 平台特征与支持能力
|
||||
|
||||
* **平台标识**:`得物`
|
||||
* **支持媒体类型**:无水印 720P 原画视频 (MP4) / 社区穿搭高清图集 / 封面 / 标题文案 / 作者信息
|
||||
* **常见链接形态**:
|
||||
* 官方分享短链:`https://dw4.co/t/A/HmfgCvBiU`
|
||||
* 社区长链:`https://m.dewu.com/rn-activity/community-share?trendId=514124663&shareId=...`
|
||||
* 网页端/PC端:`https://dewu.com/...`, `https://poizon.com/...`
|
||||
* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心逆向流程
|
||||
|
||||
1. **短链展开**:
|
||||
* 用户通过 App 分享通常获得形如 `https://dw4.co/t/A/xxxx` 的短链。
|
||||
* 系统通过 `WebFetcher.fetch_redirect_url` 跟踪 HTTP 302 重定向至 `m.dewu.com/rn-activity/community-share?trendId=...`。
|
||||
2. **Next.js SSR 数据提取**:
|
||||
* 得物移动端分享页基于 Next.js 渲染,页面源码内置 `<script>` 标签注入的 `props.pageProps.metaOGInfo.data[0]`。
|
||||
* 解析 JSON 对象:
|
||||
* **无水印视频**:遍历 `content.media.list`,提取 `mediaType == 'video'` 的原画视频 URL(`videocdn.poizon.com/app/mf/dw264_720p/...`)。若未找到则降级提取 `content.videoShareUrl`。
|
||||
* **高清图集**:遍历 `content.media.list`,提取所有 `mediaType == 'img'` 的大图 URL。
|
||||
* **元数据**:提取 `content.title`(标题)、`content.content`(正文)、`content.cover.url`(封面)以及 `userInfo.userName` / `userInfo.icon`(作者)。
|
||||
|
||||
---
|
||||
|
||||
## 3. 测试与验证
|
||||
|
||||
* **单元测试**:[tests/test_dewu_parser.py](file:///Users/leo/Projects/media-parser/tests/test_dewu_parser.py)
|
||||
* **执行命令**:`python3 -m unittest tests/test_dewu_parser.py`
|
||||
@@ -0,0 +1,41 @@
|
||||
# 网易 LOFTER (乐乎) 逆向解析指南
|
||||
|
||||
本篇详细记录网易旗下泛二次元创作者社区 **LOFTER** 的插画图集与短视频解析方案。
|
||||
|
||||
---
|
||||
|
||||
## 1. 平台特征与支持能力
|
||||
|
||||
* **平台标识**:`网易LOFTER`
|
||||
* **支持媒体类型**:无损原画大图 (raw/orign) / 创作者短视频 (MP4) / 封面 / 标题文案 / 博客作者信息
|
||||
* **常见链接形态**:
|
||||
* 博主专属二级域名:`https://emm3716958.lofter.com/post/74daebd2_34ec49f1a?incantation=...`
|
||||
* 官方主站路径:`https://www.lofter.com/front/tagChat/answer/detail?permalink=...`
|
||||
* 主域名长链:`https://lofter.com/post/...`
|
||||
* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心逆向流程
|
||||
|
||||
1. **动态泛域名通配**:
|
||||
* LOFTER 为每位创作者分配独立的个性化子域名(形如 `*.lofter.com`)。
|
||||
* `UrlParser.get_platform` 使用后缀匹配规则识别所有 `*.lofter.com` 域名为 `网易LOFTER`。
|
||||
2. **状态注入提取**:
|
||||
* 请求文章 H5 页面,定位并提取 `window.__initialize_data__` JSON 对象。
|
||||
* **博客文章 (`postData`)**:
|
||||
* **二次元/同人图集 (type 2)**:从 `postView.photoPostView.photoLinks` 提取高清大图,优先获取未经压缩的无损原图 `raw`,备选高画质 `orign`。
|
||||
* **短视频作品 (type 4)**:从 `postView.videoPostView.videoInfo` 提取 `originUrl` MP4 直链与首帧封面。
|
||||
* **作者信息**:从 `blogInfo` 提取 `blogNickName` 和头像 `bigAvaImg`。
|
||||
* **标签问答/互动讨论 (`answerDetailData`)**:
|
||||
* 从 `answerDetailData.images` 提取高清原图,从 `blogInfo` 提取作者信息。
|
||||
3. **文本清洗与容灾**:
|
||||
* 对富文本 HTML 标签(如 `<p id="...">`)进行正则净化,提取纯净标题与文案。
|
||||
* 当未渲染出 JS 数据时,使用 OpenGraph 和 HTML DOM 进行元数据与媒体流兜底提取。
|
||||
|
||||
---
|
||||
|
||||
## 3. 测试与验证
|
||||
|
||||
* **单元测试**:[tests/test_lofter_parser.py](file:///Users/leo/Projects/media-parser/tests/test_lofter_parser.py)
|
||||
* **执行命令**:`python3 -m unittest tests/test_lofter_parser.py`
|
||||
@@ -0,0 +1,129 @@
|
||||
import json
|
||||
import re
|
||||
from bs4 import BeautifulSoup
|
||||
from configs.general_constants import USER_AGENT_M
|
||||
from configs.logging_config import get_logger
|
||||
from src.parser_factory import register_parser
|
||||
from src.parsers.base_parser import BaseParser
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@register_parser("得物")
|
||||
class DewuParser(BaseParser):
|
||||
"""得物 App 社区动态解析器,支持无水印原画视频、图文图集、标题与作者信息提取。"""
|
||||
|
||||
def __init__(self, real_url):
|
||||
super().__init__(real_url)
|
||||
self.headers = {
|
||||
"User-Agent": USER_AGENT_M[0],
|
||||
"Referer": "https://m.dewu.com/",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
}
|
||||
self.title = ""
|
||||
self.cover_url = None
|
||||
self.video_url = None
|
||||
self.image_list = []
|
||||
self.author = None
|
||||
self._parse()
|
||||
|
||||
def _parse(self):
|
||||
if not self.real_url:
|
||||
return
|
||||
html = self.fetch_html_content()
|
||||
if not html:
|
||||
logger.warning("Failed to fetch HTML content for Dewu URL: %s", self.real_url)
|
||||
return
|
||||
|
||||
try:
|
||||
raw_json = self._extract_json_data(html)
|
||||
if not raw_json:
|
||||
logger.warning("Failed to locate Next.js metaOGInfo data in Dewu page")
|
||||
return
|
||||
|
||||
data = json.loads(raw_json)
|
||||
page_props = data.get("props", {}).get("pageProps", {})
|
||||
meta_data_list = page_props.get("metaOGInfo", {}).get("data", [])
|
||||
if not meta_data_list or not isinstance(meta_data_list, list):
|
||||
logger.warning("Dewu metaOGInfo data list is empty")
|
||||
return
|
||||
|
||||
target_data = meta_data_list[0]
|
||||
content_data = target_data.get("content", {})
|
||||
user_data = target_data.get("userInfo", {}) or {}
|
||||
|
||||
# 提取标题与文案
|
||||
self.title = (content_data.get("title") or content_data.get("content") or "").strip()
|
||||
|
||||
# 提取封面
|
||||
cover_info = content_data.get("cover") or {}
|
||||
if isinstance(cover_info, dict):
|
||||
self.cover_url = cover_info.get("url")
|
||||
|
||||
# 提取作者信息
|
||||
if user_data:
|
||||
author_name = user_data.get("userName")
|
||||
author_avatar = user_data.get("icon")
|
||||
if author_name or author_avatar:
|
||||
self.author = {
|
||||
"name": author_name,
|
||||
"avatar": author_avatar,
|
||||
}
|
||||
|
||||
# 提取媒体内容(优先从 media.list 提取无水印原画视频及图集)
|
||||
media_info = content_data.get("media") or {}
|
||||
media_list = media_info.get("list", []) if isinstance(media_info, dict) else []
|
||||
|
||||
for item in media_list:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
media_type = item.get("mediaType")
|
||||
url = item.get("url")
|
||||
if not url:
|
||||
continue
|
||||
|
||||
if media_type == "video" and not self.video_url:
|
||||
self.video_url = url
|
||||
elif media_type == "img":
|
||||
self.image_list.append(url)
|
||||
|
||||
# 视频兜底:若 media.list 未提取到视频,尝试提取 videoShareUrl
|
||||
if not self.video_url:
|
||||
fallback_video = content_data.get("videoShareUrl")
|
||||
if fallback_video:
|
||||
self.video_url = fallback_video
|
||||
|
||||
# 图文兜底:若是纯图文作品且 media.list 为空,使用 cover_url 作为单图
|
||||
if not self.video_url and not self.image_list and self.cover_url:
|
||||
self.image_list.append(self.cover_url)
|
||||
|
||||
except Exception as exc:
|
||||
logger.exception("Error parsing Dewu share data: %s", exc)
|
||||
|
||||
def _extract_json_data(self, html):
|
||||
"""从页面提取内嵌 Next.js 数据对象"""
|
||||
soup = BeautifulSoup(html, "lxml")
|
||||
for script in soup.find_all("script"):
|
||||
if script.string and "metaOGInfo" in script.string:
|
||||
return script.string.strip()
|
||||
|
||||
# 备用正则匹配
|
||||
match = re.search(r'<script[^>]*>(.*?metaOGInfo.*?)</script>', html, re.DOTALL)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return None
|
||||
|
||||
def get_real_video_url(self):
|
||||
return self.video_url
|
||||
|
||||
def get_title_content(self):
|
||||
return self.title or ""
|
||||
|
||||
def get_cover_photo_url(self):
|
||||
return self.cover_url
|
||||
|
||||
def get_author_info(self):
|
||||
return self.author
|
||||
|
||||
def get_image_list(self):
|
||||
return self.image_list
|
||||
@@ -0,0 +1,190 @@
|
||||
import json
|
||||
import re
|
||||
from bs4 import BeautifulSoup
|
||||
from configs.general_constants import USER_AGENT_M
|
||||
from configs.logging_config import get_logger
|
||||
from src.parser_factory import register_parser
|
||||
from src.parsers.base_parser import BaseParser
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@register_parser("网易LOFTER")
|
||||
class LofterParser(BaseParser):
|
||||
"""网易 LOFTER (乐乎) 社区解析器,支持二次元/同人/摄影等原画图集与短视频提取。"""
|
||||
|
||||
def __init__(self, real_url):
|
||||
super().__init__(real_url)
|
||||
self.headers = {
|
||||
"User-Agent": USER_AGENT_M[0],
|
||||
"Referer": "https://www.lofter.com/",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
}
|
||||
self.title = ""
|
||||
self.cover_url = None
|
||||
self.video_url = None
|
||||
self.image_list = []
|
||||
self.author = None
|
||||
self._parse()
|
||||
|
||||
def _parse(self):
|
||||
if not self.real_url:
|
||||
return
|
||||
html = self.fetch_html_content()
|
||||
if not html:
|
||||
logger.warning("Failed to fetch HTML content for LOFTER URL: %s", self.real_url)
|
||||
return
|
||||
|
||||
try:
|
||||
parsed_data = self._extract_initialize_data(html)
|
||||
if parsed_data:
|
||||
self._parse_from_init_data(parsed_data)
|
||||
else:
|
||||
self._fallback_html_parse(html)
|
||||
except Exception as exc:
|
||||
logger.exception("Error parsing LOFTER post data: %s", exc)
|
||||
self._fallback_html_parse(html)
|
||||
|
||||
if self.title:
|
||||
self.title = re.sub(r'<[^>]+>', '', self.title).strip()
|
||||
|
||||
def _extract_initialize_data(self, html):
|
||||
"""从页面提取 window.__initialize_data__ JSON 对象"""
|
||||
idx = html.find("window.__initialize_data__")
|
||||
if idx != -1:
|
||||
start = html.find("{", idx)
|
||||
end = html.find("</script>", start)
|
||||
if start != -1 and end != -1:
|
||||
json_str = html[start:end].rstrip(";").strip()
|
||||
try:
|
||||
return json.loads(json_str)
|
||||
except json.JSONDecodeError as err:
|
||||
logger.warning("Failed to decode window.__initialize_data__ JSON: %s", err)
|
||||
|
||||
match = re.search(r'window\.__initialize_data__\s*=\s*(\{.*?\})\s*(?:;|</script>)', html, re.DOTALL)
|
||||
if match:
|
||||
try:
|
||||
return json.loads(match.group(1))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return None
|
||||
|
||||
def _parse_from_init_data(self, data):
|
||||
# 1. 兼容问答/标签讨论详情 (answerDetailData)
|
||||
if "answerDetailData" in data:
|
||||
ans = data.get("answerDetailData", {}) or {}
|
||||
blog_info = ans.get("blogInfo", {}) or {}
|
||||
author_name = blog_info.get("blogNickName") or blog_info.get("blogName")
|
||||
author_avatar = blog_info.get("bigAvaImg") or blog_info.get("avaImg")
|
||||
if author_name or author_avatar:
|
||||
self.author = {
|
||||
"name": author_name,
|
||||
"avatar": author_avatar,
|
||||
}
|
||||
self.title = ans.get("barrage") or ans.get("answer") or ""
|
||||
for img in ans.get("images", []):
|
||||
if isinstance(img, dict):
|
||||
img_url = img.get("orign") or img.get("raw")
|
||||
if img_url:
|
||||
self.image_list.append(img_url)
|
||||
if self.image_list:
|
||||
self.cover_url = self.image_list[0]
|
||||
return
|
||||
|
||||
# 2. 标准博客文章详情 (postData)
|
||||
post_bundle = data.get("postData", {}).get("data", {})
|
||||
blog_info = post_bundle.get("blogInfo", {}) or {}
|
||||
post_view = post_bundle.get("postData", {}).get("postView", {}) or {}
|
||||
|
||||
# 提取作者信息
|
||||
author_name = blog_info.get("blogNickName") or blog_info.get("blogName")
|
||||
author_avatar = blog_info.get("bigAvaImg") or blog_info.get("avaImg")
|
||||
if author_name or author_avatar:
|
||||
self.author = {
|
||||
"name": author_name,
|
||||
"avatar": author_avatar,
|
||||
}
|
||||
|
||||
# 提取标题与文案
|
||||
title = post_view.get("title") or post_view.get("digest") or ""
|
||||
caption = ""
|
||||
|
||||
# 处理视频贴 (type: 4 或包含 videoPostView)
|
||||
video_post_view = post_view.get("videoPostView")
|
||||
if video_post_view and isinstance(video_post_view, dict):
|
||||
video_info = video_post_view.get("videoInfo") or {}
|
||||
self.video_url = video_info.get("originUrl") or video_info.get("flashurl")
|
||||
self.cover_url = video_info.get("video_img_url") or video_info.get("video_first_img")
|
||||
caption = video_post_view.get("caption") or ""
|
||||
|
||||
# 处理图文贴 (type: 2 或包含 photoPostView)
|
||||
photo_post_view = post_view.get("photoPostView")
|
||||
if photo_post_view and isinstance(photo_post_view, dict):
|
||||
photo_links = photo_post_view.get("photoLinks") or []
|
||||
for item in photo_links:
|
||||
if isinstance(item, dict):
|
||||
img_url = item.get("raw") or item.get("orign")
|
||||
if img_url:
|
||||
self.image_list.append(img_url)
|
||||
|
||||
first_img = photo_post_view.get("firstImage") or post_view.get("firstImage")
|
||||
if isinstance(first_img, dict):
|
||||
self.cover_url = first_img.get("raw") or first_img.get("orign")
|
||||
elif not self.cover_url and self.image_list:
|
||||
self.cover_url = self.image_list[0]
|
||||
|
||||
if not caption:
|
||||
caption = photo_post_view.get("caption") or ""
|
||||
|
||||
# 标题整理
|
||||
if not title and caption:
|
||||
clean_caption = re.sub(r'<[^>]+>', '', caption).strip()
|
||||
title = clean_caption
|
||||
|
||||
self.title = title.strip()
|
||||
|
||||
def _fallback_html_parse(self, html):
|
||||
"""备用 HTML 解析(针对老版或未渲染出初始化数据的页面)"""
|
||||
soup = BeautifulSoup(html, "lxml")
|
||||
|
||||
# 标题
|
||||
og_title = soup.find("meta", property="og:title")
|
||||
if og_title and og_title.get("content"):
|
||||
self.title = og_title["content"].strip()
|
||||
elif soup.title and soup.title.string:
|
||||
self.title = soup.title.string.strip()
|
||||
|
||||
# 封面
|
||||
og_image = soup.find("meta", property="og:image")
|
||||
if og_image and og_image.get("content"):
|
||||
self.cover_url = og_image["content"].strip()
|
||||
|
||||
# 视频
|
||||
video_tag = soup.find("video")
|
||||
if video_tag and video_tag.get("src"):
|
||||
self.video_url = video_tag["src"].strip()
|
||||
|
||||
# 图集
|
||||
if not self.image_list:
|
||||
for img in soup.find_all("img"):
|
||||
src = img.get("src") or img.get("data-src")
|
||||
if src and "lf127.net" in src and "ava" not in src:
|
||||
self.image_list.append(src)
|
||||
|
||||
if not self.cover_url and self.image_list:
|
||||
self.cover_url = self.image_list[0]
|
||||
|
||||
def get_real_video_url(self):
|
||||
return self.video_url
|
||||
|
||||
def get_title_content(self):
|
||||
return self.title or ""
|
||||
|
||||
def get_cover_photo_url(self):
|
||||
return self.cover_url
|
||||
|
||||
def get_author_info(self):
|
||||
return self.author
|
||||
|
||||
def get_image_list(self):
|
||||
return self.image_list
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"schema_version": 2,
|
||||
"description": "45 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)",
|
||||
"total_cases": 297,
|
||||
"total_cases": 301,
|
||||
"cases": [
|
||||
{
|
||||
"platform": "网易云音乐",
|
||||
@@ -4318,6 +4318,66 @@
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09 用户补充: B站 b23.tv/BV 短链"
|
||||
},
|
||||
{
|
||||
"platform": "得物",
|
||||
"url": "https://m.dewu.com/rn-activity/community-share?trendId=514124663&shareId=8xEKQBm",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: 得物无水印 720p 视频动态",
|
||||
"pattern": "UGC 潮流短视频"
|
||||
},
|
||||
{
|
||||
"platform": "得物",
|
||||
"url": "https://m.dewu.com/rn-activity/community-share?trendId=452256836&shareId=KadBoK4",
|
||||
"media_types": [
|
||||
"images"
|
||||
],
|
||||
"expected_fields": [
|
||||
"images",
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: 得物高清多图穿搭动态",
|
||||
"pattern": "UGC 穿搭图集"
|
||||
},
|
||||
{
|
||||
"platform": "网易LOFTER",
|
||||
"url": "https://emm3716958.lofter.com/post/74daebd2_34ec49f1a?incantation=rzyp3CMgofo9",
|
||||
"media_types": [
|
||||
"images"
|
||||
],
|
||||
"expected_fields": [
|
||||
"images",
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: LOFTER 二次元插画无损大图原图",
|
||||
"pattern": "二次元摄影与插画图集"
|
||||
},
|
||||
{
|
||||
"platform": "网易LOFTER",
|
||||
"url": "https://jingjiexian261.lofter.com/post/1e52d8b0_2b67492c5?incantation=rzQpNTwdCDD7",
|
||||
"media_types": [
|
||||
"video"
|
||||
],
|
||||
"expected_fields": [
|
||||
"video",
|
||||
"cover",
|
||||
"title",
|
||||
"author"
|
||||
],
|
||||
"note": "2026-09-06 已验证: LOFTER 创作者短视频",
|
||||
"pattern": "同人短视频"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
import json
|
||||
import unittest
|
||||
from unittest.mock import patch, Mock
|
||||
|
||||
from src.parsers.dewu_parser import DewuParser
|
||||
|
||||
|
||||
class DewuParserTest(unittest.TestCase):
|
||||
def test_parses_video_post_successfully(self):
|
||||
fake_html = """
|
||||
<html>
|
||||
<head>
|
||||
<script>
|
||||
{"props":{"pageProps":{"metaOGInfo":{"data":[{
|
||||
"content":{
|
||||
"contentId": 514124663,
|
||||
"title": "复古运动感跑鞋",
|
||||
"content": "好鞋分享灰绿渐变跑鞋",
|
||||
"cover": {"url": "https://image-cdn.dewu.com/cover.jpg"},
|
||||
"media": {
|
||||
"list": [
|
||||
{"mediaType": "video", "url": "https://videocdn.poizon.com/video_clean.mp4"},
|
||||
{"mediaType": "img", "url": "https://image-cdn.poizon.com/pic1.jpg"}
|
||||
]
|
||||
},
|
||||
"videoShareUrl": "https://videocdn.poizon.com/video_watermark.mp4"
|
||||
},
|
||||
"userInfo": {
|
||||
"userName": "888hdx",
|
||||
"icon": "https://image-cdn.poizon.com/avatar.jpg"
|
||||
}
|
||||
}]}}}}
|
||||
</script>
|
||||
</head>
|
||||
</html>
|
||||
"""
|
||||
with patch.object(DewuParser, "fetch_html_content", return_value=fake_html):
|
||||
parser = DewuParser("https://m.dewu.com/rn-activity/community-share?trendId=514124663")
|
||||
self.assertEqual(parser.get_title_content(), "复古运动感跑鞋")
|
||||
self.assertEqual(parser.get_real_video_url(), "https://videocdn.poizon.com/video_clean.mp4")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://image-cdn.dewu.com/cover.jpg")
|
||||
self.assertEqual(parser.get_author_info(), {"name": "888hdx", "avatar": "https://image-cdn.poizon.com/avatar.jpg"})
|
||||
self.assertEqual(parser.get_image_list(), ["https://image-cdn.poizon.com/pic1.jpg"])
|
||||
|
||||
def test_parses_image_post_successfully(self):
|
||||
fake_html = """
|
||||
<html>
|
||||
<head>
|
||||
<script>
|
||||
{"props":{"pageProps":{"metaOGInfo":{"data":[{
|
||||
"content":{
|
||||
"contentId": 452256836,
|
||||
"title": "",
|
||||
"content": "新年第一条项链",
|
||||
"cover": {"url": "https://image-cdn.poizon.com/cover.jpg"},
|
||||
"media": {
|
||||
"list": [
|
||||
{"mediaType": "img", "url": "https://image-cdn.poizon.com/img1.jpg"},
|
||||
{"mediaType": "img", "url": "https://image-cdn.poizon.com/img2.jpg"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"userInfo": {
|
||||
"userName": "YUTONG",
|
||||
"icon": "https://image-cdn.poizon.com/ava.jpg"
|
||||
}
|
||||
}]}}}}
|
||||
</script>
|
||||
</head>
|
||||
</html>
|
||||
"""
|
||||
with patch.object(DewuParser, "fetch_html_content", return_value=fake_html):
|
||||
parser = DewuParser("https://m.dewu.com/rn-activity/community-share?trendId=452256836")
|
||||
self.assertEqual(parser.get_title_content(), "新年第一条项链")
|
||||
self.assertIsNone(parser.get_real_video_url())
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://image-cdn.poizon.com/cover.jpg")
|
||||
self.assertEqual(len(parser.get_image_list()), 2)
|
||||
self.assertEqual(parser.get_image_list()[0], "https://image-cdn.poizon.com/img1.jpg")
|
||||
|
||||
def test_handles_empty_or_broken_html(self):
|
||||
with patch.object(DewuParser, "fetch_html_content", return_value="<html><body>404 Not Found</body></html>"):
|
||||
parser = DewuParser("https://m.dewu.com/rn-activity/community-share?trendId=1")
|
||||
self.assertEqual(parser.get_title_content(), "")
|
||||
self.assertIsNone(parser.get_real_video_url())
|
||||
self.assertEqual(parser.get_image_list(), [])
|
||||
self.assertIsNone(parser.get_author_info())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,107 @@
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from src.parsers.lofter_parser import LofterParser
|
||||
|
||||
|
||||
class LofterParserTest(unittest.TestCase):
|
||||
def test_parses_photo_post_successfully(self):
|
||||
fake_html = """
|
||||
<html>
|
||||
<head>
|
||||
<script>
|
||||
window.__initialize_data__ = {
|
||||
"postData": {
|
||||
"data": {
|
||||
"blogInfo": {
|
||||
"blogNickName": "画师小张",
|
||||
"bigAvaImg": "https://avaimg.lf127.net/ava.jpg"
|
||||
},
|
||||
"postData": {
|
||||
"postView": {
|
||||
"title": "原创新画",
|
||||
"type": 2,
|
||||
"photoPostView": {
|
||||
"photoLinks": [
|
||||
{"raw": "https://imglf3.lf127.net/raw1.jpg", "orign": "https://imglf3.lf127.net/orig1.jpg"},
|
||||
{"raw": "https://imglf3.lf127.net/raw2.jpg", "orign": "https://imglf3.lf127.net/orig2.jpg"}
|
||||
],
|
||||
"firstImage": {"raw": "https://imglf3.lf127.net/raw1.jpg"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
</script>
|
||||
</head>
|
||||
</html>
|
||||
"""
|
||||
with patch.object(LofterParser, "fetch_html_content", return_value=fake_html):
|
||||
parser = LofterParser("https://author.lofter.com/post/abc_123")
|
||||
self.assertEqual(parser.get_title_content(), "原创新画")
|
||||
self.assertIsNone(parser.get_real_video_url())
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://imglf3.lf127.net/raw1.jpg")
|
||||
self.assertEqual(parser.get_author_info(), {"name": "画师小张", "avatar": "https://avaimg.lf127.net/ava.jpg"})
|
||||
self.assertEqual(parser.get_image_list(), ["https://imglf3.lf127.net/raw1.jpg", "https://imglf3.lf127.net/raw2.jpg"])
|
||||
|
||||
def test_parses_video_post_successfully(self):
|
||||
fake_html = """
|
||||
<html>
|
||||
<head>
|
||||
<script>
|
||||
window.__initialize_data__ = {
|
||||
"postData": {
|
||||
"data": {
|
||||
"blogInfo": {
|
||||
"blogNickName": "剪辑君",
|
||||
"bigAvaImg": "https://avaimg.lf127.net/ava2.jpg"
|
||||
},
|
||||
"postData": {
|
||||
"postView": {
|
||||
"title": "同人短片",
|
||||
"type": 4,
|
||||
"videoPostView": {
|
||||
"videoInfo": {
|
||||
"originUrl": "https://vod.126.net/video.mp4",
|
||||
"video_img_url": "https://imglf4.lf127.net/cover.jpg"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
</script>
|
||||
</head>
|
||||
</html>
|
||||
"""
|
||||
with patch.object(LofterParser, "fetch_html_content", return_value=fake_html):
|
||||
parser = LofterParser("https://video.lofter.com/post/def_456")
|
||||
self.assertEqual(parser.get_title_content(), "同人短片")
|
||||
self.assertEqual(parser.get_real_video_url(), "https://vod.126.net/video.mp4")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://imglf4.lf127.net/cover.jpg")
|
||||
self.assertEqual(parser.get_author_info(), {"name": "剪辑君", "avatar": "https://avaimg.lf127.net/ava2.jpg"})
|
||||
self.assertEqual(parser.get_image_list(), [])
|
||||
|
||||
def test_fallback_html_parse(self):
|
||||
fake_html = """
|
||||
<html>
|
||||
<head>
|
||||
<title>老页面标题 - LOFTER</title>
|
||||
<meta property="og:image" content="https://imglf3.lf127.net/fallback_cover.jpg">
|
||||
</head>
|
||||
<body>
|
||||
<video src="https://vod.126.net/fallback.mp4"></video>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
with patch.object(LofterParser, "fetch_html_content", return_value=fake_html):
|
||||
parser = LofterParser("https://author.lofter.com/post/old_789")
|
||||
self.assertEqual(parser.get_title_content(), "老页面标题 - LOFTER")
|
||||
self.assertEqual(parser.get_real_video_url(), "https://vod.126.net/fallback.mp4")
|
||||
self.assertEqual(parser.get_cover_photo_url(), "https://imglf3.lf127.net/fallback_cover.jpg")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -247,6 +247,27 @@ class UrlParserTest(unittest.TestCase):
|
||||
with self.subTest(domain=domain):
|
||||
self.assertEqual(UrlParser.get_platform(domain), "拼多多")
|
||||
|
||||
def test_recognizes_dewu_domains(self):
|
||||
for domain in [
|
||||
"https://dw4.co/t/A/HmV2eJqiU",
|
||||
"https://dewu.com/explore",
|
||||
"https://m.dewu.com/rn-activity/community-share?trendId=123",
|
||||
"https://poizon.com/post/456",
|
||||
"https://sub.dewu.com/page",
|
||||
]:
|
||||
with self.subTest(domain=domain):
|
||||
self.assertEqual(UrlParser.get_platform(domain), "得物")
|
||||
|
||||
def test_recognizes_lofter_domains(self):
|
||||
for domain in [
|
||||
"https://lofter.com/post/123",
|
||||
"https://www.lofter.com/front/detail",
|
||||
"https://emm3716958.lofter.com/post/74daebd2_34ec49f1a",
|
||||
"https://random-author.lofter.com/post/abc_def",
|
||||
]:
|
||||
with self.subTest(domain=domain):
|
||||
self.assertEqual(UrlParser.get_platform(domain), "网易LOFTER")
|
||||
|
||||
|
||||
class WebFetcherTest(unittest.TestCase):
|
||||
@staticmethod
|
||||
@@ -341,6 +362,18 @@ class WebFetcherTest(unittest.TestCase):
|
||||
"&sruserid=2369252937&kgsscty1=link",
|
||||
)
|
||||
|
||||
def test_follows_dewu_short_link(self):
|
||||
redirect_url = (
|
||||
"https://m.dewu.com/rn-activity/community-share"
|
||||
"?trendId=514124663&shareId=8xEKQBm&noise=x"
|
||||
)
|
||||
with patch("utils.web_fetcher.requests.get", return_value=self.response(redirect_url)):
|
||||
result = WebFetcher.fetch_redirect_url("https://dw4.co/t/A/HmV2eJqiU")
|
||||
self.assertEqual(
|
||||
result,
|
||||
"https://m.dewu.com/rn-activity/community-share?trendId=514124663&shareId=8xEKQBm",
|
||||
)
|
||||
|
||||
def test_stops_before_login_or_verification_page(self):
|
||||
for blocked_path in ("/login", "/404", "/captcha", "/verify", "/error"):
|
||||
with self.subTest(blocked_path=blocked_path):
|
||||
|
||||
+29
-2
@@ -22,8 +22,8 @@ class WebFetcher:
|
||||
# 绿洲分享页会对通用桌面请求头返回 500,由解析器使用移动端请求头抓取。
|
||||
# 微博博文可通过 API 直接根据 ID 解析,直接访问网页端常触发访客系统重定向。
|
||||
domain = UrlParser.get_domain(url)
|
||||
if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan"}:
|
||||
if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI"}:
|
||||
if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan", "dw4.co"}:
|
||||
if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI", "网易LOFTER", "得物"}:
|
||||
return UrlParser.extract_video_address(url)
|
||||
try:
|
||||
current_url = url
|
||||
@@ -99,6 +99,12 @@ class UrlParser:
|
||||
if domain.endswith('.m.chenzhongtech.com'):
|
||||
return '快手'
|
||||
|
||||
if domain == 'lofter.com' or domain.endswith('.lofter.com'):
|
||||
return '网易LOFTER'
|
||||
|
||||
if domain.endswith('.dewu.com') or domain.endswith('.poizon.com'):
|
||||
return '得物'
|
||||
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
@@ -341,6 +347,24 @@ class UrlParser:
|
||||
elif platform in ("红果短剧", "番茄小说", "红果漫剧", "小云雀AI"):
|
||||
if parsed_url.query:
|
||||
address = f"{address}?{parsed_url.query}"
|
||||
elif platform == "得物":
|
||||
query_params = parse_qs(parsed_url.query)
|
||||
preserved_params = []
|
||||
for key in ("trendId", "shareId", "source", "shareType", "shareChannel", "isScreenShot"):
|
||||
value = query_params.get(key, [None])[0]
|
||||
if value is not None:
|
||||
preserved_params.append((key, value))
|
||||
if preserved_params:
|
||||
address = f"{address}?{urlencode(preserved_params)}"
|
||||
elif platform == "网易LOFTER":
|
||||
query_params = parse_qs(parsed_url.query)
|
||||
preserved_params = []
|
||||
for key in ("incantation", "permalink", "postId", "blogId"):
|
||||
value = query_params.get(key, [None])[0]
|
||||
if value is not None:
|
||||
preserved_params.append((key, value))
|
||||
if preserved_params:
|
||||
address = f"{address}?{urlencode(preserved_params)}"
|
||||
return address
|
||||
|
||||
@staticmethod
|
||||
@@ -352,6 +376,9 @@ class UrlParser:
|
||||
match_vid = re.search(r'(?:["%22]|%22)?(?:vid|video_id|content_id|material_id)(?:["%22]|%22)?\s*(?:[:=]|%3A)\s*(?:["%22]|%22)?(\d{15,22})', url)
|
||||
if match_vid:
|
||||
return match_vid.group(1)
|
||||
params_trend_id = query_params.get('trendId', [None])[0]
|
||||
if params_trend_id:
|
||||
return params_trend_id
|
||||
params_share_id = query_params.get('shareId', [None])[0]
|
||||
if params_share_id:
|
||||
return params_share_id
|
||||
|
||||
Reference in New Issue
Block a user