diff --git a/README.md b/README.md index 11e6b51..d1479a8 100644 --- a/README.md +++ b/README.md @@ -3,15 +3,15 @@ **基于 Python 的多平台媒体原生本地解析系统** -[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) [![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/) [![Flask](https://img.shields.io/badge/Framework-Flask-000000?logo=flask&logoColor=white)](https://flask.palletsprojects.com/) [![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?logo=docker&logoColor=white)](#-部署指南) [![Support](https://img.shields.io/badge/support-45%20Platforms-brightgreen.svg)](#-支持的平台矩阵) +[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) [![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/) [![Flask](https://img.shields.io/badge/Framework-Flask-000000?logo=flask&logoColor=white)](https://flask.palletsprojects.com/) [![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?logo=docker&logoColor=white)](#-部署指南) [![Support](https://img.shields.io/badge/support-47%20Platforms-brightgreen.svg)](#-支持的平台矩阵)

解析逻辑 • 支持平台 • 部署指南 • -项目结构 • -开发文档 • 接口文档 • +测试自检 • +开发文档 • 联系作者

@@ -42,55 +42,36 @@ ## 💾 支持的平台矩阵 -| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 实况 | -| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | -| **抖音** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | -| **小红书** | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | -| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | | -| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | -| **快手** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | -| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | | -| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | | -| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | | -| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **闲鱼** | | ✓ | ✓ | | ✓ | | | | -| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | | -| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | | -| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | -| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | | -| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | | -| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | -| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | | -| **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | -| **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | | | -| **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | | | -| **微视** | ✓ | ✓ | ✓ | ✓ | | | | | -| **AcFun** | ✓ | ✓ | ✓ | ✓ | | | | | -| **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | | | -| **今日头条** | ✓ | ✓ | ✓ | ✓ | | | | | -| **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | | | -| **全民K歌** | | ✓ | ✓ | ✓ | | | | | -| **新片场** | ✓ | ✓ | ✓ | ✓ | | | | | -| **好看视频** | ✓ | ✓ | ✓ | ✓ | | | | | -| **梨视频** | ✓ | ✓ | ✓ | ✓ | | | | | -| **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | | | -| **虎牙** | | ✓ | ✓ | ✓ | | | | | -| **美拍** | ✓ | ✓ | ✓ | ✓ | | | | | -| **最右** | ✓ | ✓ | | ✓ | | | | | -| **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | | | -| **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | | | -| **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | | | +| 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | 平台名称 | 作者 | 标题 | 封面 | 视频 | 图集 | 音频 | 字幕 | +| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | +| **抖音** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **剪映 / CapCut** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | +| **小红书** | ✓ | ✓ | ✓ | ✓ | **✓+** | | | **快影** | ✓ | ✓ | ✓ | ✓ | | ✓ | | +| **视频号** | ✓ | ✓ | ✓ | ✓ | | | | **皮皮搞笑** | ✓ | ✓ | ✓ | ✓ | | | | +| **微信公众号** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **微视** | ✓ | ✓ | ✓ | ✓ | | | | +| **快手** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | | **AcFun** | ✓ | ✓ | ✓ | ✓ | | | | +| **哔哩哔哩** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **西瓜视频** | ✓ | ✓ | ✓ | ✓ | | | | +| **豆包** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **今日头条** | ✓ | ✓ | ✓ | ✓ | | | | +| **即梦AI** | ✓ | ✓ | ✓ | ✓ | | | | **绿洲** | ✓ | ✓ | ✓ | ✓ | ✓ | | | +| **小云雀AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **皮皮虾** | ✓ | ✓ | ✓ | ✓ | | | | +| **可灵AI** | ✓ | ✓ | ✓ | ✓ | | | | **全民K歌** | | ✓ | ✓ | ✓ | | | | +| **海螺AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **新片场** | ✓ | ✓ | ✓ | ✓ | | | | +| **夸克AI** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **好看视频** | ✓ | ✓ | ✓ | ✓ | | | | +| **通义千问** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **梨视频** | ✓ | ✓ | ✓ | ✓ | | | | +| **腾讯元宝** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **微博** | ✓ | ✓ | ✓ | ✓ | ✓ | | | +| **闲鱼** | | ✓ | ✓ | | ✓ | | | **知乎** | ✓ | ✓ | ✓ | ✓ | ✓ | | | +| **拼多多** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **虎牙** | | ✓ | ✓ | ✓ | | | | +| **Soul** | ✓ | ✓ | ✓ | ✓ | ✓ | | | **美拍** | ✓ | ✓ | ✓ | ✓ | | | | +| **汽水音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | **最右** | ✓ | ✓ | | ✓ | | | | +| **QQ音乐** | ✓ | ✓ | ✓ | ✓ | | | | **番茄小说** | ✓ | ✓ | ✓ | ✓ | | | | +| **网易云音乐** | ✓ | ✓ | ✓ | ✓ | **✓+** | ✓ | ✓ | **红果短剧** | ✓ | ✓ | ✓ | ✓ | | | | +| **酷狗音乐** | ✓ | ✓ | ✓ | ✓ | | ✓ | | **红果漫剧** | ✓ | ✓ | ✓ | ✓ | | | | +| **配音秀** | ✓ | ✓ | ✓ | ✓ | | | | **得物** | ✓ | ✓ | ✓ | ✓ | ✓ | | | +| **松果时刻** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | **网易LOFTER** | ✓ | ✓ | ✓ | ✓ | ✓ | | | +| **腾讯频道** | ✓ | ✓ | ✓ | ✓ | | | | | | | | | | | | -*注:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。 +> 💡 **说明**: +> * **图集 `✓+`**:表示该平台支持提取 iOS/Android 原始 **LivePhoto 动态实况图**(含封面静图与关联 `.mov`/`.mp4` 实况视频原件)。 +> * **原生水印提示**:腾讯元宝等极少数平台提取的是官方存储桶原画质直链,素材保留官方原生水印。 --- @@ -147,71 +128,6 @@ pip install -r requirements.txt python app.py ``` -## 📂 项目结构 - -```text -media-parser/ -├── app.py # 应用入口 (Flask Web 服务与 API 启动) -├── configs/ # 核心配置 (域名平台映射、日志配置等) -├── docs/ # 逆向百科与开发文档 -│ ├── architecture.md # 系统分层架构与生命周期设计 -│ ├── reverse-guide.md # 通用逆向方法论 (抓包/SSR/JS签名提取) -│ ├── testing.md # 完整测试规范与回归手册 -│ └── parsers/ # 42 份逆向分析文档,覆盖 45 个平台 -├── src/ # 核心业务逻辑 -│ ├── api/ # RESTful API 路由 (/api/parse, /api/health) -│ ├── web/ # Demo 体验页与交互蓝图 -│ ├── parsers/ # 42 个解析器模块,支持 45 个平台 (核心解析逻辑) -│ └── parser_factory.py # 工厂分发器 (解析器动态发现与自动注册) -├── utils/ # 底层工具库与逆向支持 -│ ├── signer/ # JS 签名沙箱引擎 (a_bogus, x_bogus 等算法执行) -│ └── web_fetcher.py # 智能 URL 识别、重定向追踪与请求封装 -├── tests/ # 完备的双层测试体系 -│ ├── live_parser_samples.json # 45 平台真实多形态在线样本库 -│ ├── manual_verify_parsers.py # 命令行交互式冒烟与健康检查工具 -│ └── test_*_parser.py # 各平台 Mock 自动化单元测试 -├── static/ & templates/ # Web 演示页面前端静态资源 -└── docker-compose.yml # 容器化一键部署编排 -``` - -## 📖 开发者文档与逆向百科 - -本项目提供了详尽的技术架构与全平台逆向分析手册,详细内容请查阅 **[`docs/`](docs/)** 目录: - -* 🏗️ **[系统架构与生命周期设计](docs/architecture.md)**:分层设计、302 追踪与 `ParserFactory` 动态发现机制。 -* 🔍 **[通用逆向方法论](docs/reverse-guide.md)**:SSR 数据提取、H5 接口伪装、JS 签名沙箱及抓包 SOP。 -* 🧪 **[测试体系与回归验证](docs/testing.md)**:Pytest 单元测试、Mock 与线上真实样例回归。 -* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 45 个平台的技术文档。 - ---- - -## 🧪 解析有效性测试 - -本项目拥有完备的测试体系,包括基础单元测试与基于真实样本库的在线回归测试: - -### 1. 真实样本交互式验证 -样例测试库见 [`tests/live_parser_samples.json`](tests/live_parser_samples.json),用于实时验证 45 个平台的解析有效性与多形态覆盖: - -```bash -# 快速冒烟测试(每个平台测 1 条最具代表性的链接,极速完成健康检查) -python3 tests/manual_verify_parsers.py --limit 1 - -# 全量回归验证(覆盖多形态真实用例) -python3 tests/manual_verify_parsers.py - -# 仅验证单个或指定平台(如:小云雀AI) -python3 tests/manual_verify_parsers.py --platform "小云雀AI" -``` - -### 2. 自动化单元测试 -```bash -# 运行全部单元测试 -pytest - -# 运行真实线上样例自动化回归 -pytest tests/test_live_parser_samples.py -s -``` - --- ## 🔌 API 核心接口说明 @@ -292,6 +208,33 @@ pytest tests/test_live_parser_samples.py -s --- +## 🧪 自动化测试与健康自检 + +本项目拥有完备的双层测试体系(Mock 单元测试 + 47 平台真实在线样本库回归)。遇到解析异常或日常部署验证时,可一键运行健康自检: + +```bash +# 47 平台极速冒烟测试(每个平台测 1 条最具代表性的链接,秒级完成健康检查) +python3 tests/manual_verify_parsers.py --limit 1 + +# 仅验证单个或指定平台(如:小云雀AI / 抖音) +python3 tests/manual_verify_parsers.py --platform "小云雀AI" +``` + +> 💡 更多多形态全量回归测试命令与 Pytest 自动化规范,请参阅 **[测试体系与回归验证指南 (docs/testing.md)](docs/testing.md)**。 + +--- + +## 📖 开发者文档与逆向百科 + +本项目提供了详尽的技术架构与全平台逆向分析手册,详细内容请查阅 **[`docs/`](docs/)** 目录: + +* 🏗️ **[系统架构与生命周期设计](docs/architecture.md)**:分层设计、302 追踪与 `ParserFactory` 动态发现机制。 +* 🔍 **[通用逆向方法论](docs/reverse-guide.md)**:SSR 数据提取、H5 接口伪装、JS 签名沙箱及抓包 SOP。 +* 🧪 **[测试体系与回归验证](docs/testing.md)**:Pytest 单元测试、Mock 与线上真实样例回归。 +* 📚 **[平台逆向指南索引](docs/index.md)**:包含抖音、快手、小红书、B站等全部 47 个平台的技术文档。 + +--- + ## 📩 联系作者 如果您在安装、使用过程中遇到问题,或有定制需求,请通过以下方式联系: diff --git a/configs/business_config.json b/configs/business_config.json index a7007f4..f2595f2 100755 --- a/configs/business_config.json +++ b/configs/business_config.json @@ -134,7 +134,15 @@ "yangkeduo.com": "拼多多", "pinduoduo.com": "拼多多", "www.pinduoduo.com": "拼多多", - "mobile.pinduoduo.com": "拼多多" + "mobile.pinduoduo.com": "拼多多", + "dw4.co": "得物", + "dewu.com": "得物", + "www.dewu.com": "得物", + "m.dewu.com": "得物", + "poizon.com": "得物", + "www.poizon.com": "得物", + "lofter.com": "网易LOFTER", + "www.lofter.com": "网易LOFTER" }, "USER_AGENT_PC": [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36", diff --git a/docs/architecture.md b/docs/architecture.md index 95c35fd..a11cde5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -54,6 +54,33 @@ flowchart TD ParseAPI --> CommonUtils ``` +### 📂 源码目录组织与模块映射 + +```text +media-parser/ +├── app.py # 应用入口 (Flask Web 服务与 API 启动) +├── configs/ # 核心配置 (域名平台映射、日志配置等) +├── docs/ # 逆向百科与开发文档体系 +│ ├── architecture.md # 系统分层架构与生命周期设计 +│ ├── reverse-guide.md # 通用逆向方法论 (抓包/SSR/JS签名提取) +│ ├── testing.md # 完整测试规范与回归手册 +│ └── parsers/ # 平台逆向分析手册 +├── src/ # 核心业务逻辑 +│ ├── api/ # RESTful API 路由 (/api/parse, /api/health) +│ ├── web/ # Demo 体验页与交互蓝图 +│ ├── parsers/ # 各平台解析器模块 (核心解析逻辑) +│ └── parser_factory.py # 工厂分发器 (解析器动态发现与自动注册) +├── utils/ # 底层工具库与逆向支持 +│ ├── signer/ # JS 签名沙箱引擎 (a_bogus 等算法执行) +│ └── web_fetcher.py # 智能 URL 识别、重定向追踪与请求封装 +├── tests/ # 完备的双层测试体系 +│ ├── live_parser_samples.json # 平台真实多形态在线样本库 +│ ├── manual_verify_parsers.py # 命令行交互式冒烟与健康检查工具 +│ └── test_*_parser.py # 各平台 Mock 自动化单元测试 +├── static/ & templates/ # Web 演示页面前端静态资源 +└── docker-compose.yml # 容器化一键部署编排 +``` + --- ## 🔄 请求处理生命周期 (Lifecycle) diff --git a/docs/index.md b/docs/index.md index 54f968d..b831179 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,7 +2,7 @@ 欢迎查阅 **Media Parser** 开发者文档与逆向工程知识库。 -本项目是一个高性能、模块化、支持 **45 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。 +本项目是一个高性能、模块化、支持 **47 个主流媒体与 AI 内容平台** 的无水印音视频、图文及 LivePhoto 结构化解析引擎。 --- @@ -13,7 +13,7 @@ * 🧪 **[测试与回归验证 (Testing Guide)](testing.md)**:学习 Pytest 单元测试、Mock 构造与真实样本(Live Samples)测试。 * 📖 **平台实战指南 (Parser Guides)**: * **短视频与轻社区**:[抖音](parsers/douyin.md) | [快手](parsers/kuaishou.md) | [今日头条](parsers/xigua.md) | [皮皮虾](parsers/pipixia.md) | [皮皮搞笑](parsers/pipigaoxiao.md) | [最右](parsers/zuiyou.md) | [美拍](parsers/meipai.md) | [微视](parsers/weishi.md) | [绿洲](parsers/lvzhou.md) | [番茄小说/红果短剧/红果漫剧](parsers/fanqie.md) - * **图文与综合社区**:[小红书](parsers/xiaohongshu.md) | [微信公众号](parsers/wechat-mp.md) | [微博](parsers/weibo.md) | [知乎](parsers/zhihu.md) | [闲鱼](parsers/xianyu.md) | [Soul](parsers/soul.md) + * **图文与综合社区**:[小红书](parsers/xiaohongshu.md) | [微信公众号](parsers/wechat-mp.md) | [微博](parsers/weibo.md) | [知乎](parsers/zhihu.md) | [闲鱼](parsers/xianyu.md) | [Soul](parsers/soul.md) | [得物](parsers/dewu.md) | [网易LOFTER](parsers/lofter.md) * **长视频与弹幕/创作**:[哔哩哔哩](parsers/bilibili.md) | [AcFun](parsers/acfun.md) | [新片场](parsers/xinpianchang.md) | [好看视频](parsers/haokan.md) | [西瓜视频](parsers/xigua.md) | [剪映](parsers/jianying.md) | [快影](parsers/kwaiying.md) * **AI 生成与大模型**:[豆包 AI](parsers/doubao.md) | [腾讯元宝](parsers/yuanbao.md) | [即梦 AI](parsers/jimeng.md) | [可灵 AI](parsers/kling.md) | [海螺 AI](parsers/hailuo.md) | [通义千问](parsers/qianwen.md) | [夸克 AI](parsers/quark-ai.md) | [小云雀 AI](parsers/xiaoyunque.md) | [松果时刻](parsers/pinecone-moment.md) * **音频与垂直频道**:[汽水音乐](parsers/qsmusic.md) | [QQ音乐](parsers/qqmusic.md) | [网易云音乐](parsers/netease-music.md) | [酷狗音乐](parsers/kugou-music.md) | [配音秀](parsers/peiyinxiu.md) | [全民K歌](parsers/quanminkge.md) | [虎牙](parsers/huya.md) | [梨视频](parsers/lishipin.md) | [微信视频号](parsers/wechat-channels.md) | [腾讯频道](parsers/tencent-channel.md) @@ -74,6 +74,8 @@ | 43 | **酷狗音乐** | MV / 免费歌曲 / 多档 MP4 | ✅ 支持 | 🟢 公开内容免配置 | H5 公开接口签名 + 分享页数据 | [查看指南](parsers/kugou-music.md) | | 44 | **配音秀** | 配音作品视频 (MP4) | ✅ 支持 | 🟢 公开内容免配置 | 作品页初始化数据提取 | [查看指南](parsers/peiyinxiu.md) | | 45 | **松果时刻** | AI 故事多页视频 / 图集 / 配音 | ✅ 支持 | 🟢 公开内容免配置 | 公开详情接口 | [查看指南](parsers/pinecone-moment.md) | +| 46 | **得物** | 潮流穿搭 / 开箱短视频 / 高清图集 | ✅ 支持 | 🟢 免配置 | Next.js metaOGInfo 数据提取 | [查看指南](parsers/dewu.md) | +| 47 | **网易LOFTER** | 二次元插画 / 摄影图集 / 创作者短视频 | ✅ 支持 | 🟢 免配置 | window.__initialize_data__ 数据提取 | [查看指南](parsers/lofter.md) | --- diff --git a/docs/parsers/dewu.md b/docs/parsers/dewu.md new file mode 100644 index 0000000..cff237e --- /dev/null +++ b/docs/parsers/dewu.md @@ -0,0 +1,36 @@ +# 得物 (Dewu) 逆向解析指南 + +本篇详细记录潮流网购与 UGC 社区 **得物 (Poizon)** 的短视频与图集解析方案。 + +--- + +## 1. 平台特征与支持能力 + +* **平台标识**:`得物` +* **支持媒体类型**:无水印 720P 原画视频 (MP4) / 社区穿搭高清图集 / 封面 / 标题文案 / 作者信息 +* **常见链接形态**: + * 官方分享短链:`https://dw4.co/t/A/HmfgCvBiU` + * 社区长链:`https://m.dewu.com/rn-activity/community-share?trendId=514124663&shareId=...` + * 网页端/PC端:`https://dewu.com/...`, `https://poizon.com/...` +* **Cookie 依赖**:🟢 免配置,无需任何 Cookie 登录态。 + +--- + +## 2. 核心逆向流程 + +1. **短链展开**: + * 用户通过 App 分享通常获得形如 `https://dw4.co/t/A/xxxx` 的短链。 + * 系统通过 `WebFetcher.fetch_redirect_url` 跟踪 HTTP 302 重定向至 `m.dewu.com/rn-activity/community-share?trendId=...`。 +2. **Next.js SSR 数据提取**: + * 得物移动端分享页基于 Next.js 渲染,页面源码内置 `', html, re.DOTALL) + if match: + return match.group(1).strip() + return None + + def get_real_video_url(self): + return self.video_url + + def get_title_content(self): + return self.title or "" + + def get_cover_photo_url(self): + return self.cover_url + + def get_author_info(self): + return self.author + + def get_image_list(self): + return self.image_list diff --git a/src/parsers/lofter_parser.py b/src/parsers/lofter_parser.py new file mode 100644 index 0000000..76bde46 --- /dev/null +++ b/src/parsers/lofter_parser.py @@ -0,0 +1,190 @@ +import json +import re +from bs4 import BeautifulSoup +from configs.general_constants import USER_AGENT_M +from configs.logging_config import get_logger +from src.parser_factory import register_parser +from src.parsers.base_parser import BaseParser + +logger = get_logger(__name__) + + +@register_parser("网易LOFTER") +class LofterParser(BaseParser): + """网易 LOFTER (乐乎) 社区解析器,支持二次元/同人/摄影等原画图集与短视频提取。""" + + def __init__(self, real_url): + super().__init__(real_url) + self.headers = { + "User-Agent": USER_AGENT_M[0], + "Referer": "https://www.lofter.com/", + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", + } + self.title = "" + self.cover_url = None + self.video_url = None + self.image_list = [] + self.author = None + self._parse() + + def _parse(self): + if not self.real_url: + return + html = self.fetch_html_content() + if not html: + logger.warning("Failed to fetch HTML content for LOFTER URL: %s", self.real_url) + return + + try: + parsed_data = self._extract_initialize_data(html) + if parsed_data: + self._parse_from_init_data(parsed_data) + else: + self._fallback_html_parse(html) + except Exception as exc: + logger.exception("Error parsing LOFTER post data: %s", exc) + self._fallback_html_parse(html) + + if self.title: + self.title = re.sub(r'<[^>]+>', '', self.title).strip() + + def _extract_initialize_data(self, html): + """从页面提取 window.__initialize_data__ JSON 对象""" + idx = html.find("window.__initialize_data__") + if idx != -1: + start = html.find("{", idx) + end = html.find("", start) + if start != -1 and end != -1: + json_str = html[start:end].rstrip(";").strip() + try: + return json.loads(json_str) + except json.JSONDecodeError as err: + logger.warning("Failed to decode window.__initialize_data__ JSON: %s", err) + + match = re.search(r'window\.__initialize_data__\s*=\s*(\{.*?\})\s*(?:;|)', html, re.DOTALL) + if match: + try: + return json.loads(match.group(1)) + except json.JSONDecodeError: + pass + return None + + def _parse_from_init_data(self, data): + # 1. 兼容问答/标签讨论详情 (answerDetailData) + if "answerDetailData" in data: + ans = data.get("answerDetailData", {}) or {} + blog_info = ans.get("blogInfo", {}) or {} + author_name = blog_info.get("blogNickName") or blog_info.get("blogName") + author_avatar = blog_info.get("bigAvaImg") or blog_info.get("avaImg") + if author_name or author_avatar: + self.author = { + "name": author_name, + "avatar": author_avatar, + } + self.title = ans.get("barrage") or ans.get("answer") or "" + for img in ans.get("images", []): + if isinstance(img, dict): + img_url = img.get("orign") or img.get("raw") + if img_url: + self.image_list.append(img_url) + if self.image_list: + self.cover_url = self.image_list[0] + return + + # 2. 标准博客文章详情 (postData) + post_bundle = data.get("postData", {}).get("data", {}) + blog_info = post_bundle.get("blogInfo", {}) or {} + post_view = post_bundle.get("postData", {}).get("postView", {}) or {} + + # 提取作者信息 + author_name = blog_info.get("blogNickName") or blog_info.get("blogName") + author_avatar = blog_info.get("bigAvaImg") or blog_info.get("avaImg") + if author_name or author_avatar: + self.author = { + "name": author_name, + "avatar": author_avatar, + } + + # 提取标题与文案 + title = post_view.get("title") or post_view.get("digest") or "" + caption = "" + + # 处理视频贴 (type: 4 或包含 videoPostView) + video_post_view = post_view.get("videoPostView") + if video_post_view and isinstance(video_post_view, dict): + video_info = video_post_view.get("videoInfo") or {} + self.video_url = video_info.get("originUrl") or video_info.get("flashurl") + self.cover_url = video_info.get("video_img_url") or video_info.get("video_first_img") + caption = video_post_view.get("caption") or "" + + # 处理图文贴 (type: 2 或包含 photoPostView) + photo_post_view = post_view.get("photoPostView") + if photo_post_view and isinstance(photo_post_view, dict): + photo_links = photo_post_view.get("photoLinks") or [] + for item in photo_links: + if isinstance(item, dict): + img_url = item.get("raw") or item.get("orign") + if img_url: + self.image_list.append(img_url) + + first_img = photo_post_view.get("firstImage") or post_view.get("firstImage") + if isinstance(first_img, dict): + self.cover_url = first_img.get("raw") or first_img.get("orign") + elif not self.cover_url and self.image_list: + self.cover_url = self.image_list[0] + + if not caption: + caption = photo_post_view.get("caption") or "" + + # 标题整理 + if not title and caption: + clean_caption = re.sub(r'<[^>]+>', '', caption).strip() + title = clean_caption + + self.title = title.strip() + + def _fallback_html_parse(self, html): + """备用 HTML 解析(针对老版或未渲染出初始化数据的页面)""" + soup = BeautifulSoup(html, "lxml") + + # 标题 + og_title = soup.find("meta", property="og:title") + if og_title and og_title.get("content"): + self.title = og_title["content"].strip() + elif soup.title and soup.title.string: + self.title = soup.title.string.strip() + + # 封面 + og_image = soup.find("meta", property="og:image") + if og_image and og_image.get("content"): + self.cover_url = og_image["content"].strip() + + # 视频 + video_tag = soup.find("video") + if video_tag and video_tag.get("src"): + self.video_url = video_tag["src"].strip() + + # 图集 + if not self.image_list: + for img in soup.find_all("img"): + src = img.get("src") or img.get("data-src") + if src and "lf127.net" in src and "ava" not in src: + self.image_list.append(src) + + if not self.cover_url and self.image_list: + self.cover_url = self.image_list[0] + + def get_real_video_url(self): + return self.video_url + + def get_title_content(self): + return self.title or "" + + def get_cover_photo_url(self): + return self.cover_url + + def get_author_info(self): + return self.author + + def get_image_list(self): + return self.image_list diff --git a/tests/live_parser_samples.json b/tests/live_parser_samples.json index cc00bd7..eaeed0d 100644 --- a/tests/live_parser_samples.json +++ b/tests/live_parser_samples.json @@ -1,7 +1,7 @@ { "schema_version": 2, "description": "45 平台多形态全景自动化回归测试用例库 (涵盖短链/长链/图集/单视频/AI生成等)", - "total_cases": 297, + "total_cases": 301, "cases": [ { "platform": "网易云音乐", @@ -4318,6 +4318,66 @@ "author" ], "note": "2026-09 用户补充: B站 b23.tv/BV 短链" + }, + { + "platform": "得物", + "url": "https://m.dewu.com/rn-activity/community-share?trendId=514124663&shareId=8xEKQBm", + "media_types": [ + "video" + ], + "expected_fields": [ + "video", + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: 得物无水印 720p 视频动态", + "pattern": "UGC 潮流短视频" + }, + { + "platform": "得物", + "url": "https://m.dewu.com/rn-activity/community-share?trendId=452256836&shareId=KadBoK4", + "media_types": [ + "images" + ], + "expected_fields": [ + "images", + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: 得物高清多图穿搭动态", + "pattern": "UGC 穿搭图集" + }, + { + "platform": "网易LOFTER", + "url": "https://emm3716958.lofter.com/post/74daebd2_34ec49f1a?incantation=rzyp3CMgofo9", + "media_types": [ + "images" + ], + "expected_fields": [ + "images", + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: LOFTER 二次元插画无损大图原图", + "pattern": "二次元摄影与插画图集" + }, + { + "platform": "网易LOFTER", + "url": "https://jingjiexian261.lofter.com/post/1e52d8b0_2b67492c5?incantation=rzQpNTwdCDD7", + "media_types": [ + "video" + ], + "expected_fields": [ + "video", + "cover", + "title", + "author" + ], + "note": "2026-09-06 已验证: LOFTER 创作者短视频", + "pattern": "同人短视频" } ] } diff --git a/tests/test_dewu_parser.py b/tests/test_dewu_parser.py new file mode 100644 index 0000000..eec777d --- /dev/null +++ b/tests/test_dewu_parser.py @@ -0,0 +1,90 @@ +import json +import unittest +from unittest.mock import patch, Mock + +from src.parsers.dewu_parser import DewuParser + + +class DewuParserTest(unittest.TestCase): + def test_parses_video_post_successfully(self): + fake_html = """ + + + + + + """ + with patch.object(DewuParser, "fetch_html_content", return_value=fake_html): + parser = DewuParser("https://m.dewu.com/rn-activity/community-share?trendId=514124663") + self.assertEqual(parser.get_title_content(), "复古运动感跑鞋") + self.assertEqual(parser.get_real_video_url(), "https://videocdn.poizon.com/video_clean.mp4") + self.assertEqual(parser.get_cover_photo_url(), "https://image-cdn.dewu.com/cover.jpg") + self.assertEqual(parser.get_author_info(), {"name": "888hdx", "avatar": "https://image-cdn.poizon.com/avatar.jpg"}) + self.assertEqual(parser.get_image_list(), ["https://image-cdn.poizon.com/pic1.jpg"]) + + def test_parses_image_post_successfully(self): + fake_html = """ + + + + + + """ + with patch.object(DewuParser, "fetch_html_content", return_value=fake_html): + parser = DewuParser("https://m.dewu.com/rn-activity/community-share?trendId=452256836") + self.assertEqual(parser.get_title_content(), "新年第一条项链") + self.assertIsNone(parser.get_real_video_url()) + self.assertEqual(parser.get_cover_photo_url(), "https://image-cdn.poizon.com/cover.jpg") + self.assertEqual(len(parser.get_image_list()), 2) + self.assertEqual(parser.get_image_list()[0], "https://image-cdn.poizon.com/img1.jpg") + + def test_handles_empty_or_broken_html(self): + with patch.object(DewuParser, "fetch_html_content", return_value="404 Not Found"): + parser = DewuParser("https://m.dewu.com/rn-activity/community-share?trendId=1") + self.assertEqual(parser.get_title_content(), "") + self.assertIsNone(parser.get_real_video_url()) + self.assertEqual(parser.get_image_list(), []) + self.assertIsNone(parser.get_author_info()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_lofter_parser.py b/tests/test_lofter_parser.py new file mode 100644 index 0000000..1ce2438 --- /dev/null +++ b/tests/test_lofter_parser.py @@ -0,0 +1,107 @@ +import unittest +from unittest.mock import patch + +from src.parsers.lofter_parser import LofterParser + + +class LofterParserTest(unittest.TestCase): + def test_parses_photo_post_successfully(self): + fake_html = """ + + + + + + """ + with patch.object(LofterParser, "fetch_html_content", return_value=fake_html): + parser = LofterParser("https://author.lofter.com/post/abc_123") + self.assertEqual(parser.get_title_content(), "原创新画") + self.assertIsNone(parser.get_real_video_url()) + self.assertEqual(parser.get_cover_photo_url(), "https://imglf3.lf127.net/raw1.jpg") + self.assertEqual(parser.get_author_info(), {"name": "画师小张", "avatar": "https://avaimg.lf127.net/ava.jpg"}) + self.assertEqual(parser.get_image_list(), ["https://imglf3.lf127.net/raw1.jpg", "https://imglf3.lf127.net/raw2.jpg"]) + + def test_parses_video_post_successfully(self): + fake_html = """ + + + + + + """ + with patch.object(LofterParser, "fetch_html_content", return_value=fake_html): + parser = LofterParser("https://video.lofter.com/post/def_456") + self.assertEqual(parser.get_title_content(), "同人短片") + self.assertEqual(parser.get_real_video_url(), "https://vod.126.net/video.mp4") + self.assertEqual(parser.get_cover_photo_url(), "https://imglf4.lf127.net/cover.jpg") + self.assertEqual(parser.get_author_info(), {"name": "剪辑君", "avatar": "https://avaimg.lf127.net/ava2.jpg"}) + self.assertEqual(parser.get_image_list(), []) + + def test_fallback_html_parse(self): + fake_html = """ + + + 老页面标题 - LOFTER + + + + + + + """ + with patch.object(LofterParser, "fetch_html_content", return_value=fake_html): + parser = LofterParser("https://author.lofter.com/post/old_789") + self.assertEqual(parser.get_title_content(), "老页面标题 - LOFTER") + self.assertEqual(parser.get_real_video_url(), "https://vod.126.net/fallback.mp4") + self.assertEqual(parser.get_cover_photo_url(), "https://imglf3.lf127.net/fallback_cover.jpg") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_web_fetcher.py b/tests/test_web_fetcher.py index e968e77..ce1f6c4 100644 --- a/tests/test_web_fetcher.py +++ b/tests/test_web_fetcher.py @@ -247,6 +247,27 @@ class UrlParserTest(unittest.TestCase): with self.subTest(domain=domain): self.assertEqual(UrlParser.get_platform(domain), "拼多多") + def test_recognizes_dewu_domains(self): + for domain in [ + "https://dw4.co/t/A/HmV2eJqiU", + "https://dewu.com/explore", + "https://m.dewu.com/rn-activity/community-share?trendId=123", + "https://poizon.com/post/456", + "https://sub.dewu.com/page", + ]: + with self.subTest(domain=domain): + self.assertEqual(UrlParser.get_platform(domain), "得物") + + def test_recognizes_lofter_domains(self): + for domain in [ + "https://lofter.com/post/123", + "https://www.lofter.com/front/detail", + "https://emm3716958.lofter.com/post/74daebd2_34ec49f1a", + "https://random-author.lofter.com/post/abc_def", + ]: + with self.subTest(domain=domain): + self.assertEqual(UrlParser.get_platform(domain), "网易LOFTER") + class WebFetcherTest(unittest.TestCase): @staticmethod @@ -341,6 +362,18 @@ class WebFetcherTest(unittest.TestCase): "&sruserid=2369252937&kgsscty1=link", ) + def test_follows_dewu_short_link(self): + redirect_url = ( + "https://m.dewu.com/rn-activity/community-share" + "?trendId=514124663&shareId=8xEKQBm&noise=x" + ) + with patch("utils.web_fetcher.requests.get", return_value=self.response(redirect_url)): + result = WebFetcher.fetch_redirect_url("https://dw4.co/t/A/HmV2eJqiU") + self.assertEqual( + result, + "https://m.dewu.com/rn-activity/community-share?trendId=514124663&shareId=8xEKQBm", + ) + def test_stops_before_login_or_verification_page(self): for blocked_path in ("/login", "/404", "/captcha", "/verify", "/error"): with self.subTest(blocked_path=blocked_path): diff --git a/utils/web_fetcher.py b/utils/web_fetcher.py index 916d94d..9f5ab75 100755 --- a/utils/web_fetcher.py +++ b/utils/web_fetcher.py @@ -22,8 +22,8 @@ class WebFetcher: # 绿洲分享页会对通用桌面请求头返回 500,由解析器使用移动端请求头抓取。 # 微博博文可通过 API 直接根据 ID 解析,直接访问网页端常触发访客系统重定向。 domain = UrlParser.get_domain(url) - if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan"}: - if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI"}: + if domain not in {"t.cn", "b23.tv", "xhslink.cn", "xhslink.com", "hy.fan", "dw4.co"}: + if UrlParser.get_platform(url) in {"知乎", "绿洲", "新片场", "夸克AI", "通义千问", "微博", "小云雀AI", "哔哩哔哩", "快影", "微信公众号", "海螺AI", "网易LOFTER", "得物"}: return UrlParser.extract_video_address(url) try: current_url = url @@ -99,6 +99,12 @@ class UrlParser: if domain.endswith('.m.chenzhongtech.com'): return '快手' + if domain == 'lofter.com' or domain.endswith('.lofter.com'): + return '网易LOFTER' + + if domain.endswith('.dewu.com') or domain.endswith('.poizon.com'): + return '得物' + return None @staticmethod @@ -341,6 +347,24 @@ class UrlParser: elif platform in ("红果短剧", "番茄小说", "红果漫剧", "小云雀AI"): if parsed_url.query: address = f"{address}?{parsed_url.query}" + elif platform == "得物": + query_params = parse_qs(parsed_url.query) + preserved_params = [] + for key in ("trendId", "shareId", "source", "shareType", "shareChannel", "isScreenShot"): + value = query_params.get(key, [None])[0] + if value is not None: + preserved_params.append((key, value)) + if preserved_params: + address = f"{address}?{urlencode(preserved_params)}" + elif platform == "网易LOFTER": + query_params = parse_qs(parsed_url.query) + preserved_params = [] + for key in ("incantation", "permalink", "postId", "blogId"): + value = query_params.get(key, [None])[0] + if value is not None: + preserved_params.append((key, value)) + if preserved_params: + address = f"{address}?{urlencode(preserved_params)}" return address @staticmethod @@ -352,6 +376,9 @@ class UrlParser: match_vid = re.search(r'(?:["%22]|%22)?(?:vid|video_id|content_id|material_id)(?:["%22]|%22)?\s*(?:[:=]|%3A)\s*(?:["%22]|%22)?(\d{15,22})', url) if match_vid: return match_vid.group(1) + params_trend_id = query_params.get('trendId', [None])[0] + if params_trend_id: + return params_trend_id params_share_id = query_params.get('shareId', [None])[0] if params_share_id: return params_share_id