mirror of
https://github.com/ucmao/media-parser.git
synced 2026-11-20 08:04:56 +08:00
支持后台在线配置Cookie及通用动态代理,增强微博解析
This commit is contained in:
+2
-2
@@ -68,5 +68,5 @@ JIMENG_COOKIE=
|
||||
WEIBO_COOKIE=
|
||||
|
||||
|
||||
# 巨量 json2 动态代理提取地址;留空关闭代理重试,配置说明见 docs/proxy-retry.md。
|
||||
JULIANG_PROXY_API_URL=
|
||||
# 动态代理 API 提取地址(原生兼容 芝麻代理、巨量IP、青果网络、无忧代理、多米代理 等,支持 JSON 与 IP:端口 纯文本);留空关闭代理重试,配置说明见 docs/proxy-retry.md。
|
||||
DYNAMIC_PROXY_API_URL=
|
||||
|
||||
@@ -93,6 +93,8 @@ Media-Parser是一款专为短视频创作者与开发者打造的**100%原生
|
||||
<sub>注:带 `*` 的项表示该素材保留官方原生水印,未带 `*` 的项均为纯净无水印素材。</sub>
|
||||
> **Cookie凭据配置**:绝大多数平台支持免登录匿名解析;如需配置小红书、视频号等平台凭证,可参考 [`.env.example`](.env.example),详细方法见 [📖 平台Cookie配置指南](docs/cookie-config.md)。
|
||||
>
|
||||
> **动态代理配置**:配置 `DYNAMIC_PROXY_API_URL` 后,Cookie 校验失败可自动触发平台级 15 分钟代理访问,并按有效期提前储备代理。详见 [📖 动态代理配置说明](docs/proxy-retry.md)。
|
||||
>
|
||||
> **逆向与抓包SOP**:各平台详细技术分析与抓包规范,请查阅 [📖 平台逆向指南索引](docs/index.md) 与 [📖 通用逆向方法论](docs/reverse-guide.md)。
|
||||
|
||||
---
|
||||
@@ -268,9 +270,3 @@ python3 tests/manual_verify_parsers.py --platform "小云雀AI"
|
||||
### 2. 开源协议 & 法律合规
|
||||
* 本项目基于 **[MIT LICENSE](LICENSE)** 协议开源。
|
||||
* **免责声明**:本项目所有代码和文档仅用于网络技术研究、接口逆向工程学习与防御性安全交流。使用者请遵守各目标平台的《用户服务协议》与相关法律法规,不得用于任何形式的商业侵权抓取或恶意攻击行为。因使用本工具造成的任何直接或间接法律责任由使用者自行承担。
|
||||
|
||||
---
|
||||
|
||||
### Cookie 触发的动态代理
|
||||
|
||||
配置 `JULIANG_PROXY_API_URL` 后,Cookie 校验失败可自动触发平台级 15 分钟代理访问,并按有效期提前储备代理。详见 [动态代理配置与快手兼容性说明](docs/proxy-retry.md)(2026-09-28,Codex)。
|
||||
|
||||
@@ -44,7 +44,9 @@ def create_app(config=None):
|
||||
app.config['DATABASE'] = os.getenv(
|
||||
'DATABASE_PATH', os.path.join(app.instance_path, 'media_parser.db')
|
||||
)
|
||||
app.config['JULIANG_PROXY_API_URL'] = os.getenv('JULIANG_PROXY_API_URL', '').strip()
|
||||
proxy_url = (os.getenv('DYNAMIC_PROXY_API_URL') or os.getenv('JULIANG_PROXY_API_URL') or '').strip()
|
||||
app.config['DYNAMIC_PROXY_API_URL'] = proxy_url
|
||||
app.config['JULIANG_PROXY_API_URL'] = proxy_url
|
||||
app.config['MAX_CONTENT_LENGTH'] = 32 * 1024
|
||||
app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(days=30)
|
||||
app.config['JSON_SORT_KEYS'] = False
|
||||
@@ -65,7 +67,7 @@ def create_app(config=None):
|
||||
if not app.config.get('SECRET_KEY'):
|
||||
app.config['SECRET_KEY'] = _load_or_create_secret(database_dir or app.instance_path)
|
||||
init_database(app)
|
||||
manager = ProxyManager(app.config["DATABASE"], app.config["JULIANG_PROXY_API_URL"])
|
||||
manager = ProxyManager(app.config["DATABASE"], app.config["DYNAMIC_PROXY_API_URL"])
|
||||
app.extensions["proxy_manager"] = manager
|
||||
if not app.testing:
|
||||
manager.start()
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ services:
|
||||
# 纯 API / 微服务模式(开启后不挂载 Web UI,全接口免 API Key 直接调用且不写请求日志)
|
||||
- API_ONLY=${API_ONLY:-false}
|
||||
# Cookie 触发的动态代理提取地址
|
||||
- JULIANG_PROXY_API_URL=${JULIANG_PROXY_API_URL:-}
|
||||
- DYNAMIC_PROXY_API_URL=${DYNAMIC_PROXY_API_URL:-${JULIANG_PROXY_API_URL:-}}
|
||||
# 可选平台凭据(若在本地 .env 中配置将自动透传进容器):
|
||||
- XHS_COOKIE=${XHS_COOKIE:-}
|
||||
- DOUYIN_COOKIE=${DOUYIN_COOKIE:-}
|
||||
|
||||
+18
-8
@@ -27,20 +27,30 @@
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["解析器启动"] --> B{"1. 读取环境变量 (.env / 宿主机)"}
|
||||
B -->|"存在有效值"| Pass["应用该凭据"]
|
||||
B -->|"不存在"| C{"2. 读取 configs/business_config.json"}
|
||||
A["解析请求触发"] --> B{"1. 读取后台系统设置 (SQLite / /admin/settings)"}
|
||||
B -->|"存在有效值"| Pass["应用该凭据 (支持热更新)"]
|
||||
B -->|"为空/未配置"| C{"2. 读取 .env 环境变量 / 宿主机"}
|
||||
C -->|"存在有效值"| Pass
|
||||
C -->|"不存在"| D["执行默认匿名 / 游客协议提取"]
|
||||
C -->|"为空/未配置"| D{"3. 读取平台别名环境变量"}
|
||||
D -->|"存在有效值"| Pass
|
||||
D -->|"不存在"| E["执行默认匿名 / 游客协议提取"]
|
||||
```
|
||||
|
||||
### 方式 A:通过 `.env` 环境变量配置(推荐,最安全便捷)
|
||||
### 方式 A:管理员后台在线配置(推荐,即时生效无需重启)
|
||||
登录系统管理员后台,导航至 **系统设置 (`/admin/settings`)** -> **平台凭据 (Cookie)** Tab 选项卡:
|
||||
* 支持可视化配置各平台的 Cookie 凭据(如微信视频号 `cookie_wechat_channels`、小红书 `cookie_xiaohongshu` 等);
|
||||
* 此处配置具有**最高优先级**,可覆盖 `.env` 中的初始环境变量;
|
||||
* 修改并点击保存后**立即生效,无需重启应用容器/进程**;清空框内内容并保存即可自动恢复回退读取 `.env` 配置。
|
||||
|
||||
### 方式 B:通过 `.env` 环境变量配置
|
||||
在项目根目录创建或编辑 `.env` 文件(参考 [.env.example](file:///Users/leo/Projects/media-parser/.env.example)):
|
||||
```env
|
||||
# 小红书
|
||||
XHS_COOKIE="a1=xxx; webId=yyy;"
|
||||
# 微信视频号 / 腾讯元宝
|
||||
YUANBAO_COOKIE="hy_user=xxx; hy_token=xxx;"
|
||||
|
||||
# 腾讯元宝(用于视频号解析)
|
||||
# 小红书
|
||||
XHS_COOKIE="a1=xxx; webId=xxx; web_session=xxx;"
|
||||
```
|
||||
YUANBAO_COOKIE="hy_user=xxx; hy_token=yyy;"
|
||||
|
||||
# 快手
|
||||
|
||||
@@ -68,9 +68,11 @@ flowchart TD
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 风控识别与提示机制
|
||||
* **风控特征识别**:快手反爬拦截会返回 `{"result": 2}` 或 `{"data": {"result": 400002, "bizName": "ANTICRAWL_DEFAULT"}}`。
|
||||
* **熔断响应**:解析器标记 `cookie_required = True`,并在无有效 Cookie 导致无法解析时抛出 `KUAISHOU_COOKIE_REQUIRED` 错误,引导管理员更新快手 Cookie。
|
||||
### 2.4 动态代理兼容与 Cookie 隔离说明
|
||||
在系统启用动态代理(配置 `DYNAMIC_PROXY_API_URL`,详见 [动态代理配置说明](../proxy-retry.md))时,快手解析器具备以下针对性的会话隔离与容灾逻辑:
|
||||
* **结构与参数保留**:完整保留移动端/桌面端顺序、候选 URL 列表、GraphQL 请求体与回退、每条路径请求头、默认重定向行为、5 秒请求超时以及媒体字段提取规则。
|
||||
* **Session Cookie 严格隔离**:快手会话在每次顶层请求前后清空服务端 Cookie,保持独立 `requests.get/post` 不跨调用保存 Cookie 的行为(但同一调用内部的重定向仍保留 Cookie)。配置 Cookie 显式传入,请求头不写入 Session 公共字典,避免 Cookie 在多用户请求或多代理 IP 间共享污染。
|
||||
* **多路径切换逻辑**:直连模式下允许早期路径被拦截后由后续路径补救成功,只有所有路径最终均失败时才触发平台级代理窗口。代理模式检测到当前 IP 访问失败时将立即终止该 IP 的后续尝试并自动进行轮换。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
* 微博实况图集 (Live Photo, .mov / .mp4 动图视频流)
|
||||
* 微博正文内容与博主信息
|
||||
* **常见链接形态**:
|
||||
* 视频页:`https://video.weibo.com/show?fid=1034:5336219874426938`
|
||||
* 视频页 (数字或十六进制/字母数字混合 fid):`https://video.weibo.com/show?fid=1034:5336219874426938` 或 `https://video.weibo.com/show?fid=1034:cba52f9b57b69ca1a0cde0c2211e1ca1`
|
||||
* 直播/回放页:`https://weibo.com/l/wblive/p/show/1022:2321325311149536575703`
|
||||
* 实况/图文博文:`https://weibo.com/6288169783/5344438765488239`
|
||||
* 网页长链:`https://weibo.com/1234567890/Mabcdef`
|
||||
@@ -31,9 +31,10 @@
|
||||
|
||||
### 2.2 视频流、直播回放与图文分支提取
|
||||
* **分支 1 (视频专页 `1034:xxx` 与直播回放 `1022:xxx`)**:
|
||||
* 识别 `fid=1034:...`、`/tv/show/1034:...` 以及 `/l/wblive/p/show/1022:...`;
|
||||
* 初始化访客凭证后,针对视频先调用组件接口 `https://weibo.com/tv/api/component` (`Component_Play_Playinfo`);
|
||||
* 针对微博直播(`1022:...`)链接,若组件接口未返回媒体数据,进一步请求 `https://weibo.com/l/!/2/wblive/room/show_pc_live.json?live_id=1022:...` 提取直播/回放地址(`replay_origin_url` / `live_origin_hls_url`)、封面及主播信息。
|
||||
* 识别并正则匹配 `fid=1034:<media_id>`、`/tv/show/1034:<media_id>` 以及 `/l/wblive/p/show/1022:<media_id>`,同时兼容纯数字 `media_id` 与十六进制/字母数字混合 `media_id`(如 `1034:cba52f9b57b69ca1a0cde0c2211e1ca1`);
|
||||
* 初始化 Visitor 访客凭证获取合法 Session 状态,调用 PC 端视频组件接口 `https://weibo.com/tv/api/component` (`Component_Play_Playinfo`) 提取分辨率列表、直链 (`stream_url` / `urls`) 及博主/标题元数据;
|
||||
* 针对微博直播(`1022:...`)链接,若组件接口未返回媒体数据,进一步请求 `https://weibo.com/l/!/2/wblive/room/show_pc_live.json?live_id=1022:...` 提取直播/回放地址(`replay_origin_url` / `live_origin_hls_url`)、封面及主播信息;
|
||||
* 阻断误退化:当确认 URL 为视频专页,若组件 API 未返回有效数据,解析器直接终止查询并返回空字典,防止误将视频 `oid` 作为博文 `status_id` 传给 `m.weibo.cn/statuses/show` 触发无意义 404 异常。
|
||||
* **分支 2 (标准微博动态 `statuses/show`)**:
|
||||
* 从 `page_info.media_info.playback_list` 获取不同分辨率的 MP4 直链;
|
||||
* 从 `pics` 或 `pic_ids` 遍历提取原图并做去水印前缀转换;
|
||||
|
||||
+6
-13
@@ -1,12 +1,11 @@
|
||||
# Cookie 触发的动态代理重试
|
||||
|
||||
日期:2026-09-28;执行者:Codex。
|
||||
# 动态代理配置说明
|
||||
|
||||
## 配置与部署
|
||||
|
||||
在部署目录的 `.env` 设置 `JULIANG_PROXY_API_URL` 为巨量完整提取链接,保留 `num=1`、`pt=1`、`result_type=json2`、`ip_remain=1`。Docker Compose 自动透传;直接运行 Gunicorn/Python 时需要将该变量导出到进程环境。留空保持原有直连解析,不启用代理重试。
|
||||
|
||||
本次提供的最新提取链接已配置于本地 `.env`(Git 忽略文件)。部署其他机器时需同步该配置。
|
||||
在部署目录的 `.env` 中设置 `DYNAMIC_PROXY_API_URL` 为动态代理 API 提取链接。系统内置智能通用解包器,**原生兼容 芝麻代理、巨量 IP、青果网络、无忧代理、多米代理** 等主流代理服务商,支持任意返回 **JSON 结构** 或 **IP:端口 纯文本** 的 API 提取链接:
|
||||
* **巨量 IP 提取链接**:在后台生成时保留 `num=1&pt=1&result_type=json2&ip_remain=1`;
|
||||
* **芝麻 / 青果 / 无忧 / 多米等**:在服务商后台生成返回 JSON 格式或直接返回 `1.2.3.4:8080` 纯文本的提取链接均可直接填入;
|
||||
* Docker Compose 自动透传;直接运行 Gunicorn/Python 时需要将该变量导出到进程环境。留空保持原有直连解析,不启用代理重试。
|
||||
|
||||
```sh
|
||||
docker compose up -d --build
|
||||
@@ -21,7 +20,7 @@ docker compose up -d --build
|
||||
- 换 IP 时重新展开原始分享链接、创建解析器和会话,避免旧缓存或临时 Cookie 影响新尝试。用户配置的 Cookie 仍然生效。
|
||||
- 每次解析最多使用 3 个不同 IP,前台最多调用 3 次提取接口,包含直连和重试的总预算为 90 秒。每次网络调用受剩余预算限制,预算耗尽后不再发起请求。
|
||||
- 网络异常、代理失效、HTTP 401/403/407/429/5xx、登录/验证码跳转或仍需 Cookie 会剔除 IP。正常响应但无媒体不单独判定为坏代理;明确删除、私密、纯文本内容不换 IP。
|
||||
- 快手直连仍允许早期路径被拦截后由其他路径成功,只有最终失败才切换平台。代理模式检测到访问失败时结束当前 IP 的尝试。
|
||||
- 支持多路径降级机制的平台(如快手等),在直连模式下仍允许早期路径被拦截后由其他备用路径成功,只有最终全部失败才切换代理平台。代理模式检测到访问失败时结束当前 IP 的尝试。
|
||||
- 内部重试不重复预扣积分、消耗平台限流次数或写入请求日志。最终失败按原规则退款。
|
||||
|
||||
## 库存和有效期
|
||||
@@ -36,12 +35,6 @@ docker compose up -d --build
|
||||
|
||||
提前储备不能消除供应商中断:取不到有效 IP 时返回错误,窗口期间不偷偷退回直连。
|
||||
|
||||
## 快手兼容性
|
||||
|
||||
保留移动端/桌面端顺序、候选 URL、GraphQL 请求体与回退、每条路径请求头、默认重定向行为、5 秒请求超时以及媒体字段提取。
|
||||
|
||||
快手会话在每次顶层请求前后清空服务端 Cookie,保持原独立 `requests.get/post` 不跨调用保存 Cookie 的行为;同一调用内部的重定向仍能使用 Cookie。配置 Cookie 显式传入,请求头不写入会话公共字典,会话不跨用户请求或 IP 共享。
|
||||
|
||||
## 错误与排查
|
||||
|
||||
| HTTP | error_code | 含义 |
|
||||
|
||||
@@ -458,7 +458,7 @@ def get_platform_distribution(user_id=None, days=7):
|
||||
items = []
|
||||
if total_calls > 0:
|
||||
cx, cy = 100, 100
|
||||
r_out, r_in = 88, 64
|
||||
r_out, r_in = 88, 62
|
||||
current_angle = 0.0 # radians
|
||||
|
||||
for idx, row in enumerate(rows):
|
||||
|
||||
@@ -52,11 +52,11 @@ class WeiboParser(BaseParser):
|
||||
return None
|
||||
|
||||
fid = (parse_qs(urlparse(self.real_url).query).get("fid") or [""])[0]
|
||||
if re.fullmatch(r"\d+:\d+", fid):
|
||||
if re.fullmatch(r"\d+:[a-zA-Z0-9]+", fid):
|
||||
return fid
|
||||
if match := re.search(r"/(?:tv/|l/wblive/p/)?show/(\d+:\d+)", urlparse(self.real_url).path):
|
||||
if match := re.search(r"/(?:tv/|l/wblive/p/)?show/(\d+:[a-zA-Z0-9]+)", urlparse(self.real_url).path):
|
||||
return match.group(1)
|
||||
if match := re.search(r"/show/(\d+:\d+)", self.real_url):
|
||||
if match := re.search(r"/show/(\d+:[a-zA-Z0-9]+)", self.real_url):
|
||||
return match.group(1)
|
||||
return None
|
||||
|
||||
@@ -65,14 +65,14 @@ class WeiboParser(BaseParser):
|
||||
return None
|
||||
|
||||
fid = (parse_qs(urlparse(self.real_url).query).get("fid") or [""])[0]
|
||||
if match := re.fullmatch(r"\d+:(\d+)", fid):
|
||||
if match := re.fullmatch(r"\d+:([a-zA-Z0-9]+)", fid):
|
||||
return match.group(1)
|
||||
|
||||
# 视频/直播页会从 video.weibo.com/show?fid=1034:... 跳转到
|
||||
# weibo.com/tv/show/1034:... 或 weibo.com/l/wblive/p/show/1022:...
|
||||
if match := re.search(r"/(?:tv/|l/wblive/p/)?show/\d+:(\d+)", urlparse(self.real_url).path):
|
||||
if match := re.search(r"/(?:tv/|l/wblive/p/)?show/\d+:([a-zA-Z0-9]+)", urlparse(self.real_url).path):
|
||||
return match.group(1)
|
||||
if match := re.search(r"/show/\d+:(\d+)", self.real_url):
|
||||
if match := re.search(r"/show/\d+:([a-zA-Z0-9]+)", self.real_url):
|
||||
return match.group(1)
|
||||
|
||||
# PC / Mobile URL, like: weibo.com/123456789/O8yqz0I8Q or m.weibo.cn/7753941940/5332536008119716/qq
|
||||
@@ -111,6 +111,7 @@ class WeiboParser(BaseParser):
|
||||
data = self._fetch_video_page_data()
|
||||
if data:
|
||||
return data
|
||||
return {}
|
||||
|
||||
if not self.numeric_id:
|
||||
logger.error("WeiboParser: Could not extract numeric ID.")
|
||||
|
||||
+33
-12
@@ -15,31 +15,52 @@ PLATFORM_COOKIE_ALIASES = {
|
||||
}
|
||||
|
||||
|
||||
def clean_cookie_quotes(val: str) -> str:
|
||||
val = val.strip()
|
||||
if (val.startswith('"') and val.endswith('"')) or (val.startswith("'") and val.endswith("'")):
|
||||
val = val[1:-1].strip()
|
||||
return val
|
||||
|
||||
|
||||
def get_platform_cookie(platform_key: str, env_var: str | None = None) -> str:
|
||||
"""获取指定平台的环境变量 Cookie 凭据。
|
||||
"""获取指定平台的 Cookie 凭据。
|
||||
|
||||
按优先级依次检查:
|
||||
1. 指定的环境变量 (如 env_var="XHS_COOKIE");
|
||||
2. 标准环境变量 ({PLATFORM}_COOKIE);
|
||||
3. 别名列表 (如 XIAOHONGSHU_COOKIE)。
|
||||
1. 后台数据库系统配置 (system_settings 表中的 cookie_{platform});
|
||||
2. 指定的环境变量 (如 env_var="XHS_COOKIE");
|
||||
3. 标准环境变量 ({PLATFORM}_COOKIE);
|
||||
4. 别名列表 (如 XIAOHONGSHU_COOKIE)。
|
||||
"""
|
||||
key_normalized = platform_key.lower().replace("-", "_")
|
||||
|
||||
# 1. 优先读取后台系统设置 (数据库)
|
||||
db_keys = [f"cookie_{key_normalized}"]
|
||||
for alias in PLATFORM_COOKIE_ALIASES.get(key_normalized, []):
|
||||
alias_name = alias.lower().replace("_cookie", "").replace("cookie_", "")
|
||||
if alias_name and f"cookie_{alias_name}" not in db_keys:
|
||||
db_keys.append(f"cookie_{alias_name}")
|
||||
|
||||
try:
|
||||
from src.db import setting
|
||||
for db_key in db_keys:
|
||||
val = setting(db_key)
|
||||
if val is not None and str(val).strip():
|
||||
return clean_cookie_quotes(str(val))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 2. 检查指定环境变量
|
||||
if env_var is None:
|
||||
env_var = f"{key_normalized.upper()}_COOKIE"
|
||||
|
||||
# 1. 优先读取指定环境变量
|
||||
val = os.getenv(env_var, "").strip()
|
||||
if val:
|
||||
if (val.startswith('"') and val.endswith('"')) or (val.startswith("'") and val.endswith("'")):
|
||||
val = val[1:-1].strip()
|
||||
return val
|
||||
return clean_cookie_quotes(val)
|
||||
|
||||
# 2. 检查别名列表
|
||||
# 3. 检查别名列表
|
||||
for alias in PLATFORM_COOKIE_ALIASES.get(key_normalized, []):
|
||||
val = os.getenv(alias, "").strip()
|
||||
if val:
|
||||
if (val.startswith('"') and val.endswith('"')) or (val.startswith("'") and val.endswith("'")):
|
||||
val = val[1:-1].strip()
|
||||
return val
|
||||
return clean_cookie_quotes(val)
|
||||
|
||||
return ""
|
||||
|
||||
+90
-12
@@ -2,12 +2,14 @@
|
||||
|
||||
import ipaddress
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
|
||||
import requests
|
||||
from urllib3.util import Timeout
|
||||
@@ -58,9 +60,22 @@ class ProxyManager:
|
||||
with self.connection() as db:
|
||||
db.executescript(SCHEMA)
|
||||
|
||||
def get_active_api_url(self):
|
||||
"""三级优先级:后台在线配置 (system_settings) > .env 环境变量 > 留空"""
|
||||
try:
|
||||
with self.connection() as db:
|
||||
row = db.execute(
|
||||
"SELECT value FROM system_settings WHERE key IN ('dynamic_proxy_api_url', 'cookie_dynamic_proxy_api_url') AND value != '' LIMIT 1"
|
||||
).fetchone()
|
||||
if row and row["value"].strip():
|
||||
return row["value"].strip()
|
||||
except Exception:
|
||||
pass
|
||||
return (self.api_url or "").strip()
|
||||
|
||||
@property
|
||||
def enabled(self):
|
||||
return bool(self.api_url)
|
||||
return bool(self.get_active_api_url())
|
||||
|
||||
@contextmanager
|
||||
def connection(self, write=False):
|
||||
@@ -142,17 +157,73 @@ class ProxyManager:
|
||||
|
||||
@staticmethod
|
||||
def decode(payload, started):
|
||||
if not isinstance(payload, dict) or payload.get("code") != 200:
|
||||
raise ValueError("代理供应商返回失败")
|
||||
rows = payload.get("data", {}).get("proxy_list")
|
||||
if not isinstance(rows, list) or not rows:
|
||||
raise ValueError("代理列表为空")
|
||||
row = rows[0]
|
||||
ip = str(ipaddress.IPv4Address(row["ip"]))
|
||||
port = int(row["port"])
|
||||
remain = float(row["ip_remain"])
|
||||
"""通用代理解码器:支持 巨量、芝麻、青果、Data5U、多米 等各种 JSON 结构以及 TXT 文本。"""
|
||||
ip = None
|
||||
port = None
|
||||
remain = None
|
||||
|
||||
# 1. 如果输入为纯文本 TXT(或字符串格式)
|
||||
if isinstance(payload, str):
|
||||
match = re.search(r"(\d{1,3}(?:\.\d{1,3}){3}):(\d{1,5})", payload)
|
||||
if match:
|
||||
ip = match.group(1)
|
||||
port = int(match.group(2))
|
||||
remain = 300.0
|
||||
|
||||
# 2. 如果输入为 JSON (dict)
|
||||
elif isinstance(payload, dict):
|
||||
# 校验响应 Code / Status
|
||||
code = payload.get("code")
|
||||
success = payload.get("success")
|
||||
valid_code = code in (200, 0, "200", "0", "SUCCESS") or success is True
|
||||
if not valid_code and code is not None:
|
||||
raise ValueError(f"代理供应商返回失败: code={code}, msg={payload.get('msg') or payload.get('message')}")
|
||||
|
||||
# 寻找代理列表项 (兼容不同数据节点的层级)
|
||||
rows = None
|
||||
data_field = payload.get("data")
|
||||
if isinstance(data_field, dict):
|
||||
rows = data_field.get("proxy_list") or data_field.get("list") or data_field.get("proxies") or [data_field]
|
||||
elif isinstance(data_field, list):
|
||||
rows = data_field
|
||||
elif isinstance(payload.get("proxy_list"), list):
|
||||
rows = payload.get("proxy_list")
|
||||
|
||||
if isinstance(rows, list) and len(rows) > 0 and isinstance(rows[0], dict):
|
||||
row = rows[0]
|
||||
ip = str(row.get("ip") or row.get("outip") or row.get("server") or row.get("host") or "")
|
||||
port = int(row.get("port") or row.get("server_port") or 0)
|
||||
|
||||
# 解析有效期 (ip_remain / ttl / expire_seconds / expire_time / deadline)
|
||||
has_explicit_remain = "ip_remain" in row or "ttl" in row or "expire_seconds" in row
|
||||
raw_remain = row.get("ip_remain") if "ip_remain" in row else (row.get("ttl") if "ttl" in row else row.get("expire_seconds"))
|
||||
|
||||
if has_explicit_remain:
|
||||
try:
|
||||
remain = float(raw_remain)
|
||||
except (ValueError, TypeError):
|
||||
raise ValueError("代理端口或有效期无效")
|
||||
else:
|
||||
exp_val = row.get("expire_time") or row.get("deadline")
|
||||
if isinstance(exp_val, (int, float)) and exp_val > 1000000000:
|
||||
remain = float(exp_val) - started
|
||||
elif isinstance(exp_val, str):
|
||||
try:
|
||||
dt = datetime.strptime(exp_val.strip(), "%Y-%m-%d %H:%M:%S")
|
||||
remain = dt.timestamp() - started
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
if remain is None:
|
||||
remain = 300.0 # 其它未明示有效期的结构给保底默认 5 分钟
|
||||
|
||||
if not ip or not port:
|
||||
raise ValueError("未能提取到有效的代理 IP/端口")
|
||||
|
||||
ip = str(ipaddress.IPv4Address(ip))
|
||||
if not 1 <= port <= 65535 or not 10 < remain < float("inf"):
|
||||
raise ValueError("代理端口或有效期无效")
|
||||
|
||||
expires = started + remain
|
||||
if expires <= time.time() + 10:
|
||||
raise ValueError("代理剩余有效期不足")
|
||||
@@ -185,11 +256,18 @@ class ProxyManager:
|
||||
budget = min(5, deadline - time.monotonic())
|
||||
if budget <= 0:
|
||||
raise ValueError("解析预算已耗尽")
|
||||
active_url = self.get_active_api_url()
|
||||
if not active_url:
|
||||
raise ValueError("代理提取 API URL 未配置")
|
||||
response = session.get(
|
||||
self.api_url, timeout=Timeout(total=budget, connect=min(2, budget), read=budget),
|
||||
active_url, timeout=Timeout(total=budget, connect=min(2, budget), read=budget),
|
||||
)
|
||||
response.raise_for_status()
|
||||
proxy = self.decode(response.json(), now)
|
||||
try:
|
||||
payload = response.json()
|
||||
except ValueError:
|
||||
payload = response.text
|
||||
proxy = self.decode(payload, now)
|
||||
except (requests.RequestException, ValueError, TypeError, KeyError, AttributeError):
|
||||
# 不记录完整异常,避免将含签名的提取 URL 写入日志。
|
||||
logger.warning("代理提取失败:网络、供应商状态或响应字段异常")
|
||||
|
||||
@@ -146,6 +146,11 @@ def update_settings():
|
||||
set_setting("api_tip_website", (request.form.get("api_tip_website") or "").strip() or "https://github.com/ucmao/media-parser")
|
||||
set_setting("api_tip_notice", (request.form.get("api_tip_notice") or "").strip() or "本接口由开源项目 media-parser 提供服务")
|
||||
|
||||
# 保存平台 Cookie 覆盖凭据及动态代理 API 地址
|
||||
for key in request.form:
|
||||
if key.startswith("cookie_") or key == "dynamic_proxy_api_url":
|
||||
set_setting(key, (request.form.get(key) or "").strip())
|
||||
|
||||
db.commit()
|
||||
|
||||
is_ajax = request.headers.get("X-Requested-With") == "XMLHttpRequest" or "application/json" in request.headers.get("Accept", "")
|
||||
|
||||
@@ -153,8 +153,8 @@
|
||||
<div>当前时间范围内暂无调用记录</div>
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="pie-wrapper" id="admin-platform-dist">
|
||||
<div class="pie-svg-box relative w-56 h-56 mx-auto">
|
||||
<div class="pie-wrapper flex flex-col h-full justify-between" id="admin-platform-dist">
|
||||
<div class="pie-svg-box relative w-64 h-64 sm:w-[260px] sm:h-[260px] mx-auto my-2">
|
||||
<svg viewBox="0 0 200 200" class="w-full h-full">
|
||||
{% for item in pie_data['items'] %}
|
||||
<path d="{{ item['path_d'] }}" fill="{{ item['color'] }}" class="pie-slice transition-opacity hover:opacity-80 cursor-pointer"
|
||||
@@ -168,17 +168,17 @@
|
||||
</path>
|
||||
{% endfor %}
|
||||
</svg>
|
||||
<div class="pie-center-text absolute inset-0 flex flex-col items-center justify-center pointer-events-none text-center select-none px-2">
|
||||
<div class="pie-center-lbl text-xs text-slate-500 font-medium truncate max-w-[140px] transition-colors duration-150" id="admin-pie-lbl">总解析量</div>
|
||||
<div class="pie-center-val text-xl font-bold text-slate-900 mt-0.5 tracking-tight transition-all duration-150" id="admin-pie-val">{{ pie_data['total_calls_formatted'] or pie_data['total_calls'] }} 次</div>
|
||||
<div class="pie-center-sub text-[11px] text-slate-400 mt-0.5 min-h-[16px] font-medium transition-colors duration-150" id="admin-pie-sub"></div>
|
||||
<div class="pie-center-text absolute inset-0 flex flex-col items-center justify-center pointer-events-none text-center select-none px-3">
|
||||
<div class="pie-center-lbl text-xs text-slate-500 font-semibold truncate max-w-[160px] transition-colors duration-150" id="admin-pie-lbl">总解析量</div>
|
||||
<div class="pie-center-val text-2xl font-bold text-slate-900 mt-1 tracking-tight transition-all duration-150" id="admin-pie-val">{{ pie_data['total_calls_formatted'] or pie_data['total_calls'] }} 次</div>
|
||||
<div class="pie-center-sub text-xs text-slate-400 mt-0.5 min-h-[18px] font-medium transition-colors duration-150" id="admin-pie-sub"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 紧凑点状图例 -->
|
||||
<div class="flex flex-wrap items-center justify-center gap-x-1.5 gap-y-1 mt-5 pt-3 border-t border-slate-100">
|
||||
<div class="flex flex-wrap items-center justify-center gap-x-2 gap-y-1.5 mt-5 pt-4 border-t border-slate-100">
|
||||
{% for item in pie_data['items'] %}
|
||||
<div class="pie-pill-item inline-flex items-center gap-1 px-1.5 py-0.5 rounded-md hover:bg-slate-100 cursor-pointer transition text-xs font-medium text-slate-600 hover:text-slate-900 select-none"
|
||||
<div class="pie-pill-item inline-flex items-center gap-1.5 px-2 py-1 rounded-lg hover:bg-slate-100 cursor-pointer transition text-[12px] font-medium text-slate-600 hover:text-slate-900 select-none"
|
||||
data-platform="{{ item['platform'] }}"
|
||||
data-calls="{{ item['calls_formatted'] }}"
|
||||
data-percent="{{ item['percentage'] }}%"
|
||||
@@ -186,7 +186,7 @@
|
||||
data-failures="{{ item['failures_formatted'] }}"
|
||||
data-rate="{{ item['success_rate'] }}%"
|
||||
data-color="{{ item['color'] }}">
|
||||
<span class="w-2 h-2 rounded-full shrink-0" style="background:{{ item['color'] }}"></span>
|
||||
<span class="w-2.5 h-2.5 rounded-full shrink-0" style="background:{{ item['color'] }}"></span>
|
||||
<span class="truncate">{{ item['platform'] }}</span>
|
||||
</div>
|
||||
{% endfor %}
|
||||
|
||||
+253
-24
@@ -10,12 +10,23 @@
|
||||
</div>
|
||||
|
||||
<!-- 顶部 Tab 切换胶囊栏 -->
|
||||
<div class="flex items-center gap-1.5 p-1 bg-slate-200/60 rounded-xl max-w-fit border border-slate-200/80 select-none shadow-2xs">
|
||||
<div
|
||||
class="flex items-center gap-1.5 p-1 bg-slate-200/60 rounded-xl max-w-fit border border-slate-200/80 select-none shadow-2xs">
|
||||
<button type="button" onclick="switchSettingsTab('access')" id="tab-btn-access"
|
||||
class="tab-nav-btn inline-flex items-center gap-2 px-3.5 py-2 text-xs font-semibold rounded-lg transition-all bg-white text-brand shadow-xs">
|
||||
{{ icon('shield', 'w-3.5 h-3.5') }}
|
||||
<span>服务与访问控制</span>
|
||||
</button>
|
||||
<button type="button" onclick="switchSettingsTab('cookies')" id="tab-btn-cookies"
|
||||
class="tab-nav-btn inline-flex items-center gap-2 px-3.5 py-2 text-xs font-semibold rounded-lg transition-all text-slate-600 hover:text-slate-900">
|
||||
{{ icon('key', 'w-3.5 h-3.5') }}
|
||||
<span>平台凭据 (Cookie)</span>
|
||||
</button>
|
||||
<button type="button" onclick="switchSettingsTab('proxy')" id="tab-btn-proxy"
|
||||
class="tab-nav-btn inline-flex items-center gap-2 px-3.5 py-2 text-xs font-semibold rounded-lg transition-all text-slate-600 hover:text-slate-900">
|
||||
{{ icon('globe', 'w-3.5 h-3.5') }}
|
||||
<span>网络与动态代理</span>
|
||||
</button>
|
||||
<button type="button" onclick="switchSettingsTab('branding')" id="tab-btn-branding"
|
||||
class="tab-nav-btn inline-flex items-center gap-2 px-3.5 py-2 text-xs font-semibold rounded-lg transition-all text-slate-600 hover:text-slate-900">
|
||||
{{ icon('tag', 'w-3.5 h-3.5') }}
|
||||
@@ -32,7 +43,8 @@
|
||||
<section class="bg-white rounded-xl border border-slate-200/80 p-5 sm:p-6 shadow-xs space-y-5">
|
||||
<div class="border-b border-slate-100 pb-3">
|
||||
<h2 class="text-sm font-bold text-slate-900 flex items-center gap-2">
|
||||
<span class="text-brand flex items-center justify-center">{{ icon('shield', 'w-4 h-4') }}</span>
|
||||
<span class="text-brand flex items-center justify-center">{{ icon('shield', 'w-4 h-4')
|
||||
}}</span>
|
||||
服务与访问控制
|
||||
</h2>
|
||||
</div>
|
||||
@@ -60,8 +72,8 @@
|
||||
</div>
|
||||
<div class="shrink-0">
|
||||
<label class="switch-toggle" title="切换公开首页状态">
|
||||
<input type="checkbox" id="homepage-toggle" name="homepage_enabled"
|
||||
{{ 'checked' if settings.get('homepage_enabled', '1')=='1' }}
|
||||
<input type="checkbox" id="homepage-toggle" name="homepage_enabled" {{ 'checked' if
|
||||
settings.get('homepage_enabled', '1' )=='1' }}
|
||||
onchange="triggerSettingsSubmit(this)">
|
||||
<span class="switch-slider"></span>
|
||||
</label>
|
||||
@@ -89,7 +101,8 @@
|
||||
<section class="bg-white rounded-xl border border-slate-200/80 p-5 sm:p-6 shadow-xs space-y-5">
|
||||
<div class="border-b border-slate-100 pb-3">
|
||||
<h2 class="text-sm font-bold text-slate-900 flex items-center gap-2">
|
||||
<span class="text-brand flex items-center justify-center">{{ icon('user', 'w-4 h-4') }}</span>
|
||||
<span class="text-brand flex items-center justify-center">{{ icon('user', 'w-4 h-4')
|
||||
}}</span>
|
||||
用户注册与新账号配置
|
||||
</h2>
|
||||
</div>
|
||||
@@ -101,8 +114,9 @@
|
||||
</div>
|
||||
<div class="shrink-0">
|
||||
<label class="switch-toggle" title="切换用户注册开关">
|
||||
<input type="checkbox" id="registration-toggle" name="registration_enabled" {{ 'checked'
|
||||
if registration_enabled }} onchange="toggleRegistrationDetails(this)">
|
||||
<input type="checkbox" id="registration-toggle" name="registration_enabled"
|
||||
{{ 'checked' if registration_enabled }}
|
||||
onchange="toggleRegistrationDetails(this)">
|
||||
<span class="switch-slider"></span>
|
||||
</label>
|
||||
</div>
|
||||
@@ -135,7 +149,8 @@
|
||||
<div class="py-3 first:pt-0 last:pb-0 flex items-center justify-between gap-4">
|
||||
<div>
|
||||
<div class="font-medium text-slate-900">默认试用天数</div>
|
||||
<div class="text-xs text-slate-500 mt-0.5">新注册客户默认试用时长 (1 ~ 9999 天,或勾选永久有效)</div>
|
||||
<div class="text-xs text-slate-500 mt-0.5">新注册客户默认试用时长 (1 ~ 9999 天,或勾选永久有效)
|
||||
</div>
|
||||
</div>
|
||||
<div class="flex items-center gap-3 shrink-0">
|
||||
<div class="relative w-28">
|
||||
@@ -163,14 +178,15 @@
|
||||
<div class="py-3 first:pt-0 last:pb-0 flex items-center justify-between gap-4">
|
||||
<div>
|
||||
<div class="font-medium text-slate-900">默认赠送积分</div>
|
||||
<div class="text-xs text-slate-500 mt-0.5">新注册客户初始赠送的解析积分额度 (0 ~ 99999999,或勾选不限制)
|
||||
<div class="text-xs text-slate-500 mt-0.5">新注册客户初始赠送的解析积分额度 (0 ~
|
||||
99999999,或勾选不限制)
|
||||
</div>
|
||||
</div>
|
||||
<div class="flex items-center gap-3 shrink-0">
|
||||
<div class="relative w-32">
|
||||
<input type="text" inputmode="numeric" min="0" max="99999999" data-default="100"
|
||||
data-thousands="true" id="input_default_initial_credits"
|
||||
name="default_initial_credits"
|
||||
<input type="text" inputmode="numeric" min="0" max="99999999"
|
||||
data-default="100" data-thousands="true"
|
||||
id="input_default_initial_credits" name="default_initial_credits"
|
||||
value="{{ '{:,}'.format(settings.get('default_initial_credits', '100')|int) if settings.get('default_initial_credits') != '-1' else '100' }}"
|
||||
{{ 'disabled' if settings.get('default_initial_credits')=='-1' }}
|
||||
onfocus="this.value = unformatThousands(this.value)"
|
||||
@@ -182,8 +198,9 @@
|
||||
</div>
|
||||
<label
|
||||
class="inline-flex items-center gap-1.5 cursor-pointer select-none text-xs font-medium text-slate-600 hover:text-slate-900">
|
||||
<input type="checkbox" id="chk_unlimited_credits" name="chk_unlimited_credits"
|
||||
value="1" {{ 'checked' if settings.get('default_initial_credits')=='-1' }}
|
||||
<input type="checkbox" id="chk_unlimited_credits"
|
||||
name="chk_unlimited_credits" value="1" {{ 'checked' if
|
||||
settings.get('default_initial_credits')=='-1' }}
|
||||
onchange="toggleUnlimitedCredits(this)"
|
||||
class="rounded border-slate-300 text-brand focus:ring-brand/20 h-4 w-4">
|
||||
<span>不限积分</span>
|
||||
@@ -196,13 +213,162 @@
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<!-- Tab 2: API 响应水印与版权提示 -->
|
||||
<!-- Tab 2: 平台凭据 (Cookie) -->
|
||||
<div id="tab-content-cookies" class="space-y-6 tab-panel" style="display: none;">
|
||||
<section class="bg-white rounded-xl border border-slate-200/80 p-5 sm:p-6 shadow-xs space-y-5">
|
||||
<div class="border-b border-slate-100 pb-3 flex items-center justify-between">
|
||||
<div>
|
||||
<h2 class="text-sm font-bold text-slate-900 flex items-center gap-2">
|
||||
<span class="text-brand flex items-center justify-center">{{ icon('key', 'w-4 h-4')
|
||||
}}</span>
|
||||
平台 Cookie 凭据覆盖配置
|
||||
</h2>
|
||||
<p class="text-xs text-slate-500 mt-1">此处配置具有最高优先级(覆盖 <code>.env</code>
|
||||
环境变量)。保存后无需重启即可即时生效;清空输入框并保存即可自动恢复读取 <code>.env</code> 凭据。</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="space-y-5 text-xs">
|
||||
<!-- 微信视频号 / 腾讯元宝 -->
|
||||
<div class="space-y-1.5">
|
||||
<div class="flex items-center justify-between">
|
||||
<label for="cookie_wechat_channels"
|
||||
class="font-semibold text-slate-900 flex items-center gap-2">
|
||||
<span>微信视频号 / 腾讯元宝</span>
|
||||
</label>
|
||||
<span class="text-[11px] text-slate-400">后台键名: <code>cookie_wechat_channels</code> /
|
||||
降级环境变量: <code>YUANBAO_COOKIE</code></span>
|
||||
</div>
|
||||
<textarea id="cookie_wechat_channels" name="cookie_wechat_channels" rows="2"
|
||||
placeholder="格式例: hybrid_sess=...; pass_ticket=..."
|
||||
onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2 font-mono text-xs rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50 custom-scroll">{{ settings.get('cookie_wechat_channels', '') }}</textarea>
|
||||
<p class="text-[11px] text-slate-400">用于解析微信视频号无水印视频与图集内容。</p>
|
||||
</div>
|
||||
|
||||
<!-- 小红书 -->
|
||||
<div class="space-y-1.5">
|
||||
<div class="flex items-center justify-between">
|
||||
<label for="cookie_xiaohongshu"
|
||||
class="font-semibold text-slate-900 flex items-center gap-2">
|
||||
<span>小红书 (Xiaohongshu)</span>
|
||||
</label>
|
||||
<span class="text-[11px] text-slate-400">后台键名: <code>cookie_xiaohongshu</code> / 降级环境变量:
|
||||
<code>XHS_COOKIE</code></span>
|
||||
</div>
|
||||
<textarea id="cookie_xiaohongshu" name="cookie_xiaohongshu" rows="2"
|
||||
placeholder="格式例: a1=...; webId=...; web_session=..."
|
||||
onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2 font-mono text-xs rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50 custom-scroll">{{ settings.get('cookie_xiaohongshu', '') }}</textarea>
|
||||
<p class="text-[11px] text-slate-400">用于绕过小红书风控阻断与提取高清无水印图集/视频。</p>
|
||||
</div>
|
||||
|
||||
<!-- 快手 -->
|
||||
<div class="space-y-1.5">
|
||||
<div class="flex items-center justify-between">
|
||||
<label for="cookie_kuaishou"
|
||||
class="font-semibold text-slate-900 flex items-center gap-2">
|
||||
<span>快手 (Kuaishou)</span>
|
||||
</label>
|
||||
<span class="text-[11px] text-slate-400">后台键名: <code>cookie_kuaishou</code> / 降级环境变量:
|
||||
<code>KUAISHOU_COOKIE</code></span>
|
||||
</div>
|
||||
<textarea id="cookie_kuaishou" name="cookie_kuaishou" rows="2"
|
||||
placeholder="格式例: did=...; kuaishou.server.web_st=..."
|
||||
onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2 font-mono text-xs rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50 custom-scroll">{{ settings.get('cookie_kuaishou', '') }}</textarea>
|
||||
<p class="text-[11px] text-slate-400">用于解除快手风控阻断或滑块验证要求。</p>
|
||||
</div>
|
||||
|
||||
<!-- 拼多多 -->
|
||||
<div class="space-y-1.5">
|
||||
<div class="flex items-center justify-between">
|
||||
<label for="cookie_pinduoduo"
|
||||
class="font-semibold text-slate-900 flex items-center gap-2">
|
||||
<span>拼多多 (Pinduoduo)</span>
|
||||
</label>
|
||||
<span class="text-[11px] text-slate-400">后台键名: <code>cookie_pinduoduo</code> / 降级环境变量:
|
||||
<code>PINDUODUO_COOKIE</code></span>
|
||||
</div>
|
||||
<textarea id="cookie_pinduoduo" name="cookie_pinduoduo" rows="2"
|
||||
placeholder="格式例: PDDAccessToken=...; pdd_user_id=..."
|
||||
onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2 font-mono text-xs rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50 custom-scroll">{{ settings.get('cookie_pinduoduo', '') }}</textarea>
|
||||
<p class="text-[11px] text-slate-400">用于拼多多商品/视频页面详情提取。</p>
|
||||
</div>
|
||||
|
||||
<!-- 抖音 -->
|
||||
<div class="space-y-1.5">
|
||||
<div class="flex items-center justify-between">
|
||||
<label for="cookie_douyin" class="font-semibold text-slate-900 flex items-center gap-2">
|
||||
<span>抖音 (Douyin)</span>
|
||||
</label>
|
||||
<span class="text-[11px] text-slate-400">后台键名: <code>cookie_douyin</code> / 降级环境变量:
|
||||
<code>DOUYIN_COOKIE</code></span>
|
||||
</div>
|
||||
<textarea id="cookie_douyin" name="cookie_douyin" rows="2"
|
||||
placeholder="格式例: passport_csrf_token=...; sessionid=..."
|
||||
onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2 font-mono text-xs rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50 custom-scroll">{{ settings.get('cookie_douyin', '') }}</textarea>
|
||||
<p class="text-[11px] text-slate-400">可选。常规抖音内容免 Cookie,针对受限作品做风控退避。</p>
|
||||
</div>
|
||||
|
||||
<!-- 微博 -->
|
||||
<div class="space-y-1.5">
|
||||
<div class="flex items-center justify-between">
|
||||
<label for="cookie_weibo" class="font-semibold text-slate-900 flex items-center gap-2">
|
||||
<span>微博 (Weibo)</span>
|
||||
</label>
|
||||
<span class="text-[11px] text-slate-400">后台键名: <code>cookie_weibo</code> / 降级环境变量:
|
||||
<code>WEIBO_COOKIE</code></span>
|
||||
</div>
|
||||
<textarea id="cookie_weibo" name="cookie_weibo" rows="2"
|
||||
placeholder="格式例: SUB=...; SUBP=..." onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2 font-mono text-xs rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50 custom-scroll">{{ settings.get('cookie_weibo', '') }}</textarea>
|
||||
<p class="text-[11px] text-slate-400">可选。常规内容免 Cookie,遇到频繁抓包限制时提供。</p>
|
||||
</div>
|
||||
|
||||
<!-- 豆包 AI -->
|
||||
<div class="space-y-1.5">
|
||||
<div class="flex items-center justify-between">
|
||||
<label for="cookie_doubao" class="font-semibold text-slate-900 flex items-center gap-2">
|
||||
<span>豆包 AI (Doubao)</span>
|
||||
</label>
|
||||
<span class="text-[11px] text-slate-400">后台键名: <code>cookie_doubao</code> / 降级环境变量:
|
||||
<code>DOUBAO_COOKIE</code></span>
|
||||
</div>
|
||||
<textarea id="cookie_doubao" name="cookie_doubao" rows="2" placeholder="格式例: sessionid=..."
|
||||
onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2 font-mono text-xs rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50 custom-scroll">{{ settings.get('cookie_doubao', '') }}</textarea>
|
||||
<p class="text-[11px] text-slate-400">用于提取豆包 AI 对话中的视频/生成媒体。</p>
|
||||
</div>
|
||||
|
||||
<!-- 即梦 AI -->
|
||||
<div class="space-y-1.5">
|
||||
<div class="flex items-center justify-between">
|
||||
<label for="cookie_jimeng" class="font-semibold text-slate-900 flex items-center gap-2">
|
||||
<span>即梦 AI (Jimeng)</span>
|
||||
</label>
|
||||
<span class="text-[11px] text-slate-400">后台键名: <code>cookie_jimeng</code> / 降级环境变量:
|
||||
<code>JIMENG_COOKIE</code></span>
|
||||
</div>
|
||||
<textarea id="cookie_jimeng" name="cookie_jimeng" rows="2" placeholder="格式例: sessionid=..."
|
||||
onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2 font-mono text-xs rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50 custom-scroll">{{ settings.get('cookie_jimeng', '') }}</textarea>
|
||||
<p class="text-[11px] text-slate-400">用于即梦 AI 生成内容的接口凭据支持。</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<!-- Tab 3: API 响应水印与版权提示 -->
|
||||
<div id="tab-content-branding" class="space-y-6 tab-panel" style="display: none;">
|
||||
{% set tip_enabled = (settings.get('api_tip_enabled', '1') == '1') %}
|
||||
<section class="bg-white rounded-xl border border-slate-200/80 p-5 sm:p-6 shadow-xs space-y-5">
|
||||
<div class="border-b border-slate-100 pb-3">
|
||||
<h2 class="text-sm font-bold text-slate-900 flex items-center gap-2">
|
||||
<span class="text-brand flex items-center justify-center">{{ icon('tag', 'w-4 h-4') }}</span>
|
||||
<span class="text-brand flex items-center justify-center">{{ icon('tag', 'w-4 h-4')
|
||||
}}</span>
|
||||
API 响应水印与版权提示
|
||||
</h2>
|
||||
</div>
|
||||
@@ -210,7 +376,8 @@
|
||||
<div class="flex items-center justify-between gap-4">
|
||||
<div>
|
||||
<div class="font-medium text-slate-900 text-sm">API 响应版权信息</div>
|
||||
<div class="text-xs text-slate-500 mt-0.5">开启后将在所有 API 解析接口响应的 JSON 尾部包含 _tip 元数据字段</div>
|
||||
<div class="text-xs text-slate-500 mt-0.5">开启后将在所有 API 解析接口响应的 JSON 尾部包含 _tip 元数据字段
|
||||
</div>
|
||||
</div>
|
||||
<div class="shrink-0">
|
||||
<label class="switch-toggle" title="切换 API 响应版权信息">
|
||||
@@ -245,7 +412,8 @@
|
||||
</div>
|
||||
|
||||
<div class="space-y-1.5">
|
||||
<label for="api_tip_notice" class="block text-xs font-semibold text-slate-700">提示文本</label>
|
||||
<label for="api_tip_notice"
|
||||
class="block text-xs font-semibold text-slate-700">提示文本</label>
|
||||
<input type="text" id="api_tip_notice" name="api_tip_notice"
|
||||
value="{{ settings.get('api_tip_notice', '本接口由开源项目 media-parser 提供服务') }}"
|
||||
placeholder="例如: 本接口由开源项目 media-parser 提供服务" oninput="updateApiTipPreview()"
|
||||
@@ -258,9 +426,9 @@
|
||||
<summary
|
||||
class="px-4 py-2.5 bg-slate-100/70 hover:bg-slate-100 border-b border-slate-200/80 cursor-pointer select-none font-sans font-semibold text-xs text-slate-700 flex items-center justify-between transition-colors">
|
||||
<span class="flex items-center gap-2">
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor"
|
||||
stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
|
||||
class="text-slate-500">
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="none"
|
||||
stroke="currentColor" stroke-width="2" stroke-linecap="round"
|
||||
stroke-linejoin="round" class="text-slate-500">
|
||||
<polyline points="16 18 22 12 16 6" />
|
||||
<polyline points="8 6 2 12 8 18" />
|
||||
</svg>
|
||||
@@ -275,13 +443,74 @@
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<!-- Tab 4: 网络与动态代理 -->
|
||||
<div id="tab-content-proxy" class="space-y-6 tab-panel" style="display: none;">
|
||||
<section class="bg-white rounded-xl border border-slate-200/80 p-5 sm:p-6 shadow-xs space-y-5">
|
||||
<div class="border-b border-slate-100 pb-3 flex items-center justify-between">
|
||||
<h2 class="text-sm font-bold text-slate-900 flex items-center gap-2">
|
||||
<span class="text-brand flex items-center justify-center">{{ icon('globe', 'w-4 h-4')
|
||||
}}</span>
|
||||
动态代理配置 (Proxy API)
|
||||
</h2>
|
||||
{% if settings.get('dynamic_proxy_api_url') %}
|
||||
<span
|
||||
class="inline-flex items-center gap-1.5 px-2.5 py-1 text-[11px] font-semibold text-emerald-700 bg-emerald-50 rounded-full border border-emerald-200">
|
||||
<span class="w-1.5 h-1.5 rounded-full bg-emerald-500 animate-pulse"></span>
|
||||
后台在线配置已生效 (覆盖 .env)
|
||||
</span>
|
||||
{% elif config.get('DYNAMIC_PROXY_API_URL') %}
|
||||
<span
|
||||
class="inline-flex items-center gap-1.5 px-2.5 py-1 text-[11px] font-semibold text-blue-700 bg-blue-50 rounded-full border border-blue-200">
|
||||
<span class="w-1.5 h-1.5 rounded-full bg-blue-500"></span>
|
||||
已启用来自 .env 的环境变量
|
||||
</span>
|
||||
{% else %}
|
||||
<span
|
||||
class="inline-flex items-center gap-1.5 px-2.5 py-1 text-[11px] font-semibold text-slate-500 bg-slate-100 rounded-full border border-slate-200">
|
||||
<span class="w-1.5 h-1.5 rounded-full bg-slate-400"></span>
|
||||
未配置 (默认直接连网解析)
|
||||
</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<div class="space-y-4">
|
||||
<div class="text-xs text-slate-500 leading-relaxed">
|
||||
当任意平台直连解析触发风控或需要 Cookie 校验失败时,系统将自动开启 15 分钟平台级代理窗口并提取代理重试。
|
||||
系统已内置智能通用解码器,原生兼容 <strong>芝麻代理、巨量IP、青果网络、无忧代理、多米代理</strong> 等主流服务商。
|
||||
</div>
|
||||
|
||||
<div class="space-y-1.5">
|
||||
<label for="dynamic_proxy_api_url" class="block text-xs font-semibold text-slate-700">
|
||||
动态代理 API 提取 URL
|
||||
</label>
|
||||
<input type="url" id="dynamic_proxy_api_url" name="dynamic_proxy_api_url"
|
||||
value="{{ settings.get('dynamic_proxy_api_url', '') }}"
|
||||
placeholder="例如: https://v2.api.juliangip.com/dynamic/getippro?trade_no=xxx&num=1&pt=1&result_type=json2&ip_remain=1"
|
||||
onchange="triggerSettingsSubmit(this)"
|
||||
class="w-full px-3.5 py-2.5 text-xs font-mono rounded-lg border border-slate-200 focus:outline-none focus:border-brand focus:ring-2 focus:ring-brand/20 transition bg-slate-50/50">
|
||||
</div>
|
||||
|
||||
<div
|
||||
class="p-3 bg-slate-50 rounded-lg border border-slate-100 text-[11px] text-slate-500 space-y-1.5">
|
||||
<div class="font-semibold text-slate-700">📌 优先级与生效规则说明:</div>
|
||||
<ul class="list-disc list-inside space-y-1 text-slate-600">
|
||||
<li><strong>三级优先级</strong>:后台在线配置 > <code>.env</code> 环境变量
|
||||
(<code>DYNAMIC_PROXY_API_URL</code>) > 留空不启用。</li>
|
||||
<li>修改此处的提取 URL 保存后<strong>即时生效,无需重启应用或 Docker 容器</strong>。</li>
|
||||
<li>若需取消后台覆盖恢复读取 <code>.env</code> 环境变量配置,只需清空上方输入框并保存即可。</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<script>
|
||||
window.switchSettingsTab = function (tabKey) {
|
||||
const tabs = ['access', 'branding'];
|
||||
const tabs = ['access', 'cookies', 'proxy', 'branding'];
|
||||
tabs.forEach(key => {
|
||||
const btn = document.getElementById('tab-btn-' + key);
|
||||
const panel = document.getElementById('tab-content-' + key);
|
||||
@@ -303,7 +532,7 @@
|
||||
});
|
||||
try {
|
||||
history.replaceState(null, '', '#tab=' + tabKey);
|
||||
} catch (e) {}
|
||||
} catch (e) { }
|
||||
};
|
||||
|
||||
window.toggleRegistrationDetails = function (checkbox) {
|
||||
@@ -496,7 +725,7 @@
|
||||
const hash = window.location.hash;
|
||||
if (hash.includes('tab=')) {
|
||||
const tabKey = hash.split('tab=')[1].trim();
|
||||
if (['access', 'branding'].includes(tabKey)) {
|
||||
if (['access', 'cookies', 'proxy', 'branding'].includes(tabKey)) {
|
||||
switchSettingsTab(tabKey);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -626,10 +626,10 @@ function initDonutInteractions() {
|
||||
centerLbl.style.color = color;
|
||||
|
||||
centerVal.textContent = `${calls} 次`;
|
||||
centerVal.style.fontSize = calls.length > 7 ? '18px' : '20px';
|
||||
centerVal.style.fontSize = calls.length > 7 ? '20px' : '24px';
|
||||
|
||||
if (centerSub) {
|
||||
centerSub.innerHTML = `<span style="color:#94a3b8;font-size:10.5px;font-weight:400">成功率</span> <span style="color:#475569;font-weight:600">${rate}</span>`;
|
||||
centerSub.innerHTML = `<span style="color:#94a3b8;font-size:11.5px;font-weight:400">成功率</span> <span style="color:#475569;font-weight:600">${rate}</span>`;
|
||||
}
|
||||
|
||||
slices.forEach(s => {
|
||||
|
||||
@@ -173,8 +173,8 @@
|
||||
<div>当前时间范围内暂无调用记录</div>
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="pie-wrapper" id="portal-platform-dist">
|
||||
<div class="pie-svg-box relative w-56 h-56 mx-auto">
|
||||
<div class="pie-wrapper flex flex-col h-full justify-between" id="portal-platform-dist">
|
||||
<div class="pie-svg-box relative w-64 h-64 sm:w-[260px] sm:h-[260px] mx-auto my-2">
|
||||
<svg viewBox="0 0 200 200" class="w-full h-full">
|
||||
{% for item in pie_data['items'] %}
|
||||
<path d="{{ item['path_d'] }}" fill="{{ item['color'] }}" class="pie-slice transition-opacity hover:opacity-80 cursor-pointer"
|
||||
@@ -188,17 +188,17 @@
|
||||
</path>
|
||||
{% endfor %}
|
||||
</svg>
|
||||
<div class="pie-center-text absolute inset-0 flex flex-col items-center justify-center pointer-events-none text-center select-none px-2">
|
||||
<div class="pie-center-lbl text-xs text-slate-500 font-medium truncate max-w-[140px] transition-colors duration-150" id="portal-pie-lbl">总解析量</div>
|
||||
<div class="pie-center-val text-xl font-bold text-slate-900 mt-0.5 tracking-tight transition-all duration-150" id="portal-pie-val">{{ pie_data['total_calls_formatted'] or pie_data['total_calls'] }} 次</div>
|
||||
<div class="pie-center-sub text-[11px] text-slate-400 mt-0.5 min-h-[16px] font-medium transition-colors duration-150" id="portal-pie-sub"></div>
|
||||
<div class="pie-center-text absolute inset-0 flex flex-col items-center justify-center pointer-events-none text-center select-none px-3">
|
||||
<div class="pie-center-lbl text-xs text-slate-500 font-semibold truncate max-w-[160px] transition-colors duration-150" id="portal-pie-lbl">总解析量</div>
|
||||
<div class="pie-center-val text-2xl font-bold text-slate-900 mt-1 tracking-tight transition-all duration-150" id="portal-pie-val">{{ pie_data['total_calls_formatted'] or pie_data['total_calls'] }} 次</div>
|
||||
<div class="pie-center-sub text-xs text-slate-400 mt-0.5 min-h-[18px] font-medium transition-colors duration-150" id="portal-pie-sub"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 紧凑点状图例 -->
|
||||
<div class="flex flex-wrap items-center justify-center gap-x-1.5 gap-y-1 mt-5 pt-3 border-t border-slate-100">
|
||||
<div class="flex flex-wrap items-center justify-center gap-x-2 gap-y-1.5 mt-5 pt-4 border-t border-slate-100">
|
||||
{% for item in pie_data['items'] %}
|
||||
<div class="pie-pill-item inline-flex items-center gap-1 px-1.5 py-0.5 rounded-md hover:bg-slate-100 cursor-pointer transition text-xs font-medium text-slate-600 hover:text-slate-900 select-none"
|
||||
<div class="pie-pill-item inline-flex items-center gap-1.5 px-2 py-1 rounded-lg hover:bg-slate-100 cursor-pointer transition text-[12px] font-medium text-slate-600 hover:text-slate-900 select-none"
|
||||
data-platform="{{ item['platform'] }}"
|
||||
data-calls="{{ item['calls_formatted'] }}"
|
||||
data-percent="{{ item['percentage'] }}%"
|
||||
@@ -206,7 +206,7 @@
|
||||
data-failures="{{ item['failures_formatted'] }}"
|
||||
data-rate="{{ item['success_rate'] }}%"
|
||||
data-color="{{ item['color'] }}">
|
||||
<span class="w-2 h-2 rounded-full shrink-0" style="background:{{ item['color'] }}"></span>
|
||||
<span class="w-2.5 h-2.5 rounded-full shrink-0" style="background:{{ item['color'] }}"></span>
|
||||
<span class="truncate">{{ item['platform'] }}</span>
|
||||
</div>
|
||||
{% endfor %}
|
||||
|
||||
@@ -26,6 +26,18 @@ class CookieManagerTest(unittest.TestCase):
|
||||
cookie = get_platform_cookie("nonexistent_platform")
|
||||
self.assertEqual(cookie, "")
|
||||
|
||||
def test_db_setting_overrides_env_var(self):
|
||||
with patch("src.db.setting", side_effect=lambda k: "hybrid_sess=from_db;" if k == "cookie_wechat_channels" else None):
|
||||
with patch.dict(os.environ, {"YUANBAO_COOKIE": "pass_ticket=from_env;"}):
|
||||
cookie = get_platform_cookie("wechat_channels")
|
||||
self.assertEqual(cookie, "hybrid_sess=from_db;")
|
||||
|
||||
def test_db_setting_empty_falls_back_to_env_var(self):
|
||||
with patch("src.db.setting", return_value=""):
|
||||
with patch.dict(os.environ, {"YUANBAO_COOKIE": "pass_ticket=from_env;"}):
|
||||
cookie = get_platform_cookie("wechat_channels")
|
||||
self.assertEqual(cookie, "pass_ticket=from_env;")
|
||||
|
||||
def test_supported_platform_aliases(self):
|
||||
self.assertIn("xhs", PLATFORM_COOKIE_ALIASES)
|
||||
self.assertIn("douyin", PLATFORM_COOKIE_ALIASES)
|
||||
|
||||
@@ -241,7 +241,7 @@ class RetryIntegrationTest(unittest.TestCase):
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.app = create_app({'TESTING': True, 'API_ONLY': True, 'SECRET_KEY': 'test',
|
||||
'DATABASE': os.path.join(self.temp.name, 'test.db'),
|
||||
'JULIANG_PROXY_API_URL': 'http://provider.invalid'})
|
||||
'DYNAMIC_PROXY_API_URL': 'http://provider.invalid'})
|
||||
self.manager = self.app.extensions['proxy_manager']
|
||||
self.client = self.app.test_client()
|
||||
self.url = 'https://v.kuaishou.com/test'
|
||||
@@ -441,3 +441,27 @@ class RetryIntegrationTest(unittest.TestCase):
|
||||
self.assertEqual(response.status_code, 400)
|
||||
self.assertEqual(response.json['error_code'], 'KUAISHOU_COOKIE_REQUIRED')
|
||||
self.assertEqual(factory.call_count, 3)
|
||||
|
||||
def test_universal_decoders(self):
|
||||
start = time.time()
|
||||
# 1. 巨量 IP (Juliang)
|
||||
p1 = ProxyManager.decode({'code': 200, 'data': {'proxy_list': [{'ip': '1.2.3.4', 'port': 8080, 'ip_remain': 120}]}}, start)
|
||||
self.assertEqual(p1.address, '1.2.3.4:8080')
|
||||
self.assertAlmostEqual(p1.expires_at - start, 120, delta=1)
|
||||
|
||||
# 2. 芝麻代理 (Zhima)
|
||||
p2 = ProxyManager.decode({'code': 0, 'data': [{'ip': '5.6.7.8', 'port': 9090, 'expire_time': '2026-09-30 20:00:00'}]}, start)
|
||||
self.assertEqual(p2.address, '5.6.7.8:9090')
|
||||
|
||||
# 3. 青果网络 (Qingguo)
|
||||
p3 = ProxyManager.decode({'code': 'SUCCESS', 'data': [{'ip': '9.10.11.12', 'port': 7070, 'ttl': 300}]}, start)
|
||||
self.assertEqual(p3.address, '9.10.11.12:7070')
|
||||
|
||||
# 4. Data5U / 无忧代理
|
||||
p4 = ProxyManager.decode({'success': True, 'data': [{'ip': '13.14.15.16', 'port': 6060}]}, start)
|
||||
self.assertEqual(p4.address, '13.14.15.16:6060')
|
||||
|
||||
# 5. 纯文本 TXT
|
||||
p5 = ProxyManager.decode("17.18.19.20:5050\n1.1.1.1:80", start)
|
||||
self.assertEqual(p5.address, '17.18.19.20:5050')
|
||||
|
||||
|
||||
@@ -19,6 +19,12 @@ class WeiboParserTest(unittest.TestCase):
|
||||
parser.real_url = "https://weibo.com/tv/show/1034:5336275486703690"
|
||||
self.assertEqual(parser._extract_video_oid(), "1034:5336275486703690")
|
||||
|
||||
def test_extracts_alphanumeric_video_oid_and_id(self):
|
||||
parser = WeiboParser.__new__(WeiboParser)
|
||||
parser.real_url = "https://video.weibo.com/show?fid=1034:cba52f9b57b69ca1a0cde0c2211e1ca1"
|
||||
self.assertEqual(parser._extract_video_oid(), "1034:cba52f9b57b69ca1a0cde0c2211e1ca1")
|
||||
self.assertEqual(parser._extract_id(), "cba52f9b57b69ca1a0cde0c2211e1ca1")
|
||||
|
||||
def test_extracts_numeric_id_from_pc_numeric_url(self):
|
||||
parser = WeiboParser.__new__(WeiboParser)
|
||||
parser.real_url = "https://weibo.com/7928442102/5331959570240710"
|
||||
|
||||
Reference in New Issue
Block a user