Skip to content

feat(pdf): refactor MinerU parsing to the official file_parse API - #3953

Open
zonas0574 wants to merge 1 commit into
volcengine:mainfrom
zonas0574:feature/mineru-file-parse-api
Open

feat(pdf): refactor MinerU parsing to the official file_parse API#3953
zonas0574 wants to merge 1 commit into
volcengine:mainfrom
zonas0574:feature/mineru-file-parse-api

Conversation

@zonas0574

@zonas0574 zonas0574 commented Aug 12, 2026

Copy link
Copy Markdown

Description

支持最新本地mineru部署的解析API

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

#3910

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • 适配官方file_parse统计解析接口
  • 支持解析后图片入库

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

Additional Notes

不支持官方云平台https://mineru.net/apiManage/docs (只有异步接口),仅支持开源本地部署版本

@MaojiaSheng

Copy link
Copy Markdown
Collaborator

thanks, our team will review this

"pdf": {
"strategy": "auto",
"mineru_endpoint": "http://127.0.0.1:8000",
"mineru_api_key": "your-api-key",

@Tsan1024 Tsan1024 Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里方便提供下使用的是哪个版本的开源镜像服务吗?MinerU 3.4.4 提到的鉴权是MinerU -> vLLM 的下游鉴权,这里的api-key 是vllm部分的输入?

@zonas0574 zonas0574 Aug 13, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对是3.4.4,但这个其实目前没有用,我看之前在我也没删就保留,要不我先删了 = =

@Tsan1024 Tsan1024 Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对是3.4.4,但这个其实目前没有用,我看之前在我也没删就保留,要不我先删了 = =

之前的应该是有开发者支持了一版调用mineru官方api的方式,里面会有apikey存在。可以看看如何兼容官方api和自运营api方式哈,比如加个字段区分?

@Tsan1024 Tsan1024 Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

当 PDF 解析策略为 mineru 时,MinerU 是必需依赖;当策略为 auto 且配置了
mineru_endpoint 时,它也承担本地解析失败后的回退职责。目前在实际解析 PDF 时才
调用 /file_parse,因此 endpoint 配置错误、服务未启动或协议不兼容可能会延后到运
行期才发现。

这里是否可以考虑在 OpenViking 初始化完成前增加一个 MinerU 启动预检?例如:

  • 对 {mineru_endpoint}/health 做一次短时轮询。
  • 收到 HTTP 200、合法 JSON 且包含 protocol_version 时视为就绪。
  • strategy="mineru",或 strategy="auto" 且配置了 mineru_endpoint 时执行。
  • strategy="local",以及未配置 endpoint 的 auto 模式可跳过。
  • 若预检超时,启动时给出包含 endpoint、最后错误和修复建议的提示。

这样可以更早发现依赖配置问题,也避免首个 PDF 导入任务才暴露错误

示例

  import asyncio
  import time

  import httpx


  async def wait_for_mineru_ready(endpoint: str, timeout: float = 5.0) -> None:
      health_url = f"{endpoint.rstrip('/')}/health"
      deadline = time.monotonic() + timeout
      last_error = "no response"

      async with httpx.AsyncClient(timeout=0.5) as client:
          while time.monotonic() < deadline:
              try:
                  response = await client.get(health_url)
                  payload = response.json()

                  if (
                      response.status_code == 200
                      and isinstance(payload, dict)
                      and payload.get("protocol_version")
                  ):
                      return

                  last_error = (
                      f"unexpected response: HTTP {response.status_code}, "
                      f"protocol_version={payload.get('protocol_version')!r}"
                  )
              except (httpx.HTTPError, ValueError) as exc:
                  last_error = str(exc)

              await asyncio.sleep(0.2)

      raise RuntimeError(
          f"MinerU startup preflight failed for {health_url} after {timeout}s. "
          f"Last error: {last_error}. "
          "Start a compatible mineru-api service or correct
          pdf.mineru_endpoint."
      )

在服务初始化流程中、设置 _initialized = True 前调用:

  pdf_config = self._config.pdf

  should_preflight_mineru = (
      pdf_config.strategy == "mineru"
      or (
          pdf_config.strategy == "auto"
          and pdf_config.mineru_endpoint is not None
      )
  )

  if should_preflight_mineru and pdf_config.mineru_endpoint:
      await wait_for_mineru_ready(pdf_config.mineru_endpoint)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

3 participants