Skip to content

fix: buffer CLI stdin before format detection on Windows - #2351

Merged
Adam Fourney (afourney) merged 5 commits into
microsoft:mainfrom
mohui666:fix/1209-windows-stdin
Sep 2, 2026
Merged

Adam Fourney (afourney) merged 5 commits into
microsoft:mainfrom
mohui666:fix/1209-windows-stdin

Conversation

@mohui666

Copy link
Copy Markdown
Contributor

Fixes #1209

What changed

  • buffer CLI stdin into BytesIO before conversion so format probing can rewind it on Windows
  • make the existing stdin vector test actually exercise stdin
  • add a regression for a pipe that reports seekable but cannot rewind
  • run CLI subprocess tests with the active interpreter and read UTF-8 output as UTF-8

Root cause

On Windows, pipe-backed sys.stdin.buffer can report seekable() == True even though seek(0) does not rewind it. Magika consumes the pipe during format detection, leaving the converter at EOF. This is why the CLI returned an empty result even for ASCII HTML.

Verification

  • real PowerShell pipe: HTML piped to python -m markitdown -x html produces # Test HTML
  • direct raw-byte subprocess stdin: return code 0, expected stdout, empty stderr
  • targeted stdin regressions: 13 passed
  • CLI tests: 41 passed, 12 skipped
  • package tests: 307 passed, 33 skipped, with the unchanged Windows-only URI path assertion deselected
  • Black, compileall, and git diff --check pass

The only native-Windows full-suite failure is the pre-existing test_file_uris assertion, which expects the Unix path /path/to/file.txt instead of Windows C:\path\to\file.txt.

@mohui666

Copy link
Copy Markdown
Contributor Author

Mo Hui (Mo Hui (@mohui666))请阅读以下贡献者许可协议(CLA)。如果您同意该 CLA,请回复以下信息。

@microsoft-github-policy-service agree [company="{your company}"]

选项:

  • (默认 - 未指定公司)我拥有我提交作品的全部知识产权,并且我不是在为我的雇主工作期间提交作品。
@microsoft-github-policy-service agree
  • (如适用)本人提交作品系为雇主工作所致(或雇主根据合同或适用法律对本人提交的作品拥有知识产权)。本人已获得雇主许可,代表雇主提交作品并签署本协议。下方签名中的“您”一词包含本人及雇主。
@microsoft-github-policy-service agree company="Microsoft"

贡献者许可协议

@microsoft-github-policy-service agree [company="{your company}"]

@mohui666

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The Windows stdin failure is addressed with appropriate regression coverage; only a minor duplicate import remains.

Pull request overview

Buffers CLI stdin before conversion to ensure reliable format detection and rewinding on Windows.

Changes:

  • Buffers stdin using BytesIO.
  • Corrects stdin test execution and adds a Windows pipe regression test.
  • Uses the active interpreter and explicit UTF-8 file decoding.
File summaries
File Description
packages/markitdown/src/markitdown/__main__.py Buffers stdin before conversion.
packages/markitdown/tests/test_cli_misc.py Adds Windows pipe regression coverage.
packages/markitdown/tests/test_cli_vectors.py Corrects stdin and subprocess test behavior.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/markitdown/src/markitdown/__main__.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation addresses the Windows rewind failure and includes appropriate regression coverage.

Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@afourney
Adam Fourney (afourney) merged commit bebea8b into microsoft:main Sep 2, 2026
3 checks passed
@mohui666
Mo Hui (mohui666) deleted the fix/1209-windows-stdin branch September 3, 2026 05:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Basic HTML conversion via stdin not working on Windows

3 participants