Skip to content

fix: preserve Markdown hard line breaks - #2559

Open
Maciej Zmitrukiewicz (MaciejZet) wants to merge 2 commits into
microsoft:mainfrom
MaciejZet:fix/preserve-markdown-hard-breaks
Open

Maciej Zmitrukiewicz (MaciejZet) wants to merge 2 commits into
microsoft:mainfrom
MaciejZet:fix/preserve-markdown-hard-breaks

Conversation

@MaciejZet

@MaciejZet Maciej Zmitrukiewicz (MaciejZet) commented Sep 27, 2026 •

Copy link
Copy Markdown

Markdown input loses two-space hard breaks during final whitespace normalization. Preserve two or more trailing spaces on nonblank lines when PlainTextConverter handles input identified as Markdown by its extension (.md, .markdown) or MIME type (text/markdown, application/markdown).

Other formats and converter implementations retain their existing normalization. Single trailing spaces, tabs, whitespace-only lines, and blank-line collapsing are unchanged. This does not provide full whitespace preservation inside code blocks.

Related: #2643 addresses breaks generated from HTML. This PR preserves hard-break syntax already present in Markdown input.

Validation

  • Local core suite on Python 3.12.13 with GITHUB_ACTIONS=1: 1080 passed, 57 skipped. Skips include external-service tests and native Windows checks.
  • Focused API, format-detection, CLI, and normalization regressions: 30 passed.
  • Identical-input comparison: all 25 HTML, DOCX, and plain-text cases match upstream byte for byte; 12 Markdown cases retain the intended behavior.
  • Black 23.7.0 and git diff --check: pass.

@MaciejZet

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@MaciejZet

Copy link
Copy Markdown
Author

I tested this PR (75faa08) together with #2643 (7c88cb2) in a temporary checkout. The patches applied without conflicts. The initial core run had five failures, all in expected outputs added by this PR:

  • HTML with one <br>.
  • HTML with two consecutive <br> elements.
  • HTML with a .md extension and text/html MIME type.
  • DOCX with one <w:br/>.
  • DOCX with two consecutive <w:br/> elements.

These tests expect plain newlines or a paragraph break; #2643 intentionally produces backslash hard breaks. Updating those five expectations and the test names, with no further production-code changes, gave 1091 passed, 57 skipped on Python 3.12. The OCR suite also passed 108 tests, and all 37 comparison cases matched the expected combined behavior.

The patches appear compatible in this tested scope: this PR preserves two-space hard breaks in Markdown input, while #2643 handles generated breaks. Some nested <br> cases remain unresolved in #2643.

If #2643 lands, I can update these five expectations here to match its output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant