fix: convert CSV fields above the parser limit

[KennySimpson (KennyMcSimpson)]

A CSV field longer than Python's parser limit currently makes CsvConverter fail. Through the public MarkItDown API, conversion then falls back to plain text, so a valid CSV loses its Markdown table. With the standard limit, this happens at 131,073 characters.

Keep the existing limit for the first parsing attempt. If parsing raises csv.Error and the decoded input is longer than the current limit, retry with the limit bounded by the input length and restore the previous setting in finally. A module lock serializes CSV parsing across converter instances so their retries cannot overwrite each other's saved limit.

csv.field_size_limit is process-wide. The lock coordinates CsvConverter instances only: unrelated CSV readers can observe the temporary limit, and independent changes to that setting are not synchronized.

Validation

python -m pytest packages/markitdown/tests/test_csv.py -q
pre-commit run --all-files

44 tests passed, including public API checks at the standard limit, one character above it, and a long Unicode field. They also check that the previous parser limit is restored, including two simultaneous conversions with separate converter instances. The converter docstring documents the process-wide setting and the lock's scope.

Separate checks passed for custom limits, quoted multiline fields, and 36 concurrent public API conversions. The full multi-format test suite was not run.

[KennySimpson (KennyMcSimpson)]

@microsoft-github-policy-service agree

[PRABHU KIRAN VANDRANKI (VANDRANKI)]

Community review. This does not clear the merge gate.

I checked out the branch (Python 3.13) and ran tests/test_csv.py: 43 passed. I also converted CSVs through MarkItDown(...).convert_stream with a 200,000-character field and a 5,000,000-character field, and ran a second thread that polled csv.field_size_limit() during the large conversion.

The fix works for the stated case: fields above the default 131072 limit now convert, and the limit is back at 131072 afterwards (single-threaded).

The main concern is the approach, not the result. csv.field_size_limit is process-wide, and the lock only serializes MarkItDown's own callers. In my thread test, the other thread saw the limit at 5,000,007 while the conversion ran, so any other code in the process using csv (the host application, another library) briefly runs with a raised limit, which is the guard against pathological fields they were relying on. The restore in finally also writes back a stale original_limit: if another thread calls csv.field_size_limit(n) between the set and the restore, its value is overwritten. A library changing a global like this is something I would want a maintainer to decide on explicitly.

Alternatives that avoid the global:

• Parse without csv.reader for the over-limit case, for example csv.Sniffer-free manual splitting is awkward, so a small pure-Python RFC 4180 reader is one option.
• Use pandas.read_csv(..., engine="python") or the C engine, which have no csv module limit (pandas is already a dependency for xlsx, but I did not check whether it is for the base markitdown install).
• Keep the global change but document it in the converter docstring and add a test that two threads converting at once both pass.

Smaller points:

• The error path does a full parse twice for oversized content: once to discover the failure, again after the limit is raised. For a multi-MB file that is two passes over all rows.
• if len(content) <= original_limit: raise correctly avoids retrying when the failure is something else. I checked a NUL byte in a field and a long unterminated quoted field. Neither raised, so I could not exercise that branch.
• The new test covers exact limit, limit plus one and non-ASCII, and asserts the limit is restored. There is no concurrency test.