Mitigate UnicodeDecodeError due to wrong ASCII charset guess for long files

[Fabio Catalano (fcatalan92)]

This PR mitigate UnicodeDecodeError issues encountered when a non-ASCII character appears late in the file, namely after the window used to guess the charset.

The changes are the following:

• prevent IpynbConverter.accepts() from failing when a file cannot be decoded with the guessed charset. I implemented the same solution used in outlook_msg_converter
• increase the charset-detection sample from 4 KB to 64 KB, making the issue unlikely to occur
• add guidance to conversion errors caused by an incorrectly guessed charset

Another possibility is to force a retry if a UnicodeDecodeError is encountered in the converter using the same approach used when the detected charset is None. But I preferred to not change the current behavior of the converters.

markitdown/packages/markitdown/src/markitdown/converters/_plain_text_converter.py

Lines 66 to 69 in 9dc0d65

	if stream_info.charset:
	text_content = file_stream.read().decode(stream_info.charset)
	else:
	text_content = str(from_bytes(file_stream.read()).best())

The PR fixes #2359. It could also address #1505, #1894 and #1949 (for which there are no working reproducers).

Adam Fourney (@afourney) I hope you could have a look at this human-generated contribution ;-)

[Fabio Catalano (fcatalan92)]

@microsoft-github-policy-service agree

[Adam Fourney (afourney)]

Thanks! Merged!