Python File Run Unexpected Character After Line Continuation Character
Created on 2021-11-06 13:10 by pewscorner, last changed 2022-04-11 14:59 by admin. This issue is now closed.
When exec'ing code with an illegal character after the line continuation character '\', python 3.11.0a2 raises this strange exception: SystemError: Negative size passed to PyUnicode_New Here's an example where '#' is the illegal character: exec('1,\\#\n2') I expected the following exception (which is raised by Python 3.10.0, and also by Python 3.11.0a2 when using eval instead of exec, or when the string content is put into its own file and run as a normal script): SyntaxError: unexpected character after line continuation character It looks like this also affects Python 3.9.8, which makes me very suspicious of https://bugs.python.org/issue45494 as the probable cause. See https://github.com/Zac-HD/hypothesmith/issues/16 and https://github.com/psf/black/pull/2592#issuecomment-966745240 for reports.
Yep, this also break IPython command completions when users press tab in some places in multiline code. 88f4ec88e282bf861f0af2d237e9fe28fbc8deac is the first offending commit for me in the 3.9 branch: [3.9] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) (#29071) There are two errors that this commit fixes: * The parser was not correctly computing the offset and the string source for E_LINECONT errors due to the incorrect usage of strtok(). * The parser was not correctly unwinding the call stack when a tokenizer exception happened in rules involving optionals ('?', [...]) as we always make them return valid results by using the comma operator. We need to check first if we don't have an error before continuing.. (cherry picked from commit a106343f632a99c8ebb0136fa140cf189b4a6a57) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> NOTE: unlike the cherry-picked original, this commit points at a crazy location due to a bug in the tokenizer that required a big refactor in 3.10 to fix. We are leaving as-is for 3.9. Note that with the following example, exec and CommandCompiler return a different error: from codeop import CommandCompiler # identical errors #exec('1,\\#\n2') SystemError #CommandCompiler()('1,\\#\n2', symbol='exec') SystemError # one is syntax the other is System. exec('a \ \n') # SyntaxError CommandCompiler()('a \ \n', symbol='exec') # SystemError 
I will be able to take a look later in the weekend, so if someone is able to give it a go before, that would be great.
> I will be able to take a look later in the weekend Please take some time for yourself, I added you as you were the original author of the patch to make you aware. I believe most projects can workaround that for now.
New changeset 25835c518aa7446f3680b62c1fb43827e0f190d9 by Pablo Galindo Salgado in branch 'main': bpo-45738: Fix computation of error location for invalid continuation (GH-29550) https://github.com/python/cpython/commit/25835c518aa7446f3680b62c1fb43827e0f190d9
New changeset bf26a6da7aaedb526c9eb1cb56b0e46d1c10384c by Miss Islington (bot) in branch '3.10': bpo-45738: Fix computation of error location for invalid continuation (GH-29550) https://github.com/python/cpython/commit/bf26a6da7aaedb526c9eb1cb56b0e46d1c10384c
This should be fixed by now, but please, check if everything is in order in your respective test suites and confirm here.
New changeset 142fcb40b6e460fa9b4a89fe9846b1ce4176354e by Pablo Galindo Salgado in branch '3.9': bpo-45738: Fix computation of error location for invalid continuation characters in the parser (GH-29550) (GH-29552) https://github.com/python/cpython/commit/142fcb40b6e460fa9b4a89fe9846b1ce4176354e
Seems to work. Thanks!
Works for me as well. Thanks.
resolution: fixed
stage: patch review -> resolved
pull_requests: + pull_request27800
stage: patch review
pull_requests: + pull_request27799
messages: + msg406223
messages: + msg406188
versions: + Python 3.9
macrossanbeent1968.blogspot.com
Source: https://bugs.python.org/issue45738