sys.argv is wrong for unicode strings

[GiovanniBajo]

BPO	2128
Nosy	@loewis, @vstinner, @tiran, @benjaminp
Files	argv_unicode.patch wchar.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-04-05.20:42:42.184>
created_at = <Date 2008-02-16.16:27:45.539>
labels = ['interpreter-core', 'type-bug', 'OS-windows']
title = 'sys.argv is wrong for unicode strings'
updated_at = <Date 2013-01-14.09:15:31.925>
user = 'https://bugs.python.org/giovannibajo'

bugs.python.org fields:

activity = <Date 2013-01-14.09:15:31.925>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2008-04-05.20:42:42.184>
closer = 'loewis'
components = ['Interpreter Core', 'Windows']
creation = <Date 2008-02-16.16:27:45.539>
creator = 'giovannibajo'
dependencies = []
files = ['9449', '9647']
hgrepos = []
issue_num = 2128
keywords = ['patch']
message_count = 15.0
messages = ['62458', '62460', '62499', '62659', '62660', '62664', '63443', '65005', '65045', '65061', '65073', '125827', '125829', '179892', '179928']
nosy_count = 7.0
nosy_names = ['loewis', 'vstinner', 'christian.heimes', 'giovannibajo', 'benjamin.peterson', 'davidsarah', 'mherrmann.at']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2128'
versions = ['Python 2.6', 'Python 2.5', 'Python 3.0', 'Python 2.7']

[GiovanniBajo]

Under Windows, sys.argv is created through the Windows ANSI API.

When you have a file/directory which can't be represented in the
system encoding (eg: a japanese-named file or directory on a Western
Windows), Windows will encode the filename to the system encoding using
what we call the "replace" policy, and thus sys.argv[] will contain an
entry like "c:\\foo\\??????????????.dat".

My suggestion is that:

• At the Python level, we still expose a single sys.argv[], which will
  contain unicode strings. I think this exactly matches what Py3k does now.

• At the C level, I believe it involves using GetCommandLineW() and
  CommandLineToArgvW() in WinMain.c, but should Py_Main/PySys_SetArgv() be
  changed to also accept wchar_t** arguments? Or is it better to allow for
  NULL to be passed (under Windows at least), so that the Windows
  code-path in there can use GetCommandLineW()/CommandLineToArgvW() to get
  the current process' arguments?

[tiran]

The issue is related to bpo-1342

Since we have dropped support for older versions of Windows (9x, ME,
NT4) I like to get the Python interface to argv, env and files fixed.

[GiovanniBajo]

I'm attaching a simple patch that seems to work under Py3k. The trick is
that Py3k already attempts (not sure how or why) to decode argv using
utf-8. So it's sufficient to setup argv as UTF8-encoded strings.

Notice that brings the output of "python ààààà" from this:

Fatal Python error: no mem for sys.argv
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2:
invalid data

to this:

TypeError: zipimporter() argument 1 must be string without null bytes,
not str

which is expected since zipimporter_init() doesn't even know to ignore
unicode strings (let alone handle them correctly...).

[loewis]

I dislike the double decoding, and would prefer if sys.argv would be
created directly from the wide command line.

In addition, I think the patch is incorrect: it ignores the arguments to
Py_Main, which is a documented API function.

One solution might be to declare all these functions (Py_Main,
SetProgramName, GetArgcArgv) to operate on Py_UNICODE*, and then
convert the POSIX callers of Py_Main to use mbstowcs when going
from the command line to Py_Main. WinMain could then become
recompiled for Unicode directly, likewise Modules/python.c

[GiovanniBajo]

mbstowcs uses LC_CTYPE. Is that correct and consistent with the way
default encoding under UNIX is handled by Py3k?

Would a Py_MainW or similar wrapper be easier on the UNIX guys? I'm just
asking, I don't have a definite idea.

[loewis]

mbstowcs uses LC_CTYPE. Is that correct and consistent with the way
default encoding under UNIX is handled by Py3k?

It's correct, but it's not consistent with the default encoding - there
isn't really any default encoding in Py3k. More specifically,
PyUnicode_FromString uses UTF-8, but not as a (changeable) default,
but as part of its API specification.
Command line arguments are in the locale's charset, so the LC_CTYPE
must be used to convert them.

Would a Py_MainW or similar wrapper be easier on the UNIX guys? I'm just
asking, I don't have a definite idea.

See above. The current POSIX implementation is incorrect also. It should
use the locale's encoding, but doesn't.

[loewis]

Here is a patch that redoes the entire argv handling, in terms of
wchar_t. As a side effect, it also changes the sys.path handling to use
wchar_t.

[loewis]

This is now fixed in r62178 for Py3k. For 2.6, I don't think fixing it
is feasible.

[benjaminp]

MvL's recent commit creates compiler warnings for Unicode UCS4 for the
same reason as bpo-2388.

[loewis]

What warnings precisely are you seeing? I didn't see anything in the 3k
branch (not even for bpo-2388, as PyErr_Format doesn't have the GCC format
attribute in 3k, unlike 2.x).

[benjaminp]

Martin, you are right that they are not from the same reason as that issue.

gcc -c -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk/
-fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
-I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/main.o Modules/main.c
Modules/main.c: In function 'Py_Main':
Modules/main.c:478: warning: passing argument 1 of 'Py_SetProgramName'
from incompatible pointer type
Modules/main.c: In function 'Py_Main':
Modules/main.c:478: warning: passing argument 1 of 'Py_SetProgramName'
from incompatible pointer type