From f3d05bba525c048d8ca905831a7729d7f90c94bc Mon Sep 17 00:00:00 2001 From: Andrew Clayton Date: Thu, 5 Jan 2023 22:04:32 +0000 Subject: Python: Fix enabling of UTF-8 in some situations. There was a couple of reports of Python applications failing due to the following type of error File "/opt/netbox/netbox/netbox/configuration.py", line 25, in _import print(f"\U0001f9ec loaded config '{path}'") UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f9ec' in position 0: ordinal not in range(128) due to the use of Unicode text in the print() statement. This only happened for python 3.8+ when using the "home" configuration option as this meant we were going through the new PyConfig configuration. When using this new configuration method with the 'isolated' specific API (for embedded Python) UTF-8 is disabled by default, PyPreConfig->utf8_mode = 0. To fix this we need to setup the Python pre config and enable utf-8 mode. However rather than enable utf-8 unconditionally we can set to it to -1 so that it will use the LC_CTYPE environment variable to determine whether to enable utf-8 mode or not. utf-8 mode will be enabled if LC_CTYPE is either: C, POSIX or some specific UTF-8 locale. This is the default utf8_mode setting when using the non-isolated PyPreConfig API. Reported-by: Tobias Genannt Tested-by: Tobias Genannt Link: Link: Fixes: 491d0f70 ("Python: Added support for Python 3.11.") Closes: Reviewed-by: Alejandro Colomar Signed-off-by: Andrew Clayton --- src/python/nxt_python.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) (limited to 'src/python/nxt_python.c') diff --git a/src/python/nxt_python.c b/src/python/nxt_python.c index 83f2544e..7c059649 100644 --- a/src/python/nxt_python.c +++ b/src/python/nxt_python.c @@ -78,9 +78,23 @@ nxt_python3_init_config(nxt_int_t pep405) PyConfig config; PyStatus status; nxt_int_t ret; + PyPreConfig preconfig; ret = NXT_ERROR; + PyPreConfig_InitIsolatedConfig(&preconfig); + /* + * Determine whether to use UTF-8 mode or not, UTF-8 + * will be enabled if LC_CTYPE is C, POSIX or some + * specific UTF-8 locale. + */ + preconfig.utf8_mode = -1; + + status = Py_PreInitialize(&preconfig); + if (PyStatus_Exception(status)) { + return ret; + } + PyConfig_InitIsolatedConfig(&config); if (pep405) { -- cgit