Review CPython patches / tests and contribute upstream

[rth]

There is currently an active effort for adding wasm as the build target to CPython in https://bugs.python.org/issue40280 by Christian Heimes. So it would be an ideal timing to review again our CPython patches and failing tests and re-evaluate what could be contributed upstream.

(Other things were already proposed upstream when possible).

[tiran]

Hi,

I took some inspiration from your patch set. :)

• The patch files 0005-add-emscripten-host.patch and 0006-fix-Py_Sigset_Converter.patch should no longer be needed. I did not apply your patches directly, but I gave you credit.
• The patch remove-duplicate-symbols-from-cfield.c.patch is causing issues on Windows, https://bugs.python.org/issue45898 .
• Patch 0001-disable-set_inheritable.patch does not explain why the change is necessary. I would like to understand the reason for the patch first before we land it in upstream.
• I created a CONFIG_SITE override, which is IMHO a cleaner approach than your pyconfig.undefs.h.
• Eventually we will be able to replace the custom Setup.local with https://github.com/python/cpython/blob/main/Modules/Setup.stdlib.in and full dependency detection in configure. There are still some modules missing from the file, though.

[emmatyping]

I've created a repo with my build scripts to build CPython (for now for emscripten but I hope to add work on WASI eventually...): https://github.com/ethanhs/python-wasm (based on @tiran's config.site and my own attempts).

I am happy to say I've successfully gotten CPython master running in the browser with emscripten!

The main trick up my sleeve is emcc -o python.html Programs/python.o libpython3.11d.a -ldl -lm --preload-file ., which loads the local build tree and the JS Emscripten system shims.

Lots of things don't work though, like help() :P

[brettcannon]

My hope is we eventually centralize all of this WASM build knowledge somewhere so that Pyodide can focus more on the "build the scientific stack" bit and the "build CPython for WASM/WASI" becomes a shared problem.

But obviously getting this all working upstream is the first step. 😅

[rth]

Sounds great @tiran ! Thanks for writing this summary.

Patch 0001-disable-set_inheritable.patch does not explain why the change is necessary. I would like to understand the reason for the patch first before we land it in upstream.

It was added when the project started in 2018, with little documentation, difficult to determine exactly why now. Currently removing it works fine #2003 so it's probably something that was necessary for previous versions of Emecripten or CPython.

I've created a repo with my build scripts to build CPython

Great work @ethanhs !

[rth]

My hope is we eventually centralize all of this WASM build knowledge somewhere so that Pyodide can focus more on the "build the scientific stack" bit and the "build CPython for WASM/WASI" becomes a shared problem.

Looking forward to this @brettcannon ! For information, currently, we are not necessarily focusing only on scientific computing. Other areas which constitute most of the Pyodide code base, are:

• Python/JavaScript type translations / FFI. This is maybe less relevant for WASI, but when running in the browser one needs some way to communicate between the Python interpreter and the web page where it's running.
• Building packages with C extensions. This is relevant for scientific computing but not only. Currently, we are using this hack which works surprisingly well, but clearly, a better cross-compilation story would be preferrable (Cross-compile packages #819). This is also related to distutils deprecation and its replacement.
• Trying to compensate for the limitations of the WebAssembly VM in the browser which might mean re-implementing some of the stdlib functionality with Web APIs (e.g. Write http.client in terms of Web APIs #140 which in turn requires to somehow convert async JS calls into sync Python calls Synchronous IO #1503). This is also related to passing the CPython and packages test suites. There is lots of work that could be done there.
• There are still open questions for package distribution. Right now we use the emscripten package format + pure Python wheels from PyPi with micropip. One could argue that WASM is just another build target, that one could do binary wheel Add support for wasm32 platform pypi/warehouse#10416 . This would certainly be a step forward, however, there are also lots of specifics in fetching files for a web-page as opposed to for a local install. For instance, currently, to reduce download size we are,
  • extracting tests and re-exporting them as a separate package. So there are for instance, numpy and numpy-test packages.
  • (optionally) unvendoring parts of stdlib as a separate package. Currently only distutils.
  • there will likely be minimized builds in the near future Use python-minifier to minify the Python stdlib for saving about 2.4 MB #1970

Some of those topics would likely be relevant for anyone trying to run CPython in the browser, and we can certainly try to make these parts more modular and independent so that they can be more easily re-used or adapted.
It would be good to find some way to to collaborate on these subjects as well.

[brettcannon]

@rth My comment about focusing on the "build the scientific stack" was meant to imply all the work you do for general building; sorry if it came off as me shortchanging all the work you have done! 😅

One thing we will have to discuss is where is the line in terms of what CPython can/will provide out-of-the-box and what will be left to projects like Pyodide to provide themselves? For instance, packaging is not a CPython core team concern; that's left to the community. And so all of your bullet points about Pyodide's work on packaging and building extensions definitely hold as something important this project provides.

As for the JS FFI and trying to make more parts of the stdlib work via web APIs, that's an open question as to where that work would be best to live (and we should probably not try to answer until we decide how serious we all are about centralizing this work and where the centralized location should be).

[rth]

Thanks for starting this discussion @brettcannon !

One thing we will have to discuss is where is the line in terms of what CPython can/will provide out-of-the-box and what will be left to projects like Pyodide to provide themselves?

Certainly.

As for the JS FFI and trying to make more parts of the stdlib work via web APIs, that's an open question as to where that work would be best to live

Right now the JS/Python type translations is at the core of Pyodide. A good part of our API for Python and Javascript either exposes it or has a very strong coupling with it. Indeed, accessing JS from Python is a pre-requisite for finding workarounds for some of the stdlib modules. Such as using the browser event loop for asyncio or possibly in the future using WebWorkers for multiprocessing. It does make sense that better wasm support is done upstream, but as for the JS/Python type conversion API it's very specific to emscripten and the version used for the build and I imagine it would not be integrated into CPython? Even if it could, it's still actively developed, there are edge cases that need frequent fixes, and having a separate package, with faster iteration and the lighter process would still be preferable IMO. Now if it is a separate package, one can access it as,

# from Python
from pyodide import js

or

# From JS
pyodide.runPython(...)

as part of the Pyodide Python and JS packages. We have talked about it with @hoodmane and we don't really see the point of moving the JS/Python type translation code into a different project. It would pretty much mean renaming the Pyodide package to something else. Particularly in the context where it's very likely there will an effort in 2022 to add WASM builds based on the conda-forge infrastructure, where they could take CPython and the Pyodide core package and build it with a conda build setup.

We are very open to making things more modular in Pyodide #2014, or anything else that could facilitate the work you are doing though.

[brettcannon]

I imagine it would not be integrated into CPython?

Probably not. It's more of a question of whether it would be a Pyodide sort of thing or some common project we all contribute to. I basically don't know where everyone wants to draw the line on where Pyodide starts/stops compared to CPython itself and compared to some hypothetical project could sit between Pyodide and CPython.

it's very likely there will an effort in 2022 to add WASM builds based on the conda-forge infrastructure

Is that being discussed anywhere publicly? I'm curious how that would look due to some conda packages sometimes requiring arbitrary shell script execution in an activated conda environment which isn't exactly a browser feature. 😉

[SimonBiggs]

it's very likely there will an effort in 2022 to add WASM builds based on the conda-forge infrastructure

Might there instead be the possibility of having designated wasm wheel build targets?

https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/

And then have that wasm build target supported within something like the following:

https://github.com/pypa/cibuildwheel

A lot of the ecosystem is using cibuildwheel so this would mean a large number of wheels would then be distributed and downloadable from pypi in the wasm target.

---

[Edit]
I gave it a little more thought and I realised that many libraries have been needing patches and work arounds, so something like cibuildwheel won't be a magic bullet.

Personally as a package maintainer, having to support both pypi distribution and conda-forge distribution can be quite the headache. So as such I dropped conda-forge distribution of my scientific library quite some time ago
[/Edit]

[rth]

It's more of a question of whether it would be a Pyodide sort of thing or some common project we all contribute to. I basically don't know where everyone wants to draw the line on where Pyodide starts/stops compared to CPython itself and compared to some hypothetical project could sit between Pyodide and CPython.

I agree it's good to have this discussion and if you want we can organize a call to talk about it.

[possible] WASM builds based on the conda-forge infrastructure

Is that being discussed anywhere publicly?

It's been an ongoing subject for a while at conda/conda#7619 but @SylvainCorlay might have more recent information.

Might there instead be the possibility of having designated wasm wheel build targets?

Definitely, these efforts are not mutually exclusive. We are currently exploring that possibility #655, pypi/warehouse#10416 and at present, it seems likely that we might use it, at least to replace the emscripten package format. Wheels do have limitations, particularly for packaging non Python dependencies, but on the conda for WASM packaging side there are also open questions #795

[brettcannon]

I agree it's good to have this discussion and if you want we can organize a call to talk about it.

Sure! Probably @ethanhs and @tiran would be good to have from the CPython side as well as me.

We are currently exploring that possibility

As the author of packaging.tags I can sponsor the PEP at least if someone with the technical knowledge can (co-)author the PEP, but there are a lot of technical questions to work through. For instance, would the platform need to be versioned due to potential breakage from changes in emscripten? Do we have a solid grounding and definition of how to import a module compiled for WASM where if some other WASM build came about they would be able to use the wheel? Does it makes sense to reuse cp as the interpreter type or would we need a new wasm interpreter type, e.g. cp310 versus wasm310? What does the build tool support look like (I know Pyodide has a solution for setuptools, but is it robust enough for most setuptools projects to rely on? What about e.g. Meson?)? Is there some ABI we need to concern ourselves with? Basically, how do you make sure the wheel tag triple each make sense in the face of WASM wheels?