From b1e22902281f53f236ea18bf7e30ccefd095e27f Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 28 Oct 2025 15:11:23 -0700 Subject: [PATCH 01/10] PEP 818: Upstreaming the Pyodide js ffi --- .github/CODEOWNERS | 4 +- peps/pep-0818.rst | 3296 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 3299 insertions(+), 1 deletion(-) create mode 100644 peps/pep-0818.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index a0a6ab363d9..4037290d350 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -656,7 +656,6 @@ peps/pep-0775.rst @encukou peps/pep-0776.rst @hoodmane @ambv peps/pep-0777.rst @warsaw @emmatyping peps/pep-0778.rst @warsaw @emmatyping -# ... peps/pep-0779.rst @Yhg1s @colesbury @mpage peps/pep-0780.rst @lysnikolaou peps/pep-0781.rst @methane @@ -691,6 +690,9 @@ peps/pep-0811.rst @sethmlarson @gpshead peps/pep-0814.rst @vstinner @corona10 peps/pep-0815.rst @emmatyping peps/pep-0816.rst @brettcannon +peps/pep-0817.rst @warsaw @dstufft +peps/pep-0817/ @warsaw @dstufft +peps/pep-0818.rst @hoodmane @ambv # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst new file mode 100644 index 00000000000..bdfa132a6bb --- /dev/null +++ b/peps/pep-0818.rst @@ -0,0 +1,3296 @@ +PEP: 818 +Title: Upstreaming the Pyodide JavaScript Foreign Function Interface +Author: Hood Chatham +Sponsor: Łukasz Langa +Discussions-To: +Status: Draft +Type: Standards Track +Created: 10-Dec-2025 +Python-Version: 3.15 +Post-History: + +Abstract +======== + +Pyodide is a distribution of Python for JavaScript runtimes, including browsers. +Browsers are a universal computing platform. As with C for Unix family operating +systems, in the browser platform all fundamental capabilities are exposed +through the JavaScript language. For years, Pyodide has included a comprehensive +JavaScript foreign function interface. This provides the equivalent of the +``os`` module for the JavaScript world. + +This PEP proposes adding the core of the Pyodide foreign function interface to +Python. + +Motivation +========== + +The Pyodide project is a Python distribution for JavaScript runtimes. Pyodide is +a very popular project. In 2025 to date, Pyodide has received over a billion +requests on JsDelivr. The popularity is rapidly growing: usage has more than +doubled in each of the last two years. + +Pyodide includes several components: + +1. A port of CPython to the Emscripten compiler toolchain (a toolchain to + compile linux C/C++ programs to JavaScript and WebAssembly). +2. A foreign function interface for calling Python from JavaScript and + JavaScript from Python. +3. A JavaScript programmatic interface for managing the Python runtime and + package installation. +4. An ABI for native extensions. +5. A toolchain to cross-compile Python packages compatible with that ABI for use + with Pyodide. + +In the long run, we would like to upstream the runtime components (1)--(4) of +the Pyodide project into CPython. In 2022, Christian Heimes upstreamed (1) the +Emscripten port of CPython, and Emscripten is currently a tier 3 supported +platform (see :pep:`776`). :pep:`783` proposes to allow Pyodide-compatible +wheels to be uploaded to PyPI. What is needed for these to be +Emscripten-CPython-compatible wheels is to upstream (2) the Python/JavaScript +foreign function interface and (4) the ABI for native extensions. This PEP +concerns partially upstreaming (2) the Python/JavaScript foreign function +interface. + +This interface is similar to the ``os`` module for Python on linux: all IO +requires going through libc and the ``os`` module provides access to libc calls +to Python code. Similarly, in a JavaScript runtime, to do any actual work +requires making calls into JavaScript: for example, it is required to display +content to the screen, to receive user input, to handle events, to access +databases, etc. For instance, once Python has a JavaScript foreign function +interface, it will be possible to support ``urllib`` on Emscripten. Downstream, +supporting ``urllib3``, ``aiohttp``, and ``httpx`` requires the foreign function +interface. + +In order to keep the length of this PEP reasonable, we focus on the "core" of +the foreign function interface. Three areas are left to future PEPs: + +1. asyncio +2. integration between the buffer protocol and JavaScript equivalents +3. a JavaScript interface for managing the Python runtime + + +Rationale +========= + +Our goal here is to upstream Pyodide's foreign function interface, without +breaking backwards compatibility more than necessary for Pyodide's large +collection of existing users. On the other hand, the best time to making +breaking changes is now. + +With that in mind, we wish here to justify not that our design is perfect but +that the costs of any changes outweigh the benefits. + +Translating Objects +------------------- + +The most fundamental decision is how we translate objects from one language to +the other. When translating an object, we can either choose to convert the value +into a similar object in the target language, or to make a proxy that "wraps" +the original object. A few considerations apply here: + +1. Mutability: If we call a function that expects to mutate its argument, then + it is important that we proxy the argument and not convert it. Otherwise, the + function mutates a copy that we then throw away. So implicit conversion is + only a reasonable option for immutable objects. +2. Round trip behavior: It is strongly desirable that passing an object from + Python to JavaScript back to Python results in the original Python object and + vice-versa. If the object is immutable, it is okay if the result is only + equal to the original object and not the same object. If the object is + mutable, it should be the same object. +3. Performance characteristics: Converting a complex object entails a lot of up + front work. If the object is only minimally used, then it may be less + performant. On the other hand, each access to an object via a proxy is slower + than to a native object so if the object is used a lot, converting up front + is more efficient than proxying. Proxying by default and allowing the user to + explicitly convert when they want to gives the user maximum control over + performance. +4. Ergonomics: A native object is in many cases easier to work with. + +JavaScript has the following immutable types: ``string``, ``undefined``, +``boolean``, ``number`` and ``bigint``. It also has the special value ``null``. + +Of these, ``string`` and ``boolean`` directly correspond to ``str`` and +``bool``. ``number`` and ``bigint`` awkwardly correspond to ``float`` and +``int``. ``undefined`` is the default value for a missing argument so it +corresponds to ``None``. We invent a new falsey singleton Python value +``jsnull`` to act as the conversion of ``null``. We also make a new type +``JsBigInt`` to act as the conversion for ``bigint``. All other types are +proxied. + +In particular, even though ``tuples`` are immutable, they have no equivalent in +JavaScript so we proxy them. They can be manually converted to an ``Array`` with +the ``toJs()`` method if desired. + +Proxies +------- + +A ``JsProxy`` is a Python object used for accessing a JavaScript object. While +the ``JsProxy`` exists, the underlying JavaScript object is kept in a table +which keeps it from being garbage collected. + +A ``PyProxy`` is a JavaScript object used for accessing a Python object. When a +``PyProxy`` is created, the reference count of the underlying Python object is +incremented. When the ``.destroy()`` method is called, the reference count of the +underlying Python object is decremented and the proxy is disabled. Any further +attempt to use it raises an error. + +The base ``JsProxy`` implements property access, equality checks, ``__repr__``, +``__eq__``, ``__bool__``, and a handful of other convenience methods. We also +define a large number of mixins by mapping abstract Python object protocols to +abstract JavaScript object protocols (and vice-versa). The mapping described in +this PEP is as follows: + +Base proxies (properties common to all objects): + +* ``__getattribute__`` <==> ``Reflect.get`` (proxy handler) +* ``__setattr__`` <==> ``Reflect.set`` (proxy handler) +* ``__eq__`` <==> ``===`` (object identity) +* ``__repr__`` <==> ``toString`` + +For the ``__str__`` implementation, we inherit the default implementation which +uses ``__repr__``. + +We implement the following mappings between protocols as mixins. When we create +a proxy, we feature detect which of these abstract and concrete protocols it +supports and create a class for the proxy with the appropriate mixins. + +* ``__iter__`` <==> ``[Symbol.iterator]`` +* ``__next__`` <==> ``next`` +* ``__len__`` <==> ``length``, ``size`` +* ``__getitem__`` <==> ``get`` +* ``__setitem__``, ``__delitem__`` <==> ``set``, ``delete`` +* ``__contains__`` <==> ``includes``, ``has`` +* ``__call__`` <==> ``Reflect.apply`` (proxy handler) +* ``Generator`` <==> ``Generator`` +* ``Exception`` <==> ``Error`` +* ``MutableSequence`` <==> ``Array`` + +If a JavaScript object has a ``[Symbol.dispose]()`` method, we make the Python +object into a context manager, but we do not presently use context managers to +implement ``[Symbol.dispose]()``. + +JavaScript also has ``Reflect.construct`` (the ``new`` keyword). Callable +JsProxies have a method called ``new()`` which corresponds to +``Reflect.construct``. + +The following additional mappings are defined in Pyodide. It is our intention to +eventually add them to Python itself, but they are deferred to a future PEP: + +* ``__await__`` <==> ``then`` +* ``__aiter__`` <==> ``[Symbol.asyncIterator]`` +* ``__anext__`` <==> ``next`` (same as ``__next__``; check for presence of + ``[Symbol.asyncIterator]`` to distinguish) +* ``AsyncGenerator`` <==> ``AsyncGenerator`` +* buffer protocol <==> typed arrays +* Async context managers are implemented on JsProxies that implement + ``[Symbol.asyncDispose]``. + +Garbage Collection and Destruction of Proxies +--------------------------------------------- + +The most fundamental difficulty that we face is the existence of two garbage +collectors, the Python garbage collector and the JavaScript garbage collector. +Any reference loop from Python to JavaScript back to Python will be leaked. +Furthermore, even if there is no loop, the JavaScript garbage collector has no +idea how much memory a ``PyProxy`` owns nor how much memory pressure the Python +garbage collector faces. + +For this reason, we need to include a way to manually break references between +languages. In Python, destructors are run eagerly when the reference count of an +object reaches 0. Thus, if a programmer wishes to manually release a JavaScript +object, they can delete all references to it and after that the JavaScript +garbage collector will be able to reclaim it. + +On the other hand, JavaScript finalizers are not reliable. The proposal that +introduced them to the language says the following: + + If an application or library depends on GC [calling a finalizer] in a timely, + predictable manner, it's likely to be disappointed: the cleanup may happen much + later than expected, or not at all. + + ... + + It's best if [finalizers] are used as a way to avoid excess memory usage, or as a + backstop against certain bugs, rather than as a normal way to clean up external + resources. + +https://github.com/tc39/proposal-weakrefs?tab=readme-ov-file#a-note-of-caution + +A ``PyProxy`` has a ``destroy()`` method that manually detaches the ``PyProxy`` +and releases the Python reference. We consider destroying a ``PyProxy`` to be +the correct, normal way to clean it up. As recommended by the proposal, the +finalizer is treated as a backstop. In the Pyodide test suite, we require that +every ``PyProxy`` be manually destroyed in the majority of the tests. This helps +to ensure that our APIs are designed in a way that keeps this ergonomic. + +Calling Conventions +------------------- + +Calling a Python Function from JavaScript +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To call a callable ``PyProxy`` we do the following steps: + +1. Translate each argument from JavaScript to Python and place the arguments + in a C array. +2. Use ``PyObject_VectorCall`` to call the Python object. +3. If a JavaScript error is raised, this is fatal -- Python interpreter + invariants have been violated. Report the fatal error and tear down the Python interpreter. +4. If the Python error flag is set, set ``sys.last_value`` to the current + exception. Convert the Python exception to a JavaScript ``PythonError`` + object. This ``PythonError`` object records the type, the formatted traceback + of the Python exception, and a weak reference to the original Python + exception. Throw this ``PythonError``. +5. Translate the result from Python to JavaScript and return it. + +Note here that if a ``JsProxy`` is created but the Python function does not +store a reference to it, it will be released immediately. The JavaScript error +doesn't hold a strong reference the Python exception because JavaScript errors +are often leaked and Python error objects hold a reference to frame objects +which may hold a significant amount of memory. + +Calling a JavaScript Function from Python +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To call a callable ``JsProxy`` we do the following steps: + +1. Make an empty array called ``pyproxies`` +2. Translate each positional argument from Python to JavaScript and place these + arguments in a JavaScript array called ``jsargs``. If any ``PyProxy`` is + generated in this way, don't register a JavaScript finalizer for it and do + append it to ``pyproxies``. +3. If there are any keyword arguments, create an empty JavaScript object + ``jskwargs``, translate each keyword argument to JavaScript and assign + ``jskwargs[key] = jskwarg``. Append ``jskwargs`` to ``jsargs``. If any + ``PyProxy`` is generated in this way, don't register a JavaScript finalizer + for it and do append it to ``pyproxies``. +4. Call the JavaScript function and store the result into ``jsresult``. +5. If an error is thrown: + + a. If the error is a ``PythonError`` and the weak reference to the Python + exception is still alive, raise the referenced Python exception. + b. Otherwise, convert the exception from JavaScript to Python and raise the + result. Note that the ``JsException`` object holds a reference to the + original JavaScript error. + +6. If ``jsresult`` is a JavaScript generator, iterate over ``pyproxies`` and + register a JavaScript finalizer for each. Wrap the generator with a new + generator that destroys ``pyproxies`` when they are exhausted. Translate the + wrapped generator to Python and return it. +7. Otherwise, translate ``jsresult`` to Python and store it in ``pyresult``. +8. Iterate over ``pyproxies`` and destroy them. If ``jsresult`` is a + ``PyProxy``, destroy it too. +9. Return ``pyresult``. + +This is modeled on the calling convention for C Python APIs. + +Defense of the Calling Convention for a ``JsProxy`` +--------------------------------------------------- + +The calling convention from JavaScript into Python is uncontroversial so we will +not defend it. The calling convention from Python into JavaScript is more +controversial so we will explain here why we believe it is a better design than +the alternatives. + +The main disadvantage of this design is that it is not as ergonomic in cases +where the callee is going to persist its arguments. However, we argue that the +benefits outweigh this. + +The biggest advantage of this approach is that it makes it possible to use +JavaScript functions that are unaware of the existence of Python without memory +leaks. Another advantage is that registering a finalizer for a ``PyProxy`` is +somewhat expensive and so avoiding this step can substantially decrease the +overhead for certain Python to JavaScript calls. + +An Example of a Disadvantage of the Calling Convention +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We will start by illustrating the common complaint about the Python to +JavaScript calling convention. Consider the following example: + + +.. code-block:: python + + from pyodide.code import run_js + set_x = run_js("(x) => { globalThis.x = x; }") + get_x = run_js("(x) => globalThis.x") + + set_x({}) + get_x() + +This code is broken. Calling ``set_x`` creates a PyProxy but it is destroyed +when the call is done. When we call ``get_x()`` the following error is raised:: + + This borrowed proxy was automatically destroyed at the end of a function call. + +To fix it to manage memory correctly, we can change ``set_x`` to the following +function: + +.. code-block:: javascript + + (x) => { + globalThis.x?.destroy?.(); + globalThis.x = x?.copy?.() ?? x; + } + +Or we can manage the memory from Python using ``create_proxy()`` as follows: + +.. code-block:: python + + from pyodide.ffi import JsDoubleProxy + from pyodide.code import run_js + + setXJs = run_js("(x) => { globalThis.x = x; }") + def set_x(x): + orig_x = get_x() + if isinstance(orig_x, JsDoubleProxy): + orig_x.destroy() + xpx = create_proxy(x) + setXJs(xpx) + +This extra boilerplate is not too hard to get right -- it's roughly equivalent +to what is needed to assign an attribute in C. However, it does impose a +nontrivial complexity cost on the user and so we need to justify why this is +better than the alternatives. + +A Use Case That Is Made Simpler By This Calling Convention +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Suppose we have a Python function ``render()`` that returns a buffer, and a +JavaScript function ``drawImageToCanvas(buffer)`` that displays the buffer on a +canvas. If the buffer is a 1024 by 1024 bitmap with four color channels, then it +is a 4 megabyte buffer. Imagine the following code: + +.. code-block:: python + + @create_proxy + def main_loop(): + update() + buf = render() + drawImageToCanvas(buffer) + requestAnimationFrame(main_loop) + +With the calling convention described here, the buffer is released normally +after each call and memory usage stays consistent, in my tests it stays at 57 +megabytes. + +If we rely on a JavaScript finalizer to release ``buffer``, in my tests the +JavaScript finalizer doesn't run until malloc runs out of space on the +WebAssembly heap and requests more memory, with the effect that over several +minutes the WebAssembly heap gradually grows to the maximum allowed 4 gigabytes +and then a memory error is raised. + +Now a cooperating implementation of ``drawImageToCanvas()`` could destroy the +``buffer`` when it is done, but my philosophy in designing the calling +convention was that it should be possible to take care of the memory management +from Python. This necessitates something like the current approach. + +Specification +============= + +The Pseudocode in this Document +------------------------------- + +The pseudocode in this PEP is generally written in Python or JavaScript. We +leave out most resource management and exception handling except when we think +it is particularly interesting. If an error is raised, we implicitly clean up +all resources and propagate the error. A large fraction of the real code +consists of resource management and exception handling. + +For the most part the code works as written but in a few spots we directly call +a C API from Python or otherwise write code that wouldn't run but whose intent +we believe is clear. + +In Python code when we want to execute a JavaScript function inline, we write it +like: + +.. code-block:: python + + jsfunc = run_js("(x, y) => doSomething") + jsfunc(x, y) + +Conversely, when we want to execute Python code inline in JavaScript we write it +like this: + +.. code-block:: javascript + + const pyfunc = makePythonFunction(` + def pyfunc(x, y): + # do something + `); + pyfunc(x, y) + +For the most part, this code could actually be used if performance was not a +concern. In some places there may be bootstrapping issues. + +Our first task is to define the Python callable ``run_js`` and the JavaScript +callable ``makePythonFunction``. ``run_js`` is a ``JsProxy`` and +``makePythonFunction`` is a ``PyProxy``. + +To make sense of this, we need to describe + +1. how we convert values from JavaScript to Python and from Python to JavaScript +2. how to call a Python function from JavaScript and how to call a JavaScript + function from Python + +We can directly represent a ``PyObject*`` as a ``number`` in JavaScript so we +can describe the process of calling a ``PyObject*`` from JavaScript. On the +other hand, JavaScript objects are not directly representable in Python, we have +to create a ``JsProxy`` of it. We describe first the process of calling a +``JsProxy``, the process of creating it is described in the section on +JsProxies. + +Converting Values between Python and JavaScript +----------------------------------------------- + +A few primitive types are implicitly converted between Python and JavaScript. +Implicit conversions are supposed to round trip, so that when converting from +Python to JavaScript back to Python or from JavaScript to Python back to +JavaScript, the result is the same primitive as we started with. The one +exception to this is that a JavaScript ``BigInt`` that is smaller than ``2^53`` +round trips to a ``Number``. We convert ``undefined`` to ``None`` and introduce +the special falsey singleton ``pyodide.ffi.jsnull`` to convert ``null``. We also +introduce a subtype of ``int`` called ``pyodide.ffi.JsBigInt`` which converts to +and from JavaScript ``bigint``. + +Implicit conversions are done with the C functions ``_Py_python2js`` and +``_Py_js2python()``. These functions cannot be called directly from Python code +because the ``JsVal`` type is not representable in Python. + +Implicit conversion from Python to JavaScript +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``JsVal _Py_python2js_track_proxies(PyObject* pyvalue, JsVal pyproxies, bool gc_register)`` +is responsible for implicit conversions from Python to JavaScript. It does the +following steps: + +1. if ``pyvalue`` is ``None``, return ``undefined`` +2. if ``pyvalue`` is ``jsnull``, return ``null`` +3. if ``pyvalue`` is ``True``, return ``true`` +4. if ``pyvalue`` is ``False``, return ``false`` +5. if ``pyvalue`` is a ``str``, convert the string to JavaScript and return the result. +6. if ``pyvalue`` is an instance of ``JsBigInt``, convert it to a ``BigInt``. +7. if ``pyvalue`` is an ``int`` and it is less than ``2^53``, convert it to + a ``Number``. Otherwise, convert it to a ``BigInt`` +8. if ``pyvalue`` is a ``float``, convert it to a ``Number``. +9. if ``pyvalue`` is a ``JsProxy``, convert it to the wrapped JavaScript value. +10. Let ``result`` be ``createPyProxy(pyvalue, {gcRegister: gc_register})``. If + ``pyproxies`` is an array, append ``result`` to ``pyproxies``. + +We define ``JsVal _Py_python2js(PyObject* pyvalue)`` to be +``_Py_python2js_track_proxies(pyvalue, Js_undefined, true)``. + +Implicit conversion from JavaScript to Python +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``PyObject* _Py_js2python(JsVal jsvalue)`` is responsible for implicit +conversions from JavaScript to Python. + +We first define the helper function ``PyObject* _Py_js2python_immutable(JsVal jsvalue)`` +does the following steps: + +1. if ``jsvalue`` is ``undefined``, return ``None`` +2. if ``jsvalue`` is ``null`` return ``jsnull`` +3. if ``jsvalue`` is ``true`` return ``True`` +4. if ``jsvalue`` is ``false`` return ``False`` +5. if ``jsvalue`` is a ``string``, convert the string to Python and return the + result. +6. if ``jsvalue`` is a ``Number`` and ``Number.isSafeInteger(jsvalue)`` returns + ``true``, then convert ``jsvalue`` to an ``int``. Otherwise convert it to a + ``float``. +7. if ``jsvalue`` is a ``BigInt`` then convert it to an ``JsBigInt``. +8. If ``jsvalue`` is a ``PyProxy`` that has not been destroyed, convert it to + the wrapped Python value. +9. If the ``jsvalue`` is a ``PyProxy`` that has been destroyed, throw an error + indicating this. +10. Return ``NoValue``. + + +``_Py_js2python(JsVal jsvalue)`` does the following steps: + +1. Let ``result`` be ``_Py_js2python_immutable(jsvalue)``. If ``result`` is + not ``NoValue``, return ``result``. +2. Return ``create_jsproxy(jsvalue)``. + +Error handling +-------------- + +At the boundary between JavaScript and C, we have to translate errors. + +Executing JavaScript Code from C +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When we execute any JavaScript code from C, we wrap it in a try/catch block. If +an error is caught, we use ``_Py_js2python(jserror)`` to convert it into a +Python exception, set the Python error flag to this python exception, and return +the appropriate error value to signal an error. This makes it ergonomic to +create JavaScript functions that can be called from C and follow CPython's +normal conventions for C APIs. + +Executing C Code from JavaScript +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Whenever we call into C from JavaScript, we wrap the call in the following +boilerplate: + +.. code-block:: javascript + + try { + result = some_c_function(); + } catch (e) { + // If an error was thrown here, the C runtime state is corrupted. + // Signal a fatal error and tear down the interpreter. + fatal_error(e); + } + // Depending on the API, we check for -1, 0, _PyErr_Occurred(), etc to + // decide if an error occurred. + if (result === -1) { + // This function takes the error flag and converts it to a JavaScript + // exception. It leaves the error flag cleared. + throw __Py_pythonexc2js(); + } + + +Calling Conventions +------------------- + +Calling a Python Function from JavaScript +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To call a ``PyObject*`` from JavaScript we use the following code: + +.. code-block:: javascript + + function callPyObjectKwargs(pyfuncptr, jsargs, kwargs) { + const num_pos_args = jsargs.length; + const kwargs_names = Object.keys(kwargs); + const kwargs_values = Object.values(kwargs); + const num_kwargs = kwargs_names.length; + jsargs.push(...kwargs_values); + // apply the usual error handling logic for calling from JavaScript into C. + return _PyProxy_apply(pyfuncptr, jsargs, num_pos_args, kwargs_names, num_kwargs); + } + +**_PyProxy_apply(PyObject* callable, JsVal jsargs, Py_ssize_t num_pos_args, JsVal kwargs_names, Py_ssize_t num_kwargs)** + +1. Let ``total_args`` be ``num_pos_args + numkwargs``. +2. Create a C array ``pyargs`` of length ``total_args``. +3. For ``i`` ranging from ``0`` to ``total_args - 1``: + + a. Execute the JavaScript code ``jsargs[i]`` and store the result into + ``jsitem``. + b. Set ``pyargs[i]`` to ``_Py_js2python(jsitem)``. + +4. Let ``pykwnames`` be a new tuple of length ``numkwargs`` +5. For ``i`` ranging from ``0`` to ``numkwargs - 1``: + + a. Execute the JavaScript code ``jskwnames[i]`` and store the result into + ``jskey``. + b. Set the ith entry of ``pykwnames`` to ``_Py_js2python(jsitem)``. + +6. Let ``pyresult`` be ``_PyObject_Vectorcall(callable, pyargs, num_pos_args, pykwnames)``. +7. Return ``_Py_python2js(pyresult)``. + +Calling a JavaScript Function from Python +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**``JsMethod_ConvertArgs(posargs, kwargs, pyproxies)``** + +First we define the function ``JsMethod_ConvertArgs`` to convert the Python +arguments to a JavaScript array of arguments. Any ``PyProxy`` created at this +stage is not tracked by the finalization registry and is added to the JavaScript +list ``pyproxies`` so we can either destroy it or track it later. This function +performs the following steps: + +1. Let ``jsargs`` be a new empty JavaScript list. +2. For each positional argument: + + a. Set ``JsVal jsarg = _Py_python2js_track_proxies(pyarg, proxies, /*gc_register:*/false);``. + b. Call ``_PyJsvArray_Push(jsargs, arg);``. + +3. If there are any keyword arguments: + + a. Let ``jskwargs`` be a new empty JavaScript object. + b. For each keyword argument pykey, pyvalue: + + i. Set ``JsVal jskey = _Py_python2js(pykey)`` + ii. Set ``JsVal jsvalue = _Py_python2js_track_proxies(pyvalue, proxies, /*gc_register:*/false)`` + iii. Set the ``jskey`` property on ``jskwargs`` to ``jsvalue``. + + c. Call ``_PyJsvArray_Push(jsargs, jskwargs);`` + +4. Return ``jsargs`` + +**``JsMethod_Vectorcall(jsproxy, posargs, kwargs)``** + +Each ``JsProxy`` of a function has an underlying JavaScript function and an +underlying ``this`` value. + +1. Let ``jsfunc`` be the JavaScript function associated to ``jsproxy``. +2. Let ``jsthis`` be the ``this`` value associated to ``jsproxy``. +3. Let ``pyproxies`` be a new empty JavaScript list. +4. Execute ``JsMethod_ConvertArgs(posargs, kwargs, pyproxies)`` and store the + result into ``jsargs``. +5. Execute the JavaScript code ``Function.prototype.apply.apply(jsfunc, [ jsthis, jsargs ])`` + and store the result into ``jsresult``. + (Apply the usual error handling for calling from C into JavaScript.) +6. If ``jsresult`` is a ``PyProxy`` run the JavaScript code ``pyproxies.push(jsresult)`` +7. Set ``destroy_args`` to ``true`` +8. If ``jsresult`` is a ``Generator`` set ``destroy_args`` to ``false`` and set + ``jsresult`` to ``wrap_generator(jsresult, pyproxies)``. +9. Execute ``_Py_js2python(jsresult)`` and store the result into ``pyresult``. +10. If ``destroy_args`` is ``true``, then destroy all the proxies in ``pyproxies``. +11. If ``destroy_args`` is ``false``, gc register all the proxies in ``pyproxies``. +12. Return ``pyresult``. + +``wrap_generator(jsresult, pyproxies)`` is a JavaScript function that wraps a +JavaScript generator in a new generator that destroys all the proxies in +``pyproxies`` when the generator is exhausted. + +``run_js`` +---------- + +The Python object ``pyodide.code.run_js`` is defined as follows: + +1. Execute the JavaScript code ``eval`` and store the result into ``jseval``. +2. Run ``_Py_js2python(jseval)`` and store the result into ``run_js``. + +``makePythonFunction`` +---------------------- + +Unlike ``run_js``, the JavaScript object ``makePythonFunction`` is strictly for +the sake of our pseudocode and will not be included as part of the API. We +define define ``makePythonFunction`` as follows: + +.. code-block:: python + + def make_python_function(code): + mod = ast.parse(code) + if isinstance(mod.body[0], ast.FunctionDef): + d = {} + exec(code, d) + return d[mod.body[0].name] + return eval(code) + +1. Let ``make_python_function`` be the function above. +2. Run ``_Py_python2js(make_python_function)`` and store the result into + ``makePythonFunction``. + + +JsProxy +------- + +We define 14 different abstract protocols that a JavaScript object can support. +These each correspond to a ``JsProxy`` type flag. There are also two additional +flags ``IS_PY_JSON_DICT`` and ``IS_PY_JSON_SEQUENCE`` which are set by the +``JsProxy.as_py_json()`` method and do not reflect properties of the underlying +JavaScript object. + +``HAS_GET`` + Signals whether or not the JavaScript object has a ``get()`` + method. If present, used to implement ``__getitem__`` on the ``JsProxy``. + +``HAS_HAS`` + Signals whether or not the JavaScript object has a ``has()`` method. If + present, used to implement ``__contains__`` on the ``JsProxy``. + +``HAS_INCLUDES`` + Signals whether or not the JavaScript object has an ``includes()`` method. + If present, used to implement ``__contains__`` on the ``JsProxy``. We prefer + to use ``has()`` to ``includes()`` if both are present. + +``HAS_LENGTH`` + Signals whether or not the JavaScript object has a ``length`` or ``size`` + property. Used to implement ``__len__`` on the ``JsProxy``. + +``HAS_SET`` + Signals whether or not the JavaScript object has a ``set()`` method. If + present, used to implement ``__setitem__`` on the ``JsProxy``. + +``HAS_DISPOSE`` + Signals whether or not the JavaScript object has a ``[Symbol.dispose]()`` + method. If present, used to implement ``__enter__`` and ``__exit__``. + +``IS_ARRAY`` + Signals whether ``Array.isArray()`` applied to the JavaScript object returns + ``true``. If present, the ``JsProxy`` will be an instance of + ``collections.abc.MutableSequence``. + +``IS_ARRAY_LIKE`` + We set this if ``Array.isArray()`` returns ``false`` and the object has a + ``length`` property and ``IS_ITERABLE``. If present, the ``JsProxy`` will be + an instance of ``collections.abc.Sequence``. This is the case for many + interfaces defined in the webidl such as + `NodeList `_ + +``IS_CALLABLE`` + Signals whether the ``typeof`` the JavaScript object is ``"function"``. If + present, used to implement ``__call__`` on the ``JsProxy``. + +``IS_ERROR`` + Signals whether the JavaScript object is an ``Error``. If so, the + ``JsProxy`` it will subclass ``Exception`` so it can be raised. + +``IS_GENERATOR`` + Signals whether the JavaScript object is a generator. If so, the ``JsProxy`` + will be an instance of ``collections.abc.Generator``. + +``IS_ITERABLE`` + Signals whether the JavaScript object has a ``[Symbol.iterator]`` method or + the ``IS_PY_JSON_DICT`` flag is set. If so, we use it to implement + ``__iter__`` on the ``JsProxy``. + +``IS_ITERATOR`` + Signals whether the JavaScript object has a ``next()`` method and no + ``[Symbol.asyncIterator]`` method. If so, we use it to implement + ``__next__`` on the ``JsProxy``. (If there is a ``[Symbol.asyncIterator]`` + method, we assume that the ``next()`` method should be used to implement + ``__anext__``.) + +``IS_PY_JSON_DICT`` + This is set on a ``JsProxy`` by the ``as_py_json()`` method if it is not an + ``Array``. When this is set, ``__getitem__`` on the ``JsProxy`` will turn + into attribute access on the JavaScript object. Also, the return values from + iterating over the proxy or indexing it will also have ``IS_PY_JSON_DICT`` + or ``IS_PY_JSON_SEQUENCE`` set as appropriate. + +``IS_PY_JSON_SEQUENCE`` + This is set on a ``JsProxy`` by the ``as_py_json()`` method if it is an + ``Array``. When this is set, when indexing or iterating the ``JsProxy`` + we'll call ``as_py_json()`` on the result. + +``IS_MAPPING`` + We set this if the flags ``HAS_GET``, ``HAS_LENGTH``, and ``IS_ITERABLE`` + are set, or if ``IS_PY_JSON_DICT`` is set. In this case, the ``JsProxy`` + will be an instance of ``collections.abc.Mapping``. + +``IS_MUTABLE_MAPPING`` + We set this if the flags ``IS_MAPPING`` and ``HAS_SET`` are set or if + ``IS_PY_JSON_DICT`` is set. In this case, the ``JsProxy`` will be an + instance of ``collections.abc.MutableMapping``. + + +Creating a ``JsProxy`` +~~~~~~~~~~~~~~~~~~~~~~ + +To create a ``JsProxy`` from a JavaScript object and a value ``jsthis`` we do the +following steps: + +1. calculate the appropriate type flags for the JavaScript object +2. get or create and cache an appropriate ``JsProxy`` class with the mixins + appropriate for the set of type flags that are set +3. instantiate the class with a reference to the JavaScript object and the + ``jsthis`` value. + +The value ``jsthis`` is used to determine the value of ``this`` when calling a +function. If ``jsobj`` is not callable, is has no effect. + +Here is pseudocode for the functions ``create_jsproxy`` and +``create_jsproxy_with_flags``: + +.. code-block:: python + + def create_jsproxy(jsobj, jsthis=Js_undefined): + # For the definition of ``compute_type_flags``, see "Determining which flags to set". + return create_jsproxy_with_flags(compute_type_flags(jsobj), jsobj, jsthis) + + def create_jsproxy_with_flags(type_flags, jsobj, jsthis): + cls = get_jsproxy_class(type_flags) + return cls.__new__(jsobj, jsthis) + +The most important logic is for creating the classes, which works approximately +as follows: + +.. code-block:: python + + @functools.cache + def get_jsproxy_class(type_flags): + flag_mixin_pairs = [ + (HAS_GET, JsProxyHasGetMixin), + (HAS_HAS, JsProxyHasHasMixin), + # ... + (IS_PY_JSON_DICT, JsPyJsonDictMixin) + ] + bases = [mixin for flag, mixin in flag_mixin_pairs if flag & type_flags] + bases.insert(0, JsProxy) + if type_flags & IS_ERROR: + # We want JsException to be pickleable so it needs a distinct name + name = "pyodide.ffi.JsException" + bases.append(Exception) + else: + name = "pyodide.ffi.JsProxy" + ns = {"_js_type_flags": type_flags} + # Note: The actual way that we build the class does not result in the + # mixins appearing as entries on the mro. + return JsProxyMeta.__new__(JsProxyMeta, name, tuple(bases), ns) + + +The ``JsProxy`` Metaclass +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This metaclass overrides subclass checks so that if one ``JsProxy`` class has a +superset of the flags of another ``JsProxy`` class, we report it as a subclass. + +.. code:: python + + class _JsProxyMetaClass(type): + def __instancecheck__(cls, instance): + return cls.__subclasscheck__(type(instance)) + + def __subclasscheck__(cls, subcls): + if type.__subclasscheck__(cls, subcls): + return True + if not hasattr(subclass, "_js_type_flags"): + return False + + subcls_flags = subcls._js_type_flags + # Check whether the flags on subcls are a subset of the flags on cls + return cls._js_type_flags & subcls_flags == subcls_flags + + +The ``JsProxy`` Base Class +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most complicated part of the ``JsProxy`` base class is the implementation of +``__getattribute__``, ``__setattr__``, and ``__delattr__``. For +``__getattribute__``, we first check if an attribute is defined on the Python +object itself by calling ``object.__getattribute__()``. Otherwise, we look up +the attribute on the JavaScript object. + + +For ``__setattr__`` and ``__delattr__``, we set the keys "__loader__", +"__name__", "__package__", "__path__", and "__spec__" on the Python object +itself. All other values are set/deleted on the underlying JavaScript object. +This is to allow JavaScript objects to serve as Python modules without modifying +them. + +As an odd special case, if the object is an ``Array``, we filter out the +``keys`` method. We also remove it from the results of ``dir()``. This is to +ensure that ``dict.update()`` behaves correctly when passed a JavaScript +``Array``. We want the following behavior: + +.. code:: python + + d = {} + d.update(run_js("[['a', 'b'], [1, 2]]")) + assert d == {"a" : "b", 1 : 2} + # The result if we didn't filter out Array.keys would be as follows: + assert d != {1 : ['a', 'b'], 2: [1, 2]} + +A possible alternative would be to teach add special case handling for +JavaScript arrays to ``dict.update()``. + +It is common for JavaScript objects to have important methods that are named the +same thing as a Python reserved word (for example, ``Array.from``, +``Promise.then``). We access these from Python using the valid identifiers +``from_`` and ``then_``. If we want to access a JavaScript property called +``then_`` we access it from ``then__`` and so on. So if the attribute is a +Python reserved word followed by one or more underscores, we remove one +underscore from the end. The following helper function is used for this: + +.. code:: python + + def normalize_python_reserved_words(attr): + stripped = attr.strip("_") + if not is_python_reserved_word(stripped): + return attr + if stripped != attr: + return attr[:-1] + return attr + +We need the following JavaScript function to implement ``__bool__``. In +JavaScript, empty containers are truthy but in Python they should be falsey, so +we detect empty containers and return ``false``. + +.. code:: javascript + + function js_bool(val) { + // if it's a falsey JS object, return false + if (!val) { + return false; + } + // We also want to return false on container types with size 0. + if (val.size === 0) { + // Return true for HTML elements even if they have a size of zero. + if (val instanceof HTMLElement) { + return true; + } + return false; + } + // A function with zero arguments has a length property equal to + // zero. Make sure we return true for this. + if (val.length === 0 && Array.isArray(val)) { + return false; + } + // An empty buffer + if (val.byteLength === 0) { + return false; + } + return true; + + } + +The following helper function is used to implement ``__dir__``. It walks the +prototype chain and accumulates all keys, filtering out keys that start with +numbers (not valid Python identifiers) and reversing the +``normalize_python_reserved_words`` transform. We also filter out the +``Array.keys`` method. + +.. code:: javascript + + function js_dir(jsobj) { + let result = []; + let orig = jsobj; + do { + let keys = Object.getOwnPropertyNames(jsobj); + result.push(...keys); + } while ((jsobj = Object.getPrototypeOf(jsobj))); + // Filter out numbers + result = result.filter((s) => { + let c = s.charCodeAt(0); + return c < 48 || c > 57; + }); + + // Filter out "keys" key from an array + if (Array.isArray(orig)) { + result = result.filter((s) => { + return s !== "keys"; + }); + } + + // If the key is a reserved word followed by 0 or more underscores, + // add an extra underscore to reverse the transformation applied by + // normalizeReservedWords. + result = result.map((word) => + isReservedWord(word.replace(/_*$/, "")) ? word + "_" : word, + ); + + return result; + }; + + +.. code:: python + + class JsProxy: + def __getattribute__(self, attr): + try: + return object.__getattribute__(self, attr) + except AttributeError: + pass + if attr == "keys" and Array.isArray(self): + raise AttributeError(attr) + + attr = normalize_python_reserved_words(attr) + js_getattr = run_js( + """ + (jsobj, attr) => jsobj[attr] + """ + ) + js_hasattr = run_js( + """ + (jsobj, attr) => attr in jsobj + """ + ) + result = js_getattr(self, attr) + if isjsfunction(result): + result = result.__get__(self) + if result is None and not js_hasattr(self, attr): + raise AttributeError(attr) + return result + + def __setattr__(self, attr, value): + if attr in ["__loader__", "__name__", "__package__", "__path__", "__spec__"]: + return object.__setattr__(self, attr, value) + attr = normalize_python_reserved_words(attr) + js_setattr = run_js( + """ + (jsobj, attr) => { + jsobj[attr] = value; + } + """ + ) + js_setattr(self, attr, value) + + def __delattr__(self, attr): + if attr in ["__loader__", "__name__", "__package__", "__path__", "__spec__"]: + return object.__delattr__(self, attr) + attr = normalize_python_reserved_words(attr) + js_delattr = run_js( + """ + (jsobj, attr) => { + delete jsobj[attr]; + } + """ + ) + js_delattr(self, attr) + + def __dir__(self): + return object.__dir__(self) + js_dir(self) + + def __eq__(self, other): + if not isinstance(other, JsProxy): + return False + js_eq = run_js("(x, y) => x === y") + return js_eq(self, other) + + def __ne__(self, other): + if not isinstance(other, JsProxy): + return True + js_neq = run_js("(x, y) => x !== y") + return js_neq(self, other) + + def __repr__(self): + js_repr = run_js("x => x.toString()") + return js_repr(self) + + def __bool__(self): + return js_bool(self) + + @property + def js_id(self): + """ + This returns an integer with the property that jsproxy1 == jsproxy2 + if and only if jsproxy1.js_id == jsproxy2.js_id. There is no way to + express the implementation in pseudocode. + """ + raise NotImplementedError + + def as_py_json(self): + """ + This is actually a mixin method. We leave it out if any of the flags + IS_CALLABLE, IS_DOUBLE_PROXY, IS_ERROR, or IS_ITERATOR + is set. + """ + flags = self._js_type_flags + if (flags & (IS_ARRAY | IS_ARRAY_LIKE)): + flags |= IS_PY_JSON_SEQUENCE + else: + flags |= IS_PY_JSON_DICT + return create_jsproxy_with_flags(flags, self, self.jsthis) + + def to_py(self, *, depth=-1, default_converter=None): + """ + See section on deep conversions. + """ + ... + + def object_entries(self): + js_object_entries = run_js("x => Object.entries(x)") + return js_object_entries(self) + + def object_keys(self): + js_object_keys = run_js("x => Object.keys(x)") + return js_object_keys(self) + + def object_values(self): + js_object_values = run_js("x => Object.values(x)") + return js_object_values(self) + + def to_weakref(self): + js_weakref = run_js("x => new WeakRef(x)") + return js_weakref(self) + +We need the following function which calls the ``as_py_json()`` method on +``value`` if it is present: + +.. code-block:: python + + def maybe_as_py_json(value): + if ( + isinstance(value, JsProxy) + and hasattr(value, as_py_json) + ): + return value.as_py_json() + return value + + +Determining Which Flags to Set +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We need the helper function ``getTypeTag``: + +.. code:: javascript + + function getTypeTag(x) { + try { + return Object.prototype.toString.call(x); + } catch (e) { + // Catch and ignore errors + return ""; + } + } + +We use the following function to determine which flags to set: + +.. code:: javascript + + function compute_type_flags(obj, is_py_json) { + let type_flags = 0; + + const typeTag = getTypeTag(obj); + const hasLength = + isArray || (hasProperty(obj, "length") && typeof obj !== "function"); + + SET_FLAG_IF_HAS_METHOD(HAS_GET, "get"); + SET_FLAG_IF_HAS_METHOD(HAS_SET, "set"); + SET_FLAG_IF_HAS_METHOD(HAS_HAS, "has"); + SET_FLAG_IF_HAS_METHOD(HAS_INCLUDES, "includes"); + SET_FLAG_IF( + HAS_LENGTH, + hasProperty(obj, "size") || hasLength + ); + SET_FLAG_IF_HAS_METHOD(HAS_DISPOSE, Symbol.dispose); + SET_FLAG_IF(IS_CALLABLE, typeof obj === "function"); + SET_FLAG_IF(IS_ARRAY, Array.isArray(obj)); + SET_FLAG_IF( + IS_ARRAY_LIKE, + !isArray && hasLength && (type_flags & IS_ITERABLE)); + SET_FLAG_IF(IS_DOUBLE_PROXY, isPyProxy(obj)); + SET_FLAG_IF(IS_GENERATOR, typeTag === "[object Generator]"); + SET_FLAG_IF_HAS_METHOD(IS_ITERABLE, Symbol.iterator); + SET_FLAG_IF( + IS_ERROR, + hasProperty(obj, "name") && + hasProperty(obj, "message") && + (hasProperty(obj, "stack") || constructorName === "DOMException") && + !(type_flags & IS_CALLABLE) + ); + + if (is_py_json && type_flags & (IS_ARRAY | IS_ARRAY_LIKE)) { + type_flags |= IS_PY_JSON_SEQUENCE; + } else if ( + is_py_json && + !(type_flags & (IS_DOUBLE_PROXY | IS_ITERATOR | IS_CALLABLE | IS_ERROR)) + ) { + type_flags |= IS_PY_JSON_DICT; + } + const mapping_flags = HAS_GET | HAS_LENGTH | IS_ITERABLE; + const mutable_mapping_flags = mapping_flags | HAS_SET; + SET_FLAG_IF(IS_MAPPING, type_flags & (mapping_flags === mapping_flags)); + SET_FLAG_IF( + IS_MUTABLE_MAPPING, + type_flags & (mutable_mapping_flags === mutable_mapping_flags), + ); + + SET_FLAG_IF(IS_MAPPING, type_flags & IS_PY_JSON_DICT); + SET_FLAG_IF(IS_MUTABLE_MAPPING, type_flags & IS_PY_JSON_DICT); + + return type_flags; + } + +The ``HAS_GET`` Mixin +~~~~~~~~~~~~~~~~~~~~~ + +If a JavaScript ``get()`` method is present, we define ``__getitem__`` as +follows. If a ``has()`` method is also present, we'll use it to decide whether +an ``undefined`` return value should be treated as a key error or as ``None``. +If no ``has()`` method is present, ``undefined`` is treated as ``None``. + +.. code-block:: javascript + + function js_get(jsobj, item) { + const result = jsobj.get(item); + if (result !== undefined) { + return result; + } + if (hasMethod(obj, "has") && !obj.has(key)) { + throw new PythonKeyError(item); + } + return undefined; + } + +.. code-block:: python + + class JsProxyHasGetMixin: + def __getitem__(self, item): + result = js_get(self, item) + if self._js_type_flags & IS_PY_JSON_DICT: + result = maybe_as_py_json(result) + return result + +The ``HAS_SET`` Mixin +~~~~~~~~~~~~~~~~~~~~~ + +If a ``set()`` method is present, we assume a ``delete()`` method is also +present and define ``__setitem__`` and ``__delitem__`` as follows: + +.. code-block:: python + + class JsProxyHasSetMixin: + def __setitem__(self, item, value): + js_set = run_js( + """ + (jsobj, item, value) => { + jsobj.set(item, value); + } + """ + ) + js_set(self, item, value) + + def __delitem__(self, item, value): + js_delete = run_js( + """ + (jsobj, item) => { + jsobj.delete(item); + } + """ + ) + js_delete(self, item) + +The ``HAS_HAS`` Mixin +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + class JsProxyHasHasMixin: + def __contains__(self, item): + js_has = run_js( + """ + (jsobj, item) => jsobj.has(item); + """ + ) + return js_has(self, item) + + +The ``HAS_INCLUDES`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + class JsProxyHasIncludesMixin: + def __contains__(self, item): + js_includes = run_js( + """ + (jsobj, item) => jsobj.includes(item); + """ + ) + return js_includes(self, item) + + +The ``HAS_LENGTH`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~ + +We prefer to use the ``size`` attribute if present and a number and if not fall +back to returning the ``length``. If a JavaScript error is raised when looking +up either field, we allow it to propagate into Python as a +``JavaScriptException``. + +.. code-block:: python + + class JsProxyHasLengthMixin: + def __len__(self, item): + js_len = run_js( + """ + (jsobj) => { + const size = val.size; + if (typeof size === "number") { + return size; + } + return val.length + } + """ + ) + result = js_len(self) + if not isinstance(result, int): + raise TypeError("object does not have a valid length") + if result < 0: + raise ValueError("length of object is negative") + return result + +The ``HAS_DISPOSE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This makes the ``JsProxy`` into a context manager where ``__enter__`` is a no-op +and ``__exit__`` calls the ``[Symbol.dispose]()`` method. + +.. code-block:: python + + class JsProxyContextManagerMixin: + def __enter__(self): + return self + + def __exit__(self, type, value, traceback): + js_symbol_dispose = run_js( + """ + (jsobj) => jsobj[Symbol.dispose]() + """ + ) + js_symbol_dispose(self) + + +The ``IS_ARRAY`` Mixin +~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: javascript + + function js_array_slice(jsobj, length, start, stop, step) { + let result; + if (step === 1) { + result = obj.slice(start, stop); + } else { + result = Array.from({ length }, (_, i) => obj[start + i * step]); + } + return result; + } + + // we also use this for deletion by setting values to None + function js_array_slice_assign(obj, slicelength, start, stop, step, values) { + if (step === 1) { + obj.splice(start, slicelength, ...(values ?? [])); + return; + } + if (values !== undefined) { + for (let i = 0; i < slicelength; i++) { + obj.splice(start + i * step, 1, values[i]); + } + } + for (let i = slicelength - 1; i >= 0; i --) { + obj.splice(start + i * step, 1); + } + } + +.. code-block:: python + + class JsArrayMixin(MutableSequence, JsProxyHasLengthMixin): + def __getitem__(self, index): + if not isinstance(index, (int, slice)): + raise TypeError("Expected index to be an int or a slice") + length = len(self) + js_array_get = run_js( + """ + (jsobj, index) => jsobj[index] + """ + ) + if isinstance(index, int): + if index >= length: + raise IndexError(index) + if index < -length: + raise IndexError(index) + if index < 0: + index += length + result = js_array_get(self, index) + if self._js_type_flags & IS_PY_JSON_SEQUENCE: + result = maybe_as_py_json(result) + return result + start = index.start + stop = index.stop + step = index.step + slicelength = PySlice_AdjustIndices(length, &start, &stop, &step) + if (slicelength <= 0) { + return _PyJsvArray_New(); + } + result = js_array_slice(self, slicelength, start, stop, step) + if self._js_type_flags & IS_PY_JSON_SEQUENCE: + result = result.as_py_json() + return result + + + def __setitem__(self, index, value): + if not isinstance(index, (int, slice)): + raise TypeError("Expected index to be an int or a slice") + length = len(self) + js_array_set = run_js( + """ + (jsobj, index, value) => { jsobj[index] = value; } + """ + ) + if isinstance(index, int): + if index >= length: + raise IndexError(index) + if index < -length: + raise IndexError(index) + if index < 0: + index += length + result = js_array_set(self, index, value) + return + if not isinstance(value, Iterable): + raise TypeError("must assign iterable to extended slice") + seq = list(value) + start = index.start + stop = index.stop + step = index.step + slicelength = PySlice_AdjustIndices(length, &start, &stop, &step) + if step != 1 and len(seq) != slicelength: + raise TypeError( + f"attempted to assign sequence of length {len(seq)} to" + f"extended slice of length {slicelength}" + ) + if step != 1 and slicelength == 0: + return + js_array_slice_assign(self, slicelength, start, stop, step, seq) + + def __delitem__(self, index): + if not isinstance(index, (int, slice)): + raise TypeError("Expected index to be an int or a slice") + length = len(self) + js_array_delete = run_js( + """ + (jsobj, index) => { jsobj.splice(index, 1); } + """ + ) + if isinstance(index, int): + if index >= length: + raise IndexError(index) + if index < -length: + raise IndexError(index) + if index < 0: + index += length + result = js_array_delete(self, index) + return + start = index.start + stop = index.stop + step = index.step + slicelength = PySlice_AdjustIndices(length, &start, &stop, &step) + if step != 1 and slicelength == 0: + return + js_array_slice_assign(self, slicelength, start, stop, step, None) + + def insert(self, pos, value): + if not isinstance(pos, int): + raise TypeError("Expected an integer") + js_insert = run_js( + """ + (jsarr, pos, value) => { jsarr.splice(pos, value); } + """ + ) + js_insert(self, pos, value) + +The ``IS_ARRAY_LIKE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +.. code-block:: python + + class JsArrayLikeMixin(MutableSequence, JsProxyHasLengthMixin): + def __getitem__(self, index): + if not isinstance(index, int): + raise TypeError("Expected index to be an int") + JsArrayMixin.__getitem__(self, index) + + def __setitem__(self, index, value): + if not isinstance(index, int): + raise TypeError("Expected index to be an int") + JsArrayMixin.__setitem__(self, index, value) + + def __delitem__(self, index): + if not isinstance(index, int): + raise TypeError("Expected index to be an int") + JsArrayMixin.__delitem__(self, index, value) + +The ``IS_CALLABLE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~ +We already gave more accurate C code for calling a ``JsCallable``. See in +particular the definition of ``JsMethod_ConvertArgs()`` given there. + +.. code-block:: python + + class JsCallableMixin: + def __get__(self, obj): + """Return a new jsproxy bound to jsthis with the same JS object""" + return create_jsproxy(self, jsthis=obj) + + def __call__(self, *args, **kwargs): + """See the description of JsMethod_Vectorcall""" + + def new(self, *args, **kwargs): + pyproxies = [] + jsargs = JsMethod_ConvertArgs(args, kwargs, pyproxies) + + do_construct = run_js( + """ + (jsfunc, jsargs) => + Reflect.construct(jsfunc, jsargs) + """ + ) + result = do_construct(self, jsargs) + msg = ( + "This borrowed proxy was automatically destroyed " + "at the end of a function call." + ) + for px in pyproxies: + px.destroy(msg) + return result + +The ``IS_ERROR`` Mixin +~~~~~~~~~~~~~~~~~~~~~~ + +In this case, we inherit from both ``Exception`` and ``JsProxy``. We also make +sure that the resulting class is pickleable. + +The ``IS_ITERABLE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~ + +If the iterable has the ``IS_PY_JSON_DICT`` flag set, we iterate over the object +keys. Otherwise, call ``obj[Symbol.iterator]()``. If either +``IS_PY_JSON_SEQUENCE`` or ``IS_PY_JSON_DICT``, we call ``maybe_as_py_json`` on +the iteration results. + +.. code-block:: python + + def wrap_with_maybe_as_py_json(it): + try: + while val := it.next() + yield maybe_as_py_json(val) + except StopIteration(result): + return maybe_as_py_json(result) + + + class JsIterableMixin: + def __iter__(self): + pyjson = self._js_type_flags & (IS_PY_JSON_SEQUENCE | IS_PY_JSON_DICT) + pyjson_dict = self._js_type_flags & IS_PY_JSON_DICT + js_get_iter = run_js( + """ + (obj) => obj[Symbol.iterator]() + """ + ) + + if pyjson_dict: + result = iter(self.object_keys()) + else: + result = js_get_iter(self) + + if pyjson: + result = wrap_with_maybe_as_py_json(result) + return result + + +The ``IS_ITERATOR`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The JavaScript ``next`` method returns an ``IteratorResult`` which has a +``done`` field and a ``value`` field. If ``done`` is ``true``, we have to raise +a ``StopIteration`` exception to convert to the Python iterator protocol. + +.. code-block:: python + + class JsIteratorMixin: + def __iter__(self): + return self + + def send(self, arg): + js_next = run_js( + """ + (obj, arg) => obj.next(arg) + """ + ) + it_result = js_next(self, arg) + value = it_result.value + if it_result.done: + raise StopIteration(value) + return value + + def __next__(self): + return self.send(None) + + +The ``IS_GENERATOR`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Python generators have a ``close()`` method which takes no arguments instead of +a ``return()`` method. We also have to translate ``gen.throw(GeneratorExit)`` +into ``jsgen.return_()``. It is possible to call ``jsgen.return_(val)`` directly +if there is a need to return a specific value. + +.. code-block:: python + + class JsGeneratorMixin(JsIteratorMixin): + def throw(self, exc): + if isinstance(exc, GeneratorExit): + js_throw = run_js( + """ + (obj, exc) => obj.return() + """ + ) + else: + js_throw = run_js( + """ + (obj, exc) => obj.throw(exc) + """ + ) + it_result = js_throw(self, exc) + # if the error wasn't caught it will get raised back out. + # now handle the case where the error got caught. + value = it_result.value + if self._js_type_flags & IS_PY_JSON_SEQUENCE: + value = maybe_as_py_json(value) + if it_result.done: + raise StopIteration(value) + return value + + def close(self): + self.throw(GeneratorExit) + +The ``IS_MAPPING`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~ + +If the ``IS_MAPPING`` flag is set, we implement all of the ``Mapping`` methods. +We only set this flag when there are enough other flags set that the abstract +``Mapping`` methods are defined. We use the default implementations for all the +mixin methods. + +The ``IS_MUTABLE_MAPPING`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If the ``IS_MUTABLE_MAPPING`` flag is set, we implement all of the +``MutableMapping`` methods. We only set this flag when there are enough other +flags set that the abstract ``MutableMapping`` methods are defined. We use the +default implementations for all the mixin methods. + + +The ``IS_PY_JSON_SEQUENCE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This flag only ever appears with ``IS_ARRAY``. It changes the behavior of +``JsArray.__getitem__`` to apply ``maybe_as_py_json()`` to the result. + +The ``IS_PY_JSON_DICT`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + class JsPyJsonDictMixin(MutableMapping): + def __getitem__(self, key): + if not isinstance(key, str): + raise KeyError(key) + js_get = run_js( + """ + (jsobj, key) => jsobj[key] + """ + ) + result = js_get(self, key) + if result is None and not key in self: + raise KeyError(key) + return maybe_as_py_json(result) + + def __setitem__(self, key, value): + if not isinstance(key, str): + raise TypeError("only keys of type string are supported") + js_set = run_js( + """ + (jsobj, key, value) => { + jsobj[key] = value; + } + """ + ) + js_set(self, key, value) + + def __delitem__(self, key): + if not isinstance(key, str): + raise TypeError("only keys of type string are supported") + if not key in self: + raise KeyError(key) + js_delete = run_js( + """ + (jsobj, key) => { + delete jsobj[key]; + } + """ + ) + js_delete(self, key) + + def __contains__(self, key): + if not isinstance(key, str): + return False + js_contains = run_js( + """ + (jsobj, key) => key in jsobj + """ + ) + return js_contains(self, key) + + def __len__(self): + return sum(1 for _ in self) + + def __iter__(self): + # defined by IS_ITERABLE mixin, see implementation there. + + +The ``IS_DOUBLE_PROXY`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In this case the object is a ``JsProxy`` of a ``PyProxy``. We add an extra +``unwrap()`` method that returns the inner Python object. + + +PyProxy +------- + +We define 12 mixins that a Python object may support that affect the type of the +PyProxy we make from it. + +``HAS_GET`` + We set this flag if the Python object has a ``__getitem__`` method. If + present, we use it to implement a ``get()`` method on the ``PyProxy``. + +``HAS_SET`` + We set this flag if the Python object has a ``__setitem__`` method. If + present, we use it to implement a ``set()`` method on the ``PyProxy``. + +``HAS_CONTAINS`` + We set this flag if the Python object has a ``__contains__`` method. If + present, we use it to implement a ``has()`` method on the ``PyProxy``. + +``HAS_LENGTH`` + We set this flag if the Python object has a ``__len__`` method. If present, + we use it to implement a ``length`` getter on the ``PyProxy``. + +``IS_CALLABLE`` + We set this flag if the Python object has a ``__call__`` method. If present, + we make the ``PyProxy`` callable. + +``IS_DICT`` + We set this flag if the Python object is of exact type ``dict``. If present, + we will make property ``pyproxy.some_property`` fall back to + ``pyobj.__getitem__("some_property")`` if ``getattr(pyobj, "some_property")`` + raises an ``AttributeError``. + +``IS_GENERATOR`` + We set this flag if the Python object is an instance of + ``collections.abc.Generator``. If present, we make the ``PyProxy`` implement + the methods of a JavaScript generator. + +``IS_ITERABLE`` + We set this flag if the Python object has a ``__iter__`` method. If present, + we use it to implement a ``[Symbol.iterator]`` method on the ``PyProxy``. + +``IS_ITERATOR`` + We set this flag if the Python object has a ``__next__`` method. If present, + we use it to implement a ``next()`` method on the ``PyProxy``. + +``IS_SEQUENCE`` + We set this flag if the Python object is an instance of + ``collections.abc.Sequence``. If it is present, we use it to implement all + of the ``Array.prototype`` methods that don't mutate on the ``PyProxy``. + +``IS_MUTABLE_SEQUENCE`` + We set this flag if the Python object is an instance of + ``collections.abc.MutableSequence``. If it is present, we use it to implement + all ``Array.prototype`` methods on the ``PyProxy``. + +``IS_JS_JSON_DICT`` + We set this flag when the ``asJsJson()`` method is used on a dictionary. If + this flag is set, property access on the ``PyProxy`` will _only_ look at + values from ``__getitem__`` and not at attributes on the Python object. We + also will call ``asJsJson()`` on the result of indexing or iterating + the ``PyProxy``. + +``IS_JS_JSON_SEQUENCE`` + We set this flag when the ``asJsJson()`` is used on a ``Sequence``. If this + flag is set, we will call ``asJsJson()`` on the result of indexing or + iterating the ``PyProxy``. + + +A ``PyProxy`` is made up of a mixture of a JavaScript class and a collection of +ES6 ``Proxy`` handlers. Depending on which flags are present, we construct our +class out of an appropriate collection of mixins and an appropriate choice of +handlers. + +When a ``PyProxy`` is created, we increment the reference count of the wrapped +Python object. When a ``PyProxy`` is destroyed, we decrement the reference count +and mark it as destroyed. As a result, if we attempt to do anything with the +``PyProxy``, we will call ``_Py_js2python()`` on it and an error will be thrown. + +Creating a ``PyProxy`` +~~~~~~~~~~~~~~~~~~~~~~ + +Given a collection of type flags, we use the following function to generate the +``PyProxy`` class: + +.. code-block:: javascript + + let pyproxyClassMap = new Map(); + function getPyProxyClass(flags: number) { + let result = pyproxyClassMap.get(flags); + if (result) { + return result; + } + let descriptors: any = {}; + const FLAG_MIXIN_PAIRS: [number, any][] = [ + [HAS_CONTAINS, PyContainsMixin], + // ... other flag mixin pairs + [IS_MUTABLE_SEQUENCE, PyMutableSequenceMixin], + ]; + for (let [feature_flag, methods] of FLAG_MIXIN_PAIRS) { + if (flags & feature_flag) { + Object.assign( + descriptors, + Object.getOwnPropertyDescriptors(methods.prototype), + ); + } + } + // Use base constructor (just throws an error if construction is attempted). + descriptors.constructor = Object.getOwnPropertyDescriptor( + PyProxyProto, + "constructor", + ); + // $$flags static field + Object.assign( + descriptors, + Object.getOwnPropertyDescriptors({ $$flags: flags }), + ); + // We either inherit PyProxyFunction as the base class if we're callable or + // from PyProxy if we're not. + const superProto = flags & IS_CALLABLE ? PyProxyFunctionProto : PyProxyProto; + const subProto = Object.create(superProto, descriptors); + function NewPyProxyClass() {} + NewPyProxyClass.prototype = subProto; + pyproxyClassMap.set(flags, NewPyProxyClass); + return NewPyProxyClass; + } + +To create a ``PyProxy`` we also need to be able to get the appropriate handlers: + +.. code-block:: javascript + + function getPyProxyHandlers(flags) { + if (flags & IS_JS_JSON_DICT) { + return PyProxyJsJsonDictHandlers; + } + if (flags & IS_DICT) { + return PyProxyDictHandlers; + } + if (flags & IS_SEQUENCE) { + return PyProxySequenceHandlers; + } + return PyProxyHandlers; + } + +We use the following function to create the target object for the ES6 proxy: + +.. code-block:: javascript + + function createTarget(flags) { + const pyproxyClass = getPyProxyClass(flags); + if (!(flags & IS_CALLABLE)) { + return Object.create(cls.prototype); + } + // In this case we are effectively subclassing Function in order to ensure + // that the proxy is callable. With a Content Security Protocol that doesn't + // allow unsafe-eval, we can't invoke the Function constructor directly. So + // instead we create a function in the universally allowed way and then use + // `setPrototypeOf`. The documentation for `setPrototypeOf` says to use + // `Object.create` or `Reflect.construct` instead for performance reasons + // but neither of those work here. + const target = function () {}; + Object.setPrototypeOf(target, cls.prototype); + // Remove undesirable properties added by Function constructor. Note: we + // can't remove "arguments" or "caller" because they are not configurable + // and not writable + delete target.length; + delete target.name; + // prototype isn't configurable so we can't delete it but it is writable. + target.prototype = undefined; + return target; + } + +``createPyProxy`` takes the following options: + +flags + If this is passed, we use the passed flags rather than feature + detecting the object again. + +props + Information that not shared with other PyProxies of the same lifetime. + +shared + Data that is shared between all proxies with the same lifetime as this one. + +gcRegister + Should we register this with the JavaScript garbage collector? + + +.. code-block:: javascript + + const pyproxyAttrsSymbol = Symbol("pyproxy.attrs"); + function createPyProxy( + pyObjectPtr: number, + { + flags, + props, + shared, + gcRegister, + } + ) { + if (gcRegister === undefined) { + // register by default + gcRegister = true; + } + + // See the section "Determining which flags to set" for the definition of + // get_pyproxy_flags + const pythonGetFlags = makePythonFunction("get_pyproxy_flags"); + flags ??= pythonGetFlags(pyObjectPtr); + const target = createTarget(flags); + const handlers = getPyProxyHandlers(flags); + const proxy = new Proxy(target, handlers); + + props = Object.assign( + { isBound: false, captureThis: false, boundArgs: [], roundtrip: false }, + props, + ); + + // If shared was passed the new PyProxy will have a shared lifetime + // with some other PyProxy. + // This happens in asJsJson(), bind(), and captureThis(). + // It specifically does not happen in copy() + if (!shared) { + shared = { + pyObjectPtr, + destroyed_msg: undefined, + gcRegistered: false, + }; + _Py_IncRef(pyObjectPtr); + if (gcRegister) { + gcRegisterPyProxy(shared); + } + } + target[pyproxyAttrsSymbol] = { shared, props }; + return proxy; + } + +The ``PyProxy`` Base Class +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The default handlers are as follows: + +.. code-block:: javascript + + function filteredHasKey(jsobj, jskey, filterProto) { + let result = jskey in jsobj; + if (jsobj instanceof Function) { + // If we are a PyProxy of a callable we have to subclass function so that if + // someone feature detects callables with `instanceof Function` it works + // correctly. But the callable might have attributes `name` and `length` and + // we don't want to shadow them with the values from `Function.prototype`. + result &&= !( + ["name", "length", "caller", "arguments"].includes(jskey) || + // we are required by JS law to return `true` for `"prototype" in pycallable` + // but we are allowed to return the value of `getattr(pycallable, "prototype")`. + // So we filter prototype out of the "get" trap but not out of the "has" trap + (filterProto && jskey === "prototype") + ); + } + return result; + } + + const PyProxyHandlers = { + isExtensible() { + return true; + }, + has(jsobj, jskey) { + // Must report "prototype" in proxy when we are callable. + // (We can return the wrong value from "get" handler though.) + if (filteredHasKey(jsobj, jskey, false)) { + return true; + } + // hasattr will crash if given a Symbol. + if (typeof jskey === "symbol") { + return false; + } + if (jskey.startsWith("$")) { + jskey = jskey.slice(1); + } + const pythonHasAttr = makePythonFunction("hasattr"); + return pythonHasAttr(jsobj, jskey); + }, + get(jsobj, jskey) { + // Preference order: + // 1. stuff from JavaScript + // 2. the result of Python getattr + // pythonGetAttr will crash if given a Symbol. + if (typeof jskey === "symbol" || filteredHasKey(jsobj, jskey, true)) { + return Reflect.get(jsobj, jskey); + } + if (jskey.startsWith("$")) { + jskey = jskey.slice(1); + } + // 2. The result of getattr + const pythonGetAttr = makePythonFunction("getattr"); + return pythonGetAttr(jsobj, jskey); + }, + set(jsobj, jskey, jsval) { + let descr = Object.getOwnPropertyDescriptor(jsobj, jskey); + if (descr && !descr.writable && !descr.set) { + return false; + } + // pythonSetAttr will crash if given a Symbol. + if (typeof jskey === "symbol" || filteredHasKey(jsobj, jskey, true)) { + return Reflect.set(jsobj, jskey, jsval); + } + if (jskey.startsWith("$")) { + jskey = jskey.slice(1); + } + const pythonSetAttr = makePythonFunction("setattr"); + pythonSetAttr(jsobj, jskey, jsval); + return true; + }, + deleteProperty(jsobj, jskey: string | symbol): boolean { + let descr = Object.getOwnPropertyDescriptor(jsobj, jskey); + if (descr && !descr.configurable) { + // Must return "false" if "jskey" is a nonconfigurable own property. + // Strict mode JS will throw an error here saying that the property cannot + // be deleted. + return false; + } + if (typeof jskey === "symbol" || filteredHasKey(jsobj, jskey, true)) { + return Reflect.deleteProperty(jsobj, jskey); + } + if (jskey.startsWith("$")) { + jskey = jskey.slice(1); + } + const pythonDelAttr = makePythonFunction("delattr"); + pythonDelAttr(jsobj, jskey); + return true; + }, + ownKeys(jsobj) { + const pythonDir = makePythonFunction("dir"); + const result = pythonDir(jsobj).toJs(); + result.push(...Reflect.ownKeys(jsobj)); + return result; + }, + apply(jsobj: PyProxy & Function, jsthis: any, jsargs: any): any { + return jsobj.apply(jsthis, jsargs); + }, + }; + +And the base class has the following methods: + +.. code-block:: javascript + + class PyProxy { + constructor() { + throw new TypeError("PyProxy is not a constructor"); + } + get [Symbol.toStringTag]() { + return "PyProxy"; + } + static [Symbol.hasInstance](obj: any): obj is PyProxy { + return [PyProxy, PyProxyFunction].some((cls) => + Function.prototype[Symbol.hasInstance].call(cls, obj), + ); + } + get type() { + const pythonType = makePythonFunction(` + def python_type(obj): + ty = type(obj) + if ty.__module__ in ['builtins', 'main']: + return ty.__name__ + return ty.__module__ + "." + ty.__name__ + `); + return pythonType(this); + } + toString() { + const pythonStr = makePythonFunction("str"); + return pythonStr(this); + } + destroy(options) { + const { shared } = proxy[pyproxyAttrsSymbol]; + if (!shared.pyObjectPtr) { + // already destroyed + return; + } + shared.pyObjectPtr = 0; + shared.destroyed_msg = options.message ?? "Object has already been destroyed"; + _Py_DecRef(shared.pyObjectPtr); + } + [Symbol.dispose]() { + this.destroy(); + } + copy() { + const { shared, props } = proxy[pyproxyAttrsSymbol]; + // Don't pass shared as an option since we want this new PyProxy to + // have a distinct lifetime from the one we are copying. + return createPyProxy(shared.pyObjectPtr, { + flags: this.$$flags, + props: attrs.props, + }); + } + toJs(options) { + // See the definition of to_js in "Deep conversions". + } + } + +Determining Which Flags to Set +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We separate this out into a component ``get_type_flags`` that computes flags +which only depends on the type and a component that also depends on whether the +``PyProxy`` has beenJsJson + +.. code-block:: python + + def get_type_flags(ty): + from collections.abc import Generator, MutableSequence, Sequence + + flags = 0 + if hasattr(ty, "__len__"): + flags |= HAS_LENGTH + if hasattr(ty, "__getitem__"): + flags |= HAS_GET + if hasattr(ty, "__setitem__"): + flags |= HAS_SET + if hasattr(ty, "__contains__"): + flags |= HAS_CONTAINS + if ty is dict: + # Currently we don't set this on subclasses. + flags |= IS_DICT + if hasattr(ty, "__call__"): + flags |= IS_CALLABLE + if hasattr(ty, "__iter__"): + flags |= IS_ITERABLE + if hasattr(ty, "__next__"): + flags |= IS_ITERATOR + if issubclass(ty, Generator): + flags |= IS_GENERATOR + if issubclass(ty, Sequence): + flags |= IS_SEQUENCE + if issubclass(ty, MutableSequence): + flags |= IS_MUTABLE_SEQUENCE + return flags + + def get_pyproxy_flags(obj, is_js_json): + flags = get_type_flags(type(obj)) + if not is_js_json: + return flags + if flags & IS_SEQUENCE: + flags |= IS_JS_JSON_SEQUENCE + elif flags & HAS_GET: + flags |= IS_JS_JSON_DICT + return flags + + +The ``HAS_GET`` Mixin +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: javascript + + const pythonGetItem = makePythonFunction(` + def getitem(obj, key): + return obj[key] + `); + class PyProxyGetItemMixin { + get(key) { + let result = pythonGetItem(this, key); + const isJsJson = !!(this.$$flags & (IS_JS_JSON_DICT | IS_JS_JSON_SEQUENCE)); + if (isJsJson && result.asJsJson) { + result = result.asJsJson(); + } + return result; + } + asJsJson() { + const flags = this.$$flags | IS_JS_JSON_DICT; + const { shared, props } = this[pyproxyAttrsSymbol]; + // Note: The PyProxy created here has the same lifetime as the PyProxy it is + // created from. Destroying either destroys both. + return createPyProxy(shared.ptr, { flags, shared, props }); + } + } + +The ``HAS_SET`` Mixin +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: javascript + + class PyProxySetItemMixin { + set(key, value) { + const pythonSetItem = makePythonFunction(` + def setitem(obj, key, value): + obj[key] = value + `); + pythonSetItem(this, key, value); + } + delete(key) { + const pythonDelItem = makePythonFunction(` + def delitem(obj, key): + del obj[key] + `); + pythonDelItem(this, key); + } + } + +The ``HAS_CONTAINS`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: javascript + + const pythonHasItem = makePythonFunction(` + def hasitem(obj, key): + return key in obj + `); + class PyContainsMixin { + has(key) { + return pythonHasItem(this, key); + } + } + +The ``HAS_LENGTH`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: javascript + + const pythonLength = makePythonFunction("len"); + class PyLengthMixin { + get length() : number { + return pythonLength(this); + } + } + +The ``IS_CALLABLE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~ + +We have to make a custom prototype and class so that this inherits from both +``PyProxy`` and ``Function``: + +.. code-block:: javascript + + const PyProxyFunctionProto = Object.create( + Function.prototype, + Object.getOwnPropertyDescriptors(PyProxy.prototype), + ); + function PyProxyFunction() {} + PyProxyFunction.prototype = PyProxyFunctionProto; + +We use the following helper function which inserts ``this`` as the first +argument if ``captureThis`` is ``true`` and adds any bound arguments. + +.. code-block:: javascript + + function _adjustArgs(pyproxy, jsthis, jsargs) { + const { props } = this[pyproxyAttrsSymbol]; + const { captureThis, boundArgs, boundThis, isBound } = props; + if (captureThis) { + if (isBound) { + return [boundThis].concat(boundArgs, jsargs); + } else { + return [jsthis].concat(jsargs); + } + } + if (isBound) { + return boundArgs.concat(jsargs); + } + return jsargs; + } + +Then we implement the following methods. ``apply()``, ``call()``, and ``bind()`` +are methods from ``Function.prototype``. ``callKwargs()`` and ``captureThis()`` +are special to ``PyProxy`` + + +.. code-block:: javascript + + export class PyCallableMixin { + apply(thisArg, jsargs) { + // Convert jsargs to an array using ordinary .apply in order to match the + // behavior of .apply very accurately. + jsargs = function (...args) { + return args; + }.apply(undefined, jsargs); + jsargs = _adjustArgs(this, thisArg, jsargs); + const pyObjectPtr = this[pyproxyAttrsSymbol].shared.pyObjectPtr; + return callPyObjectKwargs(pyObjectPtr, jsargs, {}); + } + call(thisArg, ...jsargs) { + jsargs = _adjustArgs(this, thisArg, jsargs); + const pyObjectPtr = this[pyproxyAttrsSymbol].shared.pyObjectPtr; + return callPyObjectKwargs(pyObjectPtr, jsargs, {}); + } + + /** + * Call the function with keyword arguments. The last argument must be an + * object with the keyword arguments. + */ + callKwargs(...jsargs) { + jsargs = _adjustArgs(this, thisArg, jsargs); + if (jsargs.length === 0) { + throw new TypeError( + "callKwargs requires at least one argument (the kwargs object)", + ); + } + let kwargs = jsargs.pop(); + if ( + kwargs.constructor !== undefined && + kwargs.constructor.name !== "Object" + ) { + throw new TypeError("kwargs argument is not an object"); + } + const pyObjectPtr = this[pyproxyAttrsSymbol].shared.pyObjectPtr; + return callPyObjectKwargs(pyObjectPtr, jsargs, kwargs); + } + /** + * This is our implementation of Function.prototype.bind(). + */ + bind(thisArg, ...jsargs) { + let { shared, props } = this[pyproxyAttrsSymbol]; + const { boundArgs: boundArgsOld, boundThis: boundThisOld, isBound } = props; + let boundThis = thisArg; + if (isBound) { + boundThis = boundThisOld; + } + const boundArgs = boundArgsOld.concat(jsargs); + props = Object.assign({}, props, { + boundArgs, + isBound: true, + boundThis, + }); + return createPyProxy(shared.ptr, { + shared, + flags: this.$$flags, + props, + }); + } + /** + * This method makes a new PyProxy where ``this`` is passed as the + * first argument to the Python function. The new PyProxy has the + * same lifetime as the original. + */ + captureThis() { + let { props, shared } = this[pyproxyAttrsSymbol]; + props = Object.assign({}, props, { + captureThis: true, + }); + return createPyProxy(shared.ptr, { + shared, + flags: this.$$flags, + props, + }); + } + } + + +The ``IS_DICT`` Mixin +~~~~~~~~~~~~~~~~~~~~~ + +The ``IS_DICT`` mixin does not include any extra methods but it uses a special +set of handlers. These handlers are a hybrid between the normal handlers and the +``JS_JSON_DICT`` handlers. We first check whether ``hasattr(d, property)`` and +if so return ``d.property``. If not, we return ``d.get(property, None)``. The +other methods all work similarly. See the ``IS_JS_JSON_DICT`` flag for the +definitions of those handlers. + +.. code-block:: javascript + + const PyProxyDictHandlers = { + isExtensible(): boolean { + return true; + }, + has(jsobj: PyProxy, jskey: string | symbol): boolean { + if (PyProxyHandlers.has(jsobj, jskey)) { + return true; + } + return PyProxyJsJsonDictHandlers.has(jsobj, jskey); + }, + get(jsobj: PyProxy, jskey: string | symbol): any { + let result = PyProxyHandlers.get(jsobj, jskey); + if (result !== undefined || PyProxyHandlers.has(jsobj, jskey)) { + return result; + } + return PyProxyJsJsonDictHandlers.get(jsobj, jskey); + }, + set(jsobj: PyProxy, jskey: string | symbol, jsval: any): boolean { + if (PyProxyHandlers.has(jsobj, jskey)) { + return PyProxyHandlers.set(jsobj, jskey, jsval); + } + return PyProxyJsJsonDictHandlers.set(jsobj, jskey, jsval); + }, + deleteProperty(jsobj: PyProxy, jskey: string | symbol): boolean { + if (PyProxyHandlers.has(jsobj, jskey)) { + return PyProxyHandlers.deleteProperty(jsobj, jskey); + } + return PyProxyJsJsonDictHandlers.deleteProperty(jsobj, jskey); + }, + getOwnPropertyDescriptor(jsobj: PyProxy, prop: any) { + return ( + Reflect.getOwnPropertyDescriptor(jsobj, prop) ?? + PyProxyJsJsonDictHandlers.getOwnPropertyDescriptor(jsobj, prop) + ); + }, + ownKeys(jsobj: PyProxy): (string | symbol)[] { + const result = [ + ...PyProxyHandlers.ownKeys(jsobj), + ...PyProxyJsJsonDictHandlers.ownKeys(jsobj) + ]; + // deduplicate + return Array.from(new Set(result)); + }, + }; + + +The ``IS_ITERABLE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: javascript + + const pythonNext = makePythonFunction("next"); + const getStopIterationValue = makePythonFunction(` + def get_stop_iteration_value(): + import sys + err = sys.last_value + return err.value + `); + + function* iterHelper(iter, isJsJson) { + try { + while (true) { + let item = pythonNext(iter); + if (isJsJson && item.asJsJson) { + item = item.asJsJson(); + } + yield item; + } + } catch (e) { + if (e.type === "StopIteration") { + return getStopIterationValue(); + } + throw e; + } + } + + const pythonIter = makePythonFunction("iter"); + class PyIterableMixin { + [Symbol.iterator]() { + const isJsJson = !!(this.$$flags & (IS_JS_JSON_DICT | IS_JS_JSON_SEQUENCE)); + return iterHelper(pythonIter(this), isJsJson); + } + } + + +The ``IS_ITERATOR`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: javascript + + const pythonSend = makePythonFunction(` + def python_send(it, val): + return gen.send(val) + `); + class PyIteratorMixin { + next(x) { + try { + const result = pythonSend(this, x); + return { done: false, value: result }; + } catch (e) { + if (e.type === "StopIteration") { + const result = getStopIterationValue(); + return { done: true, value: result }; + } + throw e; + } + } + } + + +The ``IS_GENERATOR`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: javascript + + const pythonThrow = makePythonFunction(` + def python_throw(gen, val): + return gen.throw(val) + `); + const pythonClose = makePythonFunction(` + def python_close(gen): + return gen.close() + `); + class PyGeneratorMixin extends PyIteratorMixin { + throw(exc) { + try { + const result = pythonThrow(this, exc); + return { done: false, value: result }; + } catch (e) { + if (e.type === "StopIteration") { + const result = getStopIterationValue(); + return { done: true, value: result }; + } + throw e; + } + } + return(value) { + pythonClose(this); + return { done: true, value } + } + } + + +The ``IS_SEQUENCE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~ + +We define all of the ``Array.prototype`` methods that don't mutate the sequence +on ``PySequenceMixin``. For most of them, the ``Array`` prototype method works +without changes. All of these we define with boilerplate of the form: + +.. code-block:: javascript + + [methodName](...args) { + return Array.prototype[methodName].call(this, ...args) + } + +These include ``join``, ``slice``, ``indexOf``, ``lastIndexOf``, ``forEach``, +``map``, ``filter``, ``some``, ``every``, ``reduce``, ``reduceRight``, ``at``, +``concat``, ``includes``, ``entries``, ``keys``, ``values``, ``find``, and +``findIndex``. Other than these boilerplate methods, the remaining attributes on +``PySequenceMixin`` are as follows. + +.. code-block:: javascript + + class PySequenceMixin { + get [Symbol.isConcatSpreadable]() { + return true; + } + toJSON() { + return Array.from(this); + } + asJsJson() { + const flags = this.$$flags | IS_JS_JSON_SEQUENCE; + const { shared, props } = this[pyproxyAttrsSymbol]; + // Note: Because we pass shared down, the PyProxy created here has + // the same lifetime as the PyProxy it is created from. Destroying + // either destroys both. + return createPyProxy(shared.ptr, { flags, shared, props }); + } + // ... boilerplate methods + } + + +Instead of the default proxy handlers, we use the following handlers for +sequences. We don't + +.. code-block:: javascript + + const PyProxySequenceHandlers = { + isExtensible() { + return true; + }, + has(jsobj, jskey) { + if (typeof jskey === "string" && /^[0-9]+$/.test(jskey)) { + // Note: if the number was negative it didn't match the pattern + return Number(jskey) < jsobj.length; + } + return PyProxyHandlers.has(jsobj, jskey); + }, + get(jsobj, jskey) { + if (jskey === "length") { + return jsobj.length; + } + if (typeof jskey === "string" && /^[0-9]+$/.test(jskey)) { + try { + return PyProxyGetItemMixin.prototype.get.call(jsobj, Number(jskey)); + } catch (e) { + if (isPythonError(e) && e.type == "IndexError") { + return undefined; + } + throw e; + } + } + return PyProxyHandlers.get(jsobj, jskey); + }, + set(jsobj: PyProxy, jskey: any, jsval: any): boolean { + if (typeof jskey === "string" && /^[0-9]+$/.test(jskey)) { + try { + PyProxySetItemMixin.prototype.set.call(jsobj, Number(jskey), jsval); + return true; + } catch (e) { + if (isPythonError(e) && e.type == "IndexError") { + return false; + } + throw e; + } + } + return PyProxyHandlers.set(jsobj, jskey, jsval); + }, + deleteProperty(jsobj: PyProxy, jskey: any): boolean { + if (typeof jskey === "string" && /^[0-9]+$/.test(jskey)) { + try { + PyProxySetItemMixin.prototype.delete.call(jsobj, Number(jskey)); + return true; + } catch (e) { + if (isPythonError(e) && e.type == "IndexError") { + return false; + } + throw e; + } + } + return PyProxyHandlers.deleteProperty(jsobj, jskey); + }, + ownKeys(jsobj: PyProxy): (string | symbol)[] { + const result = PyProxyHandlers.ownKeys(jsobj); + result.push( + ...Array.from({ length: jsobj.length }, (_, k) => k.toString()), + ); + result.push("length"); + return result; + }, + }; + + +The ``IS_MUTABLE_SEQUENCE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This adds some additional ``Array`` methods that mutate the sequence. + +.. code-block:: javascript + + class PyMutableSequenceMixin { + reverse() { + // Same as the Python reverse method except it returns this instead of undefined + this.$reverse(); + return this; + } + push(...elts: any[]) { + for (const elt of elts) { + this.append(elt); + } + return this.length; + } + splice(start, deleteCount, ...items) { + if (deleteCount === undefined) { + // Max signed size + deleteCount = (1 << 31) - 1; + } + let stop = start + deleteCount; + if (stop > this.length) { + stop = this.length; + } + const pythonSplice = makePythonFunction(` + def splice(array, start, stop, items): + from pyodide.ffi import to_js + result = to_js(array[start:stop], depth=1) + array[start:stop] = items + return result + `); + return pythonSplice(this, start, stop, items); + } + pop() { + const pythonPop = makePythonFunction(` + def pop(array): + return array.pop() + `); + return pythonPop(this); + } + shift() { + const pythonShift = makePythonFunction(` + def pop(array): + return array.pop(0) + `); + return pythonShift(this); + } + unshift(...elts) { + elts.forEach((elt, idx) => { + this.insert(idx, elt); + }); + return this.length; + } + // Boilerplate methods + copyWithin(...args): any { + Array.prototype.copyWithin.apply(this, args); + return this; + } + fill(...args) { + Array.prototype.fill.apply(this, args); + return this; + } + } + +The ``IS_JS_JSON_DICT`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are no methods special to the ``IS_JS_JSON_DICT`` flag, but we use the +following proxy handlers. We prefer to look up a property as an item in the +dictionary with two exceptions: + +1. Symbols we always look up on the ``PyProxy`` itself. +2. We also look up the keys ``$$flags``, ``copy()``, ``constructor``, + ``destroy`` and ``toString`` on the ``PyProxy``. + +All Python dictionary methods will be shadowed by a key of the same name. + +.. code-block:: javascript + + const PyProxyJsJsonDictHandlers = { + isExtensible(): boolean { + return true; + }, + has(jsobj: PyProxy, jskey: string | symbol): boolean { + if (PyContainsMixin.prototype.has.call(jsobj, jskey)) { + return true; + } + // If it doesn't exist as a string key and it looks like a number, + // try again with the number + if (typeof jskey === "string" && /^-?[0-9]+$/.test(jskey)) { + return PyContainsMixin.prototype.has.call(jsobj, Number(jskey)); + } + return false; + }, + get(jsobj, jskey): any { + if ( + typeof jskey === "symbol" || + ["$$flags", "copy", "constructor", "destroy", "toString"].includes(jskey) + ) { + return Reflect.get(...arguments); + } + const result = PyProxyGetItemMixin.prototype.get.call(jsobj, jskey); + if ( + result !== undefined || + PyContainsMixin.prototype.has.call(jsobj, jskey) + ) { + return result; + } + if (typeof jskey === "string" && /^-?[0-9]+$/.test(jskey)) { + return PyProxyGetItemMixin.prototype.get.call(jsobj, Number(jskey)); + } + return Reflect.get(...arguments); + }, + set(jsobj, jskey, jsval): boolean { + if (typeof jskey === "symbol") { + return false; + } + if ( + !PyContainsMixin.prototype.has.call(jsobj, jskey) && + typeof jskey === "string" && + /^-?[0-9]+$/.test(jskey) + ) { + jskey = Number(jskey); + } + try { + PyProxySetItemMixin.prototype.set.call(jsobj, jskey, jsval); + return true; + } catch (e) { + if (isPythonError(e) && e.type === "KeyError") { + return false; + } + throw e; + } + }, + deleteProperty(jsobj: PyProxy, jskey: string | symbol | number): boolean { + if (typeof jskey === "symbol") { + return false; + } + if ( + !PyContainsMixin.prototype.has.call(jsobj, jskey) && + typeof jskey === "string" && + /^-?[0-9]+$/.test(jskey) + ) { + jskey = Number(jskey); + } + try { + PyProxySetItemMixin.prototype.delete.call(jsobj, jskey); + return true; + } catch (e) { + if (isPythonError(e) && e.type === "KeyError") { + return false; + } + throw e; + } + }, + getOwnPropertyDescriptor(jsobj: PyProxy, prop: any) { + if (!PyProxyJsJsonDictHandlers.has(jsobj, prop)) { + return undefined; + } + const value = PyProxyJsJsonDictHandlers.get(jsobj, prop); + return { + configurable: true, + enumerable: true, + value, + writable: true, + }; + }, + ownKeys(jsobj: PyProxy): (string | symbol)[] { + const pythonDictOwnKeys = makePythonFunction(` + def dict_own_keys(d): + from pyodide.ffi import to_js + result = set() + for key in d: + if isinstance(key, str): + result.add(key) + elif isinstance(key, (int, float)): + result.add(str(key)) + return to_js(result) + `); + return pythonDictOwnKeys(jsobj); + }, + }; + + +The ``IS_JS_JSON_SEQUENCE`` Mixin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This has no direct impact on the prototype or handlers of the proxy. However, +when indexing the list or iterating over the list we will apply ``asJsJson()`` +to the results. + +Deep Conversions +---------------- + +We define ``JsProxy.to_py()`` to make deep conversions from JavaScript to Python +and ``pyodide.ffi.to_js()`` to make deep conversions from Python to JavaScript. +Note that it is not intended that these are inverse functions to each other. + +From JavaScript to Python +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``JsProxy.to_py()`` method makes the following conversions: + +* ``Array`` ==> ``list`` +* ``Map`` ==> ``dict`` +* ``Set`` ==> ``set`` +* ``Object`` ==> ``dict`` but only if the ``constructor`` is either ``Object`` + or ``undefined``. Other objects we leave alone. + +It takes the following optional arguments: + +``depth`` + An integer, specifies the maximum depth down to which to convert. For + instance, setting ``depth=1`` allows converting exactly one level. + +``default_converter`` + A function to be called when there is no known conversion for an object. + + +The default converter takes three arguments: + +``jsobj`` + The object to convert. + +``convert`` + Allows recursing. + +``cache_conversion`` + Cache the conversion of an object to allow converting self-referential data. + +For example, if we have a JavaScript ``Pair`` class and want to convert it to a +list, we can use the following ``default_converter``: + +.. code-block:: python + + def pair_converter(jsobj, convert, cache_conversion): + if jsobj.constructor.name != "Pair": + return jsobj + result = [] + cache_conversion(jsobj, result) + result.append(convert(jsobj.first)) + result.append(convert(jsobj.second)) + return result + +By first caching the result before making any recursive calls to ``convert``, we +ensure that if ``jsobj.first`` has a transitive reference to ``jsobj``, we +convert it correctly. + +Complete pseudocode for the ``to_py`` method is as follows: + +.. code-block:: python + + def to_py(jsobj, *, depth=-1, default_converter=None): + cache = {} + return ToPyConverter(depth, default_converter).convert(jsobj) + + class ToPyConverter: + def __init__(self, depth, default_converter): + self.cache = {} + self.depth = depth + self.default_converter = default_converter + + def cache_conversion(self, jsobj, pyobj): + self.cache[jsobj.js_id] = pyobj + + def convert(self, jsobj): + if self.depth == 0 or not isinstance(jsobj, JsProxy): + return jsobj + if result := self.cache.get(jsobj.js_id): + return result + + from js import Array, Object + type_tag = getTypeTag(jsobj) + self.depth -= 1 + try: + if Array.isArray(jsobj): + return self.convert_list(jsobj) + if type_tag == "[object Map]": + return self.convert_map(jsobj, jsobj.entries()) + if type_tag == "[object Set]": + return self.convert_set(jsobj) + if type_tag == "[object Object]" and (jsobj.constructor in [None, Object]): + return self.convert_map(jsobj, Object.entries(jsobj)) + if self.default_converter is not None: + return self.default_converter(jsobj, self.convert, self.cache_conversion) + return jsobj + finally: + self.depth += 1 + + def convert_list(self, jsobj): + result = [] + self.cache_conversion(jsobj, result) + for item in jsobj: + result.append(self.convert(item)) + return result + + def convert_map(self, jsobj, entries): + result = {} + self.cache_conversion(jsobj, result) + for [key, val] in entries: + result[key] = self.convert(val) + return result + + def convert_set(self, jsobj): + result = set() + self.cache_conversion(jsobj, result) + for key in jsobj: + result.add(self.convert(key)) + return result + + +From Python to JavaScript +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + def to_js( + obj, + /, + *, + depth=-1, + pyproxies=None, + create_pyproxies=True, + dict_converter=None, + default_converter=None, + eager_converter=None, + ): + converter = ToJsConverter( + depth, + pyproxies, + create_pyproxies, + dict_converter, + default_converter, + eager_converter, + ) + result = converter.convert(obj) + converter.postprocess() + return result + + class ToJsConverter: + def __init__( + self, + depth, + pyproxies, + create_pyproxies, + dict_converter, + default_converter, + eager_converter, + ): + self.depth = depth + self.pyproxies = pyproxies + self.create_pyproxies = create_pyproxies + if dict_converter is None: + dict_converter = Object.fromEntries + self.dict_converter = dict_converter + self.default_converter = default_converter + self.eager_converter = eager_converter + self.cache = {} + self.post_process_list = [] + self.pairs_to_dict_map = {} + + def cache_conversion(self, pyobj, jsobj): + self.cache[id(pyobj)] = jsobj + + def postprocess(self): + # Replace any NoValue's that appear once we've certainly computed + # their correct conversions + for parent, key, pyobj_id in self.post_process_list: + real_value = self.cache[pyobj_id] + # If it was a dictionary, we need to lookup the actual result object + real_parent = self.pairs_to_dict_map.get(parent.js_id, parent) + real_parent[key] = real_value + + @contextmanager + def decrement_depth(self): + self.depth -= 1 + try: + yield + finally: + self.depth += 1 + + def convert(self, pyobj): + if self.depth == 0 or isinstance(pyobj, JsProxy): + return pyobj + if result := self.cache.get(id(pyobj)): + return result + + with self.decrement_depth(): + if self.eager_converter: + return self.eager_converter( + pyobj, self.convert_no_eager_public, self.cache_conversion + ) + return self.convert_no_eager(pyobj) + + def convert_no_eager_public(self, pyobj): + with self.decrement_depth(): + return self.convert_no_eager(pyobj) + + def convert_no_eager(self, pyobj): + if isinstance(pyobj, (tuple, list)): + return self.convert_sequence(pyobj) + if isinstance(pyobj, dict): + return self.convert_dict(pyobj) + if isinstance(pyobj, set): + return self.convert_set(pyobj) + if self.default_converter: + return self.default_converter( + pyobj, self.convert_no_eager_public, self.cache_conversion + ) + if not self.create_pyproxies: + raise ConversionError( + f"No conversion available for {pyobj!r} and create_pyproxies=False passed" + ) + result = create_proxy(pyobj) + if self.pyproxies is not None: + self.pyproxies.append(result) + return result + + def convert_sequence(self, pyobj): + from js import Array + + result = Array.new() + self.cache_conversion(pyobj, result) + for idx, val in enumerate(pyobj): + converted = self.convert(val) + if converted is NoValue: + self.post_process_list.append((result, idx, id(val))) + result.push(converted) + return result + + def convert_dict(self, pyobj): + from js import Array + + # Temporarily store NoValue in the cache since we only get the + # actual value from dict_converter. We'll replace these with the + # correct values in the postprocess step + self.cache_conversion(pyobj, NoValue) + pairs = Array.new() + for [key, value] in pyobj.items(): + converted = self.convert(value) + if converted is NoValue: + self.post_process_list.append((pairs, key, id(value))) + pairs.push(Array.new(key, converted)) + result = self.dict_converter(pairs) + self.pairs_to_dict_map[pairs.js_id] = result + # Update the cache to point to the actual result + self.cache_conversion(pyobj, result) + return result + + def convert_set(self, pyobj): + from js import Set + result = Set.new() + self.cache_conversion(pyobj, result) + for key in pyobj: + if isinstance(key, JsProxy): + raise ConversionError( + f"Cannot use {key!r} as a key for a JavaScript Set" + ) + result.add(key) + return result + + +The ``js`` Module +----------------- + +The ``js`` module allows us to import objects from JavaScript. The definition is +as follows: + +.. code:: python + + import sys + from _pyodide_core import run_js + from pyodide.ffi import JsProxy + from importlib.abc import Loader, MetaPathFinder + from importlib.util import spec_from_loader + + class JsLoader(Loader): + def __init__(self, jsproxy): + self.jsproxy = jsproxy + + def create_module(self, spec): + return self.jsproxy + + def exec_module(self, module): + pass + + def is_package(self, fullname): + return True + + class JsFinder(MetaPathFinder): + def _get_object(self, fullname): + [parent, _, child] = fullname.rpartition(".") + if not parent: + if child == "js": + return run_js("globalThis") + return None + + parent_module = sys.modules[parent] + if not isinstance(parent_module, JsProxy): + # Not one of us. + return None + jsproxy = getattr(parent_module, child, None) + if not isinstance(jsproxy, JsProxy): + raise ModuleNotFoundError(f"No module named {fullname!r}", name=fullname) + return jsproxy + + def find_spec( + self, + fullname, + path, + target, + ): + jsproxy = self._get_object(fullname) + loader = JsLoader(jsproxy) + return spec_from_loader(fullname, loader, origin="javascript") + + + finder = JsFinder() + sys.meta_path.insert(0, finder) + del sys.modules["js"] + import js + sys.meta_path.remove(finder) + sys.meta_path.append(finder) + +The ``pyodide`` package +----------------------- + +This has an empty ``__init__.py`` and two submodules. + +The ``pyodide.ffi`` Module +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This has the following properties: + +``create_proxy(x)``: This returns ``create_jsproxy(createPyProxy(x))``. + +``jsnull``: Special value that converts to/from the JavaScript ``null`` value. + +.. code-block:: python + + def destroy_proxies(proxies): + for proxy in proxies: + proxy.destroy() + +``to_js``: See definition in the section on deep conversions. + +``JsArray``: This is ``type(run_js("[]"))``. + +``JsCallable``: This is ``type(run_js("() => {}"))``. + +``JsDoubleProxy``: This is ``type(create_proxy({}))``. + +``JsException``: This is ``type(run_js("new Error()"))``. + +``JsGenerator``: This is ``type(run_js("(function*(){})()"))``. + +``JsIterable``: This is ``type(run_js("({[Symbol.iterator](){}})"))``. + +``JsIterator``: This is ``type(run_js("({next(){}})"))``. + +``JsMap``: This is ``type(run_js("({get(){}})"))``. + +``JsMutableMap``: This is ``type(run_js("new Map()"))``. + +``JsProxy``: This is ``type(run_js("({})"))`` + +``JsBigInt``: This is defined as follows: + +.. code-block:: python + + def _int_to_bigint(x): + if isinstance(x, int): + return JsBigInt(x) + return x + + class JsBigInt(int): + # unary ops + def __abs__(self): + return JsBigInt(int.__abs__(self)) + + def __invert__(self): + return JsBigInt(int.__invert__(self)) + + def __neg__(self): + return JsBigInt(int.__neg__(self)) + + def __pos__(self): + return JsBigInt(int.__pos__(self)) + + # binary ops + def __add__(self, other): + return _int_to_bigint(int.__add__(self, other)) + + def __and__(self, other): + return _int_to_bigint(int.__and__(self, other)) + + def __floordiv__(self, other): + return _int_to_bigint(int.__floordiv__(self, other)) + + def __lshift__(self, other): + return _int_to_bigint(int.__lshift__(self, other)) + + def __mod__(self, other): + return _int_to_bigint(int.__mod__(self, other)) + + def __or__(self, other): + return _int_to_bigint(int.__or__(self, other)) + + def __pow__(self, other, modulus = None): + return _int_to_bigint(int.__pow__(self, other, modulus)) + + def __rshift__(self, other): + return _int_to_bigint(int.__rshift__(self, other)) + + def __sub__(self, other): + return _int_to_bigint(int.__sub__(self, other)) + + def __xor__(self, other): + return _int_to_bigint(int.__xor__(self, other)) + + +The ``pyodide.code`` Module +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This exposes the ``run_js`` function. + +Changes to the ``json`` Module +------------------------------ + +The ``json`` module will be updated to serialize ``jsnull`` to ``null``. + +Backwards Compatibility +======================= + +This is strictly adding new APIs. There are backwards compatibility concerns for +Pyodide. We will try to ensure that the implementation allows Pyodide to +gradually switch to using the CPython implementation with a minimum amount of +disruption. + +Security Implications +===================== + +It improves support for one of the few fully sandboxed platforms that Python can +run on. + +How to Teach This +================= + + +Reference Implementation +======================== + +Pyodide, https://github.com/hoodmane/cpython/tree/js-ffi + + +Acknowledgments +=============== + +Mike Droettboom, Roman Yurchak, Gyeongjae Choi, Andrea Giammarchi + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 9b18f63ab39e3940caad5e802e9fd6ff1c784aa2 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Mon, 5 Jan 2026 08:53:21 -0800 Subject: [PATCH 02/10] Add discuss link --- peps/pep-0818.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index bdfa132a6bb..77fe2309585 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -7,7 +7,7 @@ Status: Draft Type: Standards Track Created: 10-Dec-2025 Python-Version: 3.15 -Post-History: +Post-History: `05-Jan-2026 `__, Abstract ======== From f6748558357277c61b9803af7874e50c7e4e0aa1 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Mon, 5 Jan 2026 08:55:26 -0800 Subject: [PATCH 03/10] Fix discussion to --- peps/pep-0818.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index 77fe2309585..d741f89dd74 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -7,7 +7,7 @@ Status: Draft Type: Standards Track Created: 10-Dec-2025 Python-Version: 3.15 -Post-History: `05-Jan-2026 `__, +Discussions-To: https://discuss.python.org/t/pep-818-upstreaming-the-pyodide-ffi/105530 Abstract ======== From 9f7005a9a7d9d82c6c775fbf11c46d44cc5aaf93 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Mon, 5 Jan 2026 08:56:27 -0800 Subject: [PATCH 04/10] try again --- peps/pep-0818.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index d741f89dd74..d021f81e458 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -2,12 +2,11 @@ PEP: 818 Title: Upstreaming the Pyodide JavaScript Foreign Function Interface Author: Hood Chatham Sponsor: Łukasz Langa -Discussions-To: +Discussions-To: https://discuss.python.org/t/pep-818-upstreaming-the-pyodide-ffi/105530 Status: Draft Type: Standards Track Created: 10-Dec-2025 Python-Version: 3.15 -Discussions-To: https://discuss.python.org/t/pep-818-upstreaming-the-pyodide-ffi/105530 Abstract ======== From 5012b5864debb1adf719d5c9b958e68e98e210ba Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Mon, 5 Jan 2026 10:02:49 -0800 Subject: [PATCH 05/10] Update tense --- peps/pep-0818.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index d021f81e458..8d080b2ce5d 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -25,9 +25,9 @@ Motivation ========== The Pyodide project is a Python distribution for JavaScript runtimes. Pyodide is -a very popular project. In 2025 to date, Pyodide has received over a billion -requests on JsDelivr. The popularity is rapidly growing: usage has more than -doubled in each of the last two years. +a very popular project. In 2025, Pyodide received over a billion requests on +JsDelivr. The popularity is rapidly growing: usage has more than doubled in each +of the last two years. Pyodide includes several components: From b70b987074b5ff086766c852dd624d309f13a7f3 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 6 Jan 2026 15:34:15 -0800 Subject: [PATCH 06/10] Change title --- peps/pep-0818.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index 8d080b2ce5d..ac54e3be95a 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -1,5 +1,5 @@ PEP: 818 -Title: Upstreaming the Pyodide JavaScript Foreign Function Interface +Title: Adding the Core of the Pyodide Foreign Function Interface to Python Author: Hood Chatham Sponsor: Łukasz Langa Discussions-To: https://discuss.python.org/t/pep-818-upstreaming-the-pyodide-ffi/105530 From 96ce020cdb75b39de3bd1fb28cc838ef24558ed6 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 7 Jan 2026 16:07:56 -0800 Subject: [PATCH 07/10] Update text in rationale describing conversions of primitive types --- peps/pep-0818.rst | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index ac54e3be95a..310413f995d 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -110,12 +110,14 @@ JavaScript has the following immutable types: ``string``, ``undefined``, ``boolean``, ``number`` and ``bigint``. It also has the special value ``null``. Of these, ``string`` and ``boolean`` directly correspond to ``str`` and -``bool``. ``number`` and ``bigint`` awkwardly correspond to ``float`` and -``int``. ``undefined`` is the default value for a missing argument so it +``bool``. We convert a ``number`` to an ``int`` if ``Number.isSafeInteger()`` +returns ``true`` and otherwise we convert it to a ``float``. Conversely we +convert ``float`` to ``number`` and we convert ``int`` to ``number`` unless it +exceeds ``2**53`` in which case we convert it to a ``bigint``. We make a new +subclass of ``int`` called ``JsBigInt`` to act as the conversion for +``bigint``.``undefined`` is the default value for a missing argument so it corresponds to ``None``. We invent a new falsey singleton Python value -``jsnull`` to act as the conversion of ``null``. We also make a new type -``JsBigInt`` to act as the conversion for ``bigint``. All other types are -proxied. +``jsnull`` to act as the conversion of ``null``. All other types are proxied. In particular, even though ``tuples`` are immutable, they have no equivalent in JavaScript so we proxy them. They can be manually converted to an ``Array`` with From 7ce4986c19daf2bf03299f14d8ebdd01ae43f360 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 7 Jan 2026 16:09:47 -0800 Subject: [PATCH 08/10] Indicate type of jsnull explicitly --- peps/pep-0818.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index 310413f995d..2d6d8e44bde 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -117,7 +117,8 @@ exceeds ``2**53`` in which case we convert it to a ``bigint``. We make a new subclass of ``int`` called ``JsBigInt`` to act as the conversion for ``bigint``.``undefined`` is the default value for a missing argument so it corresponds to ``None``. We invent a new falsey singleton Python value -``jsnull`` to act as the conversion of ``null``. All other types are proxied. +``jsnull`` of type ``JsNull`` to act as the conversion of ``null``. All other +types are proxied. In particular, even though ``tuples`` are immutable, they have no equivalent in JavaScript so we proxy them. They can be manually converted to an ``Array`` with From bf6f6399d04998c8cac740b99298363bad68b71d Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 7 Jan 2026 16:27:07 -0800 Subject: [PATCH 09/10] Add 'New Top Level Packages' section to Rationale --- peps/pep-0818.rst | 64 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index 2d6d8e44bde..f62a87e315d 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -388,6 +388,68 @@ Now a cooperating implementation of ``drawImageToCanvas()`` could destroy the convention was that it should be possible to take care of the memory management from Python. This necessitates something like the current approach. +New Top Level Packages +---------------------- + +We introduce two new top level packages, ``js`` and ``pyodide``. The ``js`` +package is the JavaScript global scope ``globalThis``. What set of values are +present on the ``js`` module depends on the JavaScript runtime and whether the +Python runtime is in the main thread or a worker thread. For instance +``from js import Buffer`` will succeed in Node but not in a browser. + +The ``pyodide`` package has two modules: ``pyodide.code`` and ``pyodide.ffi``. +``pyodide.code`` exposes the ``run_js`` function. ``pyodide.ffi`` exposes the +following functions: + +``create_proxy`` + Creates a ``PyProxy`` from Python. Used to control the lifetime of the + ``PyProxy`` from Python. + +``jsnull`` + Special value that converts to/from the JavaScript ``null`` value. + +``JsNull`` + The type of ``jsnull``. + +``JsBigInt`` + Subtype of ``int`` that converts to/from JavaScript ``bigint``. + +``to_js`` + Does a deep conversion of a Python value to JavaScript. + +We also include ``JsProxy`` and its subtypes: + +``JsProxy`` + This is ``type(run_js("({})"))`` + +``JsArray`` + This is ``type(run_js("[]"))``. + +``JsCallable`` + This is ``type(run_js("() => {}"))``. + +``JsDoubleProxy`` + This is ``type(create_proxy({}))``. + +``JsException`` + This is ``type(run_js("new Error()"))``. + +``JsGenerator`` + This is ``type(run_js("(function*(){})()"))``. + +``JsIterable`` + This is ``type(run_js("({[Symbol.iterator](){}})"))``. + +``JsIterator`` + This is ``type(run_js("({next(){}})"))``. + +``JsMap`` + This is ``type(run_js("({get(){}})"))``. + +``JsMutableMap`` + This is ``type(run_js("new Map()"))``. + + Specification ============= @@ -3169,6 +3231,8 @@ This has the following properties: ``jsnull``: Special value that converts to/from the JavaScript ``null`` value. +``JsNull``: The type of ``jsnull``. + .. code-block:: python def destroy_proxies(proxies): From 7c47f3635a508a423c87e9dc3af11c0486f5fd9f Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 21 Jan 2026 14:13:10 -0800 Subject: [PATCH 10/10] Renaming --- peps/pep-0818.rst | 364 ++++++++++++++++++++++++---------------------- 1 file changed, 187 insertions(+), 177 deletions(-) diff --git a/peps/pep-0818.rst b/peps/pep-0818.rst index f62a87e315d..ca46a3ce0f4 100644 --- a/peps/pep-0818.rst +++ b/peps/pep-0818.rst @@ -114,10 +114,10 @@ Of these, ``string`` and ``boolean`` directly correspond to ``str`` and returns ``true`` and otherwise we convert it to a ``float``. Conversely we convert ``float`` to ``number`` and we convert ``int`` to ``number`` unless it exceeds ``2**53`` in which case we convert it to a ``bigint``. We make a new -subclass of ``int`` called ``JsBigInt`` to act as the conversion for +subclass of ``int`` called ``JSBigInt`` to act as the conversion for ``bigint``.``undefined`` is the default value for a missing argument so it corresponds to ``None``. We invent a new falsey singleton Python value -``jsnull`` of type ``JsNull`` to act as the conversion of ``null``. All other +``jsnull`` of type ``JSNull`` to act as the conversion of ``null``. All other types are proxied. In particular, even though ``tuples`` are immutable, they have no equivalent in @@ -127,8 +127,8 @@ the ``toJs()`` method if desired. Proxies ------- -A ``JsProxy`` is a Python object used for accessing a JavaScript object. While -the ``JsProxy`` exists, the underlying JavaScript object is kept in a table +A ``JSProxy`` is a Python object used for accessing a JavaScript object. While +the ``JSProxy`` exists, the underlying JavaScript object is kept in a table which keeps it from being garbage collected. A ``PyProxy`` is a JavaScript object used for accessing a Python object. When a @@ -137,7 +137,7 @@ incremented. When the ``.destroy()`` method is called, the reference count of th underlying Python object is decremented and the proxy is disabled. Any further attempt to use it raises an error. -The base ``JsProxy`` implements property access, equality checks, ``__repr__``, +The base ``JSProxy`` implements property access, equality checks, ``__repr__``, ``__eq__``, ``__bool__``, and a handful of other convenience methods. We also define a large number of mixins by mapping abstract Python object protocols to abstract JavaScript object protocols (and vice-versa). The mapping described in @@ -173,7 +173,7 @@ object into a context manager, but we do not presently use context managers to implement ``[Symbol.dispose]()``. JavaScript also has ``Reflect.construct`` (the ``new`` keyword). Callable -JsProxies have a method called ``new()`` which corresponds to +JSProxies have a method called ``new()`` which corresponds to ``Reflect.construct``. The following additional mappings are defined in Pyodide. It is our intention to @@ -185,7 +185,7 @@ eventually add them to Python itself, but they are deferred to a future PEP: ``[Symbol.asyncIterator]`` to distinguish) * ``AsyncGenerator`` <==> ``AsyncGenerator`` * buffer protocol <==> typed arrays -* Async context managers are implemented on JsProxies that implement +* Async context managers are implemented on JSProxies that implement ``[Symbol.asyncDispose]``. Garbage Collection and Destruction of Proxies @@ -246,7 +246,7 @@ To call a callable ``PyProxy`` we do the following steps: exception. Throw this ``PythonError``. 5. Translate the result from Python to JavaScript and return it. -Note here that if a ``JsProxy`` is created but the Python function does not +Note here that if a ``JSProxy`` is created but the Python function does not store a reference to it, it will be released immediately. The JavaScript error doesn't hold a strong reference the Python exception because JavaScript errors are often leaked and Python error objects hold a reference to frame objects @@ -255,7 +255,7 @@ which may hold a significant amount of memory. Calling a JavaScript Function from Python ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To call a callable ``JsProxy`` we do the following steps: +To call a callable ``JSProxy`` we do the following steps: 1. Make an empty array called ``pyproxies`` 2. Translate each positional argument from Python to JavaScript and place these @@ -273,7 +273,7 @@ To call a callable ``JsProxy`` we do the following steps: a. If the error is a ``PythonError`` and the weak reference to the Python exception is still alive, raise the referenced Python exception. b. Otherwise, convert the exception from JavaScript to Python and raise the - result. Note that the ``JsException`` object holds a reference to the + result. Note that the ``JSException`` object holds a reference to the original JavaScript error. 6. If ``jsresult`` is a JavaScript generator, iterate over ``pyproxies`` and @@ -287,7 +287,7 @@ To call a callable ``JsProxy`` we do the following steps: This is modeled on the calling convention for C Python APIs. -Defense of the Calling Convention for a ``JsProxy`` +Defense of the Calling Convention for a ``JSProxy`` --------------------------------------------------- The calling convention from JavaScript into Python is uncontroversial so we will @@ -314,7 +314,7 @@ JavaScript calling convention. Consider the following example: .. code-block:: python - from pyodide.code import run_js + from jstypes.code import run_js set_x = run_js("(x) => { globalThis.x = x; }") get_x = run_js("(x) => globalThis.x") @@ -340,13 +340,13 @@ Or we can manage the memory from Python using ``create_proxy()`` as follows: .. code-block:: python - from pyodide.ffi import JsDoubleProxy - from pyodide.code import run_js + from jstypes.ffi import JSDoubleProxy + from jstypes.code import run_js setXJs = run_js("(x) => { globalThis.x = x; }") def set_x(x): orig_x = get_x() - if isinstance(orig_x, JsDoubleProxy): + if isinstance(orig_x, JSDoubleProxy): orig_x.destroy() xpx = create_proxy(x) setXJs(xpx) @@ -391,15 +391,18 @@ from Python. This necessitates something like the current approach. New Top Level Packages ---------------------- -We introduce two new top level packages, ``js`` and ``pyodide``. The ``js`` -package is the JavaScript global scope ``globalThis``. What set of values are -present on the ``js`` module depends on the JavaScript runtime and whether the -Python runtime is in the main thread or a worker thread. For instance -``from js import Buffer`` will succeed in Node but not in a browser. +We introduce a new top level package called ``jstypes``. -The ``pyodide`` package has two modules: ``pyodide.code`` and ``pyodide.ffi``. -``pyodide.code`` exposes the ``run_js`` function. ``pyodide.ffi`` exposes the -following functions: +The ``jstypes`` package three two modules: ``jstypes.code`` and ``jstypes.ffi``. +The ``jstypes.global_this`` package is the JavaScript global scope +``globalThis``. What set of values are present on the ``jstypes.global_this`` +module depends on the JavaScript runtime and whether the Python runtime is in +the main thread or a worker thread. For instance ``from jstypes.global_this +import Buffer`` will succeed in Node but fail in a browser. + +``jstypes.code`` exposes the ``run_js`` function. + +``jstypes.ffi`` exposes the following functions: ``create_proxy`` Creates a ``PyProxy`` from Python. Used to control the lifetime of the @@ -408,45 +411,45 @@ following functions: ``jsnull`` Special value that converts to/from the JavaScript ``null`` value. -``JsNull`` +``JSNull`` The type of ``jsnull``. -``JsBigInt`` +``JSBigInt`` Subtype of ``int`` that converts to/from JavaScript ``bigint``. ``to_js`` Does a deep conversion of a Python value to JavaScript. -We also include ``JsProxy`` and its subtypes: +We also include ``JSProxy`` and its subtypes: -``JsProxy`` +``JSProxy`` This is ``type(run_js("({})"))`` -``JsArray`` +``JSArray`` This is ``type(run_js("[]"))``. -``JsCallable`` +``JSCallable`` This is ``type(run_js("() => {}"))``. -``JsDoubleProxy`` +``JSDoubleProxy`` This is ``type(create_proxy({}))``. -``JsException`` +``JSException`` This is ``type(run_js("new Error()"))``. -``JsGenerator`` +``JSGenerator`` This is ``type(run_js("(function*(){})()"))``. -``JsIterable`` +``JSIterable`` This is ``type(run_js("({[Symbol.iterator](){}})"))``. -``JsIterator`` +``JSIterator`` This is ``type(run_js("({next(){}})"))``. -``JsMap`` +``JSMap`` This is ``type(run_js("({get(){}})"))``. -``JsMutableMap`` +``JSMutableMap`` This is ``type(run_js("new Map()"))``. @@ -489,7 +492,7 @@ For the most part, this code could actually be used if performance was not a concern. In some places there may be bootstrapping issues. Our first task is to define the Python callable ``run_js`` and the JavaScript -callable ``makePythonFunction``. ``run_js`` is a ``JsProxy`` and +callable ``makePythonFunction``. ``run_js`` is a ``JSProxy`` and ``makePythonFunction`` is a ``PyProxy``. To make sense of this, we need to describe @@ -501,9 +504,9 @@ To make sense of this, we need to describe We can directly represent a ``PyObject*`` as a ``number`` in JavaScript so we can describe the process of calling a ``PyObject*`` from JavaScript. On the other hand, JavaScript objects are not directly representable in Python, we have -to create a ``JsProxy`` of it. We describe first the process of calling a -``JsProxy``, the process of creating it is described in the section on -JsProxies. +to create a ``JSProxy`` of it. We describe first the process of calling a +``JSProxy``, the process of creating it is described in the section on +JSProxies. Converting Values between Python and JavaScript ----------------------------------------------- @@ -514,18 +517,18 @@ Python to JavaScript back to Python or from JavaScript to Python back to JavaScript, the result is the same primitive as we started with. The one exception to this is that a JavaScript ``BigInt`` that is smaller than ``2^53`` round trips to a ``Number``. We convert ``undefined`` to ``None`` and introduce -the special falsey singleton ``pyodide.ffi.jsnull`` to convert ``null``. We also -introduce a subtype of ``int`` called ``pyodide.ffi.JsBigInt`` which converts to +the special falsey singleton ``jstypes.ffi.jsnull`` to convert ``null``. We also +introduce a subtype of ``int`` called ``jstypes.ffi.JSBigInt`` which converts to and from JavaScript ``bigint``. Implicit conversions are done with the C functions ``_Py_python2js`` and ``_Py_js2python()``. These functions cannot be called directly from Python code -because the ``JsVal`` type is not representable in Python. +because the ``JSVal`` type is not representable in Python. Implicit conversion from Python to JavaScript ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -``JsVal _Py_python2js_track_proxies(PyObject* pyvalue, JsVal pyproxies, bool gc_register)`` +``JSVal _Py_python2js_track_proxies(PyObject* pyvalue, JSVal pyproxies, bool gc_register)`` is responsible for implicit conversions from Python to JavaScript. It does the following steps: @@ -534,24 +537,24 @@ following steps: 3. if ``pyvalue`` is ``True``, return ``true`` 4. if ``pyvalue`` is ``False``, return ``false`` 5. if ``pyvalue`` is a ``str``, convert the string to JavaScript and return the result. -6. if ``pyvalue`` is an instance of ``JsBigInt``, convert it to a ``BigInt``. +6. if ``pyvalue`` is an instance of ``JSBigInt``, convert it to a ``BigInt``. 7. if ``pyvalue`` is an ``int`` and it is less than ``2^53``, convert it to a ``Number``. Otherwise, convert it to a ``BigInt`` 8. if ``pyvalue`` is a ``float``, convert it to a ``Number``. -9. if ``pyvalue`` is a ``JsProxy``, convert it to the wrapped JavaScript value. +9. if ``pyvalue`` is a ``JSProxy``, convert it to the wrapped JavaScript value. 10. Let ``result`` be ``createPyProxy(pyvalue, {gcRegister: gc_register})``. If ``pyproxies`` is an array, append ``result`` to ``pyproxies``. -We define ``JsVal _Py_python2js(PyObject* pyvalue)`` to be +We define ``JSVal _Py_python2js(PyObject* pyvalue)`` to be ``_Py_python2js_track_proxies(pyvalue, Js_undefined, true)``. Implicit conversion from JavaScript to Python ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -``PyObject* _Py_js2python(JsVal jsvalue)`` is responsible for implicit +``PyObject* _Py_js2python(JSVal jsvalue)`` is responsible for implicit conversions from JavaScript to Python. -We first define the helper function ``PyObject* _Py_js2python_immutable(JsVal jsvalue)`` +We first define the helper function ``PyObject* _Py_js2python_immutable(JSVal jsvalue)`` does the following steps: 1. if ``jsvalue`` is ``undefined``, return ``None`` @@ -563,7 +566,7 @@ does the following steps: 6. if ``jsvalue`` is a ``Number`` and ``Number.isSafeInteger(jsvalue)`` returns ``true``, then convert ``jsvalue`` to an ``int``. Otherwise convert it to a ``float``. -7. if ``jsvalue`` is a ``BigInt`` then convert it to an ``JsBigInt``. +7. if ``jsvalue`` is a ``BigInt`` then convert it to an ``JSBigInt``. 8. If ``jsvalue`` is a ``PyProxy`` that has not been destroyed, convert it to the wrapped Python value. 9. If the ``jsvalue`` is a ``PyProxy`` that has been destroyed, throw an error @@ -571,7 +574,7 @@ does the following steps: 10. Return ``NoValue``. -``_Py_js2python(JsVal jsvalue)`` does the following steps: +``_Py_js2python(JSVal jsvalue)`` does the following steps: 1. Let ``result`` be ``_Py_js2python_immutable(jsvalue)``. If ``result`` is not ``NoValue``, return ``result``. @@ -636,7 +639,7 @@ To call a ``PyObject*`` from JavaScript we use the following code: return _PyProxy_apply(pyfuncptr, jsargs, num_pos_args, kwargs_names, num_kwargs); } -**_PyProxy_apply(PyObject* callable, JsVal jsargs, Py_ssize_t num_pos_args, JsVal kwargs_names, Py_ssize_t num_kwargs)** +**_PyProxy_apply(PyObject* callable, JSVal jsargs, Py_ssize_t num_pos_args, JSVal kwargs_names, Py_ssize_t num_kwargs)** 1. Let ``total_args`` be ``num_pos_args + numkwargs``. 2. Create a C array ``pyargs`` of length ``total_args``. @@ -659,9 +662,9 @@ To call a ``PyObject*`` from JavaScript we use the following code: Calling a JavaScript Function from Python ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -**``JsMethod_ConvertArgs(posargs, kwargs, pyproxies)``** +**``JSMethod_ConvertArgs(posargs, kwargs, pyproxies)``** -First we define the function ``JsMethod_ConvertArgs`` to convert the Python +First we define the function ``JSMethod_ConvertArgs`` to convert the Python arguments to a JavaScript array of arguments. Any ``PyProxy`` created at this stage is not tracked by the finalization registry and is added to the JavaScript list ``pyproxies`` so we can either destroy it or track it later. This function @@ -670,7 +673,7 @@ performs the following steps: 1. Let ``jsargs`` be a new empty JavaScript list. 2. For each positional argument: - a. Set ``JsVal jsarg = _Py_python2js_track_proxies(pyarg, proxies, /*gc_register:*/false);``. + a. Set ``JSVal jsarg = _Py_python2js_track_proxies(pyarg, proxies, /*gc_register:*/false);``. b. Call ``_PyJsvArray_Push(jsargs, arg);``. 3. If there are any keyword arguments: @@ -678,23 +681,23 @@ performs the following steps: a. Let ``jskwargs`` be a new empty JavaScript object. b. For each keyword argument pykey, pyvalue: - i. Set ``JsVal jskey = _Py_python2js(pykey)`` - ii. Set ``JsVal jsvalue = _Py_python2js_track_proxies(pyvalue, proxies, /*gc_register:*/false)`` + i. Set ``JSVal jskey = _Py_python2js(pykey)`` + ii. Set ``JSVal jsvalue = _Py_python2js_track_proxies(pyvalue, proxies, /*gc_register:*/false)`` iii. Set the ``jskey`` property on ``jskwargs`` to ``jsvalue``. c. Call ``_PyJsvArray_Push(jsargs, jskwargs);`` 4. Return ``jsargs`` -**``JsMethod_Vectorcall(jsproxy, posargs, kwargs)``** +**``JSMethod_Vectorcall(jsproxy, posargs, kwargs)``** -Each ``JsProxy`` of a function has an underlying JavaScript function and an +Each ``JSProxy`` of a function has an underlying JavaScript function and an underlying ``this`` value. 1. Let ``jsfunc`` be the JavaScript function associated to ``jsproxy``. 2. Let ``jsthis`` be the ``this`` value associated to ``jsproxy``. 3. Let ``pyproxies`` be a new empty JavaScript list. -4. Execute ``JsMethod_ConvertArgs(posargs, kwargs, pyproxies)`` and store the +4. Execute ``JSMethod_ConvertArgs(posargs, kwargs, pyproxies)`` and store the result into ``jsargs``. 5. Execute the JavaScript code ``Function.prototype.apply.apply(jsfunc, [ jsthis, jsargs ])`` and store the result into ``jsresult``. @@ -715,7 +718,7 @@ JavaScript generator in a new generator that destroys all the proxies in ``run_js`` ---------- -The Python object ``pyodide.code.run_js`` is defined as follows: +The Python object ``jstypes.code.run_js`` is defined as follows: 1. Execute the JavaScript code ``eval`` and store the result into ``jseval``. 2. Run ``_Py_js2python(jseval)`` and store the result into ``run_js``. @@ -742,35 +745,35 @@ define define ``makePythonFunction`` as follows: ``makePythonFunction``. -JsProxy +JSProxy ------- We define 14 different abstract protocols that a JavaScript object can support. -These each correspond to a ``JsProxy`` type flag. There are also two additional +These each correspond to a ``JSProxy`` type flag. There are also two additional flags ``IS_PY_JSON_DICT`` and ``IS_PY_JSON_SEQUENCE`` which are set by the -``JsProxy.as_py_json()`` method and do not reflect properties of the underlying +``JSProxy.as_py_json()`` method and do not reflect properties of the underlying JavaScript object. ``HAS_GET`` Signals whether or not the JavaScript object has a ``get()`` - method. If present, used to implement ``__getitem__`` on the ``JsProxy``. + method. If present, used to implement ``__getitem__`` on the ``JSProxy``. ``HAS_HAS`` Signals whether or not the JavaScript object has a ``has()`` method. If - present, used to implement ``__contains__`` on the ``JsProxy``. + present, used to implement ``__contains__`` on the ``JSProxy``. ``HAS_INCLUDES`` Signals whether or not the JavaScript object has an ``includes()`` method. - If present, used to implement ``__contains__`` on the ``JsProxy``. We prefer + If present, used to implement ``__contains__`` on the ``JSProxy``. We prefer to use ``has()`` to ``includes()`` if both are present. ``HAS_LENGTH`` Signals whether or not the JavaScript object has a ``length`` or ``size`` - property. Used to implement ``__len__`` on the ``JsProxy``. + property. Used to implement ``__len__`` on the ``JSProxy``. ``HAS_SET`` Signals whether or not the JavaScript object has a ``set()`` method. If - present, used to implement ``__setitem__`` on the ``JsProxy``. + present, used to implement ``__setitem__`` on the ``JSProxy``. ``HAS_DISPOSE`` Signals whether or not the JavaScript object has a ``[Symbol.dispose]()`` @@ -778,71 +781,71 @@ JavaScript object. ``IS_ARRAY`` Signals whether ``Array.isArray()`` applied to the JavaScript object returns - ``true``. If present, the ``JsProxy`` will be an instance of + ``true``. If present, the ``JSProxy`` will be an instance of ``collections.abc.MutableSequence``. ``IS_ARRAY_LIKE`` We set this if ``Array.isArray()`` returns ``false`` and the object has a - ``length`` property and ``IS_ITERABLE``. If present, the ``JsProxy`` will be + ``length`` property and ``IS_ITERABLE``. If present, the ``JSProxy`` will be an instance of ``collections.abc.Sequence``. This is the case for many interfaces defined in the webidl such as `NodeList `_ ``IS_CALLABLE`` Signals whether the ``typeof`` the JavaScript object is ``"function"``. If - present, used to implement ``__call__`` on the ``JsProxy``. + present, used to implement ``__call__`` on the ``JSProxy``. ``IS_ERROR`` Signals whether the JavaScript object is an ``Error``. If so, the - ``JsProxy`` it will subclass ``Exception`` so it can be raised. + ``JSProxy`` it will subclass ``Exception`` so it can be raised. ``IS_GENERATOR`` - Signals whether the JavaScript object is a generator. If so, the ``JsProxy`` + Signals whether the JavaScript object is a generator. If so, the ``JSProxy`` will be an instance of ``collections.abc.Generator``. ``IS_ITERABLE`` Signals whether the JavaScript object has a ``[Symbol.iterator]`` method or the ``IS_PY_JSON_DICT`` flag is set. If so, we use it to implement - ``__iter__`` on the ``JsProxy``. + ``__iter__`` on the ``JSProxy``. ``IS_ITERATOR`` Signals whether the JavaScript object has a ``next()`` method and no ``[Symbol.asyncIterator]`` method. If so, we use it to implement - ``__next__`` on the ``JsProxy``. (If there is a ``[Symbol.asyncIterator]`` + ``__next__`` on the ``JSProxy``. (If there is a ``[Symbol.asyncIterator]`` method, we assume that the ``next()`` method should be used to implement ``__anext__``.) ``IS_PY_JSON_DICT`` - This is set on a ``JsProxy`` by the ``as_py_json()`` method if it is not an - ``Array``. When this is set, ``__getitem__`` on the ``JsProxy`` will turn + This is set on a ``JSProxy`` by the ``as_py_json()`` method if it is not an + ``Array``. When this is set, ``__getitem__`` on the ``JSProxy`` will turn into attribute access on the JavaScript object. Also, the return values from iterating over the proxy or indexing it will also have ``IS_PY_JSON_DICT`` or ``IS_PY_JSON_SEQUENCE`` set as appropriate. ``IS_PY_JSON_SEQUENCE`` - This is set on a ``JsProxy`` by the ``as_py_json()`` method if it is an - ``Array``. When this is set, when indexing or iterating the ``JsProxy`` + This is set on a ``JSProxy`` by the ``as_py_json()`` method if it is an + ``Array``. When this is set, when indexing or iterating the ``JSProxy`` we'll call ``as_py_json()`` on the result. ``IS_MAPPING`` We set this if the flags ``HAS_GET``, ``HAS_LENGTH``, and ``IS_ITERABLE`` - are set, or if ``IS_PY_JSON_DICT`` is set. In this case, the ``JsProxy`` + are set, or if ``IS_PY_JSON_DICT`` is set. In this case, the ``JSProxy`` will be an instance of ``collections.abc.Mapping``. ``IS_MUTABLE_MAPPING`` We set this if the flags ``IS_MAPPING`` and ``HAS_SET`` are set or if - ``IS_PY_JSON_DICT`` is set. In this case, the ``JsProxy`` will be an + ``IS_PY_JSON_DICT`` is set. In this case, the ``JSProxy`` will be an instance of ``collections.abc.MutableMapping``. -Creating a ``JsProxy`` +Creating a ``JSProxy`` ~~~~~~~~~~~~~~~~~~~~~~ -To create a ``JsProxy`` from a JavaScript object and a value ``jsthis`` we do the +To create a ``JSProxy`` from a JavaScript object and a value ``jsthis`` we do the following steps: 1. calculate the appropriate type flags for the JavaScript object -2. get or create and cache an appropriate ``JsProxy`` class with the mixins +2. get or create and cache an appropriate ``JSProxy`` class with the mixins appropriate for the set of type flags that are set 3. instantiate the class with a reference to the JavaScript object and the ``jsthis`` value. @@ -871,34 +874,34 @@ as follows: @functools.cache def get_jsproxy_class(type_flags): flag_mixin_pairs = [ - (HAS_GET, JsProxyHasGetMixin), - (HAS_HAS, JsProxyHasHasMixin), + (HAS_GET, JSProxyHasGetMixin), + (HAS_HAS, JSProxyHasHasMixin), # ... - (IS_PY_JSON_DICT, JsPyJsonDictMixin) + (IS_PY_JSON_DICT, JSPyJsonDictMixin) ] bases = [mixin for flag, mixin in flag_mixin_pairs if flag & type_flags] - bases.insert(0, JsProxy) + bases.insert(0, JSProxy) if type_flags & IS_ERROR: - # We want JsException to be pickleable so it needs a distinct name - name = "pyodide.ffi.JsException" + # We want JSException to be pickleable so it needs a distinct name + name = "jstypes.ffi.JSException" bases.append(Exception) else: - name = "pyodide.ffi.JsProxy" + name = "jstypes.ffi.JSProxy" ns = {"_js_type_flags": type_flags} # Note: The actual way that we build the class does not result in the # mixins appearing as entries on the mro. - return JsProxyMeta.__new__(JsProxyMeta, name, tuple(bases), ns) + return JSProxyMeta.__new__(JSProxyMeta, name, tuple(bases), ns) -The ``JsProxy`` Metaclass +The ``JSProxy`` Metaclass ~~~~~~~~~~~~~~~~~~~~~~~~~ -This metaclass overrides subclass checks so that if one ``JsProxy`` class has a -superset of the flags of another ``JsProxy`` class, we report it as a subclass. +This metaclass overrides subclass checks so that if one ``JSProxy`` class has a +superset of the flags of another ``JSProxy`` class, we report it as a subclass. .. code:: python - class _JsProxyMetaClass(type): + class _JSProxyMetaClass(type): def __instancecheck__(cls, instance): return cls.__subclasscheck__(type(instance)) @@ -913,10 +916,10 @@ superset of the flags of another ``JsProxy`` class, we report it as a subclass. return cls._js_type_flags & subcls_flags == subcls_flags -The ``JsProxy`` Base Class +The ``JSProxy`` Base Class ~~~~~~~~~~~~~~~~~~~~~~~~~~ -The most complicated part of the ``JsProxy`` base class is the implementation of +The most complicated part of the ``JSProxy`` base class is the implementation of ``__getattribute__``, ``__setattr__``, and ``__delattr__``. For ``__getattribute__``, we first check if an attribute is defined on the Python object itself by calling ``object.__getattribute__()``. Otherwise, we look up @@ -1036,7 +1039,7 @@ numbers (not valid Python identifiers) and reversing the .. code:: python - class JsProxy: + class JSProxy: def __getattribute__(self, attr): try: return object.__getattribute__(self, attr) @@ -1093,13 +1096,13 @@ numbers (not valid Python identifiers) and reversing the return object.__dir__(self) + js_dir(self) def __eq__(self, other): - if not isinstance(other, JsProxy): + if not isinstance(other, JSProxy): return False js_eq = run_js("(x, y) => x === y") return js_eq(self, other) def __ne__(self, other): - if not isinstance(other, JsProxy): + if not isinstance(other, JSProxy): return True js_neq = run_js("(x, y) => x !== y") return js_neq(self, other) @@ -1162,7 +1165,7 @@ We need the following function which calls the ``as_py_json()`` method on def maybe_as_py_json(value): if ( - isinstance(value, JsProxy) + isinstance(value, JSProxy) and hasattr(value, as_py_json) ): return value.as_py_json() @@ -1266,7 +1269,7 @@ If no ``has()`` method is present, ``undefined`` is treated as ``None``. .. code-block:: python - class JsProxyHasGetMixin: + class JSProxyHasGetMixin: def __getitem__(self, item): result = js_get(self, item) if self._js_type_flags & IS_PY_JSON_DICT: @@ -1281,7 +1284,7 @@ present and define ``__setitem__`` and ``__delitem__`` as follows: .. code-block:: python - class JsProxyHasSetMixin: + class JSProxyHasSetMixin: def __setitem__(self, item, value): js_set = run_js( """ @@ -1307,7 +1310,7 @@ The ``HAS_HAS`` Mixin .. code-block:: python - class JsProxyHasHasMixin: + class JSProxyHasHasMixin: def __contains__(self, item): js_has = run_js( """ @@ -1322,7 +1325,7 @@ The ``HAS_INCLUDES`` Mixin .. code-block:: python - class JsProxyHasIncludesMixin: + class JSProxyHasIncludesMixin: def __contains__(self, item): js_includes = run_js( """ @@ -1342,7 +1345,7 @@ up either field, we allow it to propagate into Python as a .. code-block:: python - class JsProxyHasLengthMixin: + class JSProxyHasLengthMixin: def __len__(self, item): js_len = run_js( """ @@ -1365,12 +1368,12 @@ up either field, we allow it to propagate into Python as a The ``HAS_DISPOSE`` Mixin ~~~~~~~~~~~~~~~~~~~~~~~~~ -This makes the ``JsProxy`` into a context manager where ``__enter__`` is a no-op +This makes the ``JSProxy`` into a context manager where ``__enter__`` is a no-op and ``__exit__`` calls the ``[Symbol.dispose]()`` method. .. code-block:: python - class JsProxyContextManagerMixin: + class JSProxyContextManagerMixin: def __enter__(self): return self @@ -1416,7 +1419,7 @@ The ``IS_ARRAY`` Mixin .. code-block:: python - class JsArrayMixin(MutableSequence, JsProxyHasLengthMixin): + class JSArrayMixin(MutableSequence, JSProxyHasLengthMixin): def __getitem__(self, index): if not isinstance(index, (int, slice)): raise TypeError("Expected index to be an int or a slice") @@ -1526,40 +1529,40 @@ The ``IS_ARRAY_LIKE`` Mixin .. code-block:: python - class JsArrayLikeMixin(MutableSequence, JsProxyHasLengthMixin): + class JSArrayLikeMixin(MutableSequence, JSProxyHasLengthMixin): def __getitem__(self, index): if not isinstance(index, int): raise TypeError("Expected index to be an int") - JsArrayMixin.__getitem__(self, index) + JSArrayMixin.__getitem__(self, index) def __setitem__(self, index, value): if not isinstance(index, int): raise TypeError("Expected index to be an int") - JsArrayMixin.__setitem__(self, index, value) + JSArrayMixin.__setitem__(self, index, value) def __delitem__(self, index): if not isinstance(index, int): raise TypeError("Expected index to be an int") - JsArrayMixin.__delitem__(self, index, value) + JSArrayMixin.__delitem__(self, index, value) The ``IS_CALLABLE`` Mixin ~~~~~~~~~~~~~~~~~~~~~~~~~ -We already gave more accurate C code for calling a ``JsCallable``. See in -particular the definition of ``JsMethod_ConvertArgs()`` given there. +We already gave more accurate C code for calling a ``JSCallable``. See in +particular the definition of ``JSMethod_ConvertArgs()`` given there. .. code-block:: python - class JsCallableMixin: + class JSCallableMixin: def __get__(self, obj): """Return a new jsproxy bound to jsthis with the same JS object""" return create_jsproxy(self, jsthis=obj) def __call__(self, *args, **kwargs): - """See the description of JsMethod_Vectorcall""" + """See the description of JSMethod_Vectorcall""" def new(self, *args, **kwargs): pyproxies = [] - jsargs = JsMethod_ConvertArgs(args, kwargs, pyproxies) + jsargs = JSMethod_ConvertArgs(args, kwargs, pyproxies) do_construct = run_js( """ @@ -1579,7 +1582,7 @@ particular the definition of ``JsMethod_ConvertArgs()`` given there. The ``IS_ERROR`` Mixin ~~~~~~~~~~~~~~~~~~~~~~ -In this case, we inherit from both ``Exception`` and ``JsProxy``. We also make +In this case, we inherit from both ``Exception`` and ``JSProxy``. We also make sure that the resulting class is pickleable. The ``IS_ITERABLE`` Mixin @@ -1600,7 +1603,7 @@ the iteration results. return maybe_as_py_json(result) - class JsIterableMixin: + class JSIterableMixin: def __iter__(self): pyjson = self._js_type_flags & (IS_PY_JSON_SEQUENCE | IS_PY_JSON_DICT) pyjson_dict = self._js_type_flags & IS_PY_JSON_DICT @@ -1629,7 +1632,7 @@ a ``StopIteration`` exception to convert to the Python iterator protocol. .. code-block:: python - class JsIteratorMixin: + class JSIteratorMixin: def __iter__(self): return self @@ -1659,7 +1662,7 @@ if there is a need to return a specific value. .. code-block:: python - class JsGeneratorMixin(JsIteratorMixin): + class JSGeneratorMixin(JSIteratorMixin): def throw(self, exc): if isinstance(exc, GeneratorExit): js_throw = run_js( @@ -1707,14 +1710,14 @@ The ``IS_PY_JSON_SEQUENCE`` Mixin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This flag only ever appears with ``IS_ARRAY``. It changes the behavior of -``JsArray.__getitem__`` to apply ``maybe_as_py_json()`` to the result. +``JSArray.__getitem__`` to apply ``maybe_as_py_json()`` to the result. The ``IS_PY_JSON_DICT`` Mixin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: python - class JsPyJsonDictMixin(MutableMapping): + class JSPyJsonDictMixin(MutableMapping): def __getitem__(self, key): if not isinstance(key, str): raise KeyError(key) @@ -1774,7 +1777,7 @@ The ``IS_PY_JSON_DICT`` Mixin The ``IS_DOUBLE_PROXY`` Mixin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In this case the object is a ``JsProxy`` of a ``PyProxy``. We add an extra +In this case the object is a ``JSProxy`` of a ``PyProxy``. We add an extra ``unwrap()`` method that returns the inner Python object. @@ -2719,7 +2722,7 @@ This adds some additional ``Array`` methods that mutate the sequence. } const pythonSplice = makePythonFunction(` def splice(array, start, stop, items): - from pyodide.ffi import to_js + from jstypes.ffi import to_js result = to_js(array[start:stop], depth=1) array[start:stop] = items return result @@ -2863,7 +2866,7 @@ All Python dictionary methods will be shadowed by a key of the same name. ownKeys(jsobj: PyProxy): (string | symbol)[] { const pythonDictOwnKeys = makePythonFunction(` def dict_own_keys(d): - from pyodide.ffi import to_js + from jstypes.ffi import to_js result = set() for key in d: if isinstance(key, str): @@ -2887,14 +2890,14 @@ to the results. Deep Conversions ---------------- -We define ``JsProxy.to_py()`` to make deep conversions from JavaScript to Python -and ``pyodide.ffi.to_js()`` to make deep conversions from Python to JavaScript. +We define ``JSProxy.to_py()`` to make deep conversions from JavaScript to Python +and ``jstypes.ffi.to_js()`` to make deep conversions from Python to JavaScript. Note that it is not intended that these are inverse functions to each other. From JavaScript to Python ~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``JsProxy.to_py()`` method makes the following conversions: +The ``JSProxy.to_py()`` method makes the following conversions: * ``Array`` ==> ``list`` * ``Map`` ==> ``dict`` @@ -2959,12 +2962,12 @@ Complete pseudocode for the ``to_py`` method is as follows: self.cache[jsobj.js_id] = pyobj def convert(self, jsobj): - if self.depth == 0 or not isinstance(jsobj, JsProxy): + if self.depth == 0 or not isinstance(jsobj, JSProxy): return jsobj if result := self.cache.get(jsobj.js_id): return result - from js import Array, Object + from jstypes.global_this import Array, Object type_tag = getTypeTag(jsobj) self.depth -= 1 try: @@ -3075,7 +3078,7 @@ From Python to JavaScript self.depth += 1 def convert(self, pyobj): - if self.depth == 0 or isinstance(pyobj, JsProxy): + if self.depth == 0 or isinstance(pyobj, JSProxy): return pyobj if result := self.cache.get(id(pyobj)): return result @@ -3112,7 +3115,7 @@ From Python to JavaScript return result def convert_sequence(self, pyobj): - from js import Array + from jstypes.global_this import Array result = Array.new() self.cache_conversion(pyobj, result) @@ -3124,7 +3127,7 @@ From Python to JavaScript return result def convert_dict(self, pyobj): - from js import Array + from jstypes.global_this import Array # Temporarily store NoValue in the cache since we only get the # actual value from dict_converter. We'll replace these with the @@ -3143,11 +3146,11 @@ From Python to JavaScript return result def convert_set(self, pyobj): - from js import Set + from jstypes.global_this import Set result = Set.new() self.cache_conversion(pyobj, result) for key in pyobj: - if isinstance(key, JsProxy): + if isinstance(key, JSProxy): raise ConversionError( f"Cannot use {key!r} as a key for a JavaScript Set" ) @@ -3155,21 +3158,21 @@ From Python to JavaScript return result -The ``js`` Module ------------------ +The ``jstypes.global_this`` Module +--------------------------------- -The ``js`` module allows us to import objects from JavaScript. The definition is +The ``jstypes.global_this`` module allows us to import objects from JavaScript. The definition is as follows: .. code:: python import sys - from _pyodide_core import run_js - from pyodide.ffi import JsProxy + from jstypes.code import run_js + from jstypes.ffi import JSProxy from importlib.abc import Loader, MetaPathFinder from importlib.util import spec_from_loader - class JsLoader(Loader): + class JSLoader(Loader): def __init__(self, jsproxy): self.jsproxy = jsproxy @@ -3182,20 +3185,20 @@ as follows: def is_package(self, fullname): return True - class JsFinder(MetaPathFinder): + class JSFinder(MetaPathFinder): def _get_object(self, fullname): [parent, _, child] = fullname.rpartition(".") if not parent: - if child == "js": + if child == "jstypes": return run_js("globalThis") return None parent_module = sys.modules[parent] - if not isinstance(parent_module, JsProxy): + if not isinstance(parent_module, JSProxy): # Not one of us. return None jsproxy = getattr(parent_module, child, None) - if not isinstance(jsproxy, JsProxy): + if not isinstance(jsproxy, JSProxy): raise ModuleNotFoundError(f"No module named {fullname!r}", name=fullname) return jsproxy @@ -3206,23 +3209,23 @@ as follows: target, ): jsproxy = self._get_object(fullname) - loader = JsLoader(jsproxy) + loader = JSLoader(jsproxy) return spec_from_loader(fullname, loader, origin="javascript") - finder = JsFinder() + finder = JSFinder() sys.meta_path.insert(0, finder) - del sys.modules["js"] - import js + del sys.modules["jstypes.global_this"] + import jstypes.global_this sys.meta_path.remove(finder) sys.meta_path.append(finder) -The ``pyodide`` package +The ``jstypes`` package ----------------------- This has an empty ``__init__.py`` and two submodules. -The ``pyodide.ffi`` Module +The ``jstypes.ffi`` Module ~~~~~~~~~~~~~~~~~~~~~~~~~~ This has the following properties: @@ -3231,7 +3234,7 @@ This has the following properties: ``jsnull``: Special value that converts to/from the JavaScript ``null`` value. -``JsNull``: The type of ``jsnull``. +``JSNull``: The type of ``jsnull``. .. code-block:: python @@ -3241,48 +3244,48 @@ This has the following properties: ``to_js``: See definition in the section on deep conversions. -``JsArray``: This is ``type(run_js("[]"))``. +``JSArray``: This is ``type(run_js("[]"))``. -``JsCallable``: This is ``type(run_js("() => {}"))``. +``JSCallable``: This is ``type(run_js("() => {}"))``. -``JsDoubleProxy``: This is ``type(create_proxy({}))``. +``JSDoubleProxy``: This is ``type(create_proxy({}))``. -``JsException``: This is ``type(run_js("new Error()"))``. +``JSException``: This is ``type(run_js("new Error()"))``. -``JsGenerator``: This is ``type(run_js("(function*(){})()"))``. +``JSGenerator``: This is ``type(run_js("(function*(){})()"))``. -``JsIterable``: This is ``type(run_js("({[Symbol.iterator](){}})"))``. +``JSIterable``: This is ``type(run_js("({[Symbol.iterator](){}})"))``. -``JsIterator``: This is ``type(run_js("({next(){}})"))``. +``JSIterator``: This is ``type(run_js("({next(){}})"))``. -``JsMap``: This is ``type(run_js("({get(){}})"))``. +``JSMap``: This is ``type(run_js("({get(){}})"))``. -``JsMutableMap``: This is ``type(run_js("new Map()"))``. +``JSMutableMap``: This is ``type(run_js("new Map()"))``. -``JsProxy``: This is ``type(run_js("({})"))`` +``JSProxy``: This is ``type(run_js("({})"))`` -``JsBigInt``: This is defined as follows: +``JSBigInt``: This is defined as follows: .. code-block:: python def _int_to_bigint(x): if isinstance(x, int): - return JsBigInt(x) + return JSBigInt(x) return x - class JsBigInt(int): + class JSBigInt(int): # unary ops def __abs__(self): - return JsBigInt(int.__abs__(self)) + return JSBigInt(int.__abs__(self)) def __invert__(self): - return JsBigInt(int.__invert__(self)) + return JSBigInt(int.__invert__(self)) def __neg__(self): - return JsBigInt(int.__neg__(self)) + return JSBigInt(int.__neg__(self)) def __pos__(self): - return JsBigInt(int.__pos__(self)) + return JSBigInt(int.__pos__(self)) # binary ops def __add__(self, other): @@ -3316,7 +3319,7 @@ This has the following properties: return _int_to_bigint(int.__xor__(self, other)) -The ``pyodide.code`` Module +The ``jstypes.code`` Module ~~~~~~~~~~~~~~~~~~~~~~~~~~~ This exposes the ``run_js`` function. @@ -3330,9 +3333,16 @@ Backwards Compatibility ======================= This is strictly adding new APIs. There are backwards compatibility concerns for -Pyodide. We will try to ensure that the implementation allows Pyodide to -gradually switch to using the CPython implementation with a minimum amount of -disruption. +Pyodide. We have changed the names of several modules and types compared to +Pyodide: + +1. The ``pyodide`` package is changed to ``jstypes`` +2. The ``js`` module is changed to ``jstypes.global_this`` +3. All ``JSProxy`` variants are capitalized like ``JSProxy``. + +In the next release, Pyodide will add support for both the changed names and the +original names. We will also upload a package to PyPI that includes backwards +compatibility shims for the old names. Security Implications =====================