MyPy errors

[kkoomen]

I have downloaded mypy, typing-extensions and mypy-extensions whl files from pypy, extraced them into python_stdlib and recreated a new python_stdlib.zip, I use Pyodide 0.24.1 and when running this code:

pyodide.FS.writeFile('test.py', 'print("hello")', { encoding: "utf8" })
pyodide.runPython(`
import io, sys
sys.stdout = io.StringIO()

from mypy import api as mypy_api
mypy_api.run('test.py')

sys.stdout.getvalue()
`)

This runs into the error:

PythonError: Traceback (most recent call last):
  File "/lib/python311.zip/_pyodide/_base.py", line 499, in eval_code
    .run(globals, locals)
     ^^^^^^^^^^^^^^^^^^^^
  File "/lib/python311.zip/_pyodide/_base.py", line 340, in run
    coroutine = eval(self.code, globals, locals)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<exec>", line 3, in <module>
  File "/lib/python311.zip/mypy/api.py", line 71, in run
    return _run(
           ^^^^^
  File "/lib/python311.zip/mypy/api.py", line 58, in _run
    main_wrapper(stdout, stderr)
  File "/lib/python311.zip/mypy/api.py", line 72, in <lambda>
    lambda stdout, stderr: main(args=args, stdout=stdout, stderr=stderr, clean_exit=True)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python311.zip/mypy/main.py", line 100, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python311.zip/mypy/main.py", line 182, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python311.zip/mypy/build.py", line 192, in build
    result = _build(
             ^^^^^^^
  File "/lib/python311.zip/mypy/build.py", line 245, in _build
    manager = BuildManager(
              ^^^^^^^^^^^^^
  File "/lib/python311.zip/mypy/build.py", line 657, in __init__
    self.find_module_cache = FindModuleCache(
                             ^^^^^^^^^^^^^^^^
  File "/lib/python311.zip/mypy/modulefinder.py", line 183, in __init__
    self.stdlib_py_versions = stdlib_py_versions or load_stdlib_py_versions(
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python311.zip/mypy/modulefinder.py", line 850, in load_stdlib_py_versions
    assert os.path.isfile(versions_path), (custom_typeshed_dir, versions_path, __file__)
AssertionError: (None, '/lib/python311.zip/mypy/typeshed/stdlib/VERSIONS', '/lib/python311.zip/mypy/modulefinder.py')
pyodide.asm.js:9:104344

So the problem seems that mypy internally doing this:

def load_stdlib_py_versions(custom_typeshed_dir: str | None) -> StdlibVersions:
    typeshed_dir = custom_typeshed_dir or os.path.join(os.path.dirname(__file__), "typeshed")
    stdlib_dir = os.path.join(typeshed_dir, "stdlib")
    result = {}

    versions_path = os.path.join(stdlib_dir, "VERSIONS")
    assert os.path.isfile(versions_path), (custom_typeshed_dir, versions_path, __file__)

and the assertion fails, probably because /lib/python311.zip is a (zip) file and not a directory so /lib/python311.zip/whatever can't be checked through os.path.join(), how can this be fixed?

Any idea? I am making this as a Discussion before making an issue, because I am not sure whether this is wrong on my side or your side.

[hoodmane]

Interesting, thanks for the report @kkoomen!

[hoodmane]

See here about importing packages from zip archives:

while some Python software is written to support running directly from a zip archive, it is still common for code to be written assuming it has been fully installed. When that assumption is broken by trying to run the software from a zip archive, the failures can often be obscure and hard to diagnose (especially when they occur in third party libraries). The two most common sources of problems with this are the fact that importing C extensions from a zip archive is not supported by CPython (since doing so is not supported directly by the dynamic loading machinery on any platform) and that when running from a zip archive the file attribute no longer refers to an ordinary filesystem path, but to a combination path that includes both the location of the zip archive on the filesystem and the relative path to the module inside the archive. Even when software correctly uses the abstract resource APIs internally, interfacing with external components may still require the availability of an actual on-disk file.

Like metaclasses, monkeypatching and metapath importers, if you’re not already sure you need to take advantage of this feature, you almost certainly don’t need it. If you do decide to use it anyway, be aware that many projects will require a failure to be reproduced with a fully installed package before accepting it as a genuine bug.

https://peps.python.org/pep-0491/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

[kkoomen]

I have downloaded mypy, typing-extensions and mypy-extensions whl files from pypy, extraced them into python_stdlib and recreated a new python_stdlib.zip

This does sound like... not a good thing to do. Why not install them normally?

What do you mean with "install them normally?" I assume you mean using pyodide.loadPackage() through CDN? I don't want to have any dependencies and simply want to bundle everything I need inside the pyodide_stdlib.zip. That's a requirement for the project I'm working on.

Besides, I did this as a recommendation from another member: #4575 (reply in thread)

[kkoomen]

So, how can I fix this using the local files I am using right now?

[ryanking13]

Oh yes, I recommended bundling mypy into the zip file, but it seems like mypy requires access to these files individually.

In that case, I think you need to do the following approach instead.

1. bundle all dependencies into a separate archive
2. host it somewhere
3. download it in the runtime (or if you are in Node.js, mount the directory)
4. unpack it into site-packages directory

Related link: https://pyodide.org/en/stable/usage/faq.html#how-can-i-load-external-files-in-pyodide

[hoodmane]

What about using loadPackage with an emfs: url?

[kkoomen]

@ryanking13 Works perfectly, thanks!