Bundling all dependencies

[AKuederle]

I started to play around with Pyodide, and I couldn't find any information about, if it is possible to create a single bundle with all Python dependencies that the code requires, and then serve just that to the webbrowser, so that there is no need to load any additional packages via micropip.

Thanks in advance,
Arne

[hoodmane]

Thanks for the question. Not as of yet but @rth is working on this.

[pfebrer]

If you do:

const promise = pyodide.loadPyodide({packages: ["micropip", ... long time loading packages]})

The promise isn't resolved until all packages are loaded, right? So you can't save time if you need to use micropip (which otherwise would be used to install packages asynchronously at the same time loadPackage is loading other dependencies).

[hoodmane]

I have been exploring memory snapshots which indeed could take some of the time off, particularly the import-time package initialization time. We recommend if you always have the same set of dependencies that you make a custom lock file that includes all of your dependencies so that all of them can be downloaded at the same time.

[pfebrer]

That's cool that you are playing with the memory snapshots :) In my case the loading phase and the importing phase take a similar amount of time, so it would be very useful!

What do you mean exactly by a "lock" file? Is there a standard way to do that?

[hoodmane]

I think the easiest way to get the lockfile is with micropip.freeze(). If you use Pyodide in node with NodeFS or in Chrome with NativeFS then you can persist the generated lockfile as a file. Serve it with your other files and pass the URL as the lockFileURL argument to loadPyodide and you can use pyodide.loadPackage() to load all of your packages in one go.

[hoodmane]

On the memory snapshot front I have figured out pretty well how the implementation has to go but there are many little API details to work out. The first part of the work is here:
#4652
I've been doing the snapshots for my job putting Pyodide into the Cloudflare worker runtime. There's a discussion here (mixed in with some advertising and other stuff):
https://blog.cloudflare.com/python-workers