Future work on the Python integration

[josevalim]

Continuation of #2936:

• Rich-render Python objects, also support Jupyter representations (Dispatch Python output rendering to kino #2954)

• Add autocompletion and other editor features, currently we only offer syntax highlighting

• Refactor and unify functionality in relation to multiple language support - Livebook was originally designed with the assumption that external languages would be added via the distribution protocol. However, Pythonx showed us a different approach to integrate other runtimes, which we should now streamline.

• Change Learn section with one guide for Erlang and another for Python - the Python notebook can include examples of several libraries which we integrate and render by default

• Add support for distribution and FLAME - the goal is to be able to transfer Pythonx objects transparently across nodes. By integrating with FLAME, we can have better control over the objects lifecycle too

Under consideration:

• Importing and exporting of .ipynb files
• Export .py files (if a notebook only has Python cells)
• Explore integration with Triton and Nx
• Support Apache Arrow via ADBC and integrate with Pandas and Polars

[duongtugiang]

Hello @josevalim , I really appreciate what you and the team have done so far. Is there any timeline for "Add autocompletion and other editor features, currently we only offer syntax highlighting"? This will be super helpful. Furthermore, If you can break down the task into subtasks, i will try my best to collaborate :)
Bests

[josevalim]

We will probably resume work on this feature in ~6-10 weeks. I know there is a lang-python package for codemirror which provides some completion but I believe local completion will require work also on the backend to analyze code snippets: https://github.com/codemirror/lang-python

[jonatanklosko]

@josevalim FTR we use lang-python, it already provides some completion in the cells, but it's only very generic stuff and builtins, not aware of the environment. Once we get to implement the completions, it will supersede this :)

[cocoa-xu]

One interesting idea - from pythonx import Elixir

So we can do cool things like call Elixir functions from the embedded Python code and get the result back:

from pythonx import Elixir

bar = Elixir.MyApp.MyModule.foo()

Some POC code I've used in commutron:

class PythonxElixir:
    def __init__(self, name="Elixir", module=None, type="module"):
        self.type = type
        self.fn = None
        if type == "module":
            if module is None:
                self.module = [name]
            elif isinstance(module, list):
                self.module = list(module) + [name]
            else:
                raise TypeError(
                    f"module must be a list, not {type(module).__name__}"
                )
        elif type == "fn":
            if module is None:
                self.module = ["Elixir"]
            self.fn = name

    def __getattr__(self, name):
        if not isinstance(name, str) or len(name) == 0:
            raise ValueError(
                f"Elixir name must be a non-empty string, not {name}"
            )
        is_capital = name[0].isupper()
        if is_capital:
            return PythonxElixir(name=name, module=self.module, type="module")
        else:
            return PythonxElixir(name=name, module=self.module, type="fn")
    
    def __repr__(self):
        return ".".join(self.module)
    
    def __str__(self):
        return ".".join(self.module)
    
    def __call__(self, *args, **kwargs):
        module_name = ".".join(self.module)
        if self.fn_or_field is None:
            raise TypeError(
                f"module `{module_name}` is not callable"
            )

        # TODO: encode args and kwargs and send to Elixir and use `nif_call` to evaluate
        pid = whereis_pid("Elixir.Pythonx.Evaluator")


Elixir = PythonxElixir()

[DnOberon]

@josevalim when you have a chance, could you update where you're at in the efforts listed above? This is extremely interesting to me and I might have some time coming up I can dedicate to at least knocking out some of the low hanging fruit.

[josevalim]

The list is up to date. :)