Auto-completion when instantiating BaseModel objects

[marlonjan]

Feature Request

Hi! Currently auto-completion does not work well when calling the initializer of classes that inherit from pydantic.BaseModel, at least in IntelliJ / PyCharm.

Are there any plans to improve that? Auto-completion works nicely with dataclasses from the python 3.7 standard library. Same applies to highlighting type mismatches or missing arguments in the call. Here is a comparison of the behavior in IntelliJ:

Code example to reproduce the screenshots:

from pydantic import BaseModel
from dataclasses import dataclass


class User(BaseModel):
    id: int
    name: str
    age: float


@dataclass
class UserDataclass:
    id: int
    name: str
    age: float


# no auto-completion
user = User()

# auto-completion
user_dataclass = UserDataclass()

Thanks for creating pydantic! Best regards,
Marlon

[timonbimon]

adding auto-completion support would be huge!

[dmontagu]

I would also love to see this.

attrs seems to be able to (very flexibly) generate a mypy-compatible type-hinted __init__, so I think this might be possible, at least in theory.

However, one potential issue is that parsing is performed during initialization, so you may purposely want to avoid using annotated types as type hints on the initialization kwargs. I do think it would be nice to have a type-hinted __init__ (or other method, I suppose) for creating model instances where parsing is known to be unnecessary, even if it means that a separate method had to be used for type-hint-safe parsing (I expect this could have some performance benefits as well).

If I recall correctly, there has been some discussion of replacing the use of BaseModel.__init__ with BaseModel.parse_object (or something similar) for parsing; I would expect that would be necessary for compatibility with a type-hinted __init__. @samuelcolvin I would be interested to get your take on whether that is realistic.

[samuelcolvin]

I would love this too.

I don't know how pycharm builds suggestions, does it use mypy or it's own parsing?

A mypy plugin as suggested in #460 would fix this case in mypy, but would it help in pycharm?

Maybe @pauleveritt who I think uses pydantic and works for pycharm might know?

I'm at pycharm europe now which is sponsored by pycharm so maybe I can find a real human being to ask.

[dmontagu]

Okay, I dug into this just a little:

1. it looks like mypy ships with an attrs-specific plugin to make the __init__ checking work: https://github.com/python/mypy/blob/master/mypy/plugins/attrs.py
2. I don't have a reference on me, but I've looked into this in the past, and if I recall correctly, PyCharm has a custom extension to ensure attrs __init__s look right. (Presumably this is also how they are handling dataclass). So it might be hard to get this support in a native way (though it would be great to hear from JetBrains if they had any suggestions).

Separately, I think it might be possible to achieve this by "tricking" the IDE in a variety of ways. I played with various approaches involving TYPE_CHECKING, but I think the one I currently like best is the following:

This behaves as though undecorated at runtime, but seems to behave properly with both PyCharm and mypy (retaining all desired autocomplete behavior for the BaseModel class). @marlonjan and @timonbimon you might find this useful in the short term?

@samuelcolvin I don't know how great this trick is as a long term solution, but for the sake of "paving the way" for such a capability it would probably makes sense to push in the direction of changing the use of the __init__ method so it is never expected to be called with incorrect types (i.e., for parsing). Maybe this could be a 1.0 thing?

[samuelcolvin]

Just spoken to pycharm in person, short answer is that we would need to build a pycharm extension in Java or kotlin to do this, attrs or dataclass code is open source and would be a good starting point.

[koxudaxi]

I want to support auto-completion on PyCharm.
@samuelcolvin
Thank you for asking the person.

I have searched issues and source code for attrs and dataclass.
The issue is to support auto-completion for attrs.
https://youtrack.jetbrains.com/issue/PY-26354

Also, we can find related commit(source code) by the issue number(PY-26354) in intellij-community repository.
https://github.com/JetBrains/intellij-community/search?o=asc&p=1&q=PY-26354&s=author-date&type=Commits

JetBrains developer implemented auto-completion for attrs and dataclass by request.
However, I don't know we should request to the developers.

if you already know the information, then I'm sorry.

[gmetzker]

This would be a great feature / plugin!

[samuelcolvin]

Jet brains people were very nice but said pydantic is not popular enough for them to build the plugin themselves, however they will help if we have any issues.

[pauleveritt]

Yes, I'm watching on this ticket. I'm here at EuroPython with the PyCharm team. If anybody has any questions about compiling PyCharm from GitHub, making a plugin, or what the specifics about a pydantic plugin, let me know.

[koxudaxi]

@pauleveritt
Thank you for watching the issue.
I have a question, Should we make a plugin or patching Pycharm?
If we can merge the feature on PyCharm then, I think that it is the best way.
However, I don't know you and your company want to merge the feature on the mainstream.

Btw, I have patched PyCharm and build it for auto-completion.

It's an experimental challenge yet. Console show errors :(

[dmontagu]

@koxudaxi Would you be willing to share your implementation?

[pauleveritt]

@koxudaxi sorry for the delay in replying, I'm working the PyCharm booth at EuroPython. Kudos for getting as far as you have! I can probably get someone from the PyCharm team to look at your code if you can put it in a gist or in a fork.

As you might imagine, PyCharm tries to be careful about the things it commits to supporting. Over the years it adds up, and we have to keep supporting things even when most people stop using them. So it's better to do it as a plugin and, if it becomes popular, we could discuss a merge.

[koxudaxi]

@koxudaxi Would you be willing to share your implementation?

@dmontagu sorry, I have forgotten it.
This commit is a prototype. if we hope to publish it then we need to implement more code.
I could add pydantic.dataclass easy. but, we will need to spend time to implement BaseModel which is different from other dataclasses like attrs.

koxudaxi/intellij-community@c654365

[koxudaxi]

@pauleveritt
Thank you for your explanation. It's all clear now.
I try to create a plugin :)

Also, I use Jetbrains products every day, which increase development speed. If I contribute to your community, I'm happy. Thank you.

[marlonjan]

Thanks to all of you for taking a look at this!!

[pauleveritt]

Thank you @koxudaxi and if you want me to get someone from the team to review, let me know.

I personally am very, very happy...VERY happy...to see you work on this.

[koxudaxi]

@pauleveritt
Thank you for your kindness. If we need some help, then we ask it to you.

[dmontagu]

@koxudaxi any chance you could explain how to build pycharm to make use of your modification? (Or point me in the direction of a guide?) I've never worked with custom pycharm plugins. I see it is in the README.md of your linked repo. Do you know if it is hard to use the plugin with the pro version of PyCharm? (It looks like building is against the community version; not sure what it looks like to integrate with pro.)

Related, I've made some progress on a mypy plugin, linked on this issue: #460. It definitely doesn't behave properly under all circumstances yet, but it at least works for me to type-check BaseModel.__init__ for all of the models I use. (I probably tend to follow some conventions that keep it away from the as-yet unhandled edge cases).

[koxudaxi]

@dmontagu

Do you know if it is hard to use the plugin with the pro version of PyCharm? (It looks like building is against the community version; not sure what it looks like to integrate with pro.)

I use the plugin in the pro version.
However, the plugin is on a very experimental phase.
It happens a lot of errors. @MrMrRobat have reported (koxudaxi/pydantic-pycharm-plugin#1)

It looks like building is against the community version; not sure what it looks like to integrate with pro.

Sorry, I don't know it. I must learn how to build a plugin for each edition.
I will try to fix the plugin this week.

I have checked your mypy-plugin, I feel a great start. :)
If we cant use mypy-plugin to do auto-complete and check types in Pycharm, then I think the best way.
We can maintain the plugin easier than Kotlin-plugin.

[dmontagu]

Yeah, I was pleasantly surprised how easy it was to work with once I got started.

Related, there also appears to be some good infrastructure for calling mypy runs from within python, which I think would enable us to more easily/incrementally add mypy tests (in particular, expected fails) than the current system.

I would be surprised if the mypy plugin worked directly with PyCharm, so it would probably still be nice to build an autocomplete plugin for PyCharm.

@koxudaxi if you can think of any specific requests for the mypy plugin beyond type checking the __init__ call, please post them to #460 and I'll try to add them.

[koxudaxi]

@dmontagu
I have fixed easy bugs on my pycharm-plugin. I continue developing the plugin.
OK, I will post some request or question on the issue.
Thank you.

[samuelcolvin]

great work @koxudaxi.

Feel free to ping me here if there's anything I can do to help make plugin development easier in pydantic.

[koxudaxi]

The PyCharm plugin for Pydantic is now available on JetBrains Plugins Repository.
The Plugin can be installed from Marketplace (PyCharm's Preferences -> Plugin -> Marketplace)

The Plugin page is https://plugins.jetbrains.com/plugin/12861-pydantic

Also, I am developing the plugin on this repository https://github.com/koxudaxi/pydantic-pycharm-plugin
If you have any opinion, Would you please leave feedback?
I'm waiting for your bug reports and feature requests, PRs.

[samuelcolvin]

Fantastic, well done.

I've installed it and at first glance it seems to do everything I need.

[samuelcolvin]

@koxudaxi would you consider submitting a PR to add this to the documentation?

[koxudaxi]

@samuelcolvin
I am happy seeing your good response 😄

OK, I will create the PR.

[davideasaf]

Hey everyone,

Does anyone know if a similar solution exists within VsCode? Auto-completion when instantiating BaseModel objects would be awesome there as well.

[RafalSkolasinski]

I just hit the completion problem when using Sublime with LSP and python-language-server.

Took me a while to figure out that completion for methods in class inheriting from BaseModel are not working and it's not something wrong with my code ;)

Any good workarounds until this is resolved?

pyls logs if helpful

2020-07-19 13:54:40,852 UTC - ERROR - pyls_jsonrpc.endpoint - Failed to handle request 6
Traceback (most recent call last):
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls_jsonrpc/endpoint.py", line 113, in consume
    self._handle_request(message['id'], message['method'], message.get('params'))
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls_jsonrpc/endpoint.py", line 182, in _handle_request
    handler_result = handler(params)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls_jsonrpc/dispatchers.py", line 23, in handler
    return method(**(params or {}))
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/python_ls.py", line 322, in m_text_document__completion
    return self.completions(textDocument['uri'], position)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/python_ls.py", line 240, in completions
    completions = self._hook('pyls_completions', doc_uri, position=position)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/python_ls.py", line 156, in _hook
    return hook_handlers(config=self.config, workspace=workspace, document=doc, **kwargs)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/manager.py", line 337, in traced_hookexec
    return outcome.get_result()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 52, in from_call
    result = func()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/manager.py", line 335, in <lambda>
    outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/plugins/jedi_completion.py", line 74, in pyls_completions
    ready_completions = [
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/plugins/jedi_completion.py", line 75, in <listcomp>
    _format_completion(c, include_params)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/plugins/jedi_completion.py", line 145, in _format_completion
    'documentation': _utils.format_docstring(d.docstring()),
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/api/classes.py", line 719, in docstring
    return super(Completion, self).docstring(raw=raw, fast=fast)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/api/classes.py", line 300, in docstring
    doc = self._get_docstring()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/api/classes.py", line 728, in _get_docstring
    return super(Completion, self)._get_docstring()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/api/classes.py", line 311, in _get_docstring
    return self._name.py__doc__()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/inference/compiled/value.py", line 329, in py__doc__
    value, = self.infer()
ValueError: not enough values to unpack (expected 1, got 0)

[talarari]

i'm using vscode and would really like to have auto completion for creating models.
static type checkers (pyright for example) have no way of knowing what the arguments for init are, but they do work well with dataclasses

would it make sense to have a class inherit from BaseModel and be decorated with dataclass ?

from dataclasses import dataclass
from pydantic import BaseModel

@dataclass
class User(BaseModel):
    id: int
    name: str
    age: float

[Bobronium]

would it make sense to have a class inherit from BaseModel and be decorated with dataclass ?

I doubt that code above would work properly, but what you can try is to make fake dataclass decorator and trick type checker to believe that it's decorated with real dataclass:

from typing import TYPE_CHECKING

from pydantic import BaseModel

if TYPE_CHECKING:
    from dataclasses import dataclass
else:
    def dataclass(model):
        return model

@dataclass
class User(BaseModel):
    id: int
    name: str
    age: float

[talarari]

this does work 👍

The thing is I don't want to trick my team mates when I trick the type checker, it will make it look like they are actually dataclasses...

i tried using just the pydantic dataclass but that gives this result:

from pydantic.dataclasses import dataclass

@dataclass
class User:
    id: int
    name: str
    age: float

could be great if the pydantic dataclass would do that maybe, and then i could just use `@dataclass from pydantic possibly (when fastapi fully supports it)

[talarari]

using @MrMrRobat advice i currently went with this workaround which seems to work with vscode and is not terribly misleading:

created an auto_complete_only.py helper and aliased @ dataclass to @ autocomplete

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from dataclasses import dataclass
else:

    def dataclass(model):
        return model


autocomplete = dataclass

[StephenBrown2]

Could even be simpler (tho untested):

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from dataclasses import dataclass as autocomplete
else:
    def autocomplete(model):
        return model

🤷‍♂️

[JosXa]

using @MrMrRobat advice i currently went with this workaround which seems to work with vscode and is not terribly misleading:

created an auto_complete_only.py helper and aliased @ dataclass to @ autocomplete

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from dataclasses import dataclass
else:

    def dataclass(model):
        return model


autocomplete = dataclass

Oh god, you must be kidding me. Pyright is so incredibly buggy...

[Bobronium]

@josx, as I understand, it's more a cosmetic issue, but you can try this workaround to use workaround above 🙃 without warnings:

from typing import TYPE_CHECKING

# trick PyRight into thinking that it might be both False or True
RUNTIME = not TYPE_CHECKING

if RUNTIME:
    def autocomplete(model):
        return model
else:
    from dataclasses import dataclass as autocomplete

I didn't test it, but it should work.

[samuelcolvin]

I'm closing this as pycharm has an excellent plugin built by @koxudaxi and vscode support is discussed in microsoft/python-language-server#1898

It sounds like any further solution either requires changes to the language servers or third party plugins.

If there are any tweaks pydantic can make to make the jobs of language servers easier, please create a new issue.

If anyone succeeds in building a useful plugin, please create a PR to update the docs.

[adamal]

Very nice to find that a workaround exists even if somewhat hacky.

I came up with this simple extension of the idea – here adapted for BaseModel.

With this, you can import BaseModel from here (say type_help.py) in stead of from pydantic, and if/when a proper solution becomes available, you can just switch your import references without removing a host of @autocomplete-annotations.

from typing import TYPE_CHECKING
from pydantic import BaseModel as _BaseModel

# trick PyRight into thinking that it might be both False or True
RUNTIME = not TYPE_CHECKING

if RUNTIME:
    def base_model(model):
        return model
else:
    from dataclasses import dataclass as base_model

@base_model
class BaseModel(_BaseModel): ...

[podj]

@koxudaxi Hey man, great job!
Let me know if you need any help with the maintenance! You've saved lives, my friend!

[amir20]

For anybody following this issue, the fix has been merged and available in version 1.9.0a2.

[Tomperez98]

When will v 1.9.0 be available?

[samuelcolvin]

Next few days.

[masreplay]

@samuelcolvin now I'm using SQLModel and autocompletion it's not with Pycharm work
@tiangolo