Missing info for numpy
tekumara opened this issue · 27 comments
Environment data
- Language Server version: v2020.7.2
- OS and version: macOS
- Python version (& distribution if applicable, e.g. Anaconda): Python 3.6.10
Expected behaviour
When I hover over np.ndarray I expected to see (class) ndarray
Actual behaviour
When I hover over np.ndarray it says (import) ndarray: Unknown
Code Snippet / Additional information
import numpy as np
def f(n: np.ndarray):
passnumpy 1.18.1
Much of numpy is compiled. We currently don't have a scraper that will create type stubs for compiled modules. Therefore, for now, a fix would be for numpy to ship stubs / annotations to describe their code, which I believe that they have merged. I'd be curious if this works better in numpy 1.19 (which I think has the stubs, but I'm unsure).
Thanks for the tip @jakebailey ! I've upgraded to numpy 1.19.1 but unfortunately, I'm still encountering the above.
I'm going to retitle this and use this to deduplicate numpy issues.
Note: there's already been a PR merged into numpy (numpy/numpy#16515) that adds type stubs (these used to be in a separate repo), but the commit isn't in a release yet. So, this should be working better soon.
There are other packages with this same problem (i.e. Pysam). Is it possible to add a setting to disable is not a known member of module to Pylance?
Does this issue also including the problem that doc strings are not displayed for some numpy functions such as numpy.arange? Note: for other functions like numpy.zeros_like the doc strings are displayed correctly.
Yes. These functions are from compiled modules; there's no source code we can read to obtain them. Some of these functions are showing up only because numpy has declared they exist in their __all__ expression, but we aren't able to actually find the definition (so assume Unknown).
Numpy's next release is supposed to include stubs, which will make these functions known, however the docstrings are still only going to be in the compiled module where we can't access them without actually running Python and live importing the module to extract it.
Looks like the stubs are coming in NumPy 1.20. It's not clear to me if this will help with auto-completion in situations such as the following in VS Code with PyLance:
def test() -> np.ndarray:
return np.arange(5)
x = test()
# x. <tab> doesn't give me any ndarray completions and the hover-text for test() says it returns Any
# n.b. it doesn't help to annotate x: np.ndarray either.When I select jedi as the language server I don't have this problem, but I guess jedi must do this by
actually running Python and live importing the module to extract it.
(?)
I think this NumPy issue is especially relevant: ENH: Add annotations to ndarray and generic
We were hoping that the new numpy stubs would help immensely (as opposed to the "actually run it and see" method), but I know that a lot of the current numpy stubs right now are just Any, which won't be super helpful. For the most part, it would help in cases where there really isn't any code for us to directly analyze without scraping.
Hopefully they don't go and release with incomplete stubs...
I forgot to update this, but as of the most recent release we should be able to find many more things in numpy than previously; we'd appreciate feedback about other things that you're seeing in numpy, however.
Whether or not this will work once numpy releases their own stubs, I'm uncertain.
I have upgraded to the newest version. But it alse can't catch the type returned by frequently-used api like:
a = np.array(1)
# Pylance think the type of a is Any
May the support for numpydoc handle more problems?
The stubs are generated directly from the compiled numpy code; we don't know anything more than that and just assume Any past there. Better types will come from numpy's new stubs. Parsing docstrings is a lot tricker and not something I would expect us to try for return types.
@jakebailey Since you said feedback about problems we see in numpy is appreciated...
np.linspace(1, 2, 3).tolist()causes Pylance to report
Cannot access member "tolist" for type "tuple[Any, Any | float]"
Member "tolist" is unknownPylance (reportGeneralTypeIssues)
The next release includes some improvements to our source mapper which should help get numpy docs and navigation:
The next release includes some improvements to our source mapper which should help get numpy docs and navigation:
Does that release shows np.logical_and, np.logical_or, etc. as well?
Current version lacks it:
Are you on at least numpy 1.20? That function is defined in their stubs and should appear, but may not on older versions of numpy without type info.
thanks. upgrading to 1.20 version helped!
Much of numpy is compiled. We currently don't have a scraper that will create type stubs for compiled modules. Therefore, for now, a fix would be for numpy to ship stubs / annotations to describe their code, which I believe that they have merged. I'd be curious if this works better in numpy 1.19 (which I think has the stubs, but I'm unsure).
Hi @jakebailey , I am new to vscode, and also encounter this issue. I understand that these compiled packages are hard to get the type info. But I notice that Pycharm can detect those types accurately, any idea how they make this? Thanks in advance!
Not sure if this will be of any help, but the next 1.21 release should improve many of the annotations.
Notably this includes replacing all Any-based placeholders with explicit function definitions: numpy/numpy#18838.
Much of numpy is compiled. We currently don't have a scraper that will create type stubs for compiled modules. Therefore, for now, a fix would be for numpy to ship stubs / annotations to describe their code, which I believe that they have merged. I'd be curious if this works better in numpy 1.19 (which I think has the stubs, but I'm unsure).
Hi @jakebailey , I am new to vscode, and also encounter this issue. I understand that these compiled packages are hard to get the type info. But I notice that Pycharm can detect those types accurately, any idea how they make this? Thanks in advance!
Without reading their implementation, either by running the library to obtain info at runtime (scraping), or by hardcoding, both of which we do not do. We have prescraped stubs for some info, but largely rely on numpy itself to provide types.
Not sure if this will be of any help, but the next 1.21 release should improve many of the annotations.
Notably this includes replacing allAny-based placeholders with explicit function definitions: numpy/numpy#18838.
Hm, I could have sworn this was to be there for 1.20.3, but maybe only a small subset was backported.
Hm, I could have sworn this was to be there for 1.20.3, but maybe only a small subset was backported.
These were two different, albeit related, issues:
- The annotations for a few functions weren't picked up.
- Most objects were annotated as
Any.
The first issue was indeed fixed in 1.20.3, the second one is scheduled for 1.21.
new numpy release with improved annotations https://github.com/numpy/numpy/releases/tag/v1.21.0rc1
In my case, if a numpy array, say "A", is not created with "A=np.zeros()" or "A=np.empy()", then Pylance can't recognize the property "A.shape", e.g., if "A" is created with "np.frombuffer()".
If you select "Go to Declaration" for frombuffer, you'll be taken to the place where this function is defined in numpy. I have the latest version installed, and it looks like that function isn't properly annotated with a return type.
def frombuffer(buffer, dtype=..., count=..., offset=..., *, like=...): ...So this is bug (an omission) within the numpy library. You can report the bug (or better yet, submit a PR) in the numpy repo.
@erictraut Thanks. I see now.
FYI, annotations for from_buffer were added back in numpy/numpy#19237.
They'll be available on pypi with the 1.22 release in a few months, but they are already on the master branch.
original issue fixed.
new numpy pre-release is out https://github.com/numpy/numpy/releases/tag/v1.22.0rc1
any further missing numpy type annotations should be logged there.