palantir/conjure-python

[bug] Generated classes reference types they don't import, breaks mypy checks

tjb9dc opened this issue · 7 comments

tjb9dc commented

What happened?

Just upgraded to 3.10.2 from 3.9.0, mypy checks break due to missing imports from typing

#!/bin/bash -eo pipefail
source conda/bin/activate conda/envs/sundance-entity-extractor/
export MYPYPATH=$(cd stubs/ && pwd)
cd sundance-entity-extractor/service/app/
mypy *.py
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:7: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:51: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:143: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:211: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:247: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:254:64: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:255:60: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:299: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:306:60: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:343: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:379: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:431: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:451: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:479: error: Name 'Dict' is not defined
/home/circleci/project/stubs/net_owl_api/sundance_netowl_api/__init__.py:507: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:11: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:40: error: Name 'Any' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:50:39: error: Invalid base class
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:52: error: Name 'Any' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:57: error: Name 'Any' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:65: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:117: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:137: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:189: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:195:67: error: Name 'ListType' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:201: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:204: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:221: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:233: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:285: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:305: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:309:54: error: Name 'ListType' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:310:58: error: Name 'ListType' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:311:64: error: Name 'ListType' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:314: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:315: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:316: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:318: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:324: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:329: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:334: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:341: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:361: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:381: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:401: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:437: error: Name 'Dict' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:444:54: error: Name 'ListType' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:445:58: error: Name 'ListType' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:446:64: error: Name 'ListType' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:452: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:453: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:454: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:456: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:480: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:485: error: Name 'List' is not defined
/home/circleci/project/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:490: error: Name 'List' is not defined

What did you want to happen?

Generated classes should include everything from typing that they rely on

iamdanfox commented

@tjb9dc could you try 3.11.0? @ferozco merged a fix here: #79

tjb9dc commented

@iamdanfox @ferozco sadly only fixed part of the issue. The following is the same run after generating with 3.11.0 and copying over those stubs.

Imports for sundance_entity_extractor_api (net_owl_api is just missing Dict)

from abc import ABCMeta, abstractmethod
import builtins
from conjure_python_client import ConjureBeanType, ConjureFieldDefinition, ConjureUnionType, ListType, OptionalType
from typing import Optional, Set

Mypy errors

/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:7: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:48: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:131: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:193: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:227: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:234:64: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:235:60: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:275: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:282:60: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:316: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:350: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:398: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:418: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:445: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:472: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:11: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:40: error: Name 'Any' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:50:39: error: Invalid base class
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:52: error: Name 'Any' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:57: error: Name 'Any' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:65: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:113: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:133: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:181: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:193: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:210: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:222: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:270: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:290: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:301: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:307: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:312: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:317: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:324: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:344: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:364: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:384: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:418: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:432: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:456: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:461: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:466: error: Name 'List' is not defined

As you can see, ListType is now imported, but Any, Dict, and List are not. Also will need to look into what's wrong with ListType not being allowed to OptionalType and the Invalid base class error.

tjb9dc commented

I'll do some digging, could you all reopen?

tjb9dc commented

As for the Invalid Base class, it didn't like

class AdditionalPropertyFieldsVisitor(ABCMeta('ABC', (object,), {})):

Can that just inherit from ABCMeta? Switching to the following makes mypy happy

class AdditionalPropertyFieldsVisitor(ABCMeta):
tjb9dc commented

As for the ListType not allowed in OptionalType, the generated code it's upset about is

    @builtins.classmethod
    def _fields(cls):
        # type: () -> Dict[str, ConjureFieldDefinition]
        return {
            'ontology': ConjureFieldDefinition('ontology', str),
            'value': ConjureFieldDefinition('value', str),
            'id': ConjureFieldDefinition('id', str),
            'entity_arg': ConjureFieldDefinition('entity-arg', OptionalType(ListType(EntityArg))),
            'property': ConjureFieldDefinition('property', OptionalType(ListType(Property)))
        }

(args to both entity_arg and property), and the conjure yml is here

          entity-arg: optional<list<EntityArg>>
          property: optional<list<Property>>

Probably something weird with wrapping List in Optional

I would expect mypy to be okay with this because ListType is a child of ConjureType which is allowed there...mypy makes me sad sometime

tjb9dc commented

Just for others' SA, to run mypy on files that bring in imports from custom stubs (such as conjure-python generated classes importing stuff from conjure_python_client, or if you're running mypy on a separate file that leverage those conjure-python generated classes), you will have to copy over the packages you want mypy to traverse into a directory somewhere named stubs and set the environment variable MYPYPATH to that stubs directory.

My CircleCI check does something like

            source conda/bin/activate conda/envs/<your-env>/
            pip install mypy
            if [ -d "stubs" ]; then
              # Recreate the stubs directory
              rm -rf stubs
            fi
            mkdir stubs
            cp -r conda/envs/<your-env>/lib/python3.6/site-packages/<conjure-python-package> stubs/
            cp -r conda/envs/<your-env>/lib/python3.6/site-packages/conjure_python_client stubs/
            echo 'Added the following site-packages to stubs folder'
            ls -l stubs/

            mypy *.py <this throws any errors, etc>
robert3005 commented

this should be fixed in 4.0.0. The problem you've had is that ListType(x) is an instantiation and not a class and is not allowed as type definition