pywinrt: provide a mechanism for instantiating Python winrt types from a raw pointer to enable interop with existing Python code using frameworks such as comtypes
michaelDCurran opened this issue · 2 comments
There are situations where Python code may already have access to an ABI-level winrt COM object. E.g. a ctypes.c_void_p (void*) pointing to a COM object instantiated through a custom dll, or perhaps a comtypes COM pointer.
It would be useful if there was a way to convert this raw pointer into a true Pywinrt class instance. Similar to copy_from_abi in c++/winrt.
Similarly, going back the other way (copy_to_abi) would also be useful.
A very practical use case for this is when consuming the new Windows.UI.UIAutomation.AutomationElement or Windows.UI.UIAutomation.AutomationTextRange winrt interfaces.
Windows.UI.UIAutomation.AutomationElement is a winrt interface that is supported as an extra interface on the existing IUIAutomationElement COM interface, as part of the Microsoft UI Automation accessibility standard.
Windows.UI.UIAutomation.AutomationElement is not a class that can be activated by itself, nor does it contain all information that IUIautomationElement* interfaces do. Rather it provides a few extra methods (getAppUserModelID, isRemote etc).
There is no way to get an AutomationElement using only other objects and methods in the winrt namespace. You must always start with an IUIAutomationElement which you got from an event or method on IUIAutomation (CUIAutomation coclass, and then convert this IUIAutomationElement to the AutomationElement winrt interface, which in c++/winrt would be done with copy_to_abi.
a generic feature allowing conversion to a Pywinrt type from a winrt ABI type, would by itself allow for writing Python code to access these new methods provided by AutomationElement (getAppUserModelID) etc. But the power really comes from when we consider access to the new UI automation Remote Operations interfaces. Either the built-in Windows.UI.UIAutomation.Core.CoreAutomationRemoteOperation or the Microsoft.UI.UIAutomation objects, exposed in the microsoft/microsoft-ui-uiautomation project. This infrastructure allows you to batch logic and method calls to UI automation objects, drastically decreasing the amount of cross-process calls.
However, These remote operation interfaces require the caller to "import" existing AutomationElement and AutomationTextRange objects into the batch call so they can be referenced by the logic. Similarly, AutomationElement objects and AutomationTextRange objects may be fetched from the execution result at the end. But, in order to actually do anything useful with these objects, it is necessary to be able to both convert from a raw IUIAutomationElement* to AutomationElement and also from AutomationElement back to IUIAutomationElement*.
One solution that comes to mind for allowing conversion to a Pywinrt type is in pywinrt's pybase.h, in py::convert_to<Windows.Foundation.IInspectable): simply to support a Python object of ctypes.c_void_p, fetch the raw address from the 'value' member, and cast it to a Windows.Foundation.IInspectable.
Windows.UI.UIAutomation.AutomationElement already has a method 'fromElement' which allows passing in a Python object that is convertable to IInspectable, thus, it should then be possible to call fromElement giving it a ctypes.c_void_p instance as its argument. Note that COM interfaces generated by the Python comtypes package all inherit from ctypes.c_void_p, thus it would be possible to pass in a IUIAutomationElement comtypes COM interface pointer.
The one major risk with adding this would be that this would allow the caller to provide an arbitrary opaque pointer, and having pywinrt trust that this is truely an IInspectable. However, this is also true for the C++/winrt mechanism already I think.
To go back the other way, perhaps all winrt_base Python objects could inherit from ctypes.c_void_p, or they could contain a new method: to_abi, or a stand-alown Python function (to_abi) could be exposed on the root pywinrt package.
this request would go some way in allowing us to continue integration of UI automation Remote Operations into the NVDA Screen Reader project. Rather than having to re-write the majority of our code in C++/winrt, we could continue to stay in NVDA's native Python code.
This issue is stale because it has been open 10 days with no activity. Remove stale label or comment or this will be closed in 5 days.
Please see this repo for Python support: https://github.com/pywinrt/pywinrt