Visible Virtual Treeview causes multiple problems in other controls in Lazarus/Cocoa
Opened this issue · 61 comments
Here is a demo project which demonstrates multiple issues when having a visible virtual treeview on Lazarus Mac/Cocoa (using fpcupdeluxe "trunk" which includes newest virtual treeview)
Issue was reported here: https://bugs.freepascal.org/view.php?id=34002
And together with Lazarus developers narrowed it down to be an issue with virtual treeview and cocoa.
The attached demo project demonstrates:
- Toolbuttons becoming invisible until clicked
- TPageControl tabsheets misdrawn until clicked
- TEdit looking very weird with black background until clicked
- Some captions/labels etc. are drawn upside down some places
All only when the two virtual treeviews are set to visible in the project.
...
To see all issues simply
- run project
- doubleclick window title bar (minimizes to dock)
- open from mac dock.
VirtualTreeView does not uses cocoa directly, just LCL and lclextensions. Probably some of the LCL API used by VirtualTreeView is not mature enough on cocoa. The way to go is to isolate the API that is broken and fix it. In early days i did this for win32, qt and gtk2 widgetsets. I fixed a lot of LCL / widgetset. Unfortunately i don't have a mac.
If you want to isolate code from lclextensions, compile with USE_DELPHICOMPAT directive disabled (VTConfig.inc)
Thank you for your answer. I try see if i can find time to study LCL/widgetset implementations, Cocoa and connect the dots with Virtual Treeview. It is an extremely important control (to me at least), so kinda critical it is not working on Cocoa :(
As a start i really suggest compiling with USE_DELPHICOMPAT directive enabled. This will ensure only LCL functions are called. It should work normally but slower
I will try that now.
Meanwhile I I have tried two non-virtual-treeview demos using graphic routines that appears to work on Cocoa - a when least standalone:
- https://forum.lazarus.freepascal.org/index.php/topic,36871.msg246309.html#msg246309
Called SpriteTest2.Zip and it uses copyrect, mask, union
2) https://forum.lazarus.freepascal.org/index.php/topic,36871.msg246790.html#msg246790
Same thread, Mr.Madguy code listed uses intersect, fillrect, cliprect
Offhand not possible to comment out the USE_DELPHICOMPAT define - it results in the following errors:
VirtualTree.pas
- SetBrushOrigin
-- SetBrushOrgEx (Identifier not found) - HandleMouseDown
-- WM_LBUTTONDOWN (Identifier not found)
I did try comment out some other defines in VTConfig.inc - EnableAdvancedGraphics - but that did not make any difference.
...
Do you know what kind of graphic routines virtual treeview uses beyond those used in the two above demos? I am trying to create a demo application I can submit in my LCL cocoa bug report which reproduce the problem.
Compiled now on linux and working fine. Should at least compile on mac
WM_LBUTTONDOWN (Identifier not found)
What version are you using? Do not use master, use lazarus-v4 or lazarus_v5 with lclextensions 0.6.1
There's no WM_LBUTTONDOWN
SetBrushOrgEx (Identifier not found)
This is protected by {$ifndef INCOMPLETE_WINAPI}. Also SetBrushOrigin was removed long time ago
I am using the one that is included when using fpcupdeluxe/trunk / virtualtreeview-new (5.x)
Albeit currently lazarus now ships with virtual treeview package per default, just not installed - so fpcupdeluxe author states that is the newest version to use. (Not necessary to use fpcupdeluxe menu items for installing the virtualreeview-new (5x) package.)
You can see a problem report for lcl_extensions here:
https://bugs.freepascal.org/view.php?id=33973
http://forum.lazarus.freepascal.org/index.php/topic,42226.0.html
Where fpcupdeluxe developer tracked it down to: "Fpcupdeluxe uses online sources to install packages.However, these sources are fast changing and moving all the time.Now it seems that these vtv-sources are no longer updated due to the fact that they are included (again) with Lazarus ! So, just install the latest vtv by installing the vtv that comes with Lazarus."
(So that is what I have done)
...
But if I understand you correctly, I should probably install manually from the sources downloaded manually from here. I will do that and report back (!)
Fpcupdeluxe uses online sources to install packages.However, these sources are fast changing and moving all the time
lclextensions did not changed from a long time
But if I understand you correctly, I should probably install manually from the sources downloaded manually from here. I will do that and report back (!)
Yes. You can also use git and fork the repo. Use v4 or v5 branches:
https://github.com/blikblum/VirtualTreeView-Lazarus/tree/lazarus-v4
https://github.com/blikblum/VirtualTreeView-Lazarus/tree/lazarus-v5
This bug report still applies then:
https://bugs.freepascal.org/view.php?id=33973
delphicompat.pas(137,2) Fatal: Cannot open include file "uses.inc"
Looking in lclextensions-0/include you have subfolders for
-carbon
-generic
-gtk
-gtk2
-qt
-win32
(where each contains uses.inc)
But none for Cocoa
I suspect that is the reason and fix is simple. I will test later
Well, not so easy, but the latest version of LazarusTrunk includes a version of lclextensions where Cocoa version of delphicompat.inc (etc.) has been added.
So it would seem Lazarus people have made several additions without adding them to:
https://github.com/blikblum/luipack/tree/master/lclextensions/include
?
I uninstalled all old virtualtreeview... and fixed lclextensions
...
Then I downloaded from
https://github.com/blikblum/VirtualTreeView-Lazarus/tree/lazarus-v5
using "Download or clone"
Then go into VTConfig.ini and switch off USE_DELPHICOMPAT define
and I still get error
VirtualTrees (5282,3) Error: Identifier not found "SetBrushOrgEx"
and
VirtualTrees (22188,38) Error: Identifier not found "VM_LButtonDown"
...
I can send screenshots...
Please send screenshots of the places that the compiler warns about missing identifier
na
You are using lazarus-master branch which is not completed yet.
Try these: https://github.com/blikblum/VirtualTreeView-Lazarus/releases
Okay, so I on my Mac Lazarus Install:
- uninstalled old package
- downloaded and installed virtualtreeview-5-5.3-R1.zip
- comment off the define USE_DELPHICOMPAT
The same two issues but different line numbers. They are
VirtualTrees (5273,3) Error: Identifier not found "SetBrushOrgEx"
and
VirtualTrees (22179,38) Error: Identifier not found "VM_LButtonDown"
...
Just to confirm again I downloaded the file you suggest in Windows and opened in Notepad+ and navigated to the line numbers. I see them both there as well at the line numbers I report here.
...
And just to further confirm - I also tried download "source code (Zip)" https://github.com/blikblum/VirtualTreeView-Lazarus/archive/lazarus-5.5.3-R1.zip
just in case you meant I should download that - and again I see them both there same line numbers
Sorry, the v5 is missing some IFDEF. I will fix. Please try v4 for now: https://github.com/blikblum/VirtualTreeView-Lazarus/releases/download/lazarus-4.8.7-R4/virtualtreeview-4.8.7-R4.zip
Thank you. Personally I think I will have to wait for 5.x since it has some functionality I depend on.
However, it might be worth pursuing if 4.8.7 exhibits/causes the same drawing/clipping issue in Cocoa which drew me down a very long pursuit (took time before I got the problem isolated to be related to
virtual treeview)
...
When you fix 5.x will it be in the master branch
https://github.com/blikblum/VirtualTreeView-Lazarus
?
(just asking to make 100% sure I do not download wrong files)
...
What about the lcl-extensions issues for Cocoa? Do you want me to send you the files Lazarus distributes in its trunk for lcl-extensions-cocoa-include-files?
When you fix 5.x will it be in the master branch
Done in 1bf0494
Go to https://github.com/blikblum/VirtualTreeView-Lazarus/tree/lazarus-v5 and click on download
What about the lcl-extensions issues for Cocoa? Do you want me to send you the files Lazarus distributes in its trunk for lcl-extensions-cocoa-include-files?
I can take care of that
Thank you. While the clipping issue remains, it will hopefully help tracking down the problem being able to comment out USE_DELPHICOMPAT
I will help tracking the issue.
For start you can test the following bug on cocoa:
https://bugs.freepascal.org/view.php?id=28689
Demo at: http://bugs.freepascal.org/view.php?id=25564
I have some LCL code that i fixed on other widgetsets to make VTV work. I will look at it
You may want to test also ExcludeClipRect function
Look here for a ExcludeClipRect demo:
I will test everything today - right now latest trunk of Lazarus/Cocoa/64bit is causing problems but looking into it, so I can get Lazarus working again for testing all the things you have posted ASAP :)
I believe both your examples are buggy in Lazarus/Cocoa
I am attaching screenshots using the very latest Lazarus/Cocoa/64bit trunk
Yes they are buggy
Submitted bugs here
https://bugs.freepascal.org/view.php?id=34196
https://bugs.freepascal.org/view.php?id=34197
...
Do you think there could be more potential issues?
The original problem was that if you added a virtual treeview to a Cocoa project like in this demo project: http://forum.lazarus.freepascal.org/index.php/topic,41930.msg292282.html#msg292282
-
Toolbuttons becoming invisible until clicked (just click on the top toolbar to expose them)
-
TPageControl tabsheets misdrawn until clicked
-
TEdit looking very weird with black background until clicked
-
Captions/labels etc. are drawn upside down some places
I compiled the demo on windows and i have some remarks:
- By what i understand correctly, the issue does not exists when VTV is not visible, right?
- If i resize the window, the pink toolbar does not invalidate / repaint properly, regardless if VTV is visible or not
- The VTV components are children of the TForm with z-index on top. Try to put the VTV as child of the tabsheet
- There are many VTV components can you test with only one of them and than with no VTV component?
- The demo have many controls. Try to create a demo with only the minimal number of controls to demonstrate the issue
I simplified a bit the demo.
It has only one VTV component.
Try to test the following scenarios and post the results:
- Make the "With VTV" tab active
- Make the "No VTV" tab active
- Make the VTV component not Visible by default. Test. Toggle Visible. Test again
vtv-set-to-invisible--no-vtv-tab
vtv-set-to-visible--visible-vtv-tab--maximized
vtv-set-to-visible--visible-vtv-tab--just-resized-form
vtv-set-to-visible--visible-vtv-tab--just-resized-form--2--hover-over-button
inside-ide-reversed-text-and-icons
Here is a bunch of screenshots showing what different actions do - hopefully they will help
vtv-set-to-invisible--no-vtv-tab > VTV is set to not Visible? Everything fine right?
When VTV is visible and With VTV is active the paint at first show is fine?
The issue only manifests after minimizing / maximizing / resizing form right?
What happens if VTV is visible, but the No VTV Tab is active? Is the painting fine?
If so, you can do two tests to check if the issue manifests:
- make the With VTV tab active (select it)
- add a button that calls tvVirtual_Sitemap.Handle -> this will force control initialization but without painting
This is important to know if the problem is triggered by the initialization or the paint code.
Alternatively, put a exit call in the start of TBaseVirtualTree.PaintTree
Put a DebugLn inside TBaseVirtualTree.WMPaint and log the value of Message.PaintStruct^.rcPaint
Also check if TBaseVirtualTree.WMPaint is being called after minimizing / maximizing / resizing (when the issue occurs).
My theory is that somehow the form is not being invalidated
In CocoaPrivate inside
procedure TCocoaCustomControl.setFrame(aframe: NSRect);
begin
if NSEqualRects(aframe, frame) then Exit;
if isdrawing>0 then
faileddraw := true;
inherited setFrame(aframe);
// it actually should come from a notifcation
if Assigned(callback) then callback.frameDidChange(self);
end;
Put a breakpoint at faileddraw := true; If hit, write the callstack here
I will reply to all your questions during today and tomorrow.
...
I have also cleaned the demo a little so it is easier to see everything.
toggle-vtv-visible-then-reisze.zip
...
vtv-set-to-invisible--no-vtv-tab > VTV is set to not Visible? Everything fine right?
Adjusting the demo so it starts as invisible - no issue at startup or resize or maximize.
-The moment it is set visible the toggle button becomes invisible.
-maximixing then further causes issues in toolbar, edit etc.
...
add a button that calls tvVirtual_Sitemap.Handle
Next test:
vtv starts up as invisible, but "with vtv" tab active.
I click button which calls vtv.handle;
No visible change in anything. Also not when resized
vtv starts up as invisible, but "with vtv" tab active.
I click button which calls vtv.handle;
No visible change in anything. Also not when resized
Seems that the issue is triggered by the paint code.
The next steps are putting a breakpoint at TCocoaCustomControl.setFrame in the line that faileddraw is set to true and putting an Exit call at start of PaintTree and see what happens when VTV is Visible
Inserting "Exit;" as first line in .PaintTree solves the problem - but obviously not a very good solution :)
Faileddraw does not seem to trigger even if virtual treeview set to true, tab selected and I resize/maximize the window.
Inserting "Exit;" as first line in .PaintTree solves the problem - but obviously not a very good solution :)
But show us how to find the issue:
Put Exit calls in different parts of .PaintTree until find the offending line
You can go from start to end in the method or use a strategy similar to bisect: put exit in the middle, if no bug then try between middle and end otherwise between start and middle and go on
I suggest keeping the used exit calls commented to mark already tested places
Started doing that - then just realized I had made a grave error in my testing
...
I had inserted "Exit;" in
TBaseVirtualTree.VMPaint (solves issue)
and not in
TBaseVirtualTree.PaintTree (does not solve issue)
Just repeated the experiment multiple times to make sure it is done correct.
...
Anyhow, I will be happy to conduct more experiments - I will try see if I can diagnose it further. (If any ideas, let me know)
...
Until then I have submitted the two issues identified sofar here:
https://bugs.freepascal.org/view.php?id=34196
https://bugs.freepascal.org/view.php?id=34197
(I have also uploaded a demo and screenshot for each, but if you think anything is missing please feel free to add)
I am not sure if those two issue can explain the whole issue demonstrated in toggle-vtv-visible-then-reisze.zip
Started doing that - then just realized I had made a grave error in my testing
No problem, try to put Exit calls in TBaseVirtualTree.Paint
I am not sure if those two issue can explain the whole issue demonstrated in toggle-vtv-visible-then-reisze.zip
Probably not, but at least the SetWindowOrgEx bug must be resolved for a proper working VTV
Also add a breakpoint to TBaseVirtualTree.WMPaint and step into TWinControl.WMPaint:
Msg.DC must be different 0 and PaintHandler should be called
No problem, try to put Exit calls in TBaseVirtualTree.Paint
That solves the problem.... And seeing that it did not help exit PaintTree that really only leaves a limited amount of possible reasons left:
- ExcludeClipRect() - bug already found and reported per your guidance (the error probably ends in cocoawinapi.inc TCocoaWidgetSet.ExtSelectClipRGN and some of the stuff called there)
- FOffSetY
- FUpdateRect
Or can it be limited further?
(I do not think ComputeRTLOffset/RTLOffSet has any relevance since I am not using UseRightToLeftAlignment. I also do not think OffSetRect since its implementation is found in LCLProc and looks clean)
I guess nextup would be to confirm validity FOffSetY/FUpdateRect and then make a testcase for OffsetRect
Msg.DC must be different 0 and PaintHandler should be called
Msg.DC is different than 0 and PaintHandler is called
ExcludeClipRect() - bug already found and reported per your guidance (the error probably ends in cocoawinapi.inc TCocoaWidgetSet.ExtSelectClipRGN and some of the stuff called there)
One way to test it is unsetting Header.Options.hoVisible since is only called when header is visible
FUpdateRect
It should have a value similar (in first paint, after setting visible) to
LEFT = 0,
TOP = 0,
RIGHT = 638,
BOTTOM = 267,
It's valuabe also to check the value of FHeaderRect that is passed to ExcludeClipRect
It should be something like:
LEFT = 0,
TOP = 0,
RIGHT = 638,
BOTTOM = 17,
FUpdateRect
It should have a value similar (in first paint, after setting visible) to
LEFT = 0,
TOP = 0,
RIGHT = 638,
BOTTOM = 267,
With Header.Options.hoVisible = True; and Header.Options.hoVisible = False; values are:
LEFT = 0,
TOP = 0,
RIGHT = 618,
BOTTOM = 255,
It's valuabe also to check the value of FHeaderRect that is passed to ExcludeClipRect
It should be something like:
LEFT = 0,
TOP = 0,
RIGHT = 638,
BOTTOM = 17,
With Header.Options.hoVisible = True; values are:
LEFT = 0,
TOP = 0,
RIGHT = 618,
BOTTOM = 17,
ExcludeClipRect()
One way to test it is unsetting Header.Options.hoVisible since is only called when header is visible
Yes!
Setting Header.Options.hoVisible also solves the issue in the toggle-vtv-then-resize.zip demo application
There's a possible bug in TCocoaRegion.CombineWith when handling cc_DIFF (used by ExcludeClipRect)
If FShape is empty, a big one is created and stored in Shape var to be used in HIShapeCreateDifference, but Shape var is not used. The empty shape is used instead
cc_DIFF:
begin
if HIShapeIsEmpty(FShape) then
{HIShapeCreateDifference doesn't work properly if original shape is empty}
{to simulate "emptieness" very big shape is created }
Shape := HIShapeCreateWithRect(GetCGRect(MinCoord, MinCoord, MaxSize, MaxSize)); // create clip nothing.
Shape := HIShapeCreateDifference(FShape, ARegion.Shape);
Result := GetType;
end; Offhand this does not solve the issue:
cc_DIFF:
begin
if HIShapeIsEmpty(FShape) then
begin
{HIShapeCreateDifference doesn't work properly if original shape is empty}
{to simulate "emptieness" very big shape is created }
Shape := HIShapeCreateWithRect(GetCGRect(MinCoord, MinCoord, MaxSize, MaxSize)); // create clip nothing.
Shape := HIShapeCreateDifference(Shape, ARegion.Shape);
end
else
begin
Shape := HIShapeCreateDifference(FShape, ARegion.Shape);
end
;
Result := GetType;
end;
No sure my "fix" is correct though by any means. I usually do not deal with graphics/GDI
Offhand this does not solve the issue:
This was a blind shot, i don't ever know if in this case HIShapeIsEmpty(FShape) evaluates to true
It will be necessary to debug step by step the ExcludeClipRect code path by someone familiar with cocoa API
No sure my "fix" is correct though by any means.
It's correct
Setting Header.Options.hoVisible also solves the issue in the toggle-vtv-then-resize.zip demo application
Great. Now let's discover exactly where the problem is.
In TBaseVirtualTree.Paint, comment separately FHeader.FColumns.PaintHeader and ExcludeClipRect calls, seems that one of the two are the culprit
Inside TBaseVirtualTree.Paint:
Commenting out call to PaintHeader = problem continues
Commenting out calls to ExcludeClipRect = problem solved in the demo app (!)
...
For reference the Cocoa maintainer "skalogryz" seems active in the forum thread:
https://forum.lazarus.freepascal.org/index.php?topic=42246.msg296454
I pushed a fix disabling ExcludeClipRect in cocoa.
Probably there will be some drawing issues with header
Please do some tests with examples that shows a header, like the ones in demos folder and report here your findings.
Preliminary test seem very promising (!)
I will first get more time sunday - but hopefully I will be able to make a full report then.
It also seems the two reported bugs have been officially solved
https://bugs.freepascal.org/view.php?id=34197
https://bugs.freepascal.org/view.php?id=34196
You should test with the latest code from Lazarus repository with and without the dummy implementation of excludecliprect (comment the implementation in VirtualTrees.pas)
Regarding ExcludeClipRect the demo application in the bug report shows it as fixed on the latest Lazarus build: https://bugs.freepascal.org/view.php?id=34196
But... testing against the toggle-vtv-visible-then-reisze.zip demo the new Lazarus patch still does not work... whereas using your "dummy" ExcludeClipRect function does work...
Regarding SetWindowOrgEx the demo application now gives the expected result: https://bugs.freepascal.org/view.php?id=34197
But... testing against the toggle-vtv-visible-then-reisze.zip demo the new Lazarus patch still does not work... whereas using your "dummy" ExcludeClipRect function does work...
So we should keep it until someone find and fix the issue.
When scrolling up and down do you see the header flickering?
I need to know if is necessary to do manual clip in cocoa
When scrolling vertically the header gets hidden with the content in the virtual treeview.
Not sure if that is as intended (but does not happen in Delphi version of virtual treeview)
I am going to retest everything to make sure.
But I think virtual treeview now works much much better in Cocoa than before at least :)
Try behavior of scrolling in latest version
Your latest version solves the scrolling/header issue :)
Just downloaded the latestest version:
https://github.com/blikblum/VirtualTreeView-Lazarus/releases/
After this I had a user report back the faulty drawing again (text upside down - parts clipped off etc.)
I also updated to the latest Cocoa install.
I then reverted to my old install and see I have USE_DELPHICOMPAT directive disabled in VTConfig.inc while the download still has it enabled. I think on Cocoa this needs to be disabled.
The user confirms it works again now.. I think the public download maybe should have the VTConfig updated.
I will update the release
I don't think you ever got the official release updated? Which is problematic because Lazarus guys based their virtual treeview on the last official release I think if I am not mistaken. Or it is a kind of mish/mash :)
Done. Is the glitches fixed in cocoa?