NEWT Integration

Question

NEWT Integration

masterav10 opened this issue 12 years ago · 17 comments

I would like to know what needs to be done to integrate this library with a NEWT Window. My particular requirements are as follows:

We have some number of displays (usually more than 1).
Each display will contain a GLWindow in Full Screen Mode.
Each GLWindow would contain a mix of Swing components and "Canvases" that contain OpenGL driven content (such as a video surface).

I had minor success integrating a GLWindow and JPanel with a single button using GLG2D, but adding more components resulted in weird drawing behavior I haven't been able to track down.

Let me know how interested you are in incorporating this feature. If it seems feasible, I most certainly can offer some assistance in its development. I doubt I'm the only person interested in seeing a solution to this problem. At the very least, I think some correspondence will bring the issues to light and create a road map for the solution.

Answer 1 · 2013-01-14T00:37:11.000Z

I actually haven't done anything with NEWT yet. My experience has been
entirely with SWT and Swing. Can you send me an example of what you're
trying to do? I'd like to run it and see how I could extend glg2d.

This also aligns with issue #10, writing to an image. I need to try to
abstract out the drawing so glg2d can draw to any GL context easily.

Answer 2 · 2013-01-14T06:04:08.000Z

For reference:
http://forum.jogamp.org/GLWindow-td4027817.html

Assuming that above issue is something simple, I should be able to provide the test case by the end of the week.

Answer 3 · 2013-01-21T22:05:55.000Z

So I'm not going to worry about the multi-screen issue at the moment. The following test demonstrates basic integration between NEWT and GLG2D. I'm currently on Windows 7 x64, since that is what we deploy to.

I have a basic pipeline to handle the conversion from NEWT events to AWT events. You'll notice the upper button lights up when moused over, but the bottom button only partially renders. The console output indicates that Swing is aware of the bottom button's location however, which is interesting.

So anyway, this is a very basic idea of what I am trying to do.

Answer 4 · 2013-01-22T15:55:17.000Z

Thanks, Dan, I'll take a look at this.

Answer 5 · 2013-01-27T02:14:09.000Z

I'm including a patch of the work I did to make your NEWT test work for
me. This work was done on your 78a489b.

I did several things

the NEWT modifiers need to be mapped to AWT modifiers
the NEWT buttons need to be mapped to AWT buttons
the x,y that you're passing into the MouseEvent need to be relative to
the source component
the absolute x,y should be relative to the screen, but the line I
commented out fails if the Swing components aren't part of a visible swing
hierarchy (which they're not in this case)

Do you mind if I merge some of your code into the main glg2d trunk? I
think it would be useful.

diff --git a/src/main/java/org/jogamp/glg2d/newt/NewtAWTEventFactory.java
b/src/main/java/org/jogamp/glg2d/newt/NewtAWTEventFactory.java
index b7fda87..332de2d 100644
--- a/src/main/java/org/jogamp/glg2d/newt/NewtAWTEventFactory.java
+++ b/src/main/java/org/jogamp/glg2d/newt/NewtAWTEventFactory.java
@@ -3,7 +3,10 @@
import java.awt.AWTEvent;
import java.awt.Component;
import java.awt.Container;
+import java.awt.Point;
import java.awt.event.MouseWheelEvent;
+
+import javax.swing.SwingUtilities;

import jogamp.newt.awt.event.AWTNewtEventFactory;

@@ -22,43 +25,42 @@
*/
public class NewtAWTEventFactory extends AWTNewtEventFactory
{

/**

\* This is an inverse map {@link AWTNewtEventFactory#eventTypeAWT2NEWT}

```
*/
```
private final static IntIntHashMap EVENT_TYPE_NEWT_2_AWT;

- private final static IntIntHashMap BUTTON_MASKS_NEWT_2_AWT;

- private static final int KEY_NOT_FOUND = 0xFFFFFFFF;

static
{

   EVENT_TYPE_NEWT_2_AWT = new IntIntHashMap();

- EVENT_TYPE_NEWT_2_AWT.setKeyNotFoundValue(KEY_NOT_FOUND);

   for (Entry entry : AWTNewtEventFactory.eventTypeAWT2NEWT)

```
   {
```

       EVENT_TYPE_NEWT_2_AWT.put(entry.getValue(), entry.getKey());

- }

   BUTTON_MASKS_NEWT_2_AWT = new IntIntHashMap();

   BUTTON_MASKS_NEWT_2_AWT.setKeyNotFoundValue(KEY_NOT_FOUND);

- BUTTON_MASKS_NEWT_2_AWT.put(0, 0);

```
   addButtonMask(MouseEvent.BUTTON1);
```
```
   addButtonMask(MouseEvent.BUTTON2);
```
```
   addButtonMask(MouseEvent.BUTTON3);
```
```
   addButtonMask(MouseEvent.BUTTON4);
```
```
   addButtonMask(MouseEvent.BUTTON5);
```
```
   addButtonMask(MouseEvent.BUTTON6);
```
```
   addButtonMask(MouseEvent.BUTTON7);
```
```
   addButtonMask(MouseEvent.BUTTON8);
```
```
   addButtonMask(MouseEvent.BUTTON9);
```

- }

private static final void addButtonMask(int newtMask)
{

   BUTTON_MASKS_NEWT_2_AWT.put(getAWTButtonDownMask(newtMask),

newtMask);

}
/**

\* This is an inverse map {@link AWTNewtEventFactory#eventTypeAWT2NEWT}

```
*/
```
private final static IntIntHashMap EVENT_TYPE_NEWT_2_AWT;

private static final int KEY_NOT_FOUND = 0xFFFFFFFF;

static
{

   EVENT_TYPE_NEWT_2_AWT = new IntIntHashMap();

   EVENT_TYPE_NEWT_2_AWT.setKeyNotFoundValue(KEY_NOT_FOUND);

   for (Entry entry : AWTNewtEventFactory.eventTypeAWT2NEWT)

```
   {
```

       EVENT_TYPE_NEWT_2_AWT.put(entry.getValue(), entry.getKey());

```
   }
```
}

private static int newtModifiers2Awt(int newtMods) {
int awtMods = 0;
if ((newtMods & com.jogamp.newt.event.InputEvent.SHIFT_MASK) != 0)
awtMods |= java.awt.event.InputEvent.SHIFT_MASK;
if ((newtMods & com.jogamp.newt.event.InputEvent.CTRL_MASK) != 0)
awtMods |= java.awt.event.InputEvent.CTRL_MASK;
if ((newtMods & com.jogamp.newt.event.InputEvent.META_MASK) != 0)
awtMods |= java.awt.event.InputEvent.META_MASK;
if ((newtMods & com.jogamp.newt.event.InputEvent.ALT_MASK) != 0)
awtMods |= java.awt.event.InputEvent.ALT_MASK;
if ((newtMods & com.jogamp.newt.event.InputEvent.ALT_GRAPH_MASK) != 0)
awtMods |= java.awt.event.InputEvent.ALT_GRAPH_MASK;
return awtMods;
}

private static int newtButton2Awt(int newtButton) {
switch (newtButton) {
case com.jogamp.newt.event.MouseEvent.BUTTON1: return
java.awt.event.MouseEvent.BUTTON1;
case com.jogamp.newt.event.MouseEvent.BUTTON2: return
java.awt.event.MouseEvent.BUTTON2;
case com.jogamp.newt.event.MouseEvent.BUTTON3: return
java.awt.event.MouseEvent.BUTTON3;
}
return 0;
}

/**
Creates an AWT Mouse event from the provided Newt mouse event.
@@ -80,13 +82,16 @@

long when = event.getWhen();
int modifiers = event.getModifiers();
```
   int x, xAbs, y, yAbs;
```
```
   x = xAbs = event.getX();
```
```
   y = yAbs = event.getY();
```

   Component source = root.getComponentAt(x, y);

Point p = new Point(event.getX(), event.getY());
+// Point absP = new Point(p.x + root.getLocationOnScreen().x, p.y +
root.getLocationOnScreen().y);
Point absP = p;

   Component source = root.getComponentAt(p);
 source = source == null ? root : source;

p = SwingUtilities.convertPoint(root, p, source);
int clickCount = event.getClickCount();
boolean popupTrigger = false;

   modifiers = newtModifiers2Awt(modifiers);

 switch (event.getEventType())
 {

@@ -100,15 +105,16 @@
int wheelRotation = -1 * event.getWheelRotation();

     return new java.awt.event.MouseWheelEvent(source, id, when,

               modifiers, x, y, xAbs, yAbs, clickCount, popupTrigger,

               modifiers, p.x, p.y, absP.x, absP.y, clickCount,

popupTrigger,
scrollType, scrollAmount, wheelRotation);

 default:

     int button = event.getButton();

 button = newtButton2Awt(button);

     return new java.awt.event.MouseEvent(source, id, when,

modifiers,

               x, y, xAbs, yAbs, clickCount, popupTrigger, button);

               p.x, p.y, absP.x, absP.y, clickCount, popupTrigger,

button);
}
}

Answer 6 · 2013-01-28T15:30:00.000Z

Take whatever you need. If you need me to do anything let me know, I'm new to Git and Github.
At some point this week I'll take a look at what you did.

I might also have a solution for the non-visible swing hierarchy. I'll keep you posted here.

Answer 7 · 2013-01-28T15:59:25.000Z

What do you mean by 'non-visible swing hierarchy'? Your NEWT example
supports painting a Swing hierarchy that isn't anchored to a parent
JFrame. In my tests, I've found that before you paint a JComponent that
doesn't have a parent JFrame, you need to call the following

setSize()
doLayout()
notifyAdd()

and that tricks the JComponent into thinking it should allow painting.

Answer 8 · 2013-01-28T19:19:36.000Z

By non-visible I mean a swing hierarchy outside of a top-level container (aka Window).

I'm pretty sure the notifyAdd() generates the native peer for the hierarchy.

The peer used on a non-window hierarchy is a NullComponentPeer, which essentially provides bare bones support for the EDT. I'm afraid some Swing functionality may be lost because of this. I'll work on a test case to verify this.

Answer 9 · 2013-02-04T02:16:31.000Z

I'm closing this with commit 30cb550. You can see https://raw.github.com/brandonborkholder/glg2d/master/src/test/java/org/jogamp/glg2d/examples/NewtExample.java for an example of how I've integrated glg2d into NEWT. Thanks for your help and code! I merged one of your commits into my branch, but your masterav10 name didn't get linked properly to the commit, sorry about that.

If you have more comments or issues related to Newt, just re-open this issue.

Answer 10 · 2013-02-15T05:02:11.000Z

First, I've done some significant work with the integration of NEWT and GLG2D. See org.jogamp.glg2d.newt.

Got a couple new issues.

https://github.com/masterav10/glg2d/blob/master/src/test/java/org/jogamp/glg2d/newt/TextPanel.java

Notice the text fields have no cursors. This is likely a combination of not porting KeyEvents from NEWT over to AWT and a lack of focus management from native peers.

https://github.com/masterav10/glg2d/blob/master/src/test/java/org/jogamp/glg2d/newt/ButtonDemonstration.java

When you click on a button and then drag off it, the button remains "clicked" in. Not sure what is causing this, but I think it is GLG2D.

Both tests should be run with the following:

-Dawt.toolkit=org.jogamp.glg2d.newt.GLG2DWindowToolkit

Answer 11 · 2013-02-18T00:40:34.000Z

What JVM and version did you use to build this? I'm getting some errors with the implementation of the GLG2DWindowToolkit. Missing methods, non-existent classes and such.

I'm using Oracle's 1.6.0_30

Answer 12 · 2013-02-19T06:33:57.000Z

Oracle JDK 1.7.0_09. Platform is Windows 7 x64.

But if you are asking that question, I'm afraid this may be a lot more work than I originally intended. As you know, our current implementation uses a windowless component as the root. This is a problem because we use methods like isVisible() or isDisplayable() which do not work properly in a windowless container. So far the only solution I see is to create a custom toolkit and provide a "native" window peer, which essentially hooks into the GLWindow provided. I was hoping I could leave most methods as stubs, but it looks like the peer interfaces are used differently depending on the platform and JDK version.

In addition to the problem above, some components do not behave correctly. At this time I cannot deduce if the problem is caused by Swing or GLG2D. The following components are a problem:

Any implementation of AbstractButton
- Clicking on a button, dragging off the button, and then releasing the mouse causes the button to remain "pushed in".
JComboBox
- Dropdown does not appear, although the clicking appears to work.
Text Fields, Panes, Labels
- No cursor in the text field.
- Typing does not appear to work (need to forward key events to AWT like we did with the mouse).
Scroll panes
- Dragging the scrollbar stops working if the mouse leaves the scrollbar's draw space.

I'm planning on making individual test cases for the components specified here:
http://docs.oracle.com/javase/tutorial/uiswing/components/index.html

Answer 13 · 2013-02-19T14:01:24.000Z

Well, I'm not giving up yet. I did some work over the weekend. Somebody
else sent me an email about JPopupMenus in Newt. To fix that I had to do
something similar to what you did with a Toolkit. I haven't pushed that
yet. I'll let you know when I do.

As far as I can tell, a new Toolkit implementation is the only way to use
some of these features, integrating Swing into a Newt window.
http://openjdk.java.net/projects/caciocavallo/ might be able to answer
this, but it's not mainline yet.

Answer 14 · 2013-02-21T15:47:55.000Z

Tests for newt have been updated:

https://github.com/masterav10/glg2d/tree/master/src/test/java/org/jogamp/glg2d/newt

Whew... and now I know why caciocavallo exists. I recommend using Java 1.7 for my test cases. The peer interfaces have changed radically between 1.6 and 1.7 to point where it will become a nightmare to try and maintain compatability with both. I'll leave the stubs in there for now, however.

If I can figure out how to build OpenJDK on Windows, I'll give caciocavallo a shot.

Answer 15 · 2013-02-21T16:24:33.000Z

Agreed. it really is quite a task. Since this is really such a hack, I
think the best way to proceed is to support only Java 1.7 and only Oracle's
implementation for now.

I figured out why your button example doesn't work and that's going to take
some time to fix. The text input one I'm not sure yet.

Thanks for pushing these examples, any tests you can send me are
appreciated.

Answer 16 · 2013-02-22T18:05:31.000Z

Made a bunch of changes over the past couple of days. I moved all Newt-related code into the org.jogamp.glg2d.newt.

I've also done significant work with the peer components. The following now works:

Focus (including tabbing)
Key events.
Text components.

I recommend using this test case for identifying issues:

https://github.com/masterav10/glg2d/blob/master/src/test/java/org/jogamp/glg2d/newt/TabbedPaneTest.java

I'm currently running through my tests to see what else needs to be done. I assume when you fix that button issue it will fix some of the other funny business I'm attributing to mouse dragging (such as selecting text in a textbox).

Also, to get JTrees and Internal Frames to work, I had to modify the following class. Look for the XXX comment.

https://github.com/masterav10/glg2d/blob/master/src/main/java/org/jogamp/glg2d/impl/AbstractImageHelper.java

Answer 17 · 2013-03-12T03:34:02.000Z

I've create a new branch to address a repaint issue. GLG2D performs slow with complex structures such as JTables. TabbedPaneTest in master should demonstrate this (watch the flickering canvas).

I created a new branch for painting using the [paint immediately](http://docs.oracle.com/javase/7/docs/api/javax/swing/JComponent.html#paintImmediately(int, int, int, int) function inside of a custom repaint manager. A new listener in GLG2DWindow paints dirty components to an FBO that maintains cumulative changes, and then blits the framebuffer over to the main framebuffer as the final step during each draw. Still has some kinks in it.

In general, I think I'm missing a significant amount of AWT generated events, and that is why some of the components do not behave like I expect them to.

GLG2DWindow is no longer headless, and I'm planning on pushing all GLWindow-related functionality into the peer layer. It will extends JFrame once I am done, and behave just like one.

Take a look at the immediate-paints branch, I'm curious what you think about my approach.