Restriction of trigger playing HTML5 video

Question

Restriction of trigger playing HTML5 video

Opened this issue 6 years ago · 9 comments

Describe the bug

For now, as all we know, mobile browsers can not autoplay a video without user interactions.

As a workaround, we guide users to emit valid user interactions, such as touchend. (Full list of valid events can be found here)

I implemented similiar function with PIXI before. But, when working with Black, I used the same method to implement this function, it doesn't work.

I compared the PIXI's interactionManager and Black's Input, can not find useful information about this issue.

So I made 2 demos, one for Black, one for PIXI.

black-video-autoplay-issue - not work
pixi-video-autoplay - works

Hope that some guys can give me some tips.

To Reproduce
You can use following steps to run above 2 demos:

clone the repo
yarn install
yarn start to run the development server
Open an iPhone browser, visit http://<ip of your phone in LAN>:9000
Then you will see 2 Buttons:
- BUTTON DOM - ordinary button implemented with tag <div> (not sementic enough, I know);
- BUTTON CANVAS - text implemented with corresponding engine's Text Class;
The BUTTOM DOM works as expected, it trigger the playing of video
The BUTTON CANVAS:
- one with Black: not work, it doesn't trigger the playing of video;
- one with PIXI: works as expected, it trigger the playing of video;

Expected behavior
Make Black can be trigger the playing of video.

Screenshots
None.

Desktop

Device: iPhone6 / iPhone5 SE
OS: iOS 12.0
Browser: Chrome 69 / Safari

Some Ideas
I guess this is the problem about event bubbling. I am confirming it, also.

Answer 1 · 2018-10-07T17:54:59.000Z

Thanks, @m31271n for the detailed feedback.

The reason for this is pretty simple but let me explain to you how things work under the hood:

Game loop is more strict in Black than in other engines — it is more consistent
Black uses a fixed-time update and variable rendering strategy
Processing of input events/network packets are done before onUpdate is called

Every input, mouse, touch, network packet etc is stored till next frame and processed before update is called. In this way, you can be 100% sure you will NOT get any middle-of-the-frame calls.

So since input events are stored and processed in the next update they are not user interaction anymore and that's why video is not working for you.

The short solution is to:
document.addEventListener('touchend', () => this.yourPlayVideoFunc)

lmk if you need more details

Answer 2 · 2018-10-08T15:09:11.000Z

In a short word, "Black interrupts the browser's events, and make them be events in Black's domain. So, the event is not valid user interaction anymore", right?

The short solution you provided works in simple scene. But in some scene, it can not meeting the demand. Let me show an example:

┌─────────────────────────────────────────────────────────┐
│                                                         │
│                                                         │
│                 ┌────────────────────────┐              │
│                 │     LOGO LOGO LOGO     │              │
│                 └────────────────────────┘              │
│                                                         │
│                                                         │
│                                                         │
│                                                         │
│  ┌────────────┐      ┌─────────────┐      ┌──────────┐  │
│  │REPLAY VIDEO│      │TRIGGER MODAL│      │GO TO URL │  │
│  └────────────┘      └─────────────┘      └──────────┘  │
│                                                         │
│                                                         │
│                                                         │
└─────────────────────────────────────────────────────────┘

REPLAY VIDEO: trigger the playing of video
TRIGGER MODAL: trigger a modal created by canvas
GO TO URL: navigate to a another URL

When using with document.addEventListener('touchend', () => this.yourPlayVideoFunc), every touch will trigger the playing of video. Although, 'TRIGGER MODAL' and 'GO TO URL' works, but they also trigger the side effect - playing video.

So, I have to find a solution to catch 'touchend' event in a smaller range. Fortunately, I found it today.

I created a component called DomInteraction. Following is the code:

Before updated, it's called DomButton, as @62316e metioned below.

import { Black, Component } from 'black'

const LAYER = {
  DOM_DISPLAY: 1, // place video DOM
  CANVAS: 2, // place black canvas
  DOM_INTERACTION: 3, // place DOMs for interaction
}

function transformDOM(dom, matrix) {
  const {
    data: [a, b, c, d, tx, ty],
  } = matrix

  const transform = `matrix(${a}, ${b}, ${c}, ${d}, ${tx}, ${ty})`
  const transformOrigin = '0 0 0'

  dom.style.transform = transform
  dom.style.transformOrigin = transformOrigin

  dom.style.webkitTransform = transform
  dom.style.webkitTransformOrigin = transformOrigin
}

class DomInteraction extends Component {
  constructor(callback, { events = ['touchend'], debug = false } = {}) {
    super()

    if (!callback || typeof callback !== 'function') {
      throw new Error('[DomInteraction] invalid callback')
    }

    this.mDom = null
    this.mEvents = events
    this.mCallback = callback
    this.mDebug = debug
  }

  onAdded(gameObject) {
    const { containerElement } = Black.instance
    const { width, height } = gameObject

    const dom = document.createElement('div')
    this.mDom = dom
    dom.style.position = 'absolute'
    dom.style.width = `${width}px`
    dom.style.height = `${height}px`
    dom.style.zIndex = Layer.DOM_INTERACTION
    this.position()
    containerElement.appendChild(dom)

    if (this.mDebug) {
      dom.style.backgroundColor = 'rgba(255, 0, 0, 0.2)'
    }

    this.mEvents.forEach(event => {
      dom.addEventListener(event, this.mCallback)
    })

    Black.instance.stage.on('resize', this.position, this)
  }

  onRemoved() {
    const { containerElement } = Black.instance

    this.mEvents.forEach(event => {
      this.mDom.removeEventListener(event, this.mCallback)
    })

    containerElement.removeChild(this.mDom)

    Black.instance.stage.off('resize')
  }

  position() {
    const { gameObject, mDom: dom } = this
    const matrix = gameObject.worldTransformation
    transformDOM(dom, matrix)
  }
}

export default DomInteraction

The idea is:

create a <div>.
transform it to ensure it has same size and positon with the GameObject in Black's canvas (The transform data is provided by gameObject.worldTransformation).
place it on the top of Black's canvas.
bind native event listener.

The usage is simple:

const button = new Sprite('button-image')
const clickCallback = () => {
  /* playing video */
}
button.addComponent(new DomInteraction(clickCallback, { 
  events: ['touchend', 'click'],
}))

Now, I can catch the 'touchend' event in a smaller range.

Some thoughts

Q: The way Black does is right?
A: I think the answer is YES. As you said, "black is more consistent". Black interrupts the browser's event in very early time, and do let them spread in the scope of Black. It created the opportunity to add new Input Adaptor easily.

Q: Should this kind of function be added to Black?
A: Honestly, the need of opreating on DOMs when using a HTML5 Game Framework is rare. Maybe, it just needed in the scene which I described above. When user need it, they just add the function by theirself.

Anyway, I implemented this requirements, and the effect is plausible. :)

Last, Thanks for your help. I will try to read more code of Black.

Answer 3 · 2018-10-08T15:41:34.000Z

I like how you made the component out of this situation. Looks good.

In your case, every DOMButton is a separate DOM element. You could also try listening to document and just check if input actually happened over your game object. Not sure will this work for games inside IFrame.

document.addEventListener('touchend', () => {
	if (GameObject.intersects(this, Input.pointerPosition)) {
		// trigger the button
	} 
});

Q: Should this kind of function be added to Black?

A: Not sure about Black itself but it definitely should be the part of 'Black Component Library'

Answer 4 · 2018-10-08T16:04:37.000Z

@62316e GameObject.intersects method is better than mine, I will try it now. ;)

Answer 5 · 2018-10-08T16:34:43.000Z

@m31271n I'm not sure if listening for document will work inside IFrame.
Also, you can try experimenting with intersects and hitTest methods.

hitTest will return your only topmost touchable object.

So the best option I think is too have one shared DIV element on the top of the canvas and use hitTest

Answer 6 · 2018-10-08T17:06:56.000Z

Implement on the top of GameObject.intersects. Less code, more readable.

A: What? NativeInteraction?
B: Ya, Can't get a proper name. ;(

import { Component, GameObject, Input } from 'black'

class NativeInteraction extends Component {
  constructor(callback, { events = ['touchend'] } = {}) {
    super()

    if (!callback || typeof callback !== 'function') {
      throw new Error('[NativeInteraction] invalid callback')
    }

    this.mEvents = events
    this.mCallback = callback
  }

  eventHandler = () => {
    if (GameObject.intersects(this.gameObject, Input.pointerPosition)) {
      this.mCallback()
    }
  }

  onAdded() {
    this.mEvents.forEach(event => {
      document.addEventListener(event, this.eventHandler)
    })
  }

  onRemoved() {
    this.mEvents.forEach(event => {
      document.removeEventListener(event, this.eventHandler)
    })
  }
}

export default CurrentFrameInteraction

Call it:

const button = new Sprite('button-image')
const clickCallback = () => {
  /* playing video */
}
button.addComponent(new NativeInteraction(clickCallback, {
  events: ['touchend', 'click'],
}))

Have not been tested in iframe.

Great, have 2 solutions for now.

Answer 7 · 2018-10-08T17:39:20.000Z

Thanks @m31271n
I will keep this open so we can move it to wiki someday

Answer 8 · 2018-10-08T17:49:09.000Z

Your project, You decide. ;)

@62316e Trying to solve this problem with hitTest(), but encounter a problem:

The method of GameObject - hitTest(localPoint) refers to localPoint. Does localPoint means the coordinates related to current gameObject? If so, how to get that coordinates?

Answer 9 · 2018-10-08T18:14:40.000Z

Input.pointerPosition should work