izivkov/RemoteVideoCam

motion detection feature

Opened this issue · 34 comments

st3b1t commented

your is a amazing app I have had a chance to try it and I think it is currently the most effective open source app for monitoring realtime video streaming.

I think for your use case it is important to add a simple motion detection feature.
here:
https://github.com/jjoe64/android-motion-detection

you find a very simple example that you can integrate with: <SurfaceView i am not a kotlin programmer but looking at the code it seems to me a simple integration. I suppose here:
https://github.com/izivkov/RemoteVideoCam/blob/main/app/src/main/java/org/avmedia/remotevideocam/camera/customcomponents/WebRTCSurfaceView.kt#L12

I'm trying to do this integration myself pero I'm missing a lot of Kotlin/Android Studio development acquaintance... I managed to start in development environment RemoteVideoCam which works perfectly, connected to my local webcam so I can test.
The problem is the repository I linked, it looks like an old project and I can't start it in android studio.

you maybe can give me some pointers on what part of the code to include, I assume it is a very simple thing to do for those familiar with Kotlin.

I think this issua is also useful for other contributors who have the skills to send a PR

st3b1t commented

this functionality could start like this:
A new button on the main layot that enables or disables motion recognition that play a sound.

izivkov commented

This is a great idea, but I'm tied up with another project right now. If someone wants to tackle this feature, please create a pull request, and I'll be glad to assist

izivkov commented

@st3b1t

I'm sorry, I just don't have time to work on this project at the moment. Are you interested in contributing by creating a PR with that feature.

toliuweijing commented

@izivkov @st3b1t

I have a prototype. See details in toliuweijing#3

Highlight
Use OpenCV for better noise reduction and foreground segmentation on the binary image
Use OpenGL framebuffer to outline the detected motion.

APK
Link

Demo
output

  • 🎉1
izivkov commented

Ok, it looks very interesting, but can you describe practical use of the motion detection? Does it trigger image or video recording?

Ivo

toliuweijing commented

Ok, it looks very interesting, but can you describe practical use of the motion detection? Does it trigger image or video recording?

Ivo

Video recording would be a good use, just as a surveillance camera. I would love to explore this more.

  • 👍1
izivkov commented
  • 👍1
izivkov commented
toliuweijing commented

At least, can we have an audio tone if we detect movement? This would be useful for a baby monitor when the baby wakes up.

Baby monitor is a great idea. I would think more about the qualifications for a good baby monitor app. Some initial thoughts are

  1. better night vision for baby wakeup detection.
  2. better reliability, e.g. handle connection lost, camera open failure, etc.
  3. minimal latency between consecutive frames.

This would be useful for recruiting animals in the wild.

Sound interesting. Could you elaborate more on "recruiting the animals in the wild"? Does it require WIFI-Direct feature?

  • 👍1
izivkov commented
izivkov commented

So this app is already widely used as a baby monitor. Your improvement suggests are a great, and by all means if you have time, add them. But in the mean time, we can add small improvements like detecting movement. I think that would be useful for knowing when he baby wakes up. So, do you think we can add this feature before merging the code?

In the issue of recording wild life, you could probably use the app in your backyard as is if there is WiFi. But for longer distances, I plan to add different type of connection, not just WiFi. I worked on a similar project before, and we implemented many types of connections implementing the same interface, so the rest of the code did not have to change. But this is for another discussion if you like to work on this in the future

toliuweijing commented

do you think we can add this feature before merging the code?

That is the plan. All prototype PRs merge to the fork repo for better code quality control. It needs more time to reach feature complete for baby wakeup detection(e.g. edge case, perf tuning, testing). Once ready, will have your review on the official PR against this repo.

we implemented many types of connections implementing the same interface, so the rest of the code did not have to change

Good to know. I will keep this in mind when designing the detection logic to avoid changing the interface.

WARNING: If you are using this app as a baby monitor, please be aware that the video might freeze. To ensure the video is still running, use another method, such as placing a constantly moving object in the frame, like a clock with a second hand.

We can render a live watermark(like a timestamp or spinning circle) on the sender side. Viewer can easily know whether the sender side has technical issues by checking the watermark. WDYT?

toliuweijing commented

I have a prototype that

  1. Display can remotely toggle motion detection on/off.
  2. Display can receive detection result from Camera.
  3. Display can show the notification with custom sound optionally.

Notification is a good way to notify the user for attention. It needs a new permission for Android 13 for sure.

output

  • ❤️1
izivkov commented

Hey Jim,

Thanks for your hard work; it looks really good.

I have a couple of questions/comments we can discuss:

  1. What happens in a "noisy" environment, where there is a lot of movement? Would the notification keep popping up?
  2. If I were monitoring a baby, it would be good to know not only when there is movement but when the movement stops as well.
  3. Showing the green lines can distract and reduce the enjoyment when watching something.

Here is a possible solution to 1 & 2:

We can monitor not only movement but also lack of movement. Say, if there is little movement (some threshold) for, say, 30 seconds, we start monitoring for movement. If there is movement for more than 30 seconds, we start monitoring for a lack of movement. If movement stops for some time (again, 30 seconds), we can issue another alarm, different from the first one. We then start monitoring for movement.

What do you think?

Regards

  • 👍1
izivkov commented

Hey Jim, do you think we can merge what you have now? It brings some good new features to the app. We can think about further improvements later.

Regards

toliuweijing commented

@izivkov great suggestions. I have been thinking a lot on those questions. I would love to make this feature solid and the app more useful. Let me get back to this thread once I polished up my work. ETA is end of week.

In the meantime, if possible, do you mind sharing more use cases how the app is used, or any future plan/feature for this app? Thanks.

izivkov commented
  • 👍1
izivkov commented

Another use case is a walkie-talkie for the house.

  • 👍1
st3b1t commented

Another use case is a walkie-talkie for the house.

the wesocket stream of this app is very efficient... it's a good solution for anti-intrusion home alarm.

@toliuweijing very well done!
I will try to do some tests as soon as possible. Let's keep this issue alive

st3b1t commented

@izivkov @toliuweijing I can suggest another use case:
I think it is useful as an anti-theft system for cars and vans.
leaving a smartphone inside the vehicle and connecting with another when parking the vehicle, on a camping trip or in particular situations I think it's an excellent use case, or monitoring a Dog inside the car...
If you then add a small Wifi Access Point it is a system that can work perfectly without internet for tens of meters away

  • 👍2
izivkov commented
  • 👍1
izivkov commented

@toliuweijing @st3b1t

Here are my thoughts on how to use the motion detection feature.

  1. There should be a "motion detection mode" and a "normal mode." This could be controlled by a simple toggle button; nothing else changes from the original app. This keeps the app simple, which is one of the reasons this app is popular.

  2. In motion detection mode the screen will remain dark until motion is detected. Upon detection, an audible sound will be heard, and the screen will turn on for at least 60 seconds after motion is no longer detected.

  3. This mode can be used for security cameras, saving the phone's battery, since many phones will not have a power supply when used as a camera.

  4. In normal mode, I think the RED/GREEN dot makes sense for camera freeze. It is not entirely clear to me how we can detect a frozen camera while the app is still running. What could happen is that the whole app will freeze, and then we cannot control the DOT.

Let’s discuss.
Ivo

  • 👍1
toliuweijing commented
  1. There should be a "motion detection mode" and a "normal mode." This could be controlled by a simple toggle button; nothing else changes from the original app. This keeps the app simple, which is one of the reasons this app is popular.

Yes. This is the current behavior in latest version.

In motion detection mode the screen will remain dark until motion is detected. Upon detection, an audible sound will be heard, and the screen will turn on for at least 60 seconds after motion is no longer detected.

Current behavior sends out a notification with default system sound. User can customize the notification sound with a different alarm / ringtone in System Settings -> App -> Motion Detection Notification Channel -> Sound. See instruction.

Another option is to hardcode a custom sound per notification. I'm not sure if this is preferable because 1) the audio is hardcoded and can't be changed by user. 2) diffferent users may find it disturbing, then completely disabling this feature.

  1. This mode can be used for security cameras, saving the phone's battery, since many phones will not have a power supply when used as a camera.

I think it is the default system behavior to dim the display to save battery after being inactive for some thresholds(changable in the user).

  1. In normal mode, I think the RED/GREEN dot makes sense for camera freeze. It is not entirely clear to me how we can detect a frozen camera while the app is still running. What could happen is that the whole app will freeze, and then we cannot control the DOT.

We can solve this by rendering a blinking green dot. Users will notice a frozen app if a green dot stops blinking.

  • 👍1
izivkov commented

@toliuweijing Glad to discuss these issues.

My general philosophy is to minimize unnecessary user input and simplify app use to the absolute minimum by using judicious defaults. Here is why I am not a fan of notifications in this app:

  1. Notifications need to be canceled, and they can add up in numbers with multiple motion detections. The user will then have to spend time closing them all. I am constantly closing various notifications on my phone, and I find it annoying. Worse, the user can turn off notifications for this app or all apps, and then they will not get notified. From a usability point of view, the user is only interested in the event as it happens (i.e., motion has been detected). There is no point in keeping these notifications for the future.

  2. As far as the particular sound, I am not sure, but I think we can use the same sound as the user has defined for default notifications. Even if not, I would still prefer to hear some fixed sound rather than cancel notifications.

  3. About dimming the display: I think the app should control this explicitly and finely. Currently, the screen is kept on because it is annoying for the user to keep turning it on all the time. Conversely, when in "Motion Detection Mode," the screen should be off, according to the app function. It is better than relying on some screen timeout setting. When motion is detected, the screen should turn on immediately. Ideally, it will work like this:

    When switched to this mode, keep the screen on for, say, 3 seconds. At the same time, display a toast() at the bottom of the screen, saying something like: "Going into Motion Detection mode. Screen will blank out until motion is detected." Then blank out the screen.

  4. A minor point, but maybe this icon is a bit more descriptive for motion detection: https://www.iconarchive.com/show/material-icons-by-pictogrammers/motion-sensor-icon.html

Glad to discuss further.

  • 👍1
toliuweijing commented

@izivkov appreciate your valuable feedback. Agreed that choosing the judicious defaults makes this app stand out in the market.

The current use of notification is just like what a calendar reminder does

  1. it allows users either stand by or walk away from their phones.
  2. it allows users to understand what detection happened in the past.

I'm not sure if the other proposal can achieve this typical use case. Or should the feature assume users will somehow hear the detection sound?

Love to hear your thought. And will discuss other topic after having a better understanding on this use case.

izivkov commented

@toliuweijing

So let's look at the use case where the user wants to monitor anyone coming to their front door. Let's assume that the camera is not pointing at a busy street, and there is motion detection only if somebody is at the door.

  1. We place one phone at the door (Remote phone). The second phone is in the house and is used to monitor the door (Local phone).

  2. If we want to always see what is outside, we run the app in normal mode. The two phones are always sending video.

  3. If we want to see video only if somebody is at the door, we use motion detection mode. In this mode, the local phone is blanked. The remote phone can also be blanked because nobody is looking at it, but this can be addressed later.

  4. The user does not have to look at the local phone all the time. If there is somebody at the door, motion is detected, the local phone beeps to attract the user's attention, and the screen is turned on so the user can see who is at the door. The screen will blank out if there is no motion for 60 seconds.

  5. Let's say you go to work, and six people visit during the day. If we use notifications, you will see six notifications. However, they are not very useful because you don't know what happened—maybe the neighbor's cat walked in front of the camera (we are not recording the video). So, what is the point of having these notifications? The only thing you are interested in is the event as it happens.

This is why I think notifications are not the best option for this app.

toliuweijing commented

The only thing you are interested in is the event as it happens.

IMO, users should be notified for these events(both false alarm and true crime) as soon as possible, even when they walk away from the phone.

As soon as users come back, notification center draws attentions, even before phone unlock. User clicks the notification, which navigates to the app for video footage at the event moment. This is not implemented yet but planned in future iteration. Notification needs fine-tuned, but is a necessary stepping stone to the plan.

Is this plan / design something you prefer the app to be used? If so, can you help me understand how a custom beep works better for user being away from the phone, like in a noisy street where sound isn't enough to draw attention?

toliuweijing commented

Of course, if the app shouldn't be used when user is away from the phone, then notification is redundant.

st3b1t commented

@st3b1t @toliuweijing Again, a great idea. However, this would be useful with another type of connection, working outside of the range of the router, as I suggested earlier.

I personally did some tests and with an access point and two 2.5GHz signal repeaters I reach about 1km of distance between camera and video using two smartphones connected only via wifi, without video or motion degradation

  • 👍1
st3b1t commented

@toliuweijing @st3b1t

Here are my thoughts on how to use the motion detection feature.

  1. There should be a "motion detection mode" and a "normal mode." This could be controlled by a simple toggle button; nothing else changes from the original app. This keeps the app simple, which is one of the reasons this app is popular.

I agree entirely, About 1. Point, I'll add a detail.
It is important that it is possible to choose which device emits the sound, with three options: transmitter only, receiver only or both.
In this way the usage scenarios are broadened, surveillance devices usually have a silent mode which can be useful if used as a hidden camera.
But motion detection is a useful thing even with a single device that does not transmit video, this has various use cases as a motion or presence detector.
In the future these options may be linked to a possible video recording functionality.

  • 👍1
izivkov commented

@toliuweijing
OK, it is up to you how you want to implement it. I just want to bring up my concerns:

  1. Motion events are not like email. It is OK to miss some.

  2. In a "noisy" environment, there could be hundreds of motion detection events. For example, if the camera is pointing at a street with some traffic. In these cases, I would have to dismiss hundreds of notifications. I think I would uninstall the app if this happens.

So I propose to make notifications optional. There could be a "Notify me" checkbox with the default being OFF. What do you think?

Regards,
Ivo

  • 👍1
toliuweijing commented

This has been a great discussion. I see a few action items from this discussion, e.g.

  1. normal mode focusing for video call use case.
  2. motion detection mode with screen brightness control for one-way use case.
  3. auto-record short video footage for detection.
  4. work in WIFI-direct / hotspot connection.

Let me know if I'm missing something. We can create them in the Discussion section as backlogs and iterate with priority.

Motion events are not like email. It is OK to miss some.

Could you have ideas where to draw the lines? For baby monitor, should parent be always aware?

In a "noisy" environment, there could be hundreds of motion detection events. For example, if the camera is pointing at a street with some traffic. In these cases, I would have to dismiss hundreds of notifications. I think I would uninstall the app if this happens.

  1. I actually mean ONLY the house owner is in a load environment / room. The motion detection is still pointed at the front-door. No extra notification will bore the owner.
  2. Motion detection doesn't work for scenes where objects are constantly moving. This is obvious but regardlessly we should have a basic instruction or user education for this feature.
  • 👍1
izivkov commented

This has been a great discussion. I see a few action items from this discussion, e.g.

  1. normal mode focusing for video call use case.
  2. motion detection mode with screen brightness control for one-way use case.
  3. auto-record short video footage for detection.
  4. work in WIFI-direct / hotspot connection.

Let me know if I'm missing something. We can create them in the Discussion section as backlogs and iterate with priority.

Yes, I think these are great milestones. Long term, I like to add Internet connection as well, so people can talk over the internet or monotor their house remotely. However, this involves some infrastructure, like we Stun Server (WebRTC server) so phones can find each other, and also some Firebase communications for initial signalling. This is more than I like to commit at the moment, unless the app becomes popular enogh to generate income.

Motion events are not like email. It is OK to miss some.

Could you have ideas where to draw the lines? For baby monitor, should parent be always aware?

At the moment, we do not record video. So missed motion detection events are not useful. If we start to record video, then we would like to see these events, and maybe notifications would be useful. Another way is to keep in-app list of past motion events, so the user can review them. For now I suggest we keep it simple, and just a beep would be enpugh, but again, I leave it up to you.

In a "noisy" environment, there could be hundreds of motion detection events. For example, if the camera is pointing at a street with some traffic. In these cases, I would have to dismiss hundreds of notifications. I think I would uninstall the app if this happens.

  1. I actually mean ONLY the house owner is in a load environment / room. The motion detection is still pointed at the front-door. No extra notification will bore the owner.

I meant when the cammera is pointing otside the front door to see who is at teh door, not inside. If the camera is inside, yes, there would be much less noise.

  1. Motion detection doesn't work for scenes where objects are constantly moving. This is obvious but regardlessly we should have a basic instruction or user education for this feature.

So, in general, I would not like to impede you progress. Just go ahead and add the features as you think they should be, and we can discuss.

There is actually an urgent issue "jcenter shutdown #28", which I an working on, but maybe you can take a look as well. It prevent us from publishong the app on F-DROID.

Thanks for your work
Ivo

toliuweijing commented

Sounds good. I'm traveling and be back at end of month. Will check on the #28 issue once available.

  • 👍1