Threshold setting for Human / Vehicle classification to reduce false positives?
Closed this issue · 13 comments
First, thank you for making and maintaining this plugin – it is immensely helpful in my home automation journey!
I've got Home Assistant configured to send me a notification when a Human is detected in one of my cameras. It works when a person is there, but I'm getting a lot of false positives: it says a person is there, but there isn't. Also, SecuritySpy doesn't capture a video clip for these false positive events so SecuritySpy seems to know it's not really a person, but Home Assistant thinks it is.
I wasn't sure where the problem was coming from so I asked about this on the SecuritySpy forum, and Ben explained that the event stream sends a CLASSIFY event, with the percentage probability of both a human and vehicle is included, e.g. "CLASSIFY HUMAN 80 VEHICLE 0".
Based on that, my best guess is that the SecuritySpy for Home Assistant plugin is reporting a human when the HUMAN probability is any value over 0 – or perhaps some other value that I don't know and don't know how to adjust?
@briis could you provide any insight into how the SecuritySpy for Home Assistant plugin is handling the data from the SecuritySpy event stream? Is it registering a Human event_object for any value of HUMAN in the CLASSIFY event? If so, is there some way to set a threshold to reduce the number of false positives? For example, in SecuritySpy I have it set to only record if the probability of Human is 50 or higher (the recommended setting).
If it's possible to include the percentage probability number from SecuritySpy as an attribute in the motion binary_sensor we could potentially trigger different automations in Home Assistant based on the probability of human? e.g. if >30 turn on a light… if >60 and in the middle of the night, start flashing lights and making noise, etc. But I think just being able to set a single threshold in the settings would hugely help to decrease my false positives here.
Hi @benishs
Thank you, glad it is useful. There is a percentage submitted together with with the event, so if I can get this out, it should be possible. I will look in to this.
I just released 1.1.14-beta.1 which has this implemented. Don't build a lot of automations yet, as the event_score attribute most like will change to event_score_human and event_score_vehicle. But for now I need tested that the score is filled with a proper value.
If you installed this through HACS, you will need beta releases enabled for SecuritySpy Integration.
From the release notes:
NEW: Added anevent_scoreattribute to the binary motion sensor and the detected object sensor, that shows the percentage probability of the object being detected. Currently this will only show one score for either Human or Vehicle, depending on which is highest.
I installed 1.1.14-beta.1 and it seems to be reporting event_score as expected for humans and vehicles (see attached).
Let me know if there is anything else I can do to help test this!
It'll definitely be very helpful to have separate event_score_human and event_score_vehicle. Right now my test automation sends me the event_object and event_score in a notification. When I trigger my test automation by walking in front of the camera event_object is always "Vehicle" because my car is in the background. So the motion event is triggered by a human moving, but the Vehicle score is higher (I assume) and therefore event_object is set to Vehicle. I know this is the expected behavior at the moment.
Thank you so much for adding this – it's going to make my system much more usable to get rid of these false positives.
I released 1.1.4-beta.2, which now has the below changes. I decided to still display the event_object as either human or vehicle based on the highest score. If you have any other thoughts on that, let me know.
Please report back how this works for you.
NEW: Replaced theevent_scoreattribute with two new attributes:event_score_humanandevent_score_vehicle. Theevent_objectwill still hold the object that has the highest score of the two, but if you only want to look for a person moving, you can check on theevent_score_humanvalue.
@divemasterjm: I got a notification from Github, where you wrote: "HI, to me it doenst work, now at all on motion binary sensor after last beta" But for some reason this comment is not in here. Did you delete again? If you still have the issue, please let me know with some more details on what does not work.
@divemasterjm: I got a notification from Github, where you wrote: "HI, to me it doenst work, now at all on motion binary sensor after last beta" But for some reason this comment is not in here. Did you delete again? If you still have the issue, please let me know with some more details on what does not work.
i delete it cause i was wrong, works like a charm, thanks!!!!
I've updated my automations to use the new event_score_human attribute, and after some quick tests everything seems to be working as expected! It still triggers when someone walks into the frame, at least – and I assume it'll reduce or eliminate the false positives I was getting. I'll monitor and let you know if I see any issues. Thank you!
Side note for others who might be making this update: you need to change your Trigger type on your automation from State to Numeric state. This is probably obvious to many, but I'm new to Home Assistant and it took me a moment to sort out. Screen captures of my trigger before & after are attached.
This looks good, and yes, the event score is a numeric value - you learned something new today 😁 always nice.
I will wait for a day or two and then make this official.
I've just tried the latest beta, it seems to be working thanks.
q: currently event_score_human|vehicle retains the last value once motion stops, it should be reset to 0 after motion stops, no? would it not result in race conditions next time motion is detected if last value is retained? (sorry not sure how HA works with states etc changing)
i want to create a boolean template called hallway_human_motion_detected based on motion state and score value, so it's easier to use in blueprints etc, not sure it would work correctly if the last value is retained.
looking at security spy docs, it sends two events one for motion and one for classify so, so motion is detected first and then classify event happens (score updates after motion?), unless you are treating classify as the motion event?
Hi @luqman
Sorry for the delay in answering, I have been really busy the last few weeks and will continue so the next months.
retains the last value once motion stops
True, that is done on purpose to make sure automations have enough time to pick up the value. As you mention further down, there are two events: Motion and Classify, and the value is cleared when the next Motion event occurs, which will always be before the Classify event, so it should not be a problem - but please let me know if it is, and it can be changed.
unless you are treating classify as the motion event?
I am not. Both events are treated separately.
Just released V1.1.4, which now clears the last event object and score settings when motion ends. Please test this out in your scenario and let me know how that goes.
I assume this is solved now. Else please open a new issue.