fabriziosalmi/blacklists

ML for false positive prediction

fabriziosalmi opened this issue · 2 comments

fabriziosalmi commented

1. Hourly Cron Job

  • Use a cron job to run a Python script every hour.
0 * * * * /usr/bin/python3 /path_to_your_script/your_script.py

2. Comparison with Whitelist

  • Fetch the updated blacklist and compare it with the whitelist.
blacklist = fetch_updated_blacklist()  # Define a function to fetch the updated blacklist
whitelist = load_whitelist()  # Load the whitelist from a file or a database

false_positives = set(blacklist).intersection(whitelist)  # Find overlaps between blacklist and whitelist

3. Machine Learning Model

  • Use a pre-trained model to predict whether the identified overlaps are indeed false positives.
model = load_pretrained_model()  # Load a pre-trained model

for url in false_positives:
    is_false_positive = model.predict(url)  # Predict whether the URL is a false positive
    if is_false_positive:
        refine_blacklist(url)  # Remove the false positive from the blacklist

4. Refinement

  • Refine the blacklist by removing the confirmed false positives.
def refine_blacklist(url):
    blacklist.remove(url)  # Remove the URL from the blacklist
    save_updated_blacklist(blacklist)  # Save the updated blacklist to a file or a database

5. Alerting/Logging

  • Log the results and send alerts if necessary.
import logging

logging.basicConfig(filename='blacklist_refinement.log', level=logging.INFO)

if false_positives:
    logging.info(f"False positives identified and refined: {false_positives}")
    send_alert(false_positives)  # Define a function to send alerts, e.g., email

Additional Considerations:

  • Model Training: Regularly retrain your model with new data to ensure it stays accurate.
  • Performance Monitoring: Monitor the performance of your model and the accuracy of its predictions.
  • User Feedback: Incorporate feedback from users to identify additional false positives/negatives and improve the model.

This is a high-level overview and pseudo-code.

fabriziosalmi commented

Doing a model from scratch for this purpose.

Check wiki documentation 🍻