NetsecExplained/Attacking-and-Defending-Generative-AI

Reference notes for Attacking and Defending Generative AI presentation

Attacking-and-Defending-Generative-AI

Reference notes for Attacking and Defending Generative AI presentation

People to Follow on Twitter

• Rez0__
• Kai Greshake
• Johann Rehberger
• LLM Security
• Leon Derczynski
• Simon Willison

LLM Security Frameworks

• OWASP Top 10 for LLM Applications
• AVID - AI Vulnerability Database
• MITRE - ATLAS
• NIST - Adversarial Machine Learning
  • NIST AI - TL;DR by Rez0

LLM Security Tools

• Dropbox - LLM security
• Garak vulnerability scanner
• NeMo Guardrails

LLM Attack Technique Papers

• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild
• Understanding Prompt Injection Attacks and Mitigations
• Understanding Invisible Prompt Injections
• Invisible Prompt Injection Code
• GPT4 Technical Paper
• Netsec Explained - ChatGPT Your Red Team Ally
• Rez0 - Prompt Injection Primer for Engineers (PIPE)
• LLM Hackers Handbook
• Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Kai Greshake - (In)security and Misalignment: A modern security crisis

LLM Attacks in the Wild

• Chevy Dealership
• Teachers Finding Students Using ChatGPT with Prompt Injections
• ChatGPT Unspeakable Tokens
• Vulnerable AI Paragraph Rewriter - Harmless
• Insecure Plugin - Chat with Code
• ChatGPT Plugin Exploit Chain - Accessing Private Data

Mitigation techniques

• System Prompt to Mitigate Revealing RAG Sources
• AI Village - Threat Modeling LLMs
• NeMo Guardrails - Practical Guide
• OpenAI Moderation API