UPDATE:

SecureBERT+ is out.

SecureBERT

SecureBERT is a domain-specific language model to represent cybersecurity textual data which is trained on a large amount of in-domain text crawled from online resources.

See the presentation on YouTube

SecureBERT can be used as the base model for any downstream task including semantic analysis, text classification, NER, Seq-to-Seq, QA, etc.

SecureBERT has demonstrated significantly higher performance in predicting masked words within the text when compared to existing models like RoBERTa (base and large), SciBERT, and SecBERT.
SecureBERT has also demonstrated promising performance in preserving general English language understanding (representation).

How to use SecureBERT

SecureBERT has been uploaded to Huggingface framework. You may use the code below

from transformers import RobertaTokenizer, RobertaModel
import torch

tokenizer = RobertaTokenizer.from_pretrained("ehsanaghaei/SecureBERT")
model = RobertaModel.from_pretrained("ehsanaghaei/SecureBERT")

inputs = tokenizer("This is SecureBERT!", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state

Or just clone the repo:

git lfs install
git clone https://huggingface.co/ehsanaghaei/SecureBERT
# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:
GIT_LFS_SKIP_SMUDGE=1

Fill Mask

SecureBERT has been trained on MLM. Use the code below to predict the masked word within the given sentences:

#!pip install transformers
#!pip install torch
#!pip install tokenizers

import torch
import transformers
from transformers import RobertaTokenizer, RobertaTokenizerFast

tokenizer = RobertaTokenizerFast.from_pretrained("ehsanaghaei/SecureBERT")
model = transformers.RobertaForMaskedLM.from_pretrained("ehsanaghaei/SecureBERT")

def predict_mask(sent, tokenizer, model, topk =10, print_results = True):
    token_ids = tokenizer.encode(sent, return_tensors='pt')
    masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()
    masked_pos = [mask.item() for mask in masked_position]
    words = []
    with torch.no_grad():
        output = model(token_ids)

    last_hidden_state = output[0].squeeze()

    list_of_list = []
    for index, mask_index in enumerate(masked_pos):
        mask_hidden_state = last_hidden_state[mask_index]
        idx = torch.topk(mask_hidden_state, k=topk, dim=0)[1]
        words = [tokenizer.decode(i.item()).strip() for i in idx]
        words = [w.replace(' ','') for w in words]
        list_of_list.append(words)
        if print_results:
            print("Mask ", "Predictions : ", words)

    best_guess = ""
    for j in list_of_list:
        best_guess = best_guess + "," + j[0]

    return words


while True:
    sent = input("Text here: \t")
    print("SecureBERT: ")
    predict_mask(sent, tokenizer, model)
     
    print("===========================\n")

Sample results:

Text here: 	Gathering this information may reveal opportunities for other forms of <mask>, establishing operational resources, or initial access.
SecureBERT: 
Mask  Predictions :  ['reconnaissance', 'exploitation', 'attack', 'compromise', 'phishing', 'enumeration', 'attacks', 'penetration', 'espionage', 'intrusion']
===========================

Text here: 	Information about identities may include a variety of details, including personal data as well as <mask> details such as credentials.
SecureBERT: 
Mask  Predictions :  ['authentication', 'sensitive', 'system', 'credential', 'other', 'additional', 'security', 'technical', 'account', 'secret']
===========================

Text here: 	Adversaries may also compromise sites then include <mask> content designed to collect website authentication cookies from visitors.
SecureBERT: 
Mask  Predictions :  ['malicious', 'JavaScript', 'phishing', 'iframe', 'dynamic', 'additional', 'downloadable', 'hostile', 'embedded', 'website']
===========================

Text here: 	Adversaries may also compromise sites then include malicious content designed to collect website authentication <mask> from visitors.
SecureBERT: 
Mask  Predictions :  ['credentials', 'information', 'data', 'tokens', 'details', 'cookies', 'parameters', 'codes', 'secrets', 'keys']
===========================

Text here: 	The given website may closely resemble a <mask> site in appearance and have a URL containing elements from the real site.
SecureBERT: 
Mask  Predictions :  ['real', 'legitimate', 'trusted', 'fake', 'genuine', 'live', 'phishing', 'similar', 'known', 'different']
===========================

Text here: 	Although sensitive details may be redacted, this information may contain trends regarding <mask> such as target industries, attribution claims, and successful countermeasures.
SecureBERT: 
Mask  Predictions :  ['events', 'indicators', 'activities', 'topics', 'activity', 'trends', 'factors', 'incidents', 'information', 'campaigns']
===========================

Text here: 	Adversaries may search digital <mask> data to gather actionable information.
SecureBERT: 
Mask  Predictions :  ['forensic', 'media', 'forensics', 'asset', 'image', 'security', 'form', 'identity', 'store', 'document']
===========================

Text here: 	Once credentials are obtained, they can be used to perform <mask> movement and access restricted information.
SecureBERT: 
Mask  Predictions :  ['lateral', 'laterally', 'Lateral', 'data', 'credential', 'horizontal', 'illegal', 'money', 'physical', 'unauthorized']
===========================

Text here: 	One example of this is MS14-068, which targets <mask> and can be used to forge its tickets.
SecureBERT: 
Mask  Predictions :  ['Kerberos', 'tickets', 'RSA', 'KDC', 'LDAP']

SecureBERT outperforms the existing models in MLM testing conducted on a manually crafted dataset from the human readable descriptions of MITRE ATT&CK techniques and tactics.

This repo will be updated on a regular basis.

SecureBERT is DIFFERENT than SecBERT.

Sample Text: 	Adversaries may also compromise sites then include <mask> content designed to collect website authentication cookies from visitors.
>> SecureBERT: 
malicious |	JavaScript |	phishing |	iframe |	dynamic |	additional |	downloadable |	hostile |	embedded |	website


>> SecBERT: 
web |	exploit |	simple |	a |	social |	clipboard |	shared |	native |	malicious |	business

-------------------------------------------

Sample Text: 	One example of this is MS14-068, which targets <mask> and can be used to forge Kerberos tickets using domain user permissions.
>> SecureBERT: 
Kerberos |	authentication |	users |	Windows |	administrators |	LDAP |	PAM |	Samba |	NTLM |	AD


>> SecBERT: 
businesses |	passwords |	differ |	system |	code |	software |	sensitive |	known |	MySQL |	misconfigurations

-------------------------------------------

Sample Text: 	Paris is the <mask> of France.
>> SecureBERT: 
capital |	Republic |	Government |	province |	name |	city |	government |	language |	Capital |	Bank


>> SecBERT: 
case |	course |	leader |	peak |	Ell |	Embassy |	trademark |	Republic |	Dictionary |	ministry

-------------------------------------------

Sample Text: 	Virus causes <mask>.
>> SecureBERT: 
DoS |	crash |	reboot |	reboots |	panic |	crashes |	corruption |	DOS |	vulnerability |	XSS


>> SecBERT: 
disabled |	invisible |	vulnerabilities |	advanced |	before |	left |	too |	vector |	obviously |	vulnerable

-------------------------------------------

Sample Text: 	Sending huge amount of packets through network leads to <mask>.
>> SecureBERT: 
DoS |	congestion |	crashes |	crash |	problems |	DOS |	vulnerability |	failure |	vulnerabilities |	errors


>> SecBERT: 
monitoring |	pipes |	situations |	default |	minutes |	login |	workstations |	services |	printers |	etc

-------------------------------------------

Sample Text: 	A <mask> injection occurs when an attacker inserts malicious code into a server
>> SecureBERT: 
code |	SQL |	command |	malicious |	script |	web |	vulnerability |	server |	ql |	SQL


>> SecBERT: 
frame |	keystroke |	packet |	massive |	mass |	URL |	vector |	COM |	window |	Distributed

References

@inproceedings{aghaei2023securebert, title={SecureBERT: A Domain-Specific Language Model for Cybersecurity}, author={Aghaei, Ehsan and Niu, Xi and Shadid, Waseem and Al-Shaer, Ehab}, booktitle={Security and Privacy in Communication Networks: 18th EAI International Conference, SecureComm 2022, Virtual Event, October 2022, Proceedings}, pages={39--56}, year={2023}, organization={Springer} } [https://link.springer.com/chapter/10.1007/978-3-031-25538-0_3]