My current task is to classify the association between CVEs and CWEs. However, I’ve noticed that using BertModel.from_pretrained(‘bert-base-uncased’) in the fine-tuning stage results in lower accuracy compared to when I pretrain with more CVE-related descriptions first, and then fine-tune using the pretrained model.pt. I don’t understand why this is happening as I have ruled out compatibility issues with the model. It’s worth mentioning that in the pretraining phase, I only use the pretrained model weights for fine-tuning, and the tokenizer is consistently BertTokenizer.from_pretrained(‘bert-base-uncased’). I did not retrain or expand the tokenizer during pretraining because it is very time-consuming.
For pretraining, I am using data obtained from the NVD, encompassing around 210,000 CVE entries from the years 2000 to 2023. To ensure data cleanliness, I have also used spaCy for data cleaning.

nlp = spacy.load("en_core_web_sm")

def clean_text(text):
doc = nlp(text)
tokens = [token.text for token in doc if not token.is_stop and not token.is_punct]
return " ".join(tokens)

def read_data(file_path, column_name):
try:
df = pd.read_csv(file_path, encoding='utf-8')
except UnicodeDecodeError:
try:
df = pd.read_csv(file_path, encoding='ISO-8859-1')
except UnicodeDecodeError:
df = pd.read_csv(file_path, encoding='latin1')

    print(f"file_name：{file_path}")
    print("colums_name：", df.columns)
    print("len：", len(df))
    
    texts = df[column_name].dropna().tolist()  
    cleaned_texts = [clean_text(text) for text in texts]

Here are the hyperparameters I am using:

batch_size = 16 
num_epochs = 10 
learning_rate = 1e-4 
beta1 = 0.9 
beta2 = 0.99 
weight_decay = 0.01 
total_steps = num_epochs * len(train_loader) 
warmup_steps = total_steps // 10 
early_stopping_patience = 2

Additionally, the settings for masked language modeling (MLM) are:

mask_prob=0.15 
replace_mask_prob=0.8 
random_replace_prob=0.10 
keep_original_prob=0.10

I hope someone can answer my question. If more detailed pretrain/fine-tune code is needed, I can provide it. Thank you.

I have experimented with various combinations of hyperparameters and found that the current settings are optimal. However, the performance still falls slightly short compared to the scenario where no pre-training is used. The image shows the training loss and validation loss during my pre-training phase. The validation loss reaches its lowest point at epoch 1, and then it starts to increase from the next epoch onwards.