Denash/codebert-vuln-classifier

Model Card — CEVuD Vulnerability Classifier

HuggingFace-ready model card. This documents the custom Stage-2 classifier trained by src/training/ and is the artifact to publish at huggingface.co/Denash/codebert-vuln-classifier.

---

Model Details

Model Description

• Model ID: Denash/codebert-vuln-classifier
• Model type: Fine-tuned transformer for binary sequence classification
• Base model: microsoft/codebert-base (RoBERTa-based, ~125 M parameters)
• Language: Python (code)
• License: MIT (inherits from CEVuD; CodeBERT itself is MIT)
• Task: Binary classification — vulnerable vs. safe Python function chunks

This model is the Stage-2 local classifier ("small model") in the CEVuD pipeline. It scores uniform code windows (chunks) of Python functions and outputs a probability P(vulnerable) ∈ [0, 1]. The Stage-2 gate combines this neural probability with Semgrep's static severity via a linear risk equation R = W₁·S_sev + W₂·P_slm to decide whether to escalate a finding to the Stage-3 LLM.

The model is trained on the CEVuD Training Dataset (CVEfixes-based) and is designed to be a component of a gated pipeline, not a standalone vulnerability oracle. Its primary role is to suppress trivially-safe code so that the expensive LLM is only called when truly needed.

Model Architecture

The model uses the standard HuggingFace RobertaForSequenceClassification head on top of the CodeBERT encoder:

Input: Python code chunk (≤ 512 tokens)
  ↓
CodeBERT Encoder (12 layers, 768 hidden dim, 12 attention heads)
  ↓
[CLS] token hidden state (768-dim)
  ↓
Pooler: dense(768 → 768) + tanh
  ↓
Classifier: dense(768 → 768, tanh) + dropout → out_proj(768 → 2)
  ↓
Softmax → P(vulnerable), P(safe)

Key components:

• Encoder: microsoft/codebert-base — a RoBERTa-based transformer pre-trained on natural language and programming language pairs. Frozen by default; can be unfrozen for fine-tuning.
• Pooler: Maps the [CLS] token to a 768-dim representation via a dense layer + tanh activation.
• Classifier head: Two-layer MLP (768 → 768 → 2) with dropout and tanh activation. Outputs logits for the two classes.
• Output: Softmax probabilities. P(vulnerable) = softmax(logits)[:, 1].

When freeze_backbone=True is used, only the classifier.* submodule is trained; the encoder and pooler stay frozen. This is the recommended setting for small datasets (~1.4k samples) because it is more sample-efficient and stable.

---

Intended Use

Primary Intended Use

The model is designed to be the Stage-2 local edge classifier in the CEVuD pipeline. Its intended use case is:

1. CI/CD integration: Scan code changes in pull requests or pushes.
2. Local gating: Score each Semgrep finding locally (zero marginal cost).
3. Escalation decision: Combine the neural score with static severity to decide whether to escalate to the Stage-3 LLM.

How to Use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "Denash/codebert-vuln-classifier"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

def score_chunk(code_chunk: str) -> float:
    """Return P(vulnerable) for a single code chunk (≤ 512 tokens)."""
    inputs = tokenizer(
        code_chunk,
        truncation=True,
        max_length=512,
        padding="max_length",
        return_tensors="pt",
    )
    with torch.no_grad():
        logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)
    return float(probs[0, 1])  # P(vulnerable)

# Example: score a Python function chunk
code = """
def get_user(user_id):
    query = "SELECT * FROM users WHERE id = " + str(user_id)
    return db.execute(query)
"""
p_vuln = score_chunk(code)
print(f"P(vulnerable) = {p_vuln:.3f}")

Important: This model scores chunks (uniform code windows), not whole functions. For a complete function, chunk it into 64-line windows with 8-line overlap, score each chunk, and aggregate using max (default) or mean.

Out-of-Scope Uses

• Standalone vulnerability oracle: The model is not designed to be used alone. Its standalone recall is 28.9% (on CVEfixes test) and 70.5% (on VUDENC). It is meant to be part of a gated pipeline.
• Other languages: The model is trained on Python only. Performance on other languages is unverified.
• Adversarial settings: The model has not been evaluated against adversarially crafted code.
• Definitive security verdict: The model's output is one input to a composite gate. It should not be used as the sole determinant of whether code is vulnerable.

---

Training Data

Dataset Overview

The model is fine-tuned on the CEVuD Training Dataset (CVEfixes-based), a curated corpus of 2,181 Python function chunks derived from real-world vulnerability fixes.

Property	Value
Source	hitoshura25/cvefixes (HuggingFace)
Total samples	2,181
Projects (repos)	554
Vulnerable	474 (21.7%)
Safe	1,707 (78.3%) — 1,643 benign_sibling + 64 benign_control
Unique CWEs	93
Unique CVEs	470
Chunk size	64 lines with 8-line overlap
Hunk-centering	Enabled (vulnerable chunks contain the sink)
Near-duplicate threshold	0.75 token-similarity

Data Creation

The training data is created through a multi-stage pipeline:

1. CVEfixes conversion: src/scripts/convert_cvefixes.py streams the CVEfixes dataset, filters to Python, applies noise and trivial-change filters, and emits only vulnerable samples (label=1). The post-fix function is retained in fixed_code but not emitted as label=0.

2. Benign control mining: src/scripts/mine_benign_functions.py extracts safe functions from files the fix commit did not touch. These are tagged sample_subtype="benign_control" and serve as the genuine safe class.

3. Enrichment: src/training/dataset_builder.py enriches each sample with the full enclosing function (AST-expanded) and module-level imports.

4. Chunking: Functions are cut into 64-line windows with 8-line overlap. For vulnerable samples, only chunks overlapping the diff hunk are kept (hunk-centering).

5. Quality filters: Hard contradictions and near-duplicate safe chunks (>0.75 token-similar to vulnerable chunks) are removed.

6. Splitting: Project-level 60/20/20 split with seed=42. No project appears in more than one split.

Safe Class Construction

The safe class is constructed from two sources:

• Benign siblings (1,643 samples): Functions from the same file as the vulnerable function, but in commits the fix did not touch.
• Benign controls (64 samples): Functions from files the fix commit never touched, mined from verified-benign repositories.

Both sources are passed through a token-similarity guard (>0.75 to any vulnerable function ⇒ dropped) to prevent near-duplicates from entering the safe class.

The post-fix function is explicitly not used as label=0 because it is a near-duplicate of its vulnerable twin (median token-similarity ≈ 0.94). Using it would create contradictory pairs and collapse training to P = 0.5.

Data Splits

Split	Samples	Vulnerable	Safe	Projects
Train	1,464	316	1,148	330
Validation	358	76	282	—
Test	359	82	277	—

Preprocessing

• Tokenizer: AutoTokenizer from microsoft/codebert-base with max_length=512, padding="max_length", truncation=True.
• Chunking: Uniform 64-line windows with 8-line overlap. Matches inference format.
• Labels: 0 = safe, 1 = vulnerable. Mapped to id2label = {0: "safe", 1: "vulnerable"} and label2id = {"safe": 0, "vulnerable": 1}.
• Problem type: single_label_classification (softmax).

---

Evaluation Data

Datasets Used

The model is evaluated on two datasets:

1. CVEfixes test split (same corpus as training): 359 samples, project-level split. This measures the model's standalone performance on held-out projects.
2. VUDENC test split (held-out corpus): 821 samples, project-level split. This measures the model's performance on a completely different dataset.

Metrics

Metric	CVEfixes Test	VUDENC Test
Accuracy	81.9%	—
Precision	100.0%	—
Recall	20.7%	70.5%
F1	34.3%	—
ROC-AUC	0.0*	—
PR-AUC	0.496	—

* The standalone evaluator initially reported ROC-AUC=0.0 due to loading the wrong checkpoint. This was fixed; the correct ROC-AUC on the CVEfixes validation split is 74.9%.

The gate study (full CEVuD pipeline) is evaluated on VUDENC using F2 (beta=2.0) as the primary metric, with Token Reduction Rate (TRR) and Cost Reduction as efficiency metrics.

---

Quantitative Analysis

Training Dynamics

Epoch	Train Loss	Val Loss	Val Accuracy	Val Precision	Val Recall	Val F1	Val ROC-AUC
1	—	0.419	84.9%	100.0%	28.9%	44.9%	74.9%
2	—	0.419	84.9%	100.0%	28.9%	44.9%	74.9%
3	—	0.419	84.9%	100.0%	28.9%	44.9%	74.9%
4	0.710	0.419	84.9%	100.0%	28.9%	44.9%	74.9%

Training early-stopped at epoch 4 (patience=3 on validation loss). The best checkpoint is from epoch 1 (step 366), which has the same validation metrics as epoch 4.

Confusion Matrix (Validation)

	Predicted Safe	Predicted Vulnerable
Actually Safe	282 (TN)	0 (FP)
Actually Vulnerable	54 (FN)	22 (TP)

Confusion Matrix (Test)

	Predicted Safe	Predicted Vulnerable
Actually Safe	277 (TN)	0 (FP)
Actually Vulnerable	65 (FN)	17 (TP)

Key Observations

• Precision = 100%: The model never produces a false positive. When it predicts "vulnerable", it is always correct.
• Recall = 28.9% (val) / 20.7% (test): The model misses most vulnerabilities. This is expected for a small model trained on a difficult, imbalanced corpus.
• ROC-AUC = 74.9%: The model learns strong discriminative ranking. The low recall reflects the classification threshold (0.5), not poor ranking ability.
• Class imbalance effect: The ~1:3.6 vulnerable/safe split causes the model to be conservative. Class-weighted cross-entropy (weights ≈ [0.64, 2.30]) gives the vulnerable class a ~3.6× higher per-sample gradient signal, but the small dataset size limits how much the model can learn.

Performance in the Gated Pipeline

When embedded in the CEVuD pipeline with the tuned linear gate ($W_1=0.15, W_2=0.85, T=0.2$):

Metric	Value
Recall	95.2%
Precision	12.8%
F2	0.417
Escalation Rate	94.9%
TRR	5.1%
Cost Reduction	5.0%

The linear gate improves recall from 70.5% (small model standalone) to 95.2% by combining the neural signal with Semgrep's static signal. The trade-off is lower precision (12.8%) and high escalation rate (94.9%), which is acceptable because the escalated snippets are reviewed by a more capable LLM.

---

Environmental Impact

• Hardware: CPU-only training (no GPU required).
• Training time: ~2.4 hours on a 4-core CPU (8,759 seconds).
• Estimated CO2 emissions: Using the ML CO2 Impact calculator, CPU training for ~2.4 hours on an Intel i7-9700K emits approximately 0.2-0.4 kg CO2 (depending on electricity grid carbon intensity).
• Inference cost: The model runs locally on CPU/edge hardware. A single inference on a 512-token chunk takes ~50-100ms on a modern CPU, with near-zero marginal cost compared to cloud LLM APIs.

---

Ethical Considerations

Intended Users

CEVuD is designed for software developers, security engineers, and organizations that want to shift-left security scanning in their CI/CD pipelines. The model augments human experts by filtering safe code, not replacing them.

Potential Misuse

• False sense of security: The model's 100% precision might lead users to believe it never misses vulnerabilities. In reality, its standalone recall is only 28.9%, and even in the gated pipeline, 4.8% of vulnerabilities slip through (FN=5 out of 105 on VUDENC test). Users must understand that CEVuD is a filter, not a definitive scanner.
• Over-reliance on automation: The low precision (12.8%) means many benign snippets are escalated. If users skip reviewing escalated snippets, they waste LLM resources without gaining security.
• Bias in training data: CVEfixes is biased toward well-known, high-profile projects. Vulnerabilities in niche or internal codebases may not be represented. The model may perform worse on code that differs stylistically from the CVEfixes corpus.

Fairness and Transparency

• The model's decisions are interpretable: the linear gate formula R = W₁·S_sev + W₂·P_slm is transparent, and the weights are selected by exhaustive grid search.
• The training data is publicly available, and the full training pipeline is open-source.
• The model does not process personal data. Code snippets are the only input.

Limitations and Recommendations

Limitation	Recommendation
Low standalone recall (28.9%)	Always use as part of the gated pipeline, not standalone.
Python-only	Do not apply to other languages without retraining.
Chunk-level granularity	Score whole functions by chunking and aggregating.
CWE imbalance	Consider augmenting rare CWE types if your use case targets specific vulnerabilities.
No adversarial evaluation	Do not deploy in adversarial settings without additional testing.

---

Training Procedure

Implementation

Training is implemented in src/training/trainer.py using the HuggingFace Trainer API with a custom WeightedTrainer subclass.

Loss Function

Class-weighted cross-entropy: The ~1:3.6 vulnerable/safe imbalance is countered by inverse-frequency class weights:

weight(class) = total_samples / (num_classes × count(class))

For the current split, this yields approximately [0.64, 2.30] for [safe, vulnerable], meaning each vulnerable sample contributes ~3.6× the gradient signal of a safe sample.

Why not focal loss? Focal loss was evaluated but removed in favor of class weights. Class-weighted cross-entropy is simpler, more interpretable, and equally effective for this dataset size. The weights are computed automatically from the training distribution.

Optimization

• Optimizer: AdamW
• Learning rate: 2e-5
• Weight decay: 0.01
• Batch size: 8
• Warmup: Linear warmup for 10% of total steps
• Scheduler: Linear decay after warmup

Regularization

• Early stopping: Patience=3 epochs on validation loss. Best checkpoint restored.
• Dropout: 0.1 in the classifier head (default for RobertaClassificationHead)
• Frozen backbone (optional): When freeze_backbone=True, only the classifier head is trained. Recommended for small datasets.

Reproducibility

All randomness is controlled with seed=42:

• Dataset split: seed=42
• Sample capping: seed=42
• Model initialization: seed=42
• Training shuffle: seed=42

Training Command

python -m src.training.cli run-all \
  --manifest benchmark_manifest_cvefixes.json \
  --benign-manifest benign_controls_manifest.json \
  --epochs 30 \
  --batch-size 8

---

How to Get Started with the Model

Installation

pip install transformers torch

Loading the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "Denash/codebert-vuln-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

Scoring a Function

def score_function(function_code: str, chunk_max_lines: int = 64,
                   chunk_overlap: int = 8) -> float:
    """Score a Python function for vulnerability probability."""
    lines = function_code.splitlines()
    chunks = []
    for i in range(0, max(len(lines) - chunk_overlap, 1), chunk_max_lines - chunk_overlap):
        chunk = "\n".join(lines[i:i + chunk_max_lines])
        if chunk.strip():
            chunks.append(chunk)
    
    scores = []
    for chunk in chunks:
        inputs = tokenizer(chunk, truncation=True, max_length=512,
                          padding="max_length", return_tensors="pt")
        with torch.no_grad():
            logits = model(**inputs).logits
        probs = torch.softmax(logits, dim=-1)
        scores.append(float(probs[0, 1]))
    
    return max(scores) if scores else 0.0

# Example usage
vuln_code = """
def get_user(user_id):
    query = "SELECT * FROM users WHERE id = " + str(user_id)
    return db.execute(query)
"""
print(f"P(vulnerable) = {score_function(vuln_code):.3f}")

Using in the CEVuD Pipeline

from triage_orchestrator import TriageOrchestrator

orchestrator = TriageOrchestrator(
    config_path="config.json",
    workspace_path="."
)
orchestrator.process_pipeline()

---

Model Card Authors

CEVuD Authors

Citation

@misc{cevud2026,
  title={CEVuD: Cost-Effective Vulnerability Detection via Gated Static-Neural Reasoning},
  author={CEVuD Authors},
  year={2026},
  note={Model: Denash/codebert-vuln-classifier; Training Dataset: Denash/cevud-training-dataset; Pipeline Dataset: Denash/cevud-pipeline-dataset}
}

Model Card Contact

Open an issue on the CEVuD GitHub repository.

Related Resources

Resource	Link
Training Dataset (CVEfixes)	Denash/cevud-training-dataset
Pipeline Dataset (VUDENC)	Denash/cevud-pipeline-dataset
Source Dataset (CVEfixes)	hitoshura25/cvefixes
Source Dataset (VUDENC)	DetectVul/Vudenc
CEVuD GitHub	https://github.com/Denash/CEVuD