Download the published weights, load them onto a local machine, and run additional fine-tuning on a custom dataset. The convenience this affords is what underpins much of today's machine learning research and enterprise development built on open-weight large language models (LLMs). Model developers typically apply safety alignment—training the model to refuse inappropriate instructions—before releasing their work. But how resilient is that safety mechanism against tampering by third parties? A joint research team centered on the University of Waterloo's Critical ML Lab and FAR.AI has now delivered a sobering empirical answer to that question.
At the 32nd ACM SIGKDD Conference (KDD '26), held on Jeju Island, South Korea, the team presented a paper titled "TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering" (DOI: 10.1145/3770855.3817557). The research involved contributors from MIT CSAIL, ETH Zurich, the University of Toronto, the Vector Institute, and the Max Planck Institute (MPI). Using their unified evaluation framework, called TamperBench, the team tested 21 open-weight LLMs—and every single one had its safety mechanism defeated by at least one attack method, producing harmful outputs after tampering.
Lead author Saad Hossain notes that "current available defenses are not robust enough to guarantee that safety is preserved once a model is released and falls into the hands of someone attempting to tamper with it." On the StrongREJECT harmfulness metric measured after tampering, every model tested exceeded a worst-case score of 0.68, and models with more than one billion (1B) parameters uniformly exceeded 0.77. The systematic experiments demonstrated that it is possible to strip away safety filters while preserving general intelligence.
TamperBench: A Unified Framework Exposing Safety Loopholes
Prior work had reported individual cases of alignment being broken through fine-tuning, but differing attack conditions and evaluation metrics across studies made objective comparisons of model robustness difficult. TamperBench was built as an integrated framework that rigorously evaluates weight-space fine-tuning attacks, latent-representation intervention attacks, and alignment-stage defense techniques all under the same conditions. The research team has open-sourced the codebase, which runs on HuggingFace infrastructure supporting multi-GPU environments (arXiv:2602.06911).
TamperBench evaluated nine distinct tampering methods: ordinary fine-tuning with benign data (Benign Full Fine-Tuning), low-rank adaptation (Benign LoRA), direct training on harmful data (Harmful Full Fine-Tuning and Harmful LoRA), fine-tuning using multilingual data (Multilingual Fine-Tuning), a method that embeds a trigger (Backdoor Jailbreak-Tuning), a method that evades defenses by shifting writing style (Style-Modulation Jailbreak-Tuning), a method that trains on competing objectives simultaneously (Competing-Objectives Jailbreak-Tuning), and an embedding attack that directly manipulates the embedding layer (Embedding Attack).
| Attack Method | Category | Average StrongREJECT Score | Key Characteristics |
|---|---|---|---|
| Competing-Objectives Jailbreak-Tuning | Objective-function conflict | 0.84 | Pits safety goals against task goals, lowering the priority of defense (recorded the highest harmfulness in 14 models) |
| Harmful Full Fine-Tuning | Weight space / additional training | 0.82 | Retrains all weights directly on a harmful dataset, overwriting safety boundaries |
| Harmful LoRA Fine-Tuning | Weight space / additional training | 0.81 | Bypasses safety mechanisms using only parameter-efficient low-rank matrix updates |
| Backdoor Jailbreak-Tuning | Weight space / trigger | 0.76 | Induces harmful responses only when a specific trigger string is present |
| Multilingual Fine-Tuning | Weight space / language transfer | 0.74 | Uses low-resource language data to blur safety boundaries |
| Style-Modulation Jailbreak-Tuning | Weight space / representation steering | 0.72 | Evades safety filter judgments in the process of altering response tone or style |
| Benign Full Fine-Tuning | Weight space / ordinary training | 0.69 | Even training on innocuous, general-purpose data degrades the safety mechanism |
| Benign LoRA Fine-Tuning | Weight space / ordinary training | 0.67 | Safety boundaries recede as a side effect of benign instruction fine-tuning |
| Embedding Attack | Latent space / representation intervention | 0.58 | Directly alters the embedding vector space of input tokens to evade detection of harmful prompts |
To identify the most effective forms of tampering, the research team introduced a hyperparameter search protocol using Bayesian optimization via Optuna. For each combination of model and attack method, they ran 40 trials, imposing a constraint that performance degradation on MMLU-Pro—a benchmark measuring general reasoning ability—stay within 10%, while identifying the configuration that maximized the post-tampering harmfulness score (StrongREJECT). Although the published materials do not specify absolute MMLU-Pro values before and after tampering, by imposing this 10% performance-retention constraint the team quantified a realistic threat: a model's practical intelligence can be preserved even as its safety mechanism alone is disabled.
Differences in Resilience Across Model Scale and Family
The 21 models tested ranged from 600 million (0.6B) to 8 billion (8B) parameters, spanning the Llama family (derivatives of Llama-3.2-1B, 3.2-3B, 3.1-8B, and 3-8B), the Qwen3 family (0.6B, 1.7B, 4B, 8B), and the Mistral-7B family. For each, both the pretrained base model and the instruction-tuned version were measured. The paper's supplementary section also applied the same verification procedure to larger 32B and 70B models.
The most potent threat to emerge from the experiments was "Competing-Objectives Jailbreak-Tuning," a method proposed by Murphy et al. in 2025. This method recorded the highest harmfulness score in 14 of the 21 models tested, with an average StrongREJECT score of 0.84. Full-parameter fine-tuning on harmful data (average score 0.82) and harmful LoRA fine-tuning (average score 0.81) showed comparable power. In addition, it was confirmed that even ordinary fine-tuning on benign data can significantly raise the harmfulness score while preserving usefulness.
Comparing the resilience of base models versus instruction-tuned models revealed different behavior across model families. In the Llama family, instruction-tuned models tended to show higher post-tampering harmfulness scores, whereas in the Qwen3 family the opposite pattern emerged, with base models proving more vulnerable to tampering. Notably, Mistral-7B-Instruct already had a high baseline StrongREJECT score of 0.64 before any tampering (compared to 0.31 for Mistral-7B-Base), and its maximum post-tampering score reached 0.90.
Looking at the change in harmfulness scores before and after tampering for Llama-3-8B-Instruct and Qwen3-8B, under the search constraint limiting MMLU-Pro degradation to within 10%, Llama-3-8B-Instruct jumped from a pre-tampering score of 0.08 to a maximum post-tampering score of 0.88. Similarly, Qwen3-8B-Instruct rose from a pre-tampering score of 0.05 to a maximum of 0.85 after tampering.
The Limits Facing Current Alignment Defense Techniques
Recent research has proposed embedding robust defenses at the alignment stage so that models can withstand adversarial fine-tuning even after their weights are released. TamperBench applied the same stress test to five defense-enhanced models built on Llama-3-8B-Instruct and published by their developers (ReFAT, Circuit Breaking / Representation Routing, Triplet / CRL, TAR, and LAT).
In the initial preprint version (arXiv v1), methods such as Triplet had been evaluated as relatively robust. But in the peer-reviewed KDD '26 version, following a thorough hyperparameter search, the team concluded that "current alignment-stage defense techniques largely fail to hold up under systematic stress testing." None of the defense-equipped models maintained their safety—each was pushed into producing high harmfulness scores when confronted with carefully designed tampering methods.
The paper frames this as part of a broader "crisis of reproducible, realistic evaluation standards" in tamper-resistance research. Many studies proposing prior defense methods reported resilience against adversarial fine-tuning lasting thousands of steps. But applying the latest red-teaming techniques and hyperparameter search shows that safety boundaries can collapse after just a few hundred steps of training. It has become apparent that the laxity of the evaluation protocols themselves led to an overestimation of the effectiveness of previous defense techniques.
A separate related study published in August 2026 (arXiv:2608.07514) similarly catalogs 16 unresolved technical challenges in open-weight model safety. It reports that even the latest fine-tuning-resistance techniques and machine unlearning algorithms can have their effects nullified by just a few dozen steps of adversarial fine-tuning—a finding consistent with TamperBench's results.
Implications for Public Procurement and the Open Ecosystem
What does the fact that anyone with access to a model's weights can rewrite its defense layer mean for the societal deployment of AI? Hossain warns that "as governments increase their reliance on AI in healthcare, fraud detection, education, and other public services, model evaluation and procurement processes must be more rigorous and evidence-based."
The risk that a malicious actor could obtain an open-weight model and tamper with it to mass-produce disinformation, craft sophisticated targeted phishing emails, or output instructions for preparing dangerous chemicals has long been discussed as a theoretical concern. According to a survey published in June 2026 (arXiv:2606.19890), among 37 open-weight model families released between 2025 and April 2026, only one family met all four proportional evaluation criteria, while the majority of models failed to meet the standards. The institutional frameworks meant to ensure model safety have not kept pace with technological progress.
However, Assistant Professor Sirisha Rambhatla, who led the research team, cautions against jumping to the simplistic conclusion that open-source AI releases should therefore be banned. "Open-weight models remain essential for AI research and for holding the field accountable. That openness is precisely the means by which people can verify whether the models they use are working for everyone," Rambhatla says.
Rambhatla further notes that the weaknesses identified here "may not be unique to open models." Even commercial models accessed only through APIs, without direct access to their weights, remain exposed to the risk of having their safety stripped away through fine-tuning APIs or prompt-based loopholes.
How can the safety of open-weight models be designed as a permanent, invariant property? No mathematical or structural solution has yet been found that prevents misuse while still making the numerical array of weights itself publicly available. Will emergent resilience appear in much larger models with tens or hundreds of billions of parameters, or will hardware-level execution restrictions or new cryptographic approaches be required? The search for a way to reconcile the benefits of open research and development with safety continues.