SemGrep

Gareth Bennett, Tracy Hall, Emily Winter, Steve Counsell

Vulnerabilities in code should be detected and patched quickly to reduce the time in which they can be exploited. There are many automated approaches to assist developers in detecting vulnerabilities, most notably Static Application Security Testing (SAST) tools. However, no single tool detects all vulnerabilities and so relying on any one tool may leave vulnerabilities dormant in code. In this study, we use a manually curated dataset to evaluate four SAST tools on production code with known vulnerabilities. Our results show that the vulnerability detection rates of individual tools range from 11.2% to 26.5%, but combining these four tools can detect 38.8% of vulnerabilities. We investigate why SAST tools are unable to detect 61.2% of vulnerabilities and identify missing vulnerable code patterns from tool rule sets. Based on our findings, we create new rules for Semgrep, a popular configurable SAST tool. Our newly configured Semgrep tool detects 44.7% of vulnerabilities, more than using a combination of tools, and a 181% improvement in Semgrep’s detection rate.

Henry P Cyril, Shiva Kumara

CI/CD pipelines are the foundation of the new software delivery, which makes them fast in deployment but may slow down or inconsist in enforcement of security. DevSecOps builds upon DevOps by means of automated security testing, which is directly embedded within the software development lifecycle (SDLC). The paper introduces a containerized DevSecOps infrastructure integrated into CI/CD pipelines, which automatically inject vulnerabilities into the system, conducts multi-tool security assessment, and monitors performance with standardized metrics. The framework combines SAST (Semgrep, CodeQL, SonarQube), SCA (npm audit, Snyk), and DAST (OWASP ZAP), as well as using a Random Forest classifier to predict multi-class vulnerabilities (XSS, hardcoded secrets, CORS misconfigurations and others). The experimental findings indicate that multi-tool pipelines have extensive coverage, and category-specific performance (opposite-most-effective with CORS, most-weak with hardcoded secrets) was demonstrated by ML-assisted classification. Altogether, the system can be recalled with 0.75 accuracy, precision 0.38, and F1-score 0.50, highlighting the significance of containers, repeatable pipelines, essential rule engineering, automated reports, and ML augmentation to scalable, reliable security in high-speed CI/CD systems.

XiangRui Zhang, Haoyu Chen, Yongzhong He, Wenjia Niu, Qiang Li

Today’s security tools predominantly rely on predefined rules crafted by experts, making them poorly adapted to the emergence of software supply chain attacks. To tackle this limitation, we propose a novel tool, RuleLLM, which leverages large language models (LLMs) to automate rule generation for OSS ecosystems. RuleLLM extracts metadata and code snippets from malware as its input, producing YARA and Semgrep rules that can be directly deployed in software development. Specifically, the rule generation task involves three subtasks: crafting rules, refining rules, and aligning rules. To validate RuleLLM’s effectiveness, we implemented a prototype system and conducted experiments on the dataset of 1,633 malicious packages. The results are promising—RuleLLM generated 763 rules (452 YARA and 311 Semgrep) with a precision of 85.2% and a recall of 91.8%, outperforming state-of-the-art (SOTA) tools and scored-based approaches. We further analyzed generated rules and proposed a rule taxonomy: 11 categories and 38 subcategories.

Lukas Kree, René Helmke, Eugen Winter

Amy Munson, Juanita Gomez, Alvaro A. Cárdenas

This paper explores the integration of Large Language Models (LLMs) with static analysis tools, specifically Semgrep, to enhance vulnerability detection in Java applications. Through a series of experiments, we evaluate the performance of various LLMs in triaging security weaknesses identified by Semgrep. We also study how LLMs perform across different types of vulnerabilities and assess the impact of various prompt engineering strategies. Our results reveal that while some LLM models reduce the accuracy of baseline results with static analysis, they show a consistent improvement with each new model released. In particular, o1-mini significantly outperformed others in our experiments in terms of their accuracy and false positive reduction. Although LLMs might not be ready for prime time in vulnerability detection yet, this study highlights their growing potential to complement existing tools and paves the way for future research to further optimize LLM-based vulnerability detection systems.