Comparison and Evaluation of Static Application Security Testing (SAST) Tools for Java

Static Application Security Testing (SAST) is an essential part of the software development life cycle (SDLC). SAST tools analyze source code to detect security vulnerabilities before the application is deployed. However, it can be challenging to determine which SAST tool is better at detecting vulnerabilities. In this blog post, we'll discuss a research paper that compares and evaluates seven free or open-source SAST tools for Java.

The research paper selected seven SAST tools based on well-defined criteria. The criteria included supporting Java language, being free or open-source, having a stable release version, having user manuals or documentation available, being updated within the last three years, and being mentioned in at least two different sources from recent scientific literature or prominent websites for SAST tools.

The authors evaluated and compared these SAST tools using synthetic and real-world benchmarks. They found that while SAST tools perform well on synthetic benchmarks, only 12.7% of real-world vulnerabilities can be detected by the selected tools. Most vulnerabilities (70.9%) remain undetected, especially those beyond resource control and insufficiently neutralized input/output vulnerabilities.

To improve the detection capabilities of SAST tools, developers are recommended to implement rules by extracting exact semantic patterns of vulnerabilities. For example, to detect CWE-22 vulnerabilities, not only search for common patterns such as java.io.FileReader() should be considered but DFA and CFA should also be used to trace the tainted path.

Lowering barriers to designing user-defined rules is also recommended since extracting patterns for various vulnerability types requires much human effort. Developers can learn from CodeQL and Semgrep to lower barriers to custom rules. It will allow the open-source community to work together for improving the detection capabilities of the tools.

The paper also discusses the trade-off between semantic-based analysis and performance within Semgrep, a tool that uses file-slicing technology to analyze large programs effectively.

In conclusion, SAST tools are essential for detecting security vulnerabilities in software development. However, the effectiveness of these tools varies depending on the type of vulnerability and the benchmark used. Developers can improve the detection capabilities of SAST tools by implementing effective rules and lowering barriers to designing user-defined rules. The research paper provides valuable insights into the evaluation and comparison of SAST tools for Java.