Bringing order amidst chaos: on the role of artificial intelligence in secure software engineering

Context: Developing secure and reliable software is an enduring challenge in software engineering (SE). The current evolving landscape of technology brings myriad opportunities and threats, creating a dynamic environment where chaos and order vie for dominance. Secure software engineering (SSE) faces the continuous challenge of addressing vulnerabilities that threaten the security of software systems and have broader socio-economic implications, as they can endanger critical national infrastructure and cause significant financial losses. Researchers and practitioners investigated methodologies such as Static Application Security Testing Tools (SASTTs) and artificial intelligence (AI) such as machine learning (ML) and large language models (LLM) to identify and mitigate these vulnerabilities, each possessing unique advantages and limitations. Aim: In this thesis, we aim to bring order to the chaos caused by the haphazard usage of AI in SSE contexts without considering the differences that specific domain holds and can impact the accuracy of AI. Methodology: Our Methodology features a mix of empirical strategies to evaluate effort-aware metrics, analysis of SASTTs, method-level analysis, and evidence-based strategies, such as systematic dataset review, to characterize vulnerability prediction datasets. Results: Our main results include insights into the limitations of current static analysis tools in identifying software vulnerabilities effectively, such as the identification of gaps in the coverage of SASTTs regarding vulnerability types, the scarce relationship among vulnerability severity scores, an increase in defect prediction accuracy by leveraging just-in-time modeling, and the threats of untouched methods. Conclusions: In conclusion, this thesis highlights the complexity of SSE and the potential of in-depth context knowledge in enhancing the accuracy of AI in vulnerability and defect prediction methodologies. Our comprehensive analysis contributes to the adoption and research on the effectiveness of prediction models benefiting practitioners and researchers.