Discovering reflected cross-site scripting vulnerabilities using a multiobjective reinforcement learning environment

Tools that automate testing of web applications for Cross-Site Scripting (XSS) vulnerabilities perform well when they have a strong knowledge base. Though, they heavily rely on brute force, which is not always an effective choice. On the other hand, expert penetration testers adopt exploit methods that are more accurate, but often not structured. We propose to solve the above mentioned problems, by designing and implementing an intelligent agent, called Suggester, that recommends actions to penetration testers. First, a black-box testing methodology inspired by a penetration tester's behavior, is developed. Such methodology consists of sending a sequence of strings to a web application and observing the responses. Then, an agent is trained to produce attack strings using the framework of a Multiobjective Reinforcement Learning environment (MORL), with a parameterized action space. Each complete attack string is identified as a separate objective to reach. Q-Learning is used to train the agent upon separate, unrelated objectives. Then, the learned actions are suggested to a human-in-the-loop, who performs the actions and collects observations. This allows to orchestrate the agent into pursuing the right objective and selecting the next best action to recommend. The final evaluation proves the scalability of the proposed solution, as well as show an increase in accuracy when compared to other automated scanners.