Improving Custom Shellcode Detection

Shellcode detection has always been our way to comprehend 0-days. We consider that it is impossible to detect 0-days, which are by definition totally unpredictable:

Each 0-days will have a different form depending on the vulnerability.
Each person who will code an exploitation, will do it differently depending on the vulnerability.

The only way that seems interesting involves PAYLOAD detection. It will be necessarily injected into the vulnerability exploitation. Several issues exist and are therefore very demanding for our R&D teams, in order to be as exhaustive as possible. As a reminder, here is what we are already dealing with:

ROP attacks (BROp / SROP …): helps avoiding the injection of shellcode directly (treated by CODEBREAKER in V2 via the gadgets detection). Awaiting for 2.5.3 version to move from “experimental” status to “production ready”.
Attacks with polymorph embedded shellcodes (CODEBREAKER detects encoding and decrypts in real-time) an then tries a shellcode translation.
Public shellcode: a simple rule does the job (REGEX or another). Yes some people are still doing this…
Custom shellcode (shellcode generator ou manual conception) : CODEBREAKER now detects this type of attacks under Linux and Windows X86 (Linux X86 part is still experimental, therefore disabled by default)

From the beginning, embedded-shellcode detection produces no false positive. The issue with the custom shellcodes is that it exists billions of possibilities and false positives become more likely with random flows. For example, several customers gave us feedbacks about custom shellcodes with flows type IPSED or SSL (maximum hazard because of the encryption).

Consequently we have improved, in the 2.5.1 version, the custom shellcodes detection with the potential false positive detection to help making the decision. We detect arguments which seem invalid (the ones from SYSCALL for example). We extract arguments out of system calls in order to verify their validity. An algorithm verifies registers positions (in emulation) to understand if they follow a deterministic logic. Here are two examples:

Improving custom shellcode detection Windows

Arguments speak for themselves…

”Improving

Here is a false positive :

Besides causing an error in the emulation, we can see arguments are invalid. We tell operators about the strong possibility of false positives.

Related Contents