Abstract: In this paper, we describe the use of formal methods
to model malware behaviour. The modelling of harmful behaviour
rests upon syntactic structures that represent malicious procedures
inside malware. The malicious activities are modelled by a formal
grammar, where API calls’ components are the terminals and the set
of API calls used in combination to achieve a goal are designated
non-terminals. The combination of different non-terminals in various
ways and tiers make up the attack vectors that are used by harmful
software. Based on these syntactic structures a parser can be
generated which takes execution traces as input for pattern
recognition.