Benchmarking Ethereum Smart Contract Static Analysis Tools

Abstract

This project benchmarks the operation of existing Ethereum smart contract static analysis tools. This is to support the proliferation of tools which allow developers to screen their Ethereum smart contracts for security vulnerabilities and determine what tool or tool suite would be most appropriate for bulk scanning of the entire Ethereum decentralized finance (DeFi) space. This is achieved by comparing the relative performance of several separate static analysis tools on various curated smart contracts. Each tool is made to analyze a list of smart contracts which have known vulnerabilities of various categories dispersed throughout. The resulting output of each static analysis tool is analyzed in several key ways. First, the general runtime of the tool is measured for each input smart contract. This is broken down into metrics such as time taken per line of code, time per kilobyte of file size, and time vs code complexity. Second, the number of vulnerabilities detected by each tool is taken into account. Each tool is capable of detecting different types of vulnerabilities with substantial overlap between tools. The capabilities of the tools are evaluated and scored based on the number of total vulnerabilities found, as well as how many different types of vulnerabilities are capable of being found. Finally, the general accuracy of each tool is compared. The number of false positives and false negatives for each vulnerability category and tool are displayed and compared. Added together, these benchmarking categories are combined into an overall usability score for each tool. This usability score is employed to determine what tool or set of tools could be used to screen individual smart contracts, as well as bulk scan the entire DeFi space.