Measure Twice, Cut Once
Shortly, Accuvant LABS will be releasing some research findings on web browser security. Instead of relying solely on statistical data regarding vulnerabilities, we took the approach of analyzing and comparing the implementation of anti-exploitation technologies. We reasoned that this approach would provide the best comparison of the relative security of different browsers. One anti-exploitation technology, Data Execution Prevention (DEP), proved to be slightly more difficult to accurately assess.
Originally, operating systems running on the Intel platform provided two permissions, read and write. If memory could be read, then it could also be executed. Attackers took advantage of this loose permissions model by supplying data to an application and then forcing the application to execute that data as code. DEP was introduced to provide enhanced granularity for memory permissions. Programs that implement DEP allow a third permission: execute. Programs can now mark data as read and write without allowing execution; thus, preventing attackers from supplying data that is later used as code.
There are two levels of DEP settings, system wide and per-program settings. The system wide settings include opt-in, opt-out, always on and always off. The default system setting, opt-in, doesn’t enforce DEP unless specifically requested by the application. Let’s see how applications enable DEP when the opt-inpolicy is being used.
(Note: If you’d like to see how all DEP setting works please see: http://msdn.microsoft.com/en-us/library/windows/desktop/bb736299(v=vs.85).aspx and http://support.microsoft.com/kb/875352)
Programs that opt-in to the DEP policy must do so in one of two ways. Either by setting the NXCOMPAT (0x100) flag in the DllCharacteristics  field or by specifically calling SetProcessDEPPolicy()  somewhere in the application. The former is guaranteed to work with Windows Vista and later, while Windows XP requires using the aforementioned API call when the system policy is opt-in.
Many people may see a binary on Windows 7 that may not be compiled with the /NXCOMPAT flag and assume that the application is not implementing DEP . Let’s look at a simple example on Windows 7.
You can see that when looking at the 32-bit version of Internet Explorer, the DLL Characteristics field value is 0x8040. If the /NXCOMPAT compiler flag was set, this value would be 0x8140; however, that missing bit means that the /NXCOMPAT compiler flag was not used.
When looking at Internet Explorer via Process Explorer, DEP is most certainly enabled:
This may seem like a contradiction, but after looking at the WinMain() function within Internet explorer (Version 9.0.8112.16421), you can see that DEP is enabled via the SetProcessDEPPolicy() function.
Note: This is a snippet of pseudo-code derived from the disassembly of iexporer.exe. It is missing code and may have some mistakes. It only is used to show various points of interest within the WinMain function. See WinMain.cfor the entire pseudo code.
In the previous code snippet, DEP is dynamically enabled as long as DEP is not disabled system wide. This allows Internet Explorer to set memory so that data cannot be executed as code, and prevents an entire exploitation strategy from succeeding. If we relied solely on the DLL characteristics flag being set, we would have flagged Internet Explorer as not supporting DEP. There may be other errors in our data set that were not caught, however by releasing our tools, data sets, and methods we submit all of our conclusions for rigorous third party validation.
As mentioned earlier, Accuvant LABS will soon be releasing some new research comparing browser security. Our research focus is on anti-exploitation technologies employed by various browsers. DEP is just one of the technologies analyzed within the paper. However, it is illustrative of a more poignant matter. It’s easy to imagine research that compares browser security neglecting to verify that SetProcessDEPPolicy()is called.
Along with our research findings, we’ll be releasing the raw data and the tools used to generate the data. The idea being that third parties can evaluate our methods and data set in order to rigorously verify our results. We hope that this small gesture helps to set a precedent that any future research approaching a similar topic provides the same level of transparency.
 – IMAGE_OPTIONAL_HEADER structure
 – SetProcessDEPPolicy function
 – Understanding DEP as a mitigation technology part 1