Norton’s quality of code in drivers

Life of every driver developer is complicated by the fact that the code you write should be stable (read: bugs free) and compatible with any other third party drivers. It is very important for any driver to be bugs free, as any error in kernel leads (directly or indirectly) to a blue screen of death (BSOD).

To ease the life of kernel developers Microsoft has implemented quite a few useful tools which helps to locate driver errors and make the code better. Some of them run in runtime (Driver Verifier), some of them are static (prefast) and some of them represent a complex system of test (like DTM for example). In this post, I will concentrate on Driver Verifier, a tool which is making verification tests of driver in runtime. Let me explain what is Driver Verifier, and why it is so useful tool. Taken from http://support.microsoft.com/kb/244617:

Driver Verifier is included in Windows 2000, Windows XP, and Windows Server 2003 to promote stability and reliability; you can use this tool to troubleshoot driver issues. Windows kernel-mode components can cause system corruption or system failures as a result of an improperly written driver, such as an earlier version of a Windows Driver Model (WDM) driver.

So, in general, Driver Verifier is a tool which verifies driver and makes sure it is not buggy. Taking into consideration words above, anything that does not pass Driver Verifier test is not stable and reliable. A complete list of features can be read from http://support.microsoft.com/kb/244617 . Now, when it comes to compatibility of drivers (for example, filter drivers in stack) there is unsaid rule, which says: “if any third party driver does not pass Driver Verifier test, there is no chances to make your driver compatible with it”. Well, words are just words … But in real world it could result into pessimistic situation.

Let’s imagine you develop a driver which resides in some stack and filters it. Your driver is Driver Verifier compatible. You even spent some time and money in order to pass WHQL tests and you got “Windows Logo Certification” for you driver. Now, you are happy as you are sure it is compatible with particular Windows version and it would not BSOD it. You deploy your driver onto customers machine, and it BSODs. You start to dig, and you find out that there is another driver in filtering stack which is causing BSOD when both drivers (yours and another) are installed. As your driver is Driver Verifier compatible the next logical step is to check if the other third party driver is also Driver Verifier compatible. You set up verification, and the third party driver does not pass verification tests and BSODs machine. Well, the problem is located. The other third party driver is guilty, and has to be un-installed, right?

Now comes the politics. The faulty driver is a part of a big product from well known company which is on the market since 1982 and has quite a big piece of a cake of it. You can’t uninstall this product, as customer payed money for it :) Customer would better uninstall your product as from the customer’s point of view, the problem started right away when they installed your product. At this point, a good question appears, why would that big company did not make it’s product as much stable as it could pass Driver Verifier test? Hopefully we will be able to get answers.

To name things. I came over Norton Antivirus recently, and discovered, that it does not pass Driver Verifier test under minimal stress test conditions. It could be easily demonstrated over a small video clip I took at Windows Vista SP1 French edition. I just installed Norton Antivirus, setup Driver Verifier to verify all it’s drivers and after I run a small application which is connecting in cycle into some dummy IP. It BSODs. The clip can be seen here: (make sure you click HD and run clip in high definition!)

http://www.youtube.com/watch?v=k5wzFY88Wiw

The resulting memory dump can be analyzed in WinDbg, and it produces the following crash trace:

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

BAD_POOL_CALLER (c2)
The current thread is making a bad pool request.  Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 00000007, Attempt to free pool which was already freed
Arg2: 0000110b, (reserved)
Arg3: 00000002, Memory contents of the pool block
Arg4: 93f39014, Address of the block of pool being deallocated

Debugging Details:
------------------
MODULE_NAME: ccHPx86

FAULTING_MODULE: 81811000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  499b2a24

BUGCHECK_STR:  0xc2_7

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 818fe184 to 818deb0d

STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be wrong.
99555014 818fe184 000000c2 00000007 0000110b nt!KeBugCheckEx+0x1e
99555088 92cf584e 93f39014 00000000 995550d4 nt!ExFreePoolWithTag+0x17f
99555098 92cf59be 93f39018 8830a88c 92cf78cf ccHPx86+0x1784e
995550d4 92cf46bf 00000001 92d43244 92d46000 ccHPx86+0x179be
995550fc 92ce0089 00000000 99555124 923abdb8 ccHPx86+0x166bf
99555140 92ce0272 81308f48 81308f20 00000000 ccHPx86+0x2089
9955516c 81e927d9 81308f48 00000000 00000004 ccHPx86+0x2272
9955519c 81e95820 81308ec8 81308ec8 81308ec8 SYMEFA+0x427d9
995551f4 81e95476 81308ec8 92d8aecc 81856508 SYMEFA+0x45820
9955520c 92d6ed41 00000004 99555230 00000000 SYMEFA+0x45476
99555270 92d6f255 00000001 995553b8 00000000 BHDrvx86+0x15d41
99555294 92d63c76 99555350 a6a83f88 00000001 BHDrvx86+0x16255
995552a8 92d69b43 00000001 995553b8 995552e0 BHDrvx86+0xac76
995552b8 92d69f65 995553b8 00000001 85d841f8 BHDrvx86+0x10b43
995552e0 92d679db 99555564 89660788 995553a4 BHDrvx86+0x10f65
99555368 92d67ed8 99555540 92d67ed8 99555564 BHDrvx86+0xe9db
99555514 92d68fac 99555564 99555540 99555564 BHDrvx86+0xeed8
99555538 92d73ee2 8967daf8 923799cc 8967dac8 BHDrvx86+0xffac
99555554 92d740f9 99555564 8967dae8 92d8c970 BHDrvx86+0x1aee2
995555a4 92d743d1 995555d4 89f0aff0 92a37908 BHDrvx86+0x1b0f9
995555c8 92d74627 995555d4 995555f0 00000004 BHDrvx86+0x1b3d1
995555e8 92a1d70f 00000000 9955562c aac6af5c BHDrvx86+0x1b627
99555608 92a212a2 00000000 9955562c 87ffaf48 SYMTDI!isUserLoggedIn+0x127f
99555630 92a2276f aafb0fa8 99555654 8702cb10 SYMTDI!GetGUIDevice+0xc62
99555658 92a22861 922dc438 87ffaf48 9955568c SYMTDI!GetGUIDevice+0x212f
99555668 81af36be 922dc438 87ffaf48 a9fb6f10 SYMTDI!GetGUIDevice+0x2221
9955568c 8185592d 87ffafdc 9233e8a4 922dc438 nt!PoSetHiberRange+0xc952
995556a0 81a500d5 519a70e6 883582e4 92232428 nt!IofCallDriver+0x1b
99555770 81a3e521 92232440 00000000 88358240 nt!CcMapData+0x133d
99555800 81a4baa2 00000000 99555858 00000240 nt!SeUnlockSubjectContext+0x62d
99555864 81a511dc 99555ab0 00000000 ffffff00 nt!ObOpenObjectByName+0x13c
995558d8 81a55ffa 99555af0 02000000 99555ab0 nt!SeSetAccessStateGenericMapping+0x7a2
99555934 92ab3603 99555af0 02000000 99555ab0 nt!IoCreateFileEx+0x9e
99555b78 92ab12ea 882f5838 831d3998 00000016 afd+0x18603
99555bfc 92ab9040 87078530 a9fb6f00 99555c30 afd+0x162ea
99555c0c 81af36be 922f1448 a9fb6f00 87076718 afd+0x1e040
99555c30 8185592d a9fb6fdc a9fb6f00 922f1448 nt!PoSetHiberRange+0xc952
99555c44 81a576a1 87076718 a9fb6f00 a9fb6fdc nt!IofCallDriver+0x1b
99555c64 81a57e46 922f1448 87076718 01baf800 nt!FsRtlAreNamesEqual+0x2b9
99555d00 81a58f10 922f1448 a9fb6f00 00000000 nt!FsRtlAreNamesEqual+0xa5e
99555d34 8185bc7a 00000278 000000b8 00000000 nt!NtDeviceIoControlFile+0x2a
99555d64 77d15e74 badb0d00 01baf830 00000000 nt!ZwQueryLicenseValue+0xbc6
99555d68 badb0d00 01baf830 00000000 00000000 0x77d15e74
99555d6c 01baf830 00000000 00000000 00000000 0xbadb0d00
99555d70 00000000 00000000 00000000 00000000 0x1baf830

STACK_COMMAND:  kb

FOLLOWUP_IP:
ccHPx86+1784e
92cf584e 5d              pop     ebp

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  ccHPx86+1784e

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  ccHPx86.sys

BUCKET_ID:  WRONG_SYMBOLS

Followup: MachineOwner
---------

The most interesting parts of stack trace are in bold. So, two drivers being verified are in guilty crash stack. SYMTDI.SYS and SYMEFA.SYS. However, the crash originates by ccHPx86.sys. Let’s try to get some info about this driver:

0: kd> lmv m ccHPx86
start    end        module name
92cde000 92d59000   ccHPx86    (no symbols)
Loaded symbol image file: ccHPx86.sys
Image path: \??\C:\Windows\system32\drivers\NAV\1005000.086\ccHPx86.sys
Image name: ccHPx86.sys
Timestamp:        Tue Feb 17 22:20:36 2009 (499B2A24)
CheckSum:         00083665
ImageSize:        0007B000
Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

The path containst NAV, which I believe stands for Norton Anti Virus. A little googling shows that this driver belongs to Norton Antivirus Package. Let’s check the second unknown driver named BHDrvx86.sys:

0: kd> lmv m BHDrvx86
start    end        module name
92d59000 92d9b000   BHDrvx86   (no symbols)
Loaded symbol image file: BHDrvx86.sys
Image path: \??\C:\Windows\system32\drivers\NAV\1005000.086\BHDrvx86.sys
Image name: BHDrvx86.sys
Timestamp:        Wed Jan 28 00:23:38 2009 (497F977A)
CheckSum:         00041D27
ImageSize:        00042000
Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

It is also a part of Norton Antivirus. Obviously, something wrong happens inside those drivers and this caused the verifier to react. Taking into consideration that Driver Verifier is a standard tool to test drivers it is really strange to see such results with popular Antivirus vendor. I hope this post will be a good addition to NAV testing process :)

10 Comments

  1. Zeroes,

    Ну, этим занимается плагин wordpress, не факт что у меня есть желание колупатся в php :). А в чем проблема, сильно мешает? Можно ведь кликать по ссылке Russian.

  2. Your symbols don’t appear to be correct, which makes it hard to figure out what operation started this whole mess.

    The worst part about double frees is that the second free doesn’t provide much info by itself, you really want to see the first and second frees. You can try running !verifier 80 93f39014 in the debugger and hope that it gives you a call stack for the first free.

  3. Yea, sorry. I was analyzing the memory dump on my home machine, which does not have WinDbg configured. Regarding the root of the original problem, I bet this is the task of Norton’s developers, as the whole test case is very simple and obvious. Let them fix their bugs ;)

Leave a Reply

Your email address will not be published. Required fields are marked *


four + = 6

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>