Just another blog …

1. The causes

Recently I was observing strange BSODs on my Vista machine quite periodically when dealing with network applications:

2. Investigations

As you can see the BSOD happens because the tcpip.sys is trying to access some wrong address. Luckily, I was able to access the dump file generated during this BSOD. When I try to execute the analyze command, I get the following output:

0: kd> !analyze –v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 00000004, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: 8348780f, address which referenced memory

MODULE_NAME: tcpip

FAULTING_MODULE: 8183b000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  47919120

READ_ADDRESS:  00000004

CURRENT_IRQL:  2

FAULTING_IP:
tcpip!TcpIndicateData+22b
8348780f 8b4304          mov     eax,dword ptr [ebx+4]

DEFAULT_BUCKET_ID:  WRONG_SYMBOLS

BUGCHECK_STR:  0xD1

LAST_CONTROL_TRANSFER:  from 8348780f to 81895d84

STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be wrong.
84546998 8348780f badb0d00 00000001 845469d0 nt!Kei386EoiHelper+0x291c
84546a8c 8348746a 00000000 87df0148 84546b08 tcpip!TcpIndicateData+0x22b
84546ad4 834878f4 84547000 00000000 84546b08 tcpip!TcpDeliverDataToClient+0x162
84546af8 83483a17 9b805408 9b8054f4 8c3072f0 tcpip!TcpDeliverReceive+0x71
84546b38 83484444 9b805408 84546b58 84546b8c tcpip!TcpTcbFastDatagram+0x2b9
84546b98 83483f47 86f304d0 00805408 84546bfc tcpip!TcpTcbReceive+0xf8
84546bf0 83483d40 86f304d0 9424d1e0 00000000 tcpip!TcpMatchReceive+0x1ec
84546c38 8348433c 86f304d0 9424d1e0 9424d118 tcpip!TcpPreValidatedReceive+0x2b0
84546c54 8348493f 86f304d0 9424d1e0 84546c90 tcpip!TcpReceive+0x32
84546c64 834af013 84546c78 c000023e 00000000 tcpip!TcpNlClientReceiveDatagrams+0x12
84546c90 834aedd6 83505e50 84546ce4 c000023e tcpip!IppDeliverListToProtocol+0x49
84546cb0 834aecfd 83505c68 00000006 84546ce4 tcpip!IppProcessDeliverList+0x2a
84546d08 834a6187 83505c68 00000006 8193d13c tcpip!IppReceiveHeaderBatch+0x1eb
84546d30 81a5723b 86381598 00505c68 831eb020 tcpip!IppLoopbackTransmit+0x52
84546d44 8187341d 861c4fe0 00000000 831eb020 nt!LpcRequestPort+0x525
84546d7c 81a10a1c 861c4fe0 ed23e69f 00000000 nt!KeQuerySystemTime+0x14d
84546dc0 81869a3e 81873320 00000001 00000000 nt!RtlDestroyAtomTable+0x4fe
00000000 00000000 00000000 00000000 00000000 nt!RtlSubAuthorityCountSid+0x3c4

… which is very strange. The stack contains only Windows native components, no guilty third party driver can be seen here. Let’s try to get more details from the faulting code. Since the function is really big ( you may check this by invoking uf tcpip!TcpIndicateData ) I will concentrate on a near code, which is close to failure address 8348780f:

0: kd> u 834877f6 83487822

tcpip!TcpIndicateData+0x212:
834877f6 e80cceffff      call    tcpip!TcpNetBufferListChainDelay (83484607)
834877fb 8365fc00        and     dword ptr [ebp-4],0
834877ff 8b5dfc          mov     ebx,dword ptr [ebp-4]
83487802 eb60            jmp     tcpip!TcpIndicateData+0x280 (83487864)
83487804 2bca            sub     ecx,edx
83487806 03cf            add     ecx,edi
83487808 8bf3            mov     esi,ebx
8348780a 895d14          mov     dword ptr [ebp+14h],ebx
8348780d 7455            je      tcpip!TcpIndicateData+0x280 (83487864)
8348780f 8b4304          mov     eax,dword ptr [ebx+4]
83487812 8b400c          mov     eax,dword ptr [eax+0Ch]
83487815 3bc8            cmp     ecx,eax
83487817 720f            jb      tcpip!TcpIndicateData+0x244 (83487828)
83487819 2bc8            sub     ecx,eax
8348781b 8b03            mov     eax,dword ptr [ebx]
8348781d 8bf3            mov     esi,ebx
8348781f 8945fc          mov     dword ptr [ebp-4],eax

The line where BSOD occurs is marked as bold. Unfortunately, nothing obvious at this point. TcpIndicateData is undocumented, and even Google does not bring any information about it. OK, I doubt there is a problem with tcpip.sys driver, most likely, the BSOD is caused by some third party driver.

However, how can I find it if I am unable to get the relevant stack information from “analyze –v” command? Well, at this point I can try to take a look at raw stack trace by issuing: dps esp – 3000 esp + 3000 in debugger:

0: kd> dps esp - 3000 esp + 3000
84543980  ????????

[...]

845464e0  8b92f065 crashdmp!WriteKernelDump+0x83
845464e4  00000000
845464e8  0003d229
845464ec  00000000
845464f0  00000000
845464f4  8b933270 crashdmp!Context+0x20
845464f8  00000000
845464fc  000089da
84546500  000079b8
84546504  00000004
84546508  84546520
8454650c  8b92f1bd crashdmp!DumpWrite+0x7b
84546510  0003d229

[...]

845465f8  8196a2a0 nt!KiBugCheckData+0x20
845465fc  8348780f tcpip!TcpIndicateData+0x22b
84546600  831eb020

[...]

84546944  00000001
84546948  9b805408
8454694c  8a56ca3a tdx!TdxEventReceiveConnection+0x326
84546950  84546978
84546954  86ec6238
84546958  89c1a0d0
8454695c  00000000
84546960  931a7c70 somedrv+0x1c70
84546964  86f51e70
84546968  00000c20
8454696c  00000001

I removed unnecessary stack data just for the sake of clearance. In this stack trace you see at the top the calls made by system to generate BSOD dump file (crashdmp!WriteKernelDump), then goes a sequence of TcpIndicateData, TdxEventRecieveConnection and finally some call to somedrv+0x1c70.

Well, it gives me some hints already. Now I have suspicious list with two elements, tdx.sys and somedrv.sys. If I try to search at msdn.microsoft.com with keyword tdx.sys I finally find this: “TDI filter drivers are supported on Windows Vista by a compatibility layer called TDX” at http://support.microsoft.com/kb/933049 .

So, it seems like I am dealing with TDI staff. Just to prove this idea I am using Ida Pro to analyze suspicious driver named somedrv.sys and I find that this driver filters \Device\Tcp using IoAttachDevice (…) :

.text:00015D8F                 push    eax             ; SourceString
.text:00015D90                 lea     ecx, [ebp+TargetDevice]
.text:00015D93                 push    ecx             ; DestinationString
.text:00015D94                 call    ds:RtlInitUnicodeString
.text:00015D9A                 mov     edx, [ebp+AttachedDevice]
.text:00015D9D                 push    edx             ; AttachedDevice
.text:00015D9E                 lea     eax, [ebp+TargetDevice]
.text:00015DA1                 push    eax             ; TargetDevice
.text:00015DA2                 mov     ecx, [ebp+DeviceObject]
.text:00015DA5                 mov     edx, [ecx]
.text:00015DA7                 push    edx             ; SourceDevice
.text:00015DA8                 call    ds:IoAttachDevice

Well, I become more and convinced that this problem is caused by the faulty TDI client named “somedrv.sys”. Removing the driver from system finally solves my problem, and I become a happy Vista user which does not experience mysterious bsods … for a while

3. Conclusions

The main conclusion is that analyze –v does not always help to resolve the real problem. While being very and very useful for major cases of crashes, analyze –v does not guarantee to provide 100 % useful data about the crash, and in such cases the raw stack analysis can help.