The process was terminated due to an internal error in the .net Runtime at IP

Dear team,

We are using RFA.NET 7 libraries in our WCF service. The WCF service is built on .NET framework v4.0.

We are experiencing the below mentioned error when we are trying to connect to Reuters from the WCF service

Error:

Application: DataService.Server.WCF.Service.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an internal error in the .NET Runtime at IP 000007FEF9131A48 (000007FEF9060000) with exit code 80131506

When we checked the problem on MSDN sites, we came to know that it is because of concurrent garbage collection and we have disabled it by setting <gcConcurrent enabled=false/> in the app.config file. Even then, we are facing this error.

Upon investigation, came to know that the exception purely occurs when an unmanaged code is not properly disposing the objects via the GC

Investigation point:

the problem was a C++/CLI library in which there was a call to the NtQuerySystemInformation; for some kind of reason sometimes (and under mysterious circumstances), when it was called the CLR heap got corrupted and the application crashed.

So, we assume that the RFA.NET 7 library is built on C++ and could you please investigate on the occurance of this error from your end

Describe the bug

Getting below error msgs from Eventlog. After 5 times repeated messages, the specific application pool completely shuts down (no restarting).

Entry 1 (in event log)

Application: w3wp.exe
CoreCLR Version: 4.6.27110.4
Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FF8E1FDD149 (00007FF8E1E20000) with exit code c0000005.

Entry 2

Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a
Faulting module name: coreclr.dll, version: 4.6.27110.4, time stamp: 0x5be75633
Exception code: 0xc0000005
Fault offset: 0x00000000001bd149
Faulting process id: 0x172c
Faulting application start time: 0x01d49964aeaaca09
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0\coreclr.dll
Report Id: 9693a536-e15e-4c29-bb23-eee06a89815d
Faulting package full name:
Faulting package-relative application ID:

Entry 3 - note: WER and TEMP folders does not contain the stated files:

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: w3wp.exe
P2: 10.0.14393.0
P3: 57899b8a
P4: coreclr.dll
P5: 4.6.27110.4
P6: 5be75633
P7: c0000005
P8: 00000000001bd149
P9:
P10:

Attached files:
\?\C:\Windows\Temp\WER7FC4.tmp.appcompat.txt
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER8022.tmp.WERInternalMetadata.xml
WERGenerationLog.txt

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_9d888ac04d24e4ae9ba23c5f7762b4ddf95248f_ed19be39_cab_0b048031

Analysis symbol:
Rechecking for solution: 0
Report Id: 9693a536-e15e-4c29-bb23-eee06a89815d
Report Status: 131076
Hashed bucket:

Entry 4

Fault bucket 2071344263271118212, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: w3wp.exe
P2: 10.0.14393.0
P3: 57899b8a
P4: coreclr.dll
P5: 4.6.27110.4
P6: 5be75633
P7: c0000005
P8: 00000000001bd149
P9:
P10:

Attached files:
\?\C:\Windows\Temp\WER7FC4.tmp.appcompat.txt
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER8022.tmp.WERInternalMetadata.xml
WERGenerationLog.txt

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_w3wp.exe_9d888ac04d24e4ae9ba23c5f7762b4ddf95248f_ed19be39_11f88503

Analysis symbol:
Rechecking for solution: 0
Report Id: 9693a536-e15e-4c29-bb23-eee06a89815d
Report Status: 0
Hashed bucket: ba850d50d8e1d2fd9cbee4a20bd17584

Now after 4th entry log, I see the specific application pool is restarted. This entire process happens 3 times in total - in about 2-3 minutes time total. In this particular case, 8 hours pass and we see a similar process repeating again.

To Reproduce

Steps to reproduce the behavior:
Using ASP.NET Core 2.2 on Windows 2016 Server.
Running InProcess
Site runs normally but suddenly we get this error - don't know what code triggers the error.
From web.config:

<configuration> <system.webServer> <handlers> <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" requireAccess="Script" /> </handlers> <aspNetCore processPath=".\myproject.exe" arguments="exec &quot;C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll&quot;" stdoutLogEnabled="true" stdoutLogFile=".\logs\log" hostingModel="InProcess"> <handlerSettings> <handlerSetting name="debugLevel" value="file" /> <handlerSetting name="debugFile" value=".\logs\ancm.log" /> </handlerSettings> <environmentVariables> <environmentVariable name="ASPNETCORE_HTTPS_PORT" value="443" /> <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Production" /> </environmentVariables> </aspNetCore>

We have a bunch of these errors in the stdout log - but they have been there in ASP.NET Core 2.1.x as well:

warn: Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery[8] The 'Cache-Control' and 'Pragma' headers have been overridden and set to 'no-cache, no-store' and 'no-cache' respectively to prevent caching of this response. Any response that uses antiforgery should not be cached.

FROM ANCM log

[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'C:\inetpub\sites\www.myproject.com'
[aspnetcorev2_inprocess.dll] Starting file watcher thread
[aspnetcorev2.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 2480.. File Version: 12.2.18316.0. Description: IIS ASP.NET Core Module V2. Commit: ce8cf65.
[aspnetcorev2.dll] Resolving hostfxr parameters for application: '.\myproject.exe' arguments: 'exec "C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll"' path: 'C:\inetpub\sites\www.myproject.com'
[aspnetcorev2.dll] Known dotnet.exe location: ''
[aspnetcorev2.dll] Process path '.\myproject.exe' is not dotnet, treating application as standalone or portable with bootstrapper
[aspnetcorev2.dll] Checking application.dll at 'C:\inetpub\sites\www.myproject.com.\myproject.dll'
[aspnetcorev2.dll] Checking hostfxr.dll at 'C:\inetpub\sites\www.myproject.com.\hostfxr.dll'
[aspnetcorev2.dll] hostfxr.dll found app local at 'C:\inetpub\sites\www.myproject.com.\hostfxr.dll', treating application as standalone
[aspnetcorev2.dll] Parsed hostfxr options: dotnet location: '' hostfxr path: 'C:\inetpub\sites\www.myproject.com.\hostfxr.dll' arguments:
[aspnetcorev2.dll] Argument[0] = 'C:\inetpub\sites\www.myproject.com.\myproject.exe'
[aspnetcorev2.dll] Argument[1] = 'exec'
[aspnetcorev2.dll] Argument[2] = 'C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll'
[aspnetcorev2.dll] c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020
[aspnetcorev2.dll] Loading request handler: 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.0\aspnetcorev2_inprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_inprocess.dll] Initializing logs for 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.0\aspnetcorev2_inprocess.dll'. Process Id: 2480.. File Version: 12.2.18316.0. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: ce8cf65.
[aspnetcorev2_inprocess.dll] Waiting for initialization
[aspnetcorev2_inprocess.dll] Starting in-process worker thread
[aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: '.\myproject.exe' arguments: 'exec "C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll"' path: 'C:\inetpub\sites\www.myproject.com'
[aspnetcorev2_inprocess.dll] Known dotnet.exe location: ''
[aspnetcorev2_inprocess.dll] Process path '.\myproject.exe' is not dotnet, treating application as standalone or portable with bootstrapper
[aspnetcorev2_inprocess.dll] Checking application.dll at 'C:\inetpub\sites\www.myproject.com.\myproject.dll'
[aspnetcorev2_inprocess.dll] Checking hostfxr.dll at 'C:\inetpub\sites\www.myproject.com.\hostfxr.dll'
[aspnetcorev2_inprocess.dll] hostfxr.dll found app local at 'C:\inetpub\sites\www.myproject.com.\hostfxr.dll', treating application as standalone
[aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: '' hostfxr path: 'C:\inetpub\sites\www.myproject.com.\hostfxr.dll' arguments:
[aspnetcorev2_inprocess.dll] Argument[0] = 'C:\inetpub\sites\www.myproject.com.\myproject.exe'
[aspnetcorev2_inprocess.dll] Argument[1] = 'exec'
[aspnetcorev2_inprocess.dll] Argument[2] = 'C:\Web\myproject\myproject\bin\Release\netcoreapp2.2\myproject.dll'
[aspnetcorev2_inprocess.dll] In-process callbacks set
[aspnetcorev2_inprocess.dll] Event Log: 'Application 'C:\inetpub\sites\www.myproject.com' started the coreclr in-process successfully.'
End Event Log Message.

Attached is WER file I managed to find.
Report.wer.log

We are using the temp bug fix for getting CurrentDirectory (see #4206 (comment)) in Program.cs

using Microsoft.AspNetCore; using Microsoft.AspNetCore.Hosting; using myproject.Helpers; namespace myproject { public static class Program { public static void Main(string[] args) { CurrentDirectoryHelpers.SetCurrentDirectory(); // asp.net core 2.2 hack. In 3.0 this will be fixed: https://github.com/aspnet/AspNetCore/issues/4206#issuecomment-445612167 CreateWebHostBuilder(args).Build().Run(); } public static IWebHostBuilder CreateWebHostBuilder(string[] args) => WebHost.CreateDefaultBuilder(args) .UseStartup<Startup>(); } }

Additional context

dotnet --info

.NET Core SDK (reflecting any global.json):
Version: 2.2.100
Commit: b9f2fa0ca8

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.2.100\

Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed

.NET Core SDKs installed:
2.1.300 [C:\Program Files\dotnet\sdk]
2.1.301 [C:\Program Files\dotnet\sdk]
2.2.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]