Troubleshooting
Recover a historian dump loop after a service restart
Short answer
Deleting the dump files does not stop the loop. Preserve a representative set, measure the growth rate to know how many hours of C: drive remain, restart the whole server as the demonstrated temporary recovery, and characterize the shutdown-versus-continuous failure on build 23.1.001.000 through support — the Historian 2017 hotfix does not apply.
On this page
- What does the Event Storage unknown exception message mean?
- Why do SQL dump files appear every second after the service restart?
- How long before the dump loop fills the C: drive?
- Is deleting the dump files a fix?
- Why does restarting the whole server work when restarting the service does not?
- Does the Historian 2017 hotfix apply to build 23.1.001.000?
- How do I test whether SQL queries trigger the dump loop?
- What evidence does technical support need?
- Questions asked
A dump file every second after a historian service restart is a disk-life problem first and a fault diagnosis second, and deleting the files solves neither. In the report behind this article, an upgrade to build 23.1.001.000 was followed by the message "Historian Event Storage: Unknown exception caught [HistorianServerName; EventStorageDispatcher.cpp; 650]" on every stop-and-restart of the historian service, together with SQL dump files appearing each second and consuming C:. Restarting the entire server restored normal operation in that installation. Work the problem in that order: preserve dumps, compute the hours of disk remaining, restart the server to recover, then characterize the failure for support.
What does the Event Storage unknown exception message mean? #
It identifies an unclassified exception caught inside the historian's Event Storage dispatcher, reported from source file EventStorageDispatcher.cpp at line 650 — a code location, not a tag name, an address, or a setting you can change.
In the message, HistorianServerName is simply where the server's own name appears; the semicolon-separated bracket carries the reporting component, the source file, and the line. "Unknown exception caught" is the handler saying it could not classify what it caught, so the text alone cannot separate three different stories: a shutdown that failed to complete, an initialization that failed on restart, or a runtime fault that re-fires continuously. Which one you have decides everything downstream, and timestamps answer it.
Why do SQL dump files appear every second after the service restart? #
Because the dump is written each time the failing operation repeats, and in the reported loop the exception re-fires every second — so a per-second file stream is a continuous runtime fault, not a messy one-time shutdown.
The first question worth answering is the one support will ask: does the fault appear only at shutdown, or continuously? Microsoft's troubleshooting guidance for SQL Server dumps generated almost every second puts a product defect first and malfunctioning hardware second, with a support case as the resolution path. Before touching anything at the control level, confirm which process writes the files: a historian that cannot store shows gaps on every client while the controllers' own diagnostics stay clean — the same layer-first habit you use when you trace a SQL log error after a historian migration.
How long before the dump loop fills the C: drive? #
Divide free bytes by the measured growth rate: with 216 GB free and dumps averaging 8 MB at one per second, the drive reaches capacity in 7.5 hours, and halving the free space halves that time.
Two measurements, then arithmetic:
- growth rate = total new dump bytes ÷ observation window
- hours remaining = free bytes on C: ÷ growth rate
Worked example with illustrative numbers — measure your own. Over a 300 s window you count 300 new dump files totaling 2,400 MB. Growth rate = 2,400 MB ÷ 300 s = 8 MB/s, which is 8 MB/s × 3,600 s/h = 28,800 MB/h ≈ 28.8 GB/h. With 216,000 MB free on C:, the remaining time is 216,000 MB ÷ 8 MB/s = 27,000 s = 7.5 h. Use the units your disk tool reports; mixing binary and decimal units moves the answer a few percent, far less than dump-size variation does. The decision: with hours of margin you have time to preserve dumps and characterize the fault, and if the same arithmetic returns less than an hour, recover first and collect diagnostics on the next occurrence.
Is deleting the dump files a fix? #
No — deletion reclaims space but changes nothing in the process writing them, so a one-per-second loop refills the space in minutes; preserve a representative set first, then delete only as space management while the fault itself is characterized.
Copy the first file, one from the middle of the loop, and the newest to another volume with their timestamps before any cleanup. Dumps are diagnostic artifacts, safe to remove once preserved; do not confuse them with history block files, which hold logged data and should be moved to another directory rather than deleted when space is tight.
Why does restarting the whole server work when restarting the service does not? #
Because the two operations reset different things: a service stop and start exercises only the historian's own shutdown and initialization while the SQL engine processes and client sessions stay live, whereas a Windows restart clears every process and connection on the machine.
That is why the full restart restored service in the reported installation, and why it is a workaround rather than a fix — nothing about the reset identifies the failing component. One caution: a server restart drops every client session and every other service on that box, so run it in a window where a few minutes of lost history collection is acceptable.
| Approach | Observed result or purpose | Decision |
|---|---|---|
| Stop, disable, then restart the historian service | Triggers the exception and the per-second dump stream in the report | Do not repeat while the loop is active |
| Restart the entire server | Restored normal operation in the reported installation | Temporary recovery, once diagnostics are preserved |
| Pause querying clients, then one controlled restart | Tests whether the query workload contributes to the transition failure | Diagnostic comparison, not a fix |
| Support case on build 23.1.001.000 | Targets a release-specific correction | The permanent route |
Note the difference inside the trigger itself: stopping a running service ends the process, while disabling startup changes what happens at the next boot. Record them as separate actions, because a reproduction that mixes them cannot be diagnosed.
Does the Historian 2017 hotfix apply to build 23.1.001.000? #
No — the corrections and hotfix from the 2017 release line were built for that line, and an identical-looking message on 23.1.001.000 has no confirmed correction, so the permanent fix is a support case against your exact build.
Similar errors were reported against Historian 2017 and later corrected, with a hotfix issued for that release. Matching message text is not evidence of a matching defect or a compatible binary; fixes ship per release, and applying a 2017 package to a 23.x installation is unsupported and would overwrite the evidence trail support needs.
How do I test whether SQL queries trigger the dump loop? #
Pause the querying clients for one controlled service restart: if the exception and dump stream stay away with clients paused and return when they reconnect, you have narrowed the reproduction for support — a test outcome, not a demonstrated fix.
The hypothesis worth testing is a client hammering the historian across the shutdown window, because sessions that lose the connection mid-query retry and come back in a burst — the same behavior you manage when you reconnect an edge PC to SQL Server after connection loss.
- Record the build, the service startup configuration, free bytes on C:, and the start time; capture the exact exception text.
- List the clients that query continuously or reconnect automatically — reporting applications and automated consumers, not just operator displays.
- In a maintenance window, pause those clients' queries without changing tag configuration or driver settings, and note which ones were paused.
- Run exactly one stop/start sequence, with stop and disable recorded separately and the intended startup configuration written down.
- Mark whether the exception lands during the stop, the start, or steady operation, and whether dumps resume.
- If the loop resumes, stop testing, preserve the new diagnostics, and recover with the server restart.
A different outcome with clients paused gives support a narrow reproduction case; the same outcome tells them the workload is not the trigger.
What evidence does technical support need? #
A complete characterization of one incident: the build number, the exact stop, disable and start sequence, whether dumps appear only at shutdown or continuously, the query workload at the time, and preserved logs and dump files with their directory path and generating process.
- Build and configuration: 23.1.001.000, the service startup mode, and the services involved.
- Timeline: first exception timestamp, dump start and stop times, and the restart moment.
- Files: dump directory path, file count and sizes over the measurement window, preserved copies.
- Correlated logs: historian logs, SQL error logs, and Windows application events from the same interval.
- Workload: which clients were active or paused, and their reconnect behavior.
The decision set, then: loop active — measure the rate, preserve the dumps, restart the server. Root cause — one paused-client test, then support with the timeline. Permanent fix — a correction confirmed for 23.1.001.000, never the 2017 hotfix.
Questions asked
What is actually inside a SQL dump file?
A memory snapshot of the failing process at the moment of the exception: the call stack, loaded modules, and memory regions a support engineer reads to find the failing code path. One dump per second means the exception re-fires every second, which is why support asks for several files from the same interval rather than a single file.
How many dump files should I keep before cleaning up?
Keep a spanning set: the first file, one from the middle of the loop, and the most recent, copied to another volume with their timestamps recorded. Three files prove the repeating pattern; keep more if sizes vary widely, because an oversized dump can point to a different failure mode than a steady stream of identical ones.
Does a full C: drive damage the databases themselves?
A full volume does not corrupt data on its own, but it refuses new writes, which stops transaction logs from growing and can fail other services sharing the drive. That is why recovery order matters: copy the dumps off the volume, free space, then restart, rather than letting the loop run down to zero.
Should I upgrade the historian to a newer release during the incident?
Not as the incident response. Stabilize with the server restart first, then let support confirm whether a corrected release for this exact failure exists in your line; the 2017 corrections say nothing about 23.1.001.000. An upgrade mid-loop adds a second variable to a failure you have not yet characterized.
Operator trends show gaps but live values still update. Is the controller failing?
No. Live values updating while history grows gaps points at the storage side, because the same controller data is reaching clients in real time. A controller or communications failure shows up in that equipment's own diagnostics and blanks the live values too, so treat the gaps as a symptom of the server-side fault.