Troubleshooting
Why historian backfill runs slow from Tier 1 to Tier 2
Short answer
The 30-hour test moved 14 hours of history for 10 tags, under half real time, so the 2014-era backfill utility is the bottleneck, not the data. Verify the source history and current replication first, then fill gaps with bounded requests or a manufacturer-approved archive block copy. The SQL database export moves configuration, not samples: drop it.
On this page
A backfill that needs 30 hours to move 14 hours of history for 10 tags is replaying the past at under half its real-time rate, and that number points at the backfill utility on the 2014-era source, not at the data. Confirm the source history exists, confirm current replication delivers new samples to the Tier 2, then fill gaps with bounded backfill requests or a manufacturer-approved archive block copy. The SQL database export is the one path to drop: it moves the historian's configuration layer, not its stored samples.
The measured rate is the diagnosis #
The test bed gives three numbers: 10 tags, roughly 14 hours of stored history, and about 30 hours of wall-clock run time with aahBackFillUI. Divide history moved by time spent:
14 h of history ÷ 30 h of run time ≈ 0.47, so the utility replays history at less than half the speed the plant originally recorded it. Any healthy transfer path beats that, which places the failure in the utility, not in the tag count or the network.
Extrapolate that measured rate to a whole archive. These inputs are illustrative, not measured: assume eight years of history for the same 10 tags, which is 8 × 365 days of samples. 8 × 365 × 24 = 70,080 h of history. At 14 h per 30 h of run time: 70,080 × 30 ÷ 14 ≈ 150,171 h ≈ 6,257 days ≈ 17 years of wall-clock time. The extrapolation assumes run time scales with records moved, which one test point does not prove, but even a path ten times faster than this finishes in years. Decision: the installed utility is a spot-repair tool for small gaps, not the migration plan.
Four things move that rate:
- Record count in the interval: log-on-change and cyclic tags hold far more samples per hour than an interpolated read suggests.
- Retrieval settings on the request: raw versus interpolated retrieval changes the work per query.
- Destination ingestion: the Tier 2 write path and its buffering set a ceiling independent of the source.
- Utility generation: the 2023 release is reported to have enhanced this path, while the 2014-era utility carries the limitations you are measuring.
Prove the history exists at the source #
Query one known tag directly on the 2014 historian over a fixed interval. Record the timestamps, values and quality returned, and the retrieval settings used; that query result is the reference for everything downstream. This is the same source-first discipline as backfilling trend history from a plant historian: no transfer request can be judged until the source interval is proven non-empty. Then query the destination tag on the 2023 historian for the same interval with equivalent settings, and compare that result, not the operator trend, against the source.
| What the two direct queries show | Meaning | Action |
|---|---|---|
| Source returns nothing | Wrong tag, wrong interval, or the history was never logged | Fix the selection before requesting any transfer |
| Source has data, destination does not | Transfer problem: replication scope, backfill progress or a failed request | Continue to the replication check |
| Destination has data, the trend is blank | Display path: server connection, tag binding, displayed range | Fix the trend; rerunning backfill changes nothing |
| Both return data at different timestamps | Time-zone or retrieval interpretation differs | Reconcile the settings before judging completeness |
Prove current replication before any backfill #
In AVEVA Historian's documented tiering model, each tier replicates to the next: the tier-1 historian feeds tier 2, and a tier-2 historian feeds tier 3. The tier-2 replication and summary configuration is the authority on source, destination and replicated tag set, so read it before touching the historical request.
Simple replication forwards samples from the moment it is enabled, plus the summaries configured for it; it does not sweep the history that predates it. That is why a backfill utility exists at all, and why a stalled backfill does not by itself mean replication is broken: they are separate paths.
Test the live path. Pick a recent, identifiable source change without disturbing the process, and confirm the matching sample lands under the intended destination tag. Verify the tag's identity, not a similar display name. Then review replication diagnostics for connection, authentication, tag-selection and destination-write errors. Every configuration reference to a server name matters after the rename: a copied or restored configuration does not correct those references by itself.
Decision: if recent data is missing too, repair replication first, because backfill submitted over a broken path is wasted work. If recent data flows and only older history is missing, the remaining problem is the historical request.
Run bounded requests, not one open-ended backfill #
The retrieval side of that historian generation is a 32-bit process with 2 GB of addressable memory, and documented guidance for bulk backfill on it warns that one query spanning too many records exhausts that space; the stated fix is to find an acceptable step and work in steps. Bound every request:
- Take recoverable backups of both historians and export the current replication configuration.
- Choose one tag with known history and one day-sized interval.
- Run one approved request; capture its scope, start time, progress messages and completion or error state.
- Query the destination over the same interval with the same retrieval settings and compare it against the source result.
- Extend the interval or tag set only after a chunk completes clean; a stalled small chunk is a reproducible case for support.
Keep the query window fixed while checking progress, because sliding the trend hides samples that have just arrived. Smaller, manually triggered replication chunks are a support-assisted option on this platform: obtain the documented initiation procedure from the manufacturer instead of improvising commands against the replication service. Open the ticket now; the measured rate is exactly the evidence it needs, and the same ticket is how the block-copy procedure below is obtained.
Why the SQL export moves no history #
A historian of this type keeps its configuration in a SQL database and its time-series samples in archive block files. Exporting the 2014 database therefore moves tag definitions, identities and configuration state, not the samples, which live in the blocks the database does not contain. Importing that database onto the Tier 2 also carries references to the old server names into a server whose own configuration is already valid, with nothing to reconcile them. If something in that database must be inspected, restore it to an isolated recovery instance and read it there, never onto the live Tier 2.
The bulk path: archive block copy on procedure #
Copying history blocks from the Tier 1 directly is the faster route for a multi-year archive, and it is the operation most capable of silently ruining a destination when improvised. Archive format compatibility across the 2014-to-2023 pair, tag identity, registration of the copied blocks on the destination, and the handling of intervals where the Tier 2 already holds data are all settled by a manufacturer-approved procedure with its own backup requirements. Get that procedure through the support ticket before any block touches the Tier 2, and treat the overlap rule it states as binding.
The decision for each case #
- Recent data missing on the Tier 2: repair replication first; no backfill until a fresh source sample lands under the destination tag.
- Recent data flowing, only older history missing: bounded requests for gaps of hours to days; block copy on the approved procedure for the full archive.
- Destination holds data but the trend is blank: fix the trend's server connection, tag binding and time range; the transfer is not the fault.
- Full multi-year archive at the measured rate: block copy via the ticket; keep the installed utility for spot gaps only.
- SQL export proposed as the transfer: no, because it moves configuration rather than samples and puts the Tier 2's own configuration at risk.
Questions asked
Can aahBackFillUI be run from the Tier 2 server instead of the Tier 1?
Each historian generation ships the utility version tested against its own sources, and the known-slow configuration in this failure is the 2014-era utility on the Tier 1. Reports credit the 2023 release with improvements, but that does not establish it can read a 2014 source from a Tier 2 host. Before any full-scale rerun, confirm the supported pairing of utility version and execution host for a 2014-to-2023 pair with the manufacturer, and record both in the ticket.
Does a large backfill affect live collection while it runs?
Yes, on the destination side. Backfill replays stored samples at a rate far higher than the control system originally produced them, so the Tier 2 ingests hours or years of data in a fraction of that time, and its buffering, disk and CPU carry the load. Schedule large chunks outside critical reporting windows, watch the destination resource counters during the first chunk, and shrink the interval if ingestion lags.
What should the support ticket contain to get a usable answer?
The utility name and version with its execution host; the tag list and requested interval; elapsed time and the completion or error state of the test run; destination tag names as configured; replication status for current data; and per-host CPU, disk and network observations during a run. Add the two direct query results proving the history exists at the source and showing what the destination holds. That set lets support reproduce the slow path immediately.
What if the Tier 2 already holds some history in the interval being backfilled?
Resolve the overlap before transferring. Until the manufacturer procedure states the rule, bound every request so it stops where destination data begins, and record the boundary timestamps. Overlapping samples are either duplicated or rejected depending on destination handling, and an archive block copied over an interval that already holds data needs the procedure's explicit overlap rule. Never guess that rule on a live Tier 2.