reported_vs_observed_activity

Executive Summary

An alternative dataset is suitable for historical analysis when researchers can reconstruct not only what the data say about the past, but what information actually existed at each historical point in time.

That requires more than a long archive. Useful historical data need sufficient coverage, stable definitions, traceable provenance, timestamps that distinguish events from availability, and a way to understand revisions, backfills and changes in methodology. The dataset must also contain enough observations to evaluate the economic question being asked.

A dataset can be accurate today yet unsuitable for historical decision research if its past has been rewritten using information that became available later.

The central test is therefore reproducibility: could a researcher recreate the historical information state without allowing the future to leak backward?

Key Takeaways

• Historical depth alone does not make an alternative dataset research-ready.

• Point-in-time availability, provenance and version history are critical when testing historical signals.

• Coverage changes can create false trends even when individual observations are accurate.

• Survivorship bias can distort company, asset and infrastructure universes.

• The dataset must fit the economic mechanism and time horizon being studied.

Historical Data and Historical Information Are Not the Same Thing

A database may contain ten years of records and still fail to represent what was knowable during those ten years.

Consider a facility database assembled in 2026. It may correctly list factories that existed in 2020, assign them historical attributes and attach archived satellite observations to those sites.

That can be excellent for reconstructing what physically happened.

But if the facility identities, ownership relationships or classifications were established only in 2026, the dataset does not automatically recreate the information environment of 2020.

This distinction separates historical reconstruction from historical decision analysis.

1. Point-in-Time Availability

The first requirement is knowing when information became available.

Researchers should distinguish:

• Event time: when the underlying event occurred.

• Observation time: when it was measured.

• Processing time: when a derived value was produced.

• Availability time: when a user could actually access it.

• Revision time: when it was later changed.

If a historical test uses information before its availability time, look-ahead bias enters the experiment.

2. Stable and Understandable Coverage

Alternative datasets often improve over time.

More satellites become available. AIS reception improves. New facilities are added. Classification models become better. Geographic coverage expands.

Those improvements are valuable operationally but can create misleading historical patterns.

Suppose a dataset records twice as many industrial events in 2026 as in 2022. That might mean industrial activity doubled.

It might also mean the observation system became twice as capable of detecting events.

Historical analysis therefore needs to distinguish changes in the world from changes in the measurement system.

3. Provenance

Researchers should be able to understand where observations came from and how they were transformed.

Useful provenance can include:

• source identifier,

• collection date,

• processing version,

• geographic source,

• classification method,

• revision history.

Without provenance, a historical value may be impossible to audit when results later look surprising.

4. Consistent Definitions

A variable should mean approximately the same thing across the period being analyzed, or changes in definition should be explicitly documented.

Imagine that “port call” was defined one way until 2023 and another way afterward.

A sudden increase in port calls could reflect the new definition rather than a change in maritime activity.

The same issue appears in facility classifications, company mappings, geospatial boundaries and derived activity measures.

5. Version History and Backfills

Alternative datasets are frequently backfilled.

A provider may discover additional historical vessels, improve a satellite classifier or add facilities that existed for years before entering the database.

Backfills are not inherently problematic.

They become problematic when researchers cannot tell that they happened.

For historical reconstruction, the improved series may be preferable.

For point-in-time simulation, the original historical version may be necessary.

6. Survivorship

Historical universes should include entities that later disappeared.

If a current company universe is projected backward, failed, acquired or delisted companies can vanish from the analysis.

The same problem can affect physical assets.

Closed facilities may disappear from current databases even though they were economically relevant during the historical period.

This creates survivorship bias: the past is reconstructed using only entities that survived into the present.

7. Sufficient Observation Density

A dataset can be point-in-time clean and still lack enough observations to answer the research question.

Satellite revisit frequency, cloud cover, AIS gaps and sparse historical archives can reduce usable sample size.

The relevant quantity is not simply the number of database rows.

It is the number of independent, usable observations that satisfy the experiment's requirements.

8. An Appropriate Baseline

Many real-world signals are defined relative to normal conditions.

That requires a historical baseline.

But the baseline itself must obey the information boundary.

A signal generated in 2022 should not use a “normal” range calculated from observations extending into 2025.

Otherwise the future has entered through the reference distribution rather than the current observation.

9. Fit to the Economic Mechanism

Data quality is question-dependent.

Daily vessel positions may be excellent for studying short-term maritime disruption but poorly suited to evaluating a multi-year factory construction cycle.

Quarterly imagery may capture major infrastructure expansion while missing short-lived operational changes.

A dataset should therefore be evaluated against:

• the physical process,

• the expected transmission mechanism,

• the relevant time horizon,

• the outcome used for validation.

A Practical Historical-Readiness Test

Before beginning analysis, ask:

• Can we determine what information existed on each historical date?

• Can we identify revisions and backfills?

• Has coverage changed materially?

• Are definitions stable?

• Does the universe include entities that later disappeared?

• Can historical baselines be calculated without future data?

• Is the usable sample large enough?

• Does the observation frequency match the mechanism?

• Can another researcher reproduce the historical information state?

A “no” does not always disqualify the dataset.

It changes which claims the dataset can support.

What Can Be Observed?

Dataset metadata can directly reveal coverage periods, timestamps, source lineage, missingness and documented revisions.

These properties describe the measurement system itself.

What Can Reasonably Be Inferred?

If historical versions, availability timestamps and stable definitions are preserved, researchers can more credibly simulate how the dataset would have behaved in earlier information environments.

If those properties are missing, the dataset may still support descriptive historical research, but stronger point-in-time claims should be limited.

What Cannot Be Concluded From a Long History?

A long archive does not by itself establish:

• point-in-time integrity,

• absence of survivorship bias,

• stable measurement quality,

• economic relevance,

• predictive value.

Ten years of contaminated history can be less useful than three years of well-documented history.

Space Sat Lab Perspective

Space Sat Lab treats historical suitability as an empirical property rather than a marketing description attached to a dataset.

For physical-world signals, the research question is not only whether the past can be reconstructed today. It is whether the observation process, historical baseline and information state can be reproduced without future knowledge.

That standard is essential when moving from interesting historical imagery or activity records toward defensible signal validation.

Frequently Asked Questions

How much history does an alternative dataset need?

There is no universal minimum. The required history depends on signal frequency, economic mechanism, structural regimes and the statistical question being tested.

What is point-in-time data?

Point-in-time data preserve what was known or available at a specific historical date rather than silently replacing it with later revisions or backfills.

Are backfilled data useless for historical analysis?

No. They can be excellent for reconstructing what happened. They are more problematic when used to simulate what an analyst could have known contemporaneously.

Why does survivorship bias matter?

Removing failed or disappeared entities can make historical relationships look stronger because the analysis sees an artificially successful universe.

Does research-ready data guarantee a useful signal?

No. It creates a cleaner experiment. The economic hypothesis can still fail, and that failure is itself useful evidence.

Sources and Further Reading

CFA Institute, Backtesting and Simulation

CFA Institute, The Good, Bad and Ugly of Bias in AI