Data Collection

Understanding Data Collection

Data collection is the process of obtaining observations, together with enough context to interpret and reproduce them. Loading a table is only one step. Before choosing a source, define the question, the population of interest, and the unit of observation: what does one row represent? For a library studying borrowing activity, one row might represent a loan, not a book or a borrower. Mixing these units can produce misleading counts even when every value is valid.

Sources offer different strengths and constraints. A database query can select relevant records, but its result depends on filters, joins, and when it runs. An API provides structured access, yet pagination or request limits can leave a collection incomplete. CSV and Excel exports are convenient snapshots, but column meanings and formats may change. Web scraping depends on page structure and requires checking permission and applicable terms. Sensors and logs capture events continuously, but outages, clock differences, and repeated delivery complicate interpretation.

A collection plan should describe provenance: the source, owner, extraction method, collection time, covered period, license or authorization, and refresh cadence. Collection time is not necessarily event time. A Monday export may contain Sunday's activity. Record units and time zones too. Collect only necessary personal information, restrict access appropriately, and set retention rules. Preserving raw observations supports reproducibility, but does not justify keeping sensitive information indefinitely or without authorization.

Separate acquisition from downstream cleaning. Retain an authorized raw snapshot, then validate a working representation against an explicit schema: expected columns, types, allowed ranges, and identifier rules. The pandas user guide treats input/output, missing data, and duplicate handling as separate topics; that separation helps distinguish successfully parsing data from establishing its quality. A readable table can still contain missing measurements, repeated events, or incomplete coverage.

Finally, collection determines what conclusions are defensible. Library app records exclude loans recorded only on paper; collecting more app records does not fix that coverage gap. Faster refreshes improve timeliness but increase operational cost and may capture unfinished records. Validation detects specified problems, not every possible error. Document known exclusions and unresolved issues so later analysis does not silently treat an accessible dataset as a complete population.

Practical walkthrough

Imagine receiving a daily library loan export. Your immediate goal is not to calculate borrowing trends; it is to determine whether the batch can support that calculation. Assume one row represents one loan, loan identifiers should be unique, and duration is measured in whole days. A blank duration means unknown, not zero.

First, write a small source record. Include the export's origin, permission, covered date, and expected update schedule. In the example, all observations are invented, and the metadata explicitly identifies an in-memory teaching fixture rather than pretending that a real export was collected. Real collection records should also preserve query filters or API parameters without exposing credentials.

Next, parse the embedded CSV text. Prerequisites are Python 3.11 or newer and pandas already installed; the example uses APIs available in pandas 2.2 and 3.0. It needs no files, network connection, or additional datasets. Explicit string types preserve leading zeros in identifiers, while pandas' nullable integer type allows a missing duration without converting the column to ordinary floating-point numbers.

Then inspect the batch without editing it. Check column names, count missing durations, and count repeated identifiers after their first occurrence. Print compact scalar results rather than relying on version-dependent table formatting. Keep the original CSV string unchanged so the checks can be repeated.

Finally, interpret the findings before choosing a cleaning policy. A repeated identifier could indicate retransmission or a conflicting update. Investigate rather than automatically keeping the latest row. Likewise, request the missing duration or document its absence. This batch remains flagged for review.

Worked example

from io import StringIO
import pandas as pd

source = {
    "origin": "synthetic in-memory library export",
    "permission": "invented teaching data; no personal records",
    "covered_date": "2026-01-05",
    "refresh": "daily",
}

raw_csv = """loan_id,branch,duration_days
001,North,14
002,South,
002,South,7
003,North,21
"""

batch = pd.read_csv(
    StringIO(raw_csv),
    dtype={"loan_id": "string", "branch": "string", "duration_days": "Int64"},
)

expected = ["loan_id", "branch", "duration_days"]
if batch.columns.tolist() != expected:
    raise ValueError("Unexpected columns")

missing = int(batch["duration_days"].isna().sum())
repeated = int(batch["loan_id"].duplicated().sum())
status = "REVIEW" if missing or repeated else "PASS"

print(f"Source: {source['origin']}")
print(f"Refresh: {source['refresh']}")
print(f"Rows received: {len(batch)}")
print(f"First loan ID: {batch.loc[0, 'loan_id']}")
print(f"Missing durations: {missing}")
print(f"Repeated IDs after first occurrence: {repeated}")
print(f"Batch status: {status}")

Expected output

Source: synthetic in-memory library export
Refresh: daily
Rows received: 4
First loan ID: 001
Missing durations: 1
Repeated IDs after first occurrence: 1
Batch status: REVIEW

StringIO presents the embedded text as an in-memory input for pandas; it does not open a disk file. Four rows are parsed, and the explicit string type keeps the first identifier as 001. The empty duration becomes a missing value, so isna counts one absence. Identifier 002 occurs twice; duplicated marks only its second occurrence by default, producing one repeated identifier. Because either issue triggers review, the final status is REVIEW. No rows are removed or repaired. These checks intentionally cover only selected requirements: they do not establish complete coverage, valid branch names, or acceptable duration ranges for actual loans.

Common mistakes and how to avoid them

• Treating a successful import as proof that collection is complete.

  Check expected coverage dates, API pagination, and available source totals. Separate parser success from completeness checks; four readable rows do not prove that every loan arrived.

• Replacing missing measurements with zero during collection.

  Preserve missingness and document its meaning. Zero days is an observed duration, whereas an empty field is unknown. Any later imputation needs a justified, recorded policy.

• Removing repeated identifiers without examining their meaning.

  Inspect all records sharing the identifier. Determine whether they are retransmissions, corrections, or evidence that the proposed key is wrong before applying a documented resolution rule.

• Assuming publicly accessible data is unrestricted.

  Verify license terms, authorization, and privacy requirements before collection. Limit fields and access, and reconcile raw-data retention with applicable deletion requirements.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide