Data Science · Chapter 35 of 43
Time Series Basics
TIME SERIES data is indexed by time (daily sales, hourly temperature). Order matters — splits must be chronological.
Common tasks: forecasting, anomaly detection, trend/seasonality decomposition.
Example 1 (python)
import pandas as pd
df = pd.read_csv('sales.csv', parse_dates=['date'], index_col='date')
print(df.resample('M').sum().head())
Monthly totals.
Example 2 (python)
df['rolling_7'] = df['sales'].rolling(7).mean()
7-day moving average.
Key points
- Order matters — split chronologically.
- Common patterns: trend + seasonality.
- Resampling changes frequency.
- Rolling stats smooth noise.
💡 Note: Never randomly shuffle time-series data — you'll leak the future into the past and get unrealistically great scores.
Understanding Time Series Basics
A time series records a measurement together with when it occurred. Daily sales, hourly temperatures, and monthly account balances are examples. Time is not merely a label: neighboring observations often resemble one another, and the spacing between observations affects interpretation. Before calculating anything, define what each value measures, its units, its reporting interval, and when it becomes available.
Distinguish three useful patterns. A trend is a sustained change in level over time. Seasonality is a recurring pattern tied to a regular period, such as differences between weekdays and weekends. Irregular variation is what these descriptions do not explain. These are interpretive categories, not proof of underlying causes. A short upward sequence does not establish a durable trend, and identifying weekly seasonality requires observing multiple weeks.
A trustworthy time index needs parsed timestamps, chronological ordering, and checks for duplicates and gaps. Duplicate dates might represent separate transactions that should be aggregated, or accidental repeated records. Missing dates are not automatically zero sales: they could indicate a closed shop or a failed data feed. For hourly observations, time zones and daylight-saving changes can also alter what a calendar day contains.
Resampling groups observations into time bins and summarizes each bin. The aggregation must match the measurement: sum daily units sold, average regularly sampled temperatures, or take the last recorded account balance. In current pandas, `resample('ME')` forms calendar-month bins labeled at month end. A partial month's total is still partial, regardless of its label. Rolling calculations instead use overlapping windows, retaining a local summary at successive timestamps. They smooth variation but can hide spikes and respond slowly to changes.
Forecasting adds an information boundary: predict using only what was available at the forecast time. Evaluate on later observations, not randomly mixed past and future rows. Fit any learned scaling or imputation on training data only. Chronological ordering alone does not prevent leakage if a feature includes future values. Start with a simple baseline, such as repeating the latest observed value, and state the forecast horizon. Its errors describe performance for the evaluated dates, not a guarantee about future periods or evidence that one variable causes another.
Practical walkthrough
Begin with a concrete question: after January 6 closes, what sales would we predict for January 7 and 8? Assume each observation is a complete day's unit sales, available immediately after closing. The example uses eight invented observations, no missing dates, and one shop. It requires pandas but reads no files.
Create a Series with a daily DatetimeIndex and sort it. With real records, inspect timestamp parsing, duplicate dates, missing values, and expected frequency before proceeding. An integer rolling window counts observations: `rolling(3)` means three rows, not necessarily three calendar days. Here they coincide because the daily index has no gaps.
Calculate a monthly sum for descriptive reporting. The eight observations fall into one January bin, but they do not cover the whole month. Keep this retrospective summary separate from forecasting: it includes observations unavailable on January 6. Next, calculate the trailing three-observation mean with `min_periods=3`, leaving the first two results undefined rather than silently averaging shorter windows.
Split the first six dates into training history and the final two into a test period. Repeat the final training value for both predictions. This is a fixed-origin, two-day forecast: neither prediction uses test observations. Updating the second prediction after seeing January 7 would be a different evaluation protocol.
Finally, calculate mean absolute error by averaging absolute prediction differences. Report it in units sold per day and inspect individual errors too. Two test dates demonstrate the procedure, not reliable general performance. For a serious comparison, evaluate several later windows without tuning on the final test period.
Worked example
import pandas as pd
sales = pd.Series(
[10, 12, 11, 13, 12, 14, 16, 18],
index=pd.date_range('2025-01-01', periods=8, freq='D'),
name='units',
).sort_index()
monthly = sales.resample('ME').sum()
trailing = sales.rolling(3, min_periods=3).mean()
train, test = sales.iloc[:6], sales.iloc[6:]
forecast = pd.Series(float(train.iloc[-1]), index=test.index)
mae = (test - forecast).abs().mean()
print(f'January observed total (8 days): {monthly.iloc[0]:.0f}')
print(f'Trailing 3-day mean on Jan 6: {trailing.loc["2025-01-06"]:.2f}')
for date in test.index:
print(f'{date:%Y-%m-%d}: actual={test.loc[date]:.0f}, forecast={forecast.loc[date]:.2f}')
print(f'Test MAE: {mae:.2f} units/day')
Expected output
January observed total (8 days): 106
Trailing 3-day mean on Jan 6: 13.00
2025-01-07: actual=16, forecast=14.00
2025-01-08: actual=18, forecast=14.00
Test MAE: 3.00 units/day
The observed total is 106 units across eight days, not a complete January total. The January 6 rolling mean averages January 4–6: (13 + 12 + 14) / 3 = 13. This describes recent sales after January 6 closes; it is not automatically a forecast. The baseline instead repeats the latest training observation, 14. Absolute errors are 2 and 4 units, giving an MAE of 3 units per day. Both predictions underestimate the held-out observations, but two errors cannot establish systematic bias. The example illustrates aggregation, smoothing, and honest evaluation without claiming seasonality or fitting a complex model to insufficient history.
Common mistakes and how to avoid them
Treating missing dates as confirmed zero sales.
Check the reporting process first. Reindex to expose absent dates, preserve unknown values, and insert zero only when no sales are confirmed.
Using a same-day rolling mean to predict that day's unknown total.
For predictions made before the day begins, use `sales.shift(1).rolling(3).mean()`. Avoid centered windows, which can include future observations.
Comparing partial-month totals with complete-month totals.
Report coverage alongside totals. Compare equivalent periods or suitable daily rates, while accounting for differences in weekdays and operating days.
Choosing the rolling window using final test errors.
Compare candidate windows on earlier chronological validation periods. Reserve the final test period for evaluating the already chosen procedure.
Review questions
How do resampling and rolling averages differ?
Resampling summarizes time bins, such as calendar months. Rolling averages summarize overlapping local windows. Both simplify data, but answer different questions about totals and recent levels.
Why might seven observations not represent seven days?
Records may be missing or irregularly spaced. An integer window counts rows. Check the index frequency or deliberately use a time-based window with explicit coverage requirements.
Can January 7's actual sales inform the January 8 forecast?
Yes, if forecasting after January 7 closes and its total is available. No, if both predictions must be issued on January 6, as in this example.
What does an MAE of 3 units/day tell us?
Across these test dates, predictions missed actual daily sales by 3 units on average in absolute terms. MAE does not show error direction, worst-case error, or future reliability.
