pandas Basics

Understanding pandas Basics

A pandas DataFrame is a two-dimensional table whose rows and columns have labels. Think of a small quiz register: each row represents one learner, while columns hold names, groups, and scores. Unlike a spreadsheet, pandas expresses changes as Python operations that you can rerun. The usual import, import pandas as pd, gives the library a short name; pd is a convention, not a separate package.

A Series is a one-dimensional labelled collection. Selecting df['score'] returns a Series, whereas df[['score']] returns a DataFrame with one column. This distinction matters because operations can return different shapes. Each column has a dtype describing how its values are represented. Numeric scores support arithmetic; text that looks numeric may not. Inspect df.head(), df.shape, and df.dtypes before calculating. They reveal sample records, the row-and-column count, and column types, respectively, but do not establish that the data is correct.

The index supplies row labels. If none is provided, pandas normally creates consecutive integer labels starting at zero. Labels are not necessarily current row positions: filtering preserves the original labels. Use .loc for label-based selection and .iloc for position-based selection. pandas also aligns Series by their index labels during many operations, rather than simply pairing the first values, then the second. This helps match records correctly, but unexpected labels can produce missing results.

Boolean filtering answers a question for every row. The expression df['score'] > 80 produces a Series of truth values, called a mask. Passing that mask to .loc keeps matching rows. Column arithmetic similarly works across values without an explicit Python loop. These vectorized expressions usually communicate tabular intent more clearly than manually iterating through records.

Missing values require a policy, not just a command. An unrecorded score is not automatically zero. Methods such as isna(), dropna(), and fillna() help detect or handle absence, but choosing among them changes the analysis. Many reductions, including mean(), skip missing values by default. State which observations contribute to a result. pandas is convenient for in-memory analysis, but large tables and intermediate results can exceed available memory; concise syntax does not eliminate resource costs.

Practical walkthrough

Use Python 3.11 or later with pandas 3.0.x already installed. You should understand dictionaries, lists, comparisons, and function calls. The example constructs every value in memory; it needs no files, network access, or scikit-learn installation. Run the complete code in one notebook cell or Python session.

First, construct four quiz records from equal-length lists. The dictionary keys become column labels. Give the score column the nullable integer dtype Int64, with a capital I. This explicitly permits whole-number scores alongside pd.NA, pandas' missing-value marker. The other columns contain text. One row deliberately lacks a score so that the selection rule must account for incomplete information.

Next, inspect the table's shape and score dtype. Four rows and three columns confirm the intended structure, while Int64 confirms that scores were not stored as text. Count missing scores with isna().sum(): isna() creates Boolean indicators, and summing them counts the True entries.

Build a mask requiring both membership in group red and a score of at least 80. Parentheses separate the comparisons; & combines them element by element. A missing score can produce an unknown comparison result. Calling fillna(False) on the mask makes the decision explicit: unknown eligibility is excluded, not interpreted as a zero score.

Finally, select only names and scores, then create a points_to_100 column using assign(). Sort the result by score in descending order. This short method chain reads as a sequence of transformations and leaves df unchanged. Named intermediate variables remain useful when debugging longer chains. Display records as dictionaries for compact, predictable output.

Worked example

import pandas as pd

df = pd.DataFrame({
    'name': ['A', 'B', 'C', 'D'],
    'group': ['red', 'blue', 'red', 'red'],
    'score': pd.Series([80, 92, pd.NA, 95], dtype='Int64'),
})

print('Shape:', df.shape)
print('Score dtype:', df['score'].dtype)
print('Missing scores:', int(df['score'].isna().sum()))

mask = ((df['group'] == 'red') & (df['score'] >= 80)).fillna(False)
result = (
    df.loc[mask, ['name', 'score']]
    .assign(points_to_100=lambda table: 100 - table['score'])
    .sort_values('score', ascending=False)
)

print('Selected labels:', result.index.tolist())
print('Records:', result.to_dict(orient='records'))
print(f"Selected mean: {result['score'].mean():.1f}")

Expected output

Shape: (4, 3)
Score dtype: Int64
Missing scores: 1
Selected labels: [3, 0]
Records: [{'name': 'D', 'score': 95, 'points_to_100': 5}, {'name': 'A', 'score': 80, 'points_to_100': 20}]
Selected mean: 87.5

D and A qualify because both belong to red and have recorded scores of at least 80. B meets the score threshold but belongs to blue; C has unknown eligibility and is excluded. Descending sorting places D before A while preserving their original index labels, 3 and 0. Subtracting scores from 100 produces 5 and 20. Their mean is (95 + 80) / 2 = 87.5; it describes only the selected learners, not the entire register. The calculation assumes scores use a 100-point scale. pandas does not enforce that assumption, so real data would also need range validation before interpretation.

Common mistakes and how to avoid them

• Using Python's and or or to combine Series comparisons.

  Use & for elementwise AND and | for elementwise OR, with parentheses around each comparison. Python's and expects a single truth value, not a separate decision for every row.

• Assuming index label 0 always identifies the first displayed row.

  After sorting this result, result.loc[0] selects A, while result.iloc[0] selects D. Choose label or position deliberately; reset_index(drop=True) only when new consecutive labels are actually useful.

• Updating a table through chained selection.

  Avoid df[mask]['score'] = 100. Under pandas 3.0 Copy-on-Write, chained assignment does not update df. To intentionally modify the original table, use one assignment: df.loc[mask, 'score'] = 100.

• Replacing every missing score with zero before summarizing.

  First determine what absence means. A missed recording and an earned zero are different observations. Preserve missingness or document a justified replacement policy, and report the number of usable scores.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide