Ethics & Privacy

Understanding Ethics & Privacy

Ethics in data science concerns whether a system's purpose, evidence, decisions, and consequences are defensible. Privacy concerns how information about people is collected, used, protected, and disclosed. They overlap but are not interchangeable: a secure system can still make unfair decisions, and an accurate model can still use data for an inappropriate purpose. Responsible practice therefore starts before model training and continues after deployment.

Consider a service that recommends applicants for a limited training programme. Purpose limitation means using application data for the stated selection process, not silently repurposing it for advertising. Minimisation means collecting only what that purpose requires. Where consent is the basis for processing, it must be meaningful and informed; a checkbox does not make every subsequent use acceptable. Applicable legal requirements depend on jurisdiction and context, so an ethical checklist is not legal advice.

Security requires practical controls: restricted access, protected storage and transmission, retention limits, and accountable deletion procedures. Removing names does not necessarily anonymise records. Combinations such as location, age, and employment history may identify someone when linked with other information. Aggregation reduces detail but can expose individuals in small groups. Differential privacy offers a formal way to limit an individual's influence on released results, but requires carefully chosen mechanisms and privacy budgets, not arbitrary noise.

Fairness auditing asks who receives benefits and who bears errors. Overall accuracy combines correct decisions across everyone, potentially hiding unequal error rates. For eligible applicants, recall measures the proportion recommended: true positives divided by all actual positives. Comparing recall across groups can reveal unequal access among eligible people. Selection rate instead measures recommendations among everyone in a group; it answers a different question. Neither metric alone establishes fairness.

Bias can arise from sampling, unreliable labels, proxy features, objectives, thresholds, and deployment conditions—not only training data. Historical decisions are not automatically trustworthy labels. Sensitive attributes may be needed for a controlled audit even when excluded from prediction. This creates a privacy trade-off: collect and retain justified audit information under restricted access, rather than assuming either unlimited collection or complete removal is always responsible. Metrics support investigation; they do not prove discrimination, its cause, or legal compliance.

Practical walkthrough

Begin by stating the decision and its possible harm. Here, a positive prediction recommends an applicant for training. A false negative excludes someone labelled eligible. Assume eligibility was assessed consistently and independently; in practice, verify that assumption with domain experts and affected people. Our fictional groups A and B stand for audit categories, not actual populations.

Use Python 3.10 or later; no third-party packages are required. The example stores eight invented records in memory, each containing only a group, eligibility label, and recommendation. There are no names or contact details. This illustrates minimisation, not proof that equivalent real records would be anonymous. Predictions are hand-specified to expose a metric difference; no model is trained and no population performance is estimated.

Run the example and inspect each denominator. Accuracy uses all records, selection rate uses all members of a group, and recall uses only eligible members. The code prints group sizes and eligible counts alongside rates. If a group has no eligible records, recall becomes undefined rather than misleadingly becoming zero. The grouping follows the split-and-summarise idea covered by the pandas user guide, implemented here with basic Python.

For an actual model, keep final evaluation data separate from development. Fit preprocessing only on training data and choose thresholds on validation data, consistent with scikit-learn's guidance on leakage and evaluation. Investigate gaps before release; record uncertainties, mitigation decisions, and an appeal route. Recheck after deployment as populations and processes change. Share real audit results through access-controlled reports; publishing tiny group summaries can undermine privacy.

Worked example

# Python 3.10+; synthetic records only.
# Each tuple is (audit_group, eligible, recommended).
records = [
    ("A", 1, 1),
    ("A", 1, 1),
    ("A", 0, 0),
    ("A", 0, 0),
    ("B", 1, 1),
    ("B", 1, 0),
    ("B", 0, 1),
    ("B", 0, 0),
]

correct = sum(actual == predicted for _, actual, predicted in records)
print(f"Overall accuracy: {correct / len(records):.2f}")

for group in sorted({row[0] for row in records}):
    members = [row for row in records if row[0] == group]
    eligible = sum(actual for _, actual, _ in members)
    selected = sum(predicted for _, _, predicted in members)
    true_positives = sum(
        actual == 1 and predicted == 1
        for _, actual, predicted in members
    )
    recall = f"{true_positives / eligible:.2f}" if eligible else "undefined"
    print(
        f"{group}: n={len(members)}, eligible={eligible}, "
        f"selection={selected / len(members):.2f}, recall={recall}"
    )

Expected output

Overall accuracy: 0.75
A: n=4, eligible=2, selection=0.50, recall=1.00
B: n=4, eligible=2, selection=0.50, recall=0.50

Six of eight recommendations match eligibility, giving overall accuracy 0.75. Both groups receive two recommendations among four applicants, so their selection rates are identical. However, both eligible applicants in A are recommended, while only one of the two eligible applicants in B is recommended. Recall is therefore 1.00 for A and 0.50 for B. B also has one recommendation for an ineligible applicant. Equal selection rates thus coexist with unequal error patterns. These deliberately constructed counts demonstrate arithmetic, not evidence about real groups. With only two eligible applicants per group, changing one decision changes recall by 0.50.

Common mistakes and how to avoid them

• Treating removal of identifiers as guaranteed anonymity.

  Check whether remaining attributes permit linkage or singling out. Restrict access, minimise retained detail, and assess disclosure risks before releasing even aggregated results.

• Calling equal selection rates a complete fairness pass.

  Inspect error rates, denominators, and relevant intersections of groups. Select metrics according to harms; different fairness criteria can conflict, especially when eligibility prevalence differs.

• Assuming observed eligibility is unquestionable ground truth.

  Review how labels were assigned and whose outcomes are missing. If labels reflect historical exclusion, agreement with them may reward reproducing that exclusion.

• Repeatedly adjusting a model against its final audit set.

  Use training and validation data for changes, then evaluate on untouched test data. Repeated adaptation to test results makes reported performance overly optimistic.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide