Module 18: Data Cleaning and Preprocessing

Lesson Flow

Learn

Goals and Concepts

Start with the capability target and concept set for this module.

Practice

Studio Activity

Apply the ideas in a guided activity tied to realistic outputs.

Check

Assessment Rubric

Use the rubric to verify competency and identify improvement targets.

Teach

Slides and Worksheets

Open the teaching deck, worksheet, and editable slide source.

Interactive Lab

Practice in short loops: checkpoint quiz, microtask decision, and competency progress tracking.

Checkpoint Quiz

Q1. Which output most clearly demonstrates module competency? A measurable artifact linked to an explicit method A general reflection about interest in neuroscience A list of future goals without evidence

Competency is shown through measurable, method-linked evidence.

Q2. What should always accompany a technical claim in this curriculum? A limitation or uncertainty statement A motivational quote A citation count

Every claim should include boundaries and uncertainty.

Q3. What is the best next step after identifying a gap in understanding? Define a concrete practice task and verification criterion Skip to a harder module Ignore the gap if progress is slow

Progress improves when gaps become explicit practice targets.

Microtask Decision

Choose the action that best improves scientific reliability.

Record method, version, and decision rationale before sharing results Share results quickly and document details later Rely on memory for why decisions were made

Progress Tracker

State is saved locally in your browser for this module.

Read capability target
Complete studio activity
Review assessment rubric
Pass checkpoint quiz
Complete microtask
Complete annotation challenge

0% complete

Annotation Challenge

Click the hotspot with the strongest evidence for the requested feature.

Selected hotspot: none

Capability target

Produce a reproducible preprocessing release that transforms raw or intermediate connectomics outputs into analysis-ready data, with explicit quality gates and full provenance.

Why this module matters

Most downstream failures in connectome analysis are not model failures first; they are data-quality and preprocessing failures. This module teaches how to clean data without erasing signal, and how to document each transformation so conclusions remain defensible.

Concept set

1) Cleaning vs distortion

Technical: preprocessing should reduce known artifacts/noise while preserving biologically meaningful structure.
Plain language: fix mistakes, do not “polish away” the biology.
Misconception guardrail: more filtering is not always better.

2) Provenance as a scientific requirement

Technical: every transform should be traceable (input version, parameters, timestamp, owner, output hash).
Plain language: if you cannot explain how the file was made, you cannot trust the result.
Misconception guardrail: version-control notes alone are insufficient without data lineage.

3) QC metrics must be decision-linked

Technical: metrics (missingness, merge/split rates, consistency checks) should trigger concrete accept/rework decisions.
Plain language: a dashboard is useful only if it changes what you do.
Misconception guardrail: reporting metrics without thresholds is not quality control.

Core workflow: preprocessing for connectomics

Ingest and integrity validation
- Confirm file completeness, schema conformance, and version compatibility.
- Log dataset identifiers and checksums.
Artifact and anomaly screening
- Identify missing values, label conflicts, geometric outliers, and suspicious connectivity spikes.
- Triage issues by likely biological impact.
Cleaning transforms
- Apply deterministic corrections (schema normalization, unit harmonization, explicit missing-value policy).
- Isolate heuristic transforms for extra review.
QC and drift checks
- Compare pre/post distributions and topology statistics.
- Verify no unacceptable biological-signal loss.
Release packaging
- Publish analysis-ready tables/volumes plus transform log, metric report, and known limitations.

Studio activity: preprocessing release simulation

Scenario: Your team receives a mixed-quality connectomics export with missing labels, duplicated IDs, and inconsistent units.

Tasks

Define cleaning policy for each issue category.
Implement a preprocessing pipeline (pseudocode or notebook-level steps).
Run pre/post QC metrics and justify any tradeoffs.
Produce a release note that includes lineage metadata and known residual risks.

Expected outputs

Preprocessing decision table.
QC metric summary with thresholds and pass/fail calls.
Release note (inputs, transforms, outputs, limitations).

Assessment rubric

Minimum pass
- Cleaning decisions are explicit and reproducible.
- QC metrics include thresholds tied to actions.
- Release package includes provenance metadata.
Strong performance
- Distinguishes low-risk cleanup from biologically sensitive transforms.
- Quantifies and explains pre/post changes clearly.
- Documents limitations and unresolved risks transparently.
Common failure modes
- Silent ad-hoc edits with no transform log.
- Aggressive filtering that removes biologically meaningful variation.
- Metrics reported without operational thresholds.

Teaching resources

Lesson context: Volume Reconstruction Infrastructure
QC context: Segmentation and Proofreading
Slides: Infrastructure deck draft
Practice dataset workflow: Workflow overview
Quality framework: Connectome Quality tool

60-minute tutorial run-of-show

Materials

One noisy connectomics table (missing values, duplicated IDs, inconsistent units).
Shared preprocessing decision sheet.
QC dashboard template (pre/post metrics).

Timing and flow

**00:00-08:00 Setup and target**
- Define release objective and non-negotiable quality gates.
**08:00-18:00 Instructor modeling**
- Live demonstration of ingest checks and anomaly triage logic.
**18:00-32:00 Team preprocessing design**
- Teams draft cleaning rules and escalation criteria.
**32:00-44:00 QC pass**
- Teams compute/estimate pre-post metrics and decide release/no-release.
**44:00-54:00 Cross-team review**
- Teams audit each other’s transform logs for reproducibility gaps.
**54:00-60:00 Competency checkpoint**
- Submit one release note with provenance, thresholds, and residual risk.

Success criteria for this session

Cleaning decisions are deterministic and documented.
QC thresholds are tied to operational actions.
Release note exposes at least one unresolved interpretation risk.

Evidence anchors from connectomics practice

Key papers to use in this module

Key datasets to practice on

Competency checks

Can you trace every preprocessing transform from input version to release artifact?
Can you justify your QC thresholds and their operational consequences?
Can you state one unresolved data-risk that could still affect interpretation?

Quick practice prompt

Take one connectomics table (real or mock) and write:

Three cleaning rules with rationale.
Two QC thresholds and associated actions.
One limitation that remains after preprocessing.

Teaching Materials

Slide Deck

Classroom-ready deck links for teaching and delivery.

Open slide deck page

Open rendered HTML deck

Download PowerPoint (.pptx)

Activity Worksheet

Learner worksheet aligned to the studio activity and rubric.

Open worksheet

Slide Source

Marp source file for editing and rendering.

course/decks/marp/modules/module18.marp.md