Read slides as a page
H01 electron microscopy with object segmentation and original 2 µm scale bar Nanoscale · Module 12

Big Data in Connectomics

Teaching Deck

Human cortex · H01
Object segmentation over electron microscopy

H01 release · Lichtman Lab / Harvard & Connectomics at Google · CC BY 4.0
Shapson-Coe et al. (2024) · doi:10.1126/science.adk4858

Module 12 · Nanoscale

Learning Objectives

  • Describe core architecture patterns for petascale connectomics data
  • Plan compute, storage, and indexing strategies for large EM volumes
  • Implement query workflows that preserve provenance and reproducibility
  • Identify bottlenecks and failure modes in large-scale analysis pipelines
Module 12 · Nanoscale

Capability Target

Produce a scalable, reproducible query-and-analysis plan for a large connectomics dataset, including storage assumptions, indexing strategy, and provenance capture. Concretely: size a dataset from its imaging parameters before anyone quotes you a price, choose a chunk and shard layout from your actual access pattern rather than from the format everyone else uses, predict which query will dominate your bill, and pin every published number to a segmentation version a stranger can re-query a year from now.

Module 12 · Nanoscale

Concept Focus

1) Storage layout is chosen by access pattern, not by format popularity

  • Technical: chunked formats (Zarr, N5, Neuroglancer precomputed) store a volume as independent compressed blocks, commonly 64³ to 256³ voxels.
Module 12 · Nanoscale

Concept Focus (continued)

  • Plain language: the chunk is the smallest thing you can read, so shape it like the reads you will actually do.
  • Misconception guardrail: the format everyone else uses is automatically the right layout for your access pattern.
Module 12 · Nanoscale

Core Workflow

  • Write the analysis question as a sentence naming the table, the filter, and the unit of the answer — for example, "count synapses between layer 2/3 pyramidal cells and basket cells, per neuron pair, at cleft score above threshold."
  • Estimate the working set: how many rows, how many objects, how many bytes must move, and whether that fits in memory on the machine you have.
  • Choose storage and index strategy from the access pattern — chunk shape for volumetric reads, sharding if object counts exceed roughly 10^6, a pre-joined extract if the same join recurs.
Module 12 · Nanoscale

Core Workflow (continued)

  • Pin the segmentation: record the materialization version or timestamp, and refuse to proceed if it is unknown.
  • Prototype on a 0.1% sample, profile, and extrapolate the full runtime before running it once at full scale.
  • Add provenance fields to the output artifact itself, not to the surrounding notebook.
  • Validate reproducibility by having a second person re-run the query package from the recorded version and compare row counts and summary statistics.
Module 12 · Nanoscale

Run of Show (60 min)

  • 00:00-08:00 | Architecture framing and failure examples
  • 08:00-20:00 | Access-pattern to index mapping exercise
  • 20:00-34:00 | Query profiling and bottleneck diagnosis
  • 34:00-46:00 | Provenance logging implementation
Module 12 · Nanoscale

Run of Show (60 min) (continued)

  • 46:00-56:00 | Team review of reproducibility gaps
  • 56:00-60:00 | Competency check and next-step assignment
Module 12 · Nanoscale

Misconceptions to Watch

  • Misconception guardrail: the format everyone else uses is automatically the right layout for your access pattern.
  • Misconception guardrail: storage cost is the storage line on the invoice.
  • Misconception guardrail: the dataset size is the petabyte figure quoted for the raw imagery.
  • Misconception guardrail: an object ID refers to the same neuron next month.
Module 12 · Nanoscale

Misconceptions to Watch (continued)

  • Misconception guardrail: "it runs eventually" is acceptable for iterative science.
  • Misconception guardrail: notebook history is sufficient provenance.
Module 12 · Nanoscale

Studio Activity

Scenario: Your team delivers a weekly motif-analysis report from a store holding a ~5 x 10^8-row synapse table, a 120,000-row segment table, and cell-type annotations for about 8,400 neurons. The volume is ~1 mm³, the bytes live in cloud object storage, and your analysis cluster is on-premises. The report is regenerated every Monday and will be cited in a manuscript. Last week's run took nine hours and produced numbers that do not match the report from three weeks ago; nobody knows why.

Module 12 · Nanoscale

Activity Output Checklist

  • Evidence-linked artifact submitted.
  • At least one limitation or uncertainty stated.
  • Revision point captured from feedback.
Module 12 · Nanoscale

Assessment Rubric

Minimum pass

  • Query design matches analysis goal and data shape, with at least one quantitative estimate.
  • Provenance requirements are explicit, actionable, and attached to the artifact rather than the notebook.
  • Bottlenecks are identified with one realistic mitigation.
Module 12 · Nanoscale

Assessment Rubric

Strong performance

  • Separates exploratory and production query paths and says which rules apply to each.
  • Quantifies tradeoffs across latency, dollar cost, and reproducibility, and names the assumption behind each number.
  • Identifies version drift as the first hypothesis for the discrepancy, before code bugs.
Module 12 · Nanoscale

Assessment Rubric (continued)

  • Anticipates failure recovery and rollback needs, including what happens when a materialization is superseded mid-analysis.
Module 12 · Nanoscale

Assessment Rubric

Common failure modes

  • Index choices disconnected from query workload.
  • Missing version metadata in outputs.
  • Optimization attempts without a benchmark baseline.
  • Sizing that counts only the raw volume and ignores derived products and egress.
Module 12 · Nanoscale

Exit Ticket

Document one query you use with:

  1. data source/version,
  2. expected runtime class,
  3. one provenance field you currently miss.
Module 12 · Nanoscale

References (Instructor)

  • H01 human cortical fragment release and infrastructure notes.
  • MICrONS data platform documentation.
  • Januszewski et al. (2018) for scalable reconstruction context.
Module 12 · Nanoscale

Teaching Materials

  • Module page: /modules/module12/
  • Session kit: /teaching/sessions/module12/
  • Worksheet: /assets/worksheets/module12/module12-activity.md
Module 12 · Nanoscale

Read, review, or teach

Read these slides as a page

Open the complete lesson

The reading version includes figure descriptions and discussion guidance.

Nanoscale text: CC BY 4.0. Scientific figures retain their individual credits.

Module 12 · Nanoscale

Further teaching detail from the module: **Technical:** chunked formats (Zarr, N5, Neuroglancer precomputed) store a volume as independent compressed blocks, commonly 64³ to 256³ voxels. The chunk is the unit of I/O, so you pay for the whole chunk even when you want one plane of it. A 512 x 512 section-plane view touches 4 chunks at 256³ and 64 at 64³, yet moves about 67 MB against about 17 MB, because each 256³ chunk carries z-depth you did not ask for. Anisotropic chunks such as 128 x 128 x 16 improve plane reads and worsen z-traversal. EM compresses 2-10x; label volumes compress far better.

00:00-08:00 | Architecture framing and failure examples Open with two failure shapes: the eleven-hour query and the unreproducible figure. Both are design decisions made before any analysis, not accidents. 08:00-20:00 | Access-pattern to index mapping exercise Learners size a 1 mm³ volume by hand, then compute chunk counts at 64³, 128³, and 256³ and the byte cost of one 512 x 512 plane view at each. Instructor challenge: "Which is right, and what did you assume about how people read this volume?" 20:00-34:00 | Query profiling and bottleneck diagnosis Run a supplied query on a 0.1% sample, record wall time, extrapolate, then run the pre-joined version and write down the ratio. 34:00-46:00 | Provenance logging implementation Each learner adds a provenance block — dataset, version, query hash, thresholds, commit, date — to one of their own outputs and shows it to a neighbor. 46:00-56:00 | Team review of reproducibility gaps Pairs swap query packages and attempt to state, from the artifact alone, which segmentation version produced it. Any package that fails this test is marked and repaired. 56:00-60:00 | Competency check and next-step assignment Each learner names the single query that will dominate their own project's cost, and the mitigation they will try first.