Slide 1 of 21
Nanoscale · Module 12
Big Data in Connectomics
Teaching Deck
Human cortex · H01
Object segmentation over electron microscopy
H01 release · Lichtman Lab / Harvard & Connectomics at Google · CC BY 4.0
Shapson-Coe et al. (2024) · doi:10.1126/science.adk4858
Slide 2 of 21
Learning Objectives
- Describe core architecture patterns for petascale connectomics data
- Plan compute, storage, and indexing strategies for large EM volumes
- Implement query workflows that preserve provenance and reproducibility
- Identify bottlenecks and failure modes in large-scale analysis pipelines
Slide 3 of 21
Capability Target
Produce a scalable, reproducible query-and-analysis plan for a large connectomics dataset, including storage assumptions, indexing strategy, and provenance capture. Concretely: size a dataset from its imaging parameters before anyone quotes you a price, choose a chunk and shard layout from your actual access pattern rather than from the format everyone else uses, predict which query will dominate your bill, and pin every published number to a segmentation version a stranger can re-query a year from now.
Slide 4 of 21
Concept Focus
1) Storage layout is chosen by access pattern, not by format popularity
- Technical: chunked formats (Zarr, N5, Neuroglancer precomputed) store a volume as independent compressed blocks, commonly 64³ to 256³ voxels.
Figure description and teaching notes
Further teaching detail from the module:
**Technical:** chunked formats (Zarr, N5, Neuroglancer precomputed) store a volume as independent compressed blocks, commonly 64³ to 256³ voxels. The chunk is the unit of I/O, so you pay for the whole chunk even when you want one plane of it. A 512 x 512 section-plane view touches 4 chunks at 256³ and 64 at 64³, yet moves about 67 MB against about 17 MB, because each 256³ chunk carries z-depth you did not ask for. Anisotropic chunks such as 128 x 128 x 16 improve plane reads and worsen z-traversal. EM compresses 2-10x; label volumes compress far better.
Slide 5 of 21
Concept Focus (continued)
- Plain language: the chunk is the smallest thing you can read, so shape it like the reads you will actually do.
- Misconception guardrail: the format everyone else uses is automatically the right layout for your access pattern.
Slide 6 of 21
Core Workflow
- Write the analysis question as a sentence naming the table, the filter, and the unit of the answer — for example, "count synapses between layer 2/3 pyramidal cells and basket cells, per neuron pair, at cleft score above threshold."
- Estimate the working set: how many rows, how many objects, how many bytes must move, and whether that fits in memory on the machine you have.
- Choose storage and index strategy from the access pattern — chunk shape for volumetric reads, sharding if object counts exceed roughly 10^6, a pre-joined extract if the same join recurs.
Slide 7 of 21
Core Workflow (continued)
- Pin the segmentation: record the materialization version or timestamp, and refuse to proceed if it is unknown.
- Prototype on a 0.1% sample, profile, and extrapolate the full runtime before running it once at full scale.
- Add provenance fields to the output artifact itself, not to the surrounding notebook.
- Validate reproducibility by having a second person re-run the query package from the recorded version and compare row counts and summary statistics.
Slide 8 of 21
Run of Show (60 min)
- 00:00-08:00 | Architecture framing and failure examples
- 08:00-20:00 | Access-pattern to index mapping exercise
- 20:00-34:00 | Query profiling and bottleneck diagnosis
- 34:00-46:00 | Provenance logging implementation
Slide 9 of 21
Run of Show (60 min) (continued)
- 46:00-56:00 | Team review of reproducibility gaps
- 56:00-60:00 | Competency check and next-step assignment
Slide 10 of 21
Misconceptions to Watch
- Misconception guardrail: the format everyone else uses is automatically the right layout for your access pattern.
- Misconception guardrail: storage cost is the storage line on the invoice.
- Misconception guardrail: the dataset size is the petabyte figure quoted for the raw imagery.
- Misconception guardrail: an object ID refers to the same neuron next month.
Slide 11 of 21
Misconceptions to Watch (continued)
- Misconception guardrail: "it runs eventually" is acceptable for iterative science.
- Misconception guardrail: notebook history is sufficient provenance.
Slide 12 of 21
Studio Activity
Scenario: Your team delivers a weekly motif-analysis report from a store holding a ~5 x 10^8-row synapse table, a 120,000-row segment table, and cell-type annotations for about 8,400 neurons. The volume is ~1 mm³, the bytes live in cloud object storage, and your analysis cluster is on-premises. The report is regenerated every Monday and will be cited in a manuscript. Last week's run took nine hours and produced numbers that do not match the report from three weeks ago; nobody knows why.
Slide 13 of 21
Activity Output Checklist
- Evidence-linked artifact submitted.
- At least one limitation or uncertainty stated.
- Revision point captured from feedback.
Slide 14 of 21
Assessment Rubric
Minimum pass
- Query design matches analysis goal and data shape, with at least one quantitative estimate.
- Provenance requirements are explicit, actionable, and attached to the artifact rather than the notebook.
- Bottlenecks are identified with one realistic mitigation.
Slide 15 of 21
Assessment Rubric
Strong performance
- Separates exploratory and production query paths and says which rules apply to each.
- Quantifies tradeoffs across latency, dollar cost, and reproducibility, and names the assumption behind each number.
- Identifies version drift as the first hypothesis for the discrepancy, before code bugs.
Slide 16 of 21
Assessment Rubric (continued)
- Anticipates failure recovery and rollback needs, including what happens when a materialization is superseded mid-analysis.
Slide 17 of 21
Assessment Rubric
Common failure modes
- Index choices disconnected from query workload.
- Missing version metadata in outputs.
- Optimization attempts without a benchmark baseline.
- Sizing that counts only the raw volume and ignores derived products and egress.
Slide 18 of 21
Exit Ticket
Document one query you use with:
- data source/version,
- expected runtime class,
- one provenance field you currently miss.
Slide 19 of 21
References (Instructor)
- H01 human cortical fragment release and infrastructure notes.
- MICrONS data platform documentation.
- Januszewski et al. (2018) for scalable reconstruction context.
Slide 20 of 21
Teaching Materials
- Module page: /modules/module12/
- Session kit: /teaching/sessions/module12/
- Worksheet: /assets/worksheets/module12/module12-activity.md
Slide 21 of 21
Read, review, or teach
The reading version includes figure descriptions and discussion guidance.
Nanoscale text: CC BY 4.0. Scientific figures retain their individual credits.