NeuroTrailblazers banner

Computer Vision & ML Journal Papers

The methods that made petascale connectomics possible. Read roughly in order — the sequence below is close to the field’s actual chronology, and each paper is a response to a limitation of the previous ones.

Companion unit: Unit 08 — Segmentation and Proofreading. Read Unit 08 §1–2 first if you want the conceptual frame before the primary literature.

A framing to carry through all ten papers. Every method here is engineered around one asymmetry: merge errors cost far more than split errors. Watershed thresholds, agglomeration criteria, loss functions, and evaluation metrics all reflect that choice. When you read a paper’s design decisions, ask which error it is buying protection from.

1. Turaga et al. (2010) — Convolutional Networks Can Learn to Generate Affinity Graphs

Citation: Turaga SC, Murray JF, Jain V, Roth F, Helmstaedter M, Briggman K, Denk W, Seung HS. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation. 2010;22(2):511-538. DOI: 10.1162/neco.2009.10-08-881

Tags: computer-vision-ml:affinity-prediction computer-vision-ml:boundary-detection computer-vision-ml:3D-convolution neuroai:deep-learning

Summaries

Beginner: Before this paper, computers found neuron boundaries by looking for dark lines in the image and hoping they formed closed shapes. This work proposed something different: instead of asking “is this pixel a boundary?”, ask “do these two neighbouring pixels belong to the same neuron?” That question turns out to be much easier for a network to answer well, and the answers can be assembled into whole neurons afterwards.

Intermediate: Turaga et al. introduced the affinity graph formulation for EM segmentation. A convolutional network predicts, for each voxel, the affinity to its neighbours along each axis. Segmentation then reduces to a graph-partitioning problem over those affinities — typically watershed followed by agglomeration. The key methodological contribution is training the network with an objective aligned to the segmentation, rather than to per-pixel boundary classification, which decouples the network’s job from the downstream clustering.

Advanced: The affinity representation remains the backbone of most production connectomics pipelines fifteen years on. Its durability comes from two properties: it is local (so inference parallelizes over blocks), and it separates the learned component from the combinatorial one, which lets each be improved independently. The paper also identified the central evaluation problem — that per-pixel accuracy is a poor proxy for segmentation quality, because a single wrong voxel at a thin neck causes a topological error while thousands of wrong voxels in an object interior cause none. That observation motivates the structured losses in paper 8 and the metrics in Metrics and QA.

Key figures: Affinity graph formulation; comparison of boundary-trained vs affinity-trained networks; segmentation results on SBEM data.

Discussion prompts:

Related content: Reconstruction pipeline, Unit 08

2. Ciresan et al. (2012) — Deep Neural Networks Segment Neuronal Membranes

Citation: Ciresan D, Giusti A, Gambardella LM, Schmidhuber J. Deep neural networks segment neuronal membranes in electron microscopy images. Advances in Neural Information Processing Systems (NIPS). 2012;25:2843-2851.

Tags: computer-vision-ml:boundary-detection computer-vision-ml:semantic-segmentation neuroai:deep-learning methodology:benchmark

Summaries

Beginner: This paper won the first major public competition for finding neuron boundaries in EM images, and it did so using a deep neural network at a time when most groups were using hand-designed image filters. It was one of the results that convinced the field that learned methods would win.

Intermediate: Ciresan et al. applied a sliding-window deep CNN to the ISBI 2012 membrane segmentation challenge, classifying each pixel from a surrounding patch. The approach was computationally wasteful — overlapping windows recompute the same features many times — but it substantially outperformed the hand-engineered feature pipelines of the day, and it established that the limiting factor was model capacity rather than feature design.

Advanced: Historically important rather than currently practical. Its inefficiency is precisely the problem U-Net (paper 3) solved by making prediction fully convolutional. Read it for two things: the demonstration that learned features dominate engineered ones in this domain, and as a case study in how a benchmark victory can redirect a field. The ISBI 2012 task itself is 2D and small, and — as the atlas notes — benchmark performance on clean small volumes systematically overstates performance on production data.

Key figures: Network architecture; ISBI 2012 challenge results; comparison against contemporaneous methods.

Discussion prompts:

Related content: EM principles, Error taxonomy

3. Ronneberger, Fischer & Brox (2015) — U-Net

Citation: Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI). 2015;9351:234-241. DOI: 10.1007/978-3-319-24574-4_28

Tags: computer-vision-ml:U-Net computer-vision-ml:semantic-segmentation computer-vision-ml:data-augmentation neuroai:deep-learning

Summaries

Beginner: U-Net is the network shape that most biomedical image segmentation still uses. It works by first shrinking the image down to understand large-scale context, then expanding it back up to make a precise per-pixel prediction — with shortcuts across the middle so fine detail is not lost on the way. It was designed for problems with very little training data, which describes almost all biological imaging.

Intermediate: The encoder-decoder architecture with skip connections lets the network combine coarse semantic context from the contracting path with high-resolution localization from the skip connections. The paper also emphasized heavy elastic data augmentation, which is what made training from small annotated datasets feasible — a decisive practical point in EM, where dense ground truth is enormously expensive to produce.

Advanced: U-Net is not a connectomics paper, but 3D variants of it are the workhorse backbone for affinity and boundary prediction across the field. Two design points matter for EM specifically: the receptive field must be large enough to disambiguate thin processes from context, which drives depth; and the anisotropy of typical EM data motivates anisotropic pooling schedules that downsample xy earlier and more aggressively than z. The augmentation strategy also matters more than usual here, because the dominant distribution shift in production is between tissue preparations, not within one.

Key figures: Fig. 1 (the U-shaped architecture); elastic deformation augmentation examples; segmentation results.

Discussion prompts:

Related content: Reconstruction pipeline, Artifact taxonomy

4. Berning, Boergens & Helmstaedter (2015) — SegEM

Citation: Berning M, Boergens KM, Helmstaedter M. SegEM: efficient image analysis for high-resolution connectomics. Neuron. 2015;87(6):1193-1206. DOI: 10.1016/j.neuron.2015.09.003

Tags: computer-vision-ml:segmentation computer-vision-ml:agglomeration proofreading:proofreading-strategy methodology:ground-truth

Summaries

Beginner: This paper is about building a complete, usable segmentation system rather than a single clever algorithm — including how to generate the training data, how to check the result, and how to hand it to human annotators efficiently.

Intermediate: SegEM presents an end-to-end pipeline for SBEM data with a deliberate design choice: tune for over-segmentation, then rely on human agglomeration. The authors quantify the tradeoff between automated segmentation quality and the human effort required to finish the job, and they introduce practical training-data generation and validation procedures.

Advanced: The lasting contribution is the explicit framing of segmentation quality as a human-effort variable rather than an accuracy score. That framing — how many corrections per millimetre of traced neurite does this segmentation cost? — is the correct optimization target for a production project, and it is the frame Unit 08 builds on. The paper’s error analysis by process calibre is also worth studying: error rates are strongly non-uniform, concentrating in thin processes, so aggregate accuracy numbers systematically flatter a method.

Key figures: Pipeline overview; segmentation-quality vs human-effort tradeoff; error rates by neurite calibre.

Discussion prompts:

Related content: Proofreading strategies, Metrics and QA

5. Lee, Zung, Li, Jain & Seung (2017) — Superhuman Accuracy on SNEMI3D

Citation: Lee K, Zung J, Li P, Jain V, Seung HS. Superhuman accuracy on the SNEMI3D connectomics challenge. arXiv. 2017. arXiv: 1706.00120

Tags: computer-vision-ml:affinity-prediction computer-vision-ml:3D-convolution methodology:benchmark methodology:ground-truth

Summaries

Beginner: This paper reported that an automated system beat human annotators on a standard neuron-segmentation benchmark. That headline is true and needs careful reading: it was one benchmark, on a small, clean, well-prepared volume, and it does not mean proofreading became unnecessary.

Intermediate: The authors combined a deep 3D residual U-Net predicting affinities with careful training practice — large receptive fields, extensive augmentation, and inference-time strategies — to exceed the reported human-annotator error rate on SNEMI3D. The paper is a useful record of which engineering choices mattered and by how much.

Advanced: The interesting question this paper raises is what “superhuman” means when the benchmark’s ground truth was itself produced by humans. Exceeding the reported human error rate on a small isotropic volume with excellent staining tells you relatively little about performance on a petascale anisotropic volume containing folds, charging, and rare morphologies — which is where all the residual proofreading cost lives. Read alongside the atlas’s caution on benchmark transfer, and note that FlyWire and MICrONS both still required extensive human proofreading years after this result.

Key figures: Architecture; SNEMI3D leaderboard comparison; ablation of training choices.

Discussion prompts:

Related content: Atlas: benchmarks, Metrics and QA

6. Dorkenwald et al. (2017) — Automated Synaptic Connectivity Inference (SyConn)

Citation: Dorkenwald S, Schubert PJ, Killinger MF, Urban G, Mikula S, Svara F, Kornfeld J. Automated synaptic connectivity inference for volume electron microscopy. Nature Methods. 2017;14:435-442. DOI: 10.1038/nmeth.4206

Tags: computer-vision-ml:synapse-detection computer-vision-ml:object-detection cell-types:connectivity-based-classification neuroai:deep-learning

Summaries

Beginner: Segmenting neurons is only half the job — you also have to find the synapses between them, and work out which neuron is sending and which is receiving. This paper automated that step, including inferring cell type and synapse type from the image.

Intermediate: SyConn combines neuron segmentation with learned classifiers for synapse detection, synaptic polarity (which partner is presynaptic), and cell-type prediction, producing a connectivity graph rather than just a segmentation. Treating synapse detection and partner assignment as separate learned problems — rather than as a by-product of segmentation — is the architectural point.

Advanced: Partner assignment is where the field’s directional errors are manufactured, and this paper is where it became an explicit learned task with its own evaluation. Two implications follow. First, synapse detection must be evaluated on its own metrics (precision and recall over synapses and over partner assignments), not folded into segmentation scores. Second, storing partner identity against the immutable supervoxel layer rather than against neuron IDs is what keeps those assignments valid through proofreading — see Unit 04 §2.

Key figures: SyConn pipeline; synapse detection performance; cell-type classification from morphology.

Discussion prompts:

Related content: Synapse classification, Graph representations

7. Januszewski et al. (2018) — Flood-Filling Networks

Citation: Januszewski M, Kornfeld J, Li PH, Pope A, Blakely T, Lindsey L, Maitin-Shepard J, Tyka M, Denk W, Jain V. High-precision automated reconstruction of neurons with flood-filling networks. Nature Methods. 2018;15:605-610. DOI: 10.1038/s41592-018-0049-4

Tags: computer-vision-ml:flood-filling-network computer-vision-ml:instance-segmentation neuroai:deep-learning case-studies:zebrafish

Summaries

Beginner: Rather than labelling every pixel at once and then grouping, a flood-filling network grows one neuron at a time from a starting point, repeatedly asking “does the next bit belong to this neuron?” Because it always knows what it has built so far, it makes far fewer mistakes at the difficult places.

Intermediate: FFNs perform recurrent, object-centred segmentation: a network maintains an evolving object mask as an input channel and iteratively extends it. The paper reports order-of-magnitude improvements in expected run length over prior methods on songbird and zebrafish data. The method also introduced practical machinery for large-volume application, including seed selection and consistency-based agglomeration.

Advanced: The conditioning on the current mask is what gives FFNs their advantage at ambiguous membrane contacts — the network has context that a purely feedforward per-voxel predictor lacks. The cost is that inference is sequential per object and expensive, which drove substantial engineering to make it tractable at petascale (see Macrina et al. 2021 and the MICrONS reconstruction). FFNs produced the FlyWire base segmentation, so this paper is upstream of a large fraction of the field’s current data. Note the paper’s use of expected run length as its headline metric, and why that choice suits a tracing-oriented method — Unit 08 §3 discusses what ERL is blind to.

Key figures: FFN iterative filling schematic; expected run length comparisons; large-volume reconstruction results.

Discussion prompts:

Related content: Reconstruction pipeline, Metrics and QA

8. Funke et al. (2019) — Structured Loss for Connectome Reconstruction

Citation: Funke J, Tschopp F, Grisaitis W, Sheridan A, Singh C, Saalfeld S, Turaga SC. Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019;41(7):1669-1680. DOI: 10.1109/TPAMI.2018.2835450

Tags: computer-vision-ml:affinity-prediction computer-vision-ml:agglomeration computer-vision-ml:watershed neuroai:deep-learning

Summaries

Beginner: Not all mistakes are equally bad. Getting one pixel wrong in the middle of a big neuron changes nothing; getting one pixel wrong at a thin neck can split a neuron in two. This paper changed the training objective so the network is punished in proportion to how much a mistake would damage the final reconstruction.

Intermediate: The structured (MALIS-style) loss weights errors by their effect on the topology of the resulting segmentation rather than treating all voxels equally. Combined with an efficient 3D U-Net and a well-tuned watershed and agglomeration stage, this produced strong results on Drosophila data and a practical, widely reused pipeline.

Advanced: This is the clearest expression in the literature of the principle that the loss should match the evaluation metric, and that in connectomics the evaluation metric is topological. The paper also gives an unusually clear account of the full stack — affinity prediction, watershed thresholds, agglomeration criteria — and how the parameters interact, which makes it the best single reference for someone tuning a pipeline rather than proposing a new architecture. Note how each stage’s parameters are chosen to protect against merges at the cost of splits.

Key figures: Structured loss illustration; pipeline stages; VI comparisons across parameter settings.

Discussion prompts:

Related content: Metrics and QA, Error taxonomy

9. Buhmann et al. (2021) — Automatic Detection of Synaptic Partners in a Whole-Brain Dataset

Citation: Buhmann J, Sheridan A, Malin-Mayor C, Schlegel P, Gerhard S, Kazimiers T, Krause R, Nguyen TM, Heinrich L, Lee WCA, Wilson R, Saalfeld S, Jefferis GSXE, Bock DD, Turaga SC, Cook M, Funke J. Automatic detection of synaptic partners in a whole-brain Drosophila electron microscopy data set. Nature Methods. 2021;18:771-774. DOI: 10.1038/s41592-021-01183-7

Tags: computer-vision-ml:synapse-detection computer-vision-ml:object-detection case-studies:FAFB case-studies:Drosophila case-studies:whole-brain

Summaries

Beginner: Finding every synapse in a whole fly brain — tens of millions of them — and correctly identifying which neuron is on each side. This is the step that turns a set of neuron shapes into an actual wiring diagram.

Intermediate: The method predicts, per presynaptic site, a vector pointing to the postsynaptic partner, which turns partner assignment into a dense regression problem that scales to whole-brain volumes. The resulting synapse predictions across FAFB underpin FlyWire’s connectivity.

Advanced: The design choice worth studying is the reformulation of a combinatorial matching problem as a local dense prediction, which is what makes it tractable at 10⁸-synapse scale. Read the evaluation carefully: precision and recall are reported for detection and for partner assignment separately, and both matter differently downstream. Detection recall failures deflate degree non-uniformly, biased against small synapses; partner assignment failures produce directional errors of exactly the kind Unit 06 §4 shows are not statistically recoverable. Any analysis applying a synapse-count threshold is implicitly making a decision about which of these errors to tolerate.

Key figures: Partner-vector prediction schematic; whole-brain synapse statistics; precision-recall for detection and partner assignment.

Discussion prompts:

Related content: FlyWire whole-brain, Graph representations, Unit 09

10. Sheridan et al. (2023) — Local Shape Descriptors for Neuron Segmentation

Citation: Sheridan A, Nguyen TM, Deb D, Lee WCA, Saalfeld S, Turaga SC, Manor U, Funke J. Local shape descriptors for neuron segmentation. Nature Methods. 2023;20:295-303. DOI: 10.1038/s41592-022-01711-z

Tags: computer-vision-ml:affinity-prediction computer-vision-ml:agglomeration computer-vision-ml:segmentation neuroai:deep-learning

Summaries

Beginner: Alongside predicting whether two pixels belong together, this method also predicts a compact description of the local shape of the object at each point. Shape information helps the system decide whether a proposed merge would produce something that actually looks like a neurite.

Intermediate: Local shape descriptors (LSDs) are per-voxel statistics — size, centre-of-mass offset, and second-moment-like terms of the local object — used as an auxiliary prediction target. Trained jointly with affinities, they improve agglomeration quality substantially while remaining local and therefore cheap and parallelizable, which is the practical advantage over methods requiring global context.

Advanced: LSDs are a good example of an auxiliary task that regularizes the primary one: predicting shape forces the network to represent object extent, which is exactly the information a purely local affinity predictor lacks at ambiguous contacts. The paper’s comparison against FFNs is worth reading closely — it argues that a substantial part of the accuracy gap can be closed by adding shape information while keeping the cheap parallel feedforward structure, which changes the compute economics at petascale considerably. Evaluate the claim against your own throughput constraints rather than the reported benchmark numbers.

Key figures: LSD components illustrated; joint training setup; accuracy vs compute comparison against FFN.

Discussion prompts:

Related content: Reconstruction pipeline, Unit 08

Reading it as a sequence

Paper The limitation it addresses
Turaga 2010 Boundary detection is the wrong objective
Ciresan 2012 Hand-engineered features are the bottleneck
Ronneberger 2015 Sliding-window inference is wasteful; data is scarce
Berning 2015 Accuracy is not the right optimization target; human effort is
Lee 2017 How far can careful engineering push a feedforward affinity model?
Dorkenwald 2017 Segmentation alone is not a connectome; synapses and partners need their own models
Januszewski 2018 Local feedforward prediction lacks object context
Funke 2019 The loss does not match the topological evaluation metric
Buhmann 2021 Partner assignment must scale to whole brains
Sheridan 2023 Object context can be added without abandoning cheap parallel inference