# Speaker Uncertainty Evaluation - Apollo 13

Date: 2026-05-19

Variant: `uncertainty_auto_fallback_v2`

Mapping used for evaluation: `{'SPEAKER_00': 'SPACECRAFT', 'SPEAKER_01': 'HOUSTON'}`

Modal worker wall time: `24.431s`; VAD fallback used: `True`

## Output Summary

- Words with uncertainty fields: `350`
- Words marked `speaker_uncertain=true`: `8` (`2.3%`)
- Mean speaker confidence: `0.9422`
- Boolean rule: `speaker_confidence <= 0.50`
- Reason counts: `{'low_cluster_margin': 6, 'overlapping_windows_disagree': 3, 'low_confidence': 1, 'near_speaker_boundary': 1}`

## Ground-Truth Comparison

- Matched exact words in alignment: `281`
- Actually wrong speaker words among matched words: `8`
- Flagged uncertain among matched words: `8`
- True positives: `6`
- False positives: `2`
- False negatives: `2`
- True negatives: `271`
- Precision: `0.750`
- Recall: `0.750`
- F1: `0.750`

## Wrong Speaker Words

| Time | Word | Ground truth | Predicted | Uncertain | Confidence | Margin | Vote | Reasons |
|---|---|---|---|---:|---:|---:|---:|---|
| 14.56-14.96 | `okay` | HOUSTON | SPACECRAFT | false | 0.652 | 0.095 | 1.00 | - |
| 40.14-40.62 | `roger` | HOUSTON | SPACECRAFT | true | 0.450 | -0.011 | 1.00 | low_cluster_margin |
| 40.68-40.88 | `fred` | HOUSTON | SPACECRAFT | true | 0.450 | -0.011 | 1.00 | low_cluster_margin |
| 48.24-48.72 | `roger` | HOUSTON | SPACECRAFT | true | 0.450 | -0.013 | 1.00 | low_cluster_margin |
| 64.78-65.26 | `roger` | HOUSTON | SPACECRAFT | false | 0.604 | 0.070 | 1.00 | - |
| 107.26-107.78 | `roger` | HOUSTON | SPACECRAFT | true | 0.450 | -0.036 | 1.00 | low_cluster_margin |
| 167.10-167.56 | `we` | SPACECRAFT | HOUSTON | true | 0.500 | 0.064 | 1.00 | - |
| 167.56-167.68 | `are` | SPACECRAFT | HOUSTON | true | 0.345 | 0.064 | 0.67 | overlapping_windows_disagree, near_speaker_boundary |

## Flagged But Speaker-Correct Words

| Time | Word | Role | Confidence | Margin | Vote | Reasons |
|---|---|---|---:|---:|---:|---|
| 149.00-149.46 | `we` | HOUSTON | 0.300 | 0.012 | 0.50 | overlapping_windows_disagree, low_cluster_margin |
| 149.46-149.66 | `copy` | HOUSTON | 0.300 | 0.012 | 0.50 | overlapping_windows_disagree, low_cluster_margin |

## Interpretation

The uncertainty score is useful but not perfect. On this sample it caught 6 of 8 speaker-label errors among exactly matched words, with 2 false positives. The two missed errors were not acoustically ambiguous according to the current cluster diagnostics; they are cases where the diarizer was confidently wrong.

This is therefore a good review signal, not a correction mechanism. It should be shown as "needs review" rather than used to automatically flip speakers.