Making Sense of P(doom)

Summary

Clarifies P(doom) probability estimates for AI existential risk. Objective probability (QBU): measure of quantum timelines where doom occurs—ontologically real but practically inaccessible. Subjective probability (credence): best-informed estimate based on evidence—pragmatically crucial for decisions. Temporal bounds essential: must specify timeframe (e.g., “within 100 years”). P(doom, t) is monotonically increasing, approaches asymptote (not certainty). Common error: extending timeframe doesn’t increase probability in QBU—quantum branching structure fixed at t=0. Analogy: Cancer risk exists objectively but unknowable; subjective estimate guides health choices. Similarly, subjective P(doom) informs AI alignment research priorities despite inaccessible objective probability.

Tags

• ai-alignment
• quantum
• bayesianism
• risk
• epistemology

Cross-References

• Related: Objective vs. Subjective Probability
• Related: QBU framework

Notes

• Applies rigorous QBU/conditionalism to practical AI safety debate
• Resolves confusion about probability increase over time
• Shows value of subjective estimates despite unknowable objective reality
• Anticipates later alignment work