Summary

Comprehensive AI alignment analysis. P(doom) estimate: ~25% under maximal epistemic uncertainty (feasibility + alignment both uncertain), contrasts with Yudkowsky’s near-certainty. Orthogonality Thesis skepticism: Evolutionary, embodied, semantic constraints may limit viable goal-space (20-60% OT correctness plausible); if true, simplifies alignment. Other existential risks: Nuclear war (10-20%), engineered pandemics (20-30%) are immediate threats; climate = risk multiplier. AGI’s effect: Aligned AGI reduces risks (surveillance, crisis management); misaligned AGI amplifies them. Human values problem: Minimal universal agreement; proposes Value-9s metric (like “five nines” reliability): Level 0 (<50%), Level 1 (90%), Level 2 (99%), Level 3 (99.9%), Level 4+ (99.99%+). Recommends targeting Level 3+ values (autonomy, consent, harm avoidance). Practical framework for value selection in alignment work.

Tags

Cross-References

  • Related: Orthogonality thesis, Deutsch, Friston, Yudkowsky

Notes

  • Comprehensive alignment survey
  • Novel “Value-9s” metric (practical tool)
  • Moderate P(doom) (vs Yudkowsky pessimism)
  • Engages multiple x-risks
  • Cites Deutsch, Friston, Clark, Bach, Dennett