This piece has been replaced by Deep Anti-Determinism.
Go read that instead!
Say there’s some event you care about, E. You want E to happen.
Initially, your life plan is π_0, and you estimate P(E | π_0) < 1. But since you want E to happen, you come up with a new plan, π_1, to make E more likely. Now you calculate P(E | π_1) and check: are you satisfied? If not, you devise a new plan, π_2, and calculate P(E | π_2).
If each plan is better than the last, you’ll obtain a monotone increasing sequence of probability forecasts, bounded above by 1. By Monotone Convergence Theorem, this sequence will converge to sup(P(E | π_k)).
This isn’t how things work IRL, of course. Perhaps at some point, you’re unable to come up with a better plan; P(E | π_k+n) ≤ P(E | π_k) ∀n ≥ 1. Or you just run out of time and stop at some step K yielding P(E | π_K) < 1.
IRL, of course, the process is often less iterative. Model it this way: upon realizing it’s dissatisfied with P(E | π_0), the agent generates some top-k policies {π_1, … π_k} drawing from Π(u), the set of actionable policies given the agent’s u(resources).
Smarter and better-resourced agents are able to generate better plans. Let π_A be Agent A’s top-1 policy. If Agent A is smarter than Agent B, both are equi-resourced, and both want E to happen, then P(E | π_A) ≥ P(E | π_B).1
There’s an intuition going around that sup(P(E∣π))=1, considering all π∈Π(u).
To complicate matters further, you typically don’t care only about one event E. There’s typically a whole set of events {E_1, E_2, … E_N} that you’re considering influencing. For the sake of simplicity, let’s suppose you care about influencing the likelihood of one event as much as possible, and you’re willing to throw all your resources at doing so.
You have a budget; you’ll throw some constant 0 < x < 1 of your resources at generating the best plan you possibly can, argmax_π(P(E∣π)) from the set of all π∈Π(1-x)(u), and 1 - x on actually carrying out the plan.
All this means that I am sceptical of most forecasting. If you model something well enough to be confident in your prediction, you should understand the levers affecting it so well that you’d be able to move them and thus alter your prediction.2 Only if you truly don’t care about the outcome would you not. Or if you endorse the forecast staying at, say, 0.6, and are content with that. Otherwise, you’ll set off one of these non-stationary spirals.
Forecasts should be viewed as commitments / endorsements. And when you give one, you should say what you’re doing and why, i.e. “I think the elasticity to me acting would be very low”, or “I think the elasticity to me acting would be fairly high, but I’m not going to allocate attention towards acting, because I’m instead going to allocate my action mostly towards xyz”.
Forecasts should be indexed by policies to be coherent.
And a probability forecast is the supremum over a set of policies you’re willing to take to nudge the forecast in your preferred direction.
epistemic status: notes, scratchpad, “started looking into but not planning to continue further rn”. formalization doesn’t feel worthwhile here — too ‘theoryslop’ — i’m going to pause this here unless something re-surfaces it.
This is related to a broader notion of ‘intelligence as tree search’ that I’d like to flesh out at some point.
There’s a weird phenomenon in x-risk reduction where you’d want to find the lever where one small team can seriously move the needle / avert catastrophe, but if you were to find one, that would be fairly terrifying, not least because it would imply the existence of more.
Fortunately, you can apply a transform on x-risk reduction and convert it into ‘good future design’:
https://guzey.com/existential-risk/
> If you model something well enough to be confident in your prediction, you should understand the levers affecting it so well that you’d be able to move them and thus alter your prediction.
I confidently predict that the sun will rise tomorrow. Does that mean I can stop it from rising?
> If each plan is better than the last, you’ll obtain a monotone increasing sequence of probability forecasts, bounded above by 1. By Monotone Convergence Theorem, this sequence will converge to sup(P(E | π_k)) = 1.
> This isn’t how things work IRL, of course. Perhaps at some point, you’re unable to come up with a better plan; P(E | π_k+n) ≤ P(E | π_k) ∀n ≥ 1. Or you just run out of time and stop at some step K yielding P(E | π_K) < 1.
Small mathematical quibble, but it's also possible that sup(P(E | π_k))<1, but there is still in infinite sequence of increasingly better plans that converge to, say, 0.8.