The doctrine assumes that each side has enough nuclear weaponry to destroy the other side and that either side, if attacked for any reason by the other, would retaliate with equal or greater force. The expected result is an immediate escalation resulting in both combatants' total and assured destruction. It is now generally assumed that the nuclear fallout or nuclear winter resulting from a large scale nuclear war would bring about worldwide devastation, though this was not a critical assumption to the theory of MAD.National defense as commonly envisioned is not excludable. Armies, air forces, and so on aren't capable of selectively defending small areas -- they can either defend a large area or no area at all. This is a problem for libertarians. Without the ability to exclude non-customers, a market cannot form, and the libertarian society cannot protect itself. The resulting equilibrium is government-provided national defense.
The doctrine further assumes that neither side will dare to launch a first strike because the other side will launch on warning (also called fail-deadly) or with secondary forces (second strike) resulting in the destruction of both parties. The payoff of this doctrine is expected to be a tense but stable peace.
Retaliatory approaches, like MAD, however, are easily excludable. If a customer doesn't pay, and if his land is invaded or destroyed by some opposing force, the retaliation business simply doesn't retaliate.
It is thus possible to privatize national defense. People contract with a retaliation company, and attackers avoid them for fear of annihilation. People not contracting may still be attacked, so they have an incentive to pay for the service, and national defense is provided efficiently, on the market.
Furthermore, this approach doesn't necessarily require nuclear weapons. Perhaps a company could retaliate with assassins who kill the person that gave the order to invade a customer's property (and maybe his family, too). There are plenty of creative possibilities waiting to be explored by a society allowing experimentation.