There is a genuine widespread fallacy that lactic acid is the causal culprit to delayed onset muscle soreness (DOMS). While the full details of how DOMS happens hasn't been figured out yet, we do know that it's likely part of the inflammatory process (which is why asprin and ibuprofen taken before and after exercise can modify the DOMS sensation). Lactic acid is a bi-product of anaerobic cell metabolism. Its production is not limited to muscle cells and cells have adapted to deal with lactic acid in such a way that it's not just wildly secreted all over the place like some out of control fire hose.
What may be happening in DOMS is that training in and of itself may not be the mechanism by which muscles are "broken down", but rather the stimulus. So, you're not doing any "breaking down" _while_ you exercise (unless you're going for extremely long bouts), but after you're done, the cells in your muscles and tendons react to the exercise stimulus and actually start to break things down. This may involve the degradation of collagen fibres and other tissues by these cell types. As a result of this cell-mediated break-down, noxious substances may be secreted which interact with the the sensory nerve endings in the area, which then send pain signals to your brain. The reasons this takes a few hours to 2 days is that 1) the rate of "breaking down" varies from person to person, dependent mostly on genetics; and 2) the amount of time in which a threshold concentrion of noxious molecules is achieved also varies, dependent upon genetics, the efficency of one's body in getting rid of these substances (ex. if the body clears these biproducts at a lower rate then they are produced), and perhaps the extent of overload on the system.
Of course, this is all theoretical conjecture.