Mufflers work on the principle that if two opposite sound waves collide they cancel each other out. This phenomenon, known as destructive interference, is also used in noise-cancelling headphones and soundproofing.
Sound waves create local changes in atmospheric pressure, which is why you can feel the bass thump if you stand close to a speaker. A collision between two sound waves that changes the pressure the same amount but in opposite directions will nullify both waves, silencing the noise.
Mufflers use a combination of complex tubing and chambers in order to bounce the sound waves and create opposites which will interfere with one another. The positioning of the holes, the distance between the walls of the chambers and the volume of air that the muffler holds are all carefully calibrated to offset sound waves at the frequency most used by the car engine.
Mufflers are fixed to a narrow set of frequencies, but cars produce a wide range of sounds, especially as they change speed. Mufflers are therefore also heavily insulated, providing physical noise dampening on top of destructive interference.