Is your graphics card from Nvidia or ATI? Their closed-source binary drivers need to recompile certain small portions that extremely depend on the kernel that is installed. You touch the kernel == you break the drivers.
Special "thank you" to Nvidia and ATI .... the problem wouldn't exist if they opensourced their stupid drivers just like everyone else (Intel, Matrox, SiS, many others ....)
I at least give Nvidia the credit for producing top-notch Linux drivers. Thanks guys. But it's a pain in the a...... (you know where