So I eventually went around trying UP4 this morning. Appart from the awful gnome shell (easy to replace with cinnamon thankfully), I stumbled accross another problem.
When booting kernel 3.2 (3.0 works fine), my comp goes mad: it ends up in terminal/recovery mode and locks up. Upon investigating I found out is that there is tons of Xorg and libgdm3-something-slave processes being created, so much the loadaverage goes to 66/25/12 within a few seconds and keep rising until the comp is locked up. Sounds like a buggy infinite loop to me. Tried reinstalling nvidia drivers, no luck. Any idea on how to process to find out the issue and/or fix it?
P.S. as the comp locks up I can't really do anything like pulling logs up, I barely have the time to run one command and usually do not get the full output.