Here's a thread that might help regarding Nautilus taking over the desktop:http://ubuntuforums.org/showthread.php?t=351744
I use Konqueror instead of Nautilus anyway, so I'm having no problems. I think I read somewhere that when you start up Nautilus it starts up all the GNOME stuff and leads to problems. Refer to the thread above for more info.
Regarding making Openbox the default, I may be misunderstanding your set-up but here when I log into Openbox, I get the option of making it for this session only or making it the default session. What exactly happens when you log in and select Openbox?
As for Locrian's other comments, yes Openbox takes some work to get it set up to your liking, but once it's set up it's beautiful, definitely as cool as Fluxbox, if not better. I have it installed in Mint Elyssa as well as in Ubuntu Hardy. I've used Fluxbox a lot, too, and I love it, but I haven't tried the Fluxbox community edition. I would say keep on working with Openbox until you get it to work as you want, because it's worth the extra effort. And once you've gotten things squared away, if you've kept good notes, it's not so hard at all to do it the second time around. I used my notes from Hardy to install it and set it up in Elyssa, and it things went smoothly.