I have also got a similar problem with LMDE edition trying to use nautilus. The annoying thing is that I found a solution online a couple of months ago but had to delete my Firefox folder as it became corrupted and now it doesn't work again and I have forgotten what I did. I have searched online and cannot find what I did before, It was not the 'add 3 lines to about:config' solution or associate 'file' type with nautilus in preferences (this means nautilus opens when you use 'open' or 'open containing folder'). I had found another solution that worked perfectly and involved either editing or creating a file in gedit. I guess this file must have been in the firefox folder that I had to delete.
Does anyone know what this solution was? I have tried googling for an hour or two and can't find it again. I will definitely keep a note of it this time!