If you haven't do so already, I'd recommend you enable the backports repository, so you get the newer Cinnamon and Nemo as a file manager:
http://blog.linuxmint.com/?p=2225. With the default Linux Mint 13 Cinnamon you will have Nautilus as a file manager, which has removed functionality that you need.
Anyway, assuming you are using Firefox as your browser, instead of opening a .torrent file with Transmission, select to download the .torrent file. Then right-click it in your file manager Nemo, and select Properties from the context menu. Then go to the Open With tab, and change the default, or add a program first, that you want to handle these kind of files by default. I think that is how Firefox figures out with which program to open the file. As I couldn't find it in Firefox's Preferences > Applications settings.