You can reduce the swappiness setting, and this will delay using of swap till there is less free memory available:
http://community.linuxmint.com/tutorial/view/998. By default the kernel swaps early, so there is always a larger amount of free memory available--handy when you start a new application or load a big file, as it won't have to swap first as that was already done earlier. Reducing swappiness makes the kernel wait longer, so if you start that new application or load that big file it may take a little longer as it may have to swap first to make the needed free memory available.
As for caching and buffering, don't concern yourself with that. Read this for understanding how you and the Linux kernel speak different languages on that:
http://www.linuxatemyram.com/