Part of the reason for swap is it is needed for the computer to hibernate. If you do not hibernate then I have found that with 4GB RAM I never touch swap with what I do. If you run multiple VM's or crunch a lot of data you very well could.
Think about this though, with today's drive space, what is 4-5GB to most people? Better safe than sorry and you will never miss this. I set my swaps to 4GB just because on all my computers except my big rig desktop, which does not run a swap. Never had any problems either way.