I have used most every Linux distro that has Xfce as it's Default desktop.
The transfer lag is minimal (at least, on my machine) but the new Xfce-4.8 release has addressed this, as well as, several other slightly annoying issues.
Some of these 'bugs' and previous issues are spoken of, in the Xfce Forums. The Xfce Devs as working at providing 'fixes' to these.
You can ask your questions in the Xfce Forums, they are better apt to help, with providing answers for Xfce specific topics.
Perhaps, you might look into upgrading your version of Xfce from 4.6.2 to 4.8 pre2 . . . not been released as 'stable' yet.
But, it works really well. I have my Xfce upgraded to 4.8pre2 in my Debian 6.0 Squeeze, beta2 install.
Xfce04.8 release >> http://www.xfce.org/about/news?id=25
Xfce Forums >> http://forum.xfce.org/
Xfce Wiki >> http://wiki.xfce.org/
Xfce 'bugs' page >> http://bugzilla.xfce.org/
Come on over to the Xfce Forums . . . we has cookies !!