Any sane package manager would complain about the foreign architecture, though there is always the option of forcing the install. The ever so big catch is that a 64-bit kernel would likely go looking for /lib64, or 64-bit versions of various other things that you just wouldn't have if you install only a 64-bit kernel. I REALLY don't recommend that you do this - I'm fairly adventurous, and I'd NEVER try it myself. I mean, you'll still have the other kernels to boot from if the 64-bit one won't work, but the best-case scenario results in a bit of a schizoid build. Best by far to install a 64-bit version from the ground up.