The two biggest differences I noticed when I tried XeTeX was 1) the pdf files were much smaller than from pdflatex, and 2) I couldn't use microtype [1] with XeTeX.
I stuck with pdflatex to use microtype (which is awesome), and use pdfsizeopt [2] to crush the size of my pdfs.
(More details: LuaTeX also makes small pdfs and might be the way to go, but I haven't tried it much. The size advantage over pdflatex depends on the fonts in use. There is apparently microtype support for XeTeX in the works.)
I stuck with pdflatex to use microtype (which is awesome), and use pdfsizeopt [2] to crush the size of my pdfs.
(More details: LuaTeX also makes small pdfs and might be the way to go, but I haven't tried it much. The size advantage over pdflatex depends on the fonts in use. There is apparently microtype support for XeTeX in the works.)
[1] ftp://tug.ctan.org/pub/tex-archive/macros/latex/contrib/microtype/microtype.pdf
[2] http://code.google.com/p/pdfsizeopt/ and my notes on it: http://homepages.inf.ed.ac.uk/imurray2/compnotes/latex.html#...