Does XFCE try to remember what you had open last time you logged out? This would account for most of the logout time in particular. You want to make sure...