By Android convention:
Home key is same as "minimize" is in Windows -- it preserves UI state on next "restart" (== "restore from tray" so to speak).
To really "exit" please use the Back button -- then the next time the app will re-start in a "fresh" state.
Note that this is only for the app's UI -- the "background part" will keep working either way, checking mail, notifying, as you would expect.