Author Topic: Migrated Mails - New Mails not shown  (Read 9497 times)

Breece

  • Newbie
  • *
  • Posts: 6
Migrated Mails - New Mails not shown
« on: May 06, 2017, 05:55:42 pm »
As suggested here:
https://www.aqua-mail.com/forum/index.php?topic=1325.msg34167#msg34167 [nofollow]
I migrated one Gmail account to another. Copying the mails in the correct new folders was a bit of a hassle but everything works now, except in AquaMail.
The mails are in the correct folders with the correct labels (in thunderbird, online in gmail and in the gmail android app) BUT in AquaMail only old mails are getting synced.
E.g. the newest mail in the inbox is from today but aquamail insists on only showing mails from 2015... At the same time Aquamail tells me that 1/519 mails is new but I cannot open it within aquamail.
I even tried to archive old mails (older than 2017) from one of my subfolders. AquaMail shows emails up to february but not the newest ones..

I think the timestamps might be broken, tried to change the settings from "from Mailserver" to "from header" but it does not work. I think Aquamail loads "old" mails before new ones, as if the dates are completely reversed..
Completely new mails since setting the new account up in Aquamail show up.

Anyone got an idea? Thank you in advance
« Last Edit: May 06, 2017, 05:58:22 pm by Breece »

Breece

  • Newbie
  • *
  • Posts: 6
Re: Migrated Mails - New Mails not shown
« Reply #1 on: May 06, 2017, 06:14:40 pm »
Thank you! That works, but why do I have to use this "workaround" ? Is there any way to "fix" this "problem" (except copying all the mails back and forth to another account in a different order)?

[edit]
The "problem" now is that I cannot use my archive folders because they are shown as empty.

Breece

  • Newbie
  • *
  • Posts: 6
Re: Migrated Mails - New Mails not shown
« Reply #2 on: May 06, 2017, 06:59:15 pm »
Alright, thank you for the quick replies.

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Migrated Mails - New Mails not shown
« Reply #3 on: May 07, 2017, 03:17:38 pm »
Re:  That works, but why do I have to use this "workaround" ?

Because IMAP normally files messages by date/time -- that's the "natural" order.

A migration or upload screws this order up - if the "uploading" into the new account is not done in proper order.

And unfortunately, I don't know any mail system or email app that would consider this "little detail" when "migrating" or "uploading".

When you set the "messages to sync" to a "time range value", Aqua Mail stops relying on the "natural order" and instead searches the server for "messages from last N days".

---

Desktop mail apps can afford to just load "everything", and Gmail app or web mail use custom protocols (not IMAP).

By comparison, the issue doesn't exist in the EWS (Exchange) protocol where a mail app can ask for messages to be returned sorted by date (and stop when it gets enough).
« Last Edit: May 07, 2017, 03:21:31 pm by Kostya Vasilyev, Aqua Mail »
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

Breece

  • Newbie
  • *
  • Posts: 6
Re: Migrated Mails - New Mails not shown
« Reply #4 on: May 10, 2017, 08:33:27 pm »
Thank you for the in depth explanation!

StR

  • Hero Member
  • *****
  • Posts: 1558
Re: Migrated Mails - New Mails not shown
« Reply #5 on: May 12, 2017, 03:40:59 pm »
PS. Next time you copy an account into another, begin with OLDEST mail to the NEWEST (first come in server, first copied to new account)
I've migrated several accounts via IMAP.

Most recently, just a few months ago, I was migrating an account that was about 9 years old, had about 13GB worth of messages, and went through being handled by different server-side software combined with the direct access by clients. That history yielded some glitches in the way messages were indexed. As a result, migration software was choking on some messages, and I've tried various ways, and had to do some manual intervention to assure all (or rather most) messages were migrated.

Unfortunately, with some methods, when you do the migration, you have no explicit way of controlling the sequence in which messages are moved. E.g. one of the relatively easy ways is to copy folders from one account to another one in Thunderbird. Then you are hoping that Thunderbird would copy the messages in their sequence of appearance, which, in principle, is not guaranteed. (But surprisingly, it seems to do that, at least most of the time.)

And unfortunately, I don't know any mail system or email app that would consider this "little detail" when "migrating" or "uploading".
@Kostya: Actually, <i>offlineimap</i> copies messages according to their sequence provided by the IMAP server, -- as long as there are no problems/glitches. If it chokes on the specific folder, then, during the subsequent re-run, you might get some artifacts in the sequence. That's my personal observation.


Long click on your account name, in main view -> Options and folders -> Account options -> Messages to sync -> By time period -> 30 days
I actually wonder if that always produces the result that you are thinking about.
Each message accessible via IMAP server has several dates associated with it:
1. Date in the "Date:" header set by the mail client, or by the first MSA (mail submission agent) server if it was missing.
2. "Internal date", set in the database (index) of messages by the SMTP or IMAP server. Typically it is set to the date when the message arrived.
3. I am not 100% certain, but I suspect that some mail clients might also be able to used the last "Received" header to derive the date when the message arrived. Maybe none of the modern mail programs is using that.

I suspect that there could be situations when the Internal Date (#2) is not copied over when the messages are migrated from one server to another (even though according to RFCs it should be). I wouldn't be surprised that at least in some scenarios of migrating messages, this date can be lost. In that case the "new" IMAP server needs to set that date to some value. I see two possibilities: it could be set as the date when the message was migrated ("first seen by the IMAP server"), or, in some implementations, it could be set to one of the dates #1 or #3 above, if the IMAP server is smart.

RFC 3501 has this description::
Quote
   In  the case of messages delivered via [SMTP], this SHOULD be the
   date and time of final delivery of the message as defined by
   [SMTP].  In the case of messages delivered by the IMAP4rev1 COPY
   command, this SHOULD be the internal date and time of the source
   message.  In the case of messages delivered by the IMAP4rev1
   APPEND command, this SHOULD be the date and time as specified in
   the APPEND command description.  All other cases are
   implementation defined.

Furthermore, in case of APPEND:
Quote
If a date-time is specified, the internal date SHOULD be set in
      the resulting message; otherwise, the internal date of the
      resulting message is set to the current date and time by default.

@Kostya, your response implicitly suggests that choosing "30 days" always works...
But it sounds that there are situations (most likely rare) that the internal date could be lost during some migration. This is not a criticism. Aquamail does what it can do.
I am just saying that this recipe also has limitations.

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Migrated Mails - New Mails not shown
« Reply #6 on: May 12, 2017, 10:34:06 pm »
Re: But it sounds that there are situations (most likely rare) that the internal date could be lost during some migration.

Possibly, I personally haven't come across that, but --

- Migrations and uploads *should* go "oldest to newest". Sometimes they don't.

- Migrations and uploads *should* set INTERNALDATE to Date header. Sometimes they don't.

- Mail servers should correctly parse the date set on uploads (APPEND). Sometimes they don't.

---

It is possible in Aqua Mail to set the "which date/time to download" to "date/time sent, Date header" and also "sync by ... days".

And then our IMAP code will do a server search on the Date header.

Alas, there are mail servers that cannot perform searches like that (I'm thinking one specifically, a very large and popular Russian Internet portal and its email service).
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/