Mail vanishes after short listing

[Ralf Landgraf]

Sorry if this is already discussed. The search terms I got did not lead to an answer. I also understand that this is not strictly an Aquamail Bug, but can possibly addressed by settings in Aquamail.

I have used Aquamail Pro for a long time and love it. However, a few months ago my company switched to office365 as their mail system. I set up the EWS interface with Aquamail and it works fine, but I am not comfortable with the degree of control I am handing over to IT when it comes to my private hone. So I prefer to stick to IMAP.

IMAP works great except for a rare but very serious problem. Occasionally when I check my mail on my phone using Aquamail I see a message, even the summary content when I go over it (without opening it). When I get back later to try and read the message it has vanished. Completely. It is not in any junk, trash or other folder, not even on the server. When I get to work and check with thunderbird, it is as if the message never existed. This happens to individual messages. One out of 20 in a given morning and seems to be random. When I call people and ask them to resend, it makes it without problem.

From reading up on this, I understand that this is an exchange server sync problem (or similar). However, it does not happen with Thunderbird. Is there a setting in Aquamail that I can change to avoid this. I am otherwise stuck with a scenario where opening my mail with Aquamail risks obliterating an important message and I may never know. My companies demands when using EWS are very intrusive (dictated login procedure, password type and change, right to wipe the phone remotely ....).

Thanks for any suggestions
RL


On

[StR]

IMAP works great except for a rare but very serious problem. Occasionally when I check my mail on my phone using Aquamail I see a message, even the summary content when I go over it (without opening it). When I get back later to try and read the message it has vanished. Completely. It is not in any junk, trash or other folder, not even on the server. When I get to work and check with thunderbird, it is as if the message never existed. This happens to individual messages. One out of 20 in a given morning and seems to be random. When I call people and ask them to resend, it makes it without problem.

From reading up on this, I understand that this is an exchange server sync problem (or similar). However, it does not happen with Thunderbird. Is there a setting in Aquamail that I can change to avoid this.

From this description, I am confused, - the problem of the disappearing messages, - does it happen in an IMAP account or in an EWS account?

My companies demands when using EWS are very intrusive (dictated login procedure, password type and change, right to wipe the phone remotely ....).

Those rules are draconian, IMHO. I've heard of such companies. I know some developers who refused setting up e-mail access on their personal phones for that reason. I wouldn't give up my own phone to such a company. If I were required to do that, I'd probably consider using a separate phone if bought and paid by the company.

[Ralf Landgraf]

Thanks for agreeing with my rejection of the EWS option. Those rules are extreme but there are healthcare databases involved. So I just stick to IMAP with fewer options (no address database access) as a compromise

In answer to the clarification request. The problem occurs with IMAP. I only mention the EWS because that would (presumably) fix the issue, but I don't want to go that route.

I talked to our IT, and the only answer they come up with is to drop aquamail in favor of outlook for android (but that also insists on running EWS)

[Kostya Vasilyev]

So just to be clear: your account is at Office 365 and is set up in Aqua Mail as IMAP?

I also have an Office 365 account, and have tried using it as IMAP, did not see this issue.

The best way to diagnose this is for you to capture the issue in the app's debug log with "raw session data" enabled so I can see the app "talking" to the mail server.

Please see the link in my signature, below.

[StR]

It doesn't sound like it is a problem of the app. Rather, it sounds as a combination of some server settings and some other action happening from a different e-mail client.
Basically, Aquamail (and any IMAP mail client) is a "window" into what is on the e-mail server. It shows messages on the server, and if the message is deleted on the server, it disappears in Aquamail. So, in the situation you've described, something deletes the message on the server between the time when Aquamail first downloaded it (and you saw it in passing), and when you look at it again.

So, hence an additional question: do you have any other e-mail client that might be connecting via POP3 (cell phone, tablet, laptop/desktop, webmail, etc.)? If you do, - once it downloads e-mails, they may be deleted on the server. (Either it is configured in the client to delete downloaded messages, or in some cases the POP3 server will delete or hide messages ones downloaded via POP3, even when the e-mail client would be configured otherwise.) This type of scenario has been frequently seen among e-mail users, and it is not related to Aquamail per se, but would be the case with any client.

And it sounds like this mysterious disappearing happens only with the unread messages. This would be consistent, if the server is configured to serve to the POP client only unread messages. I'd expect that that additional POP3 client is configured to check messages very infrequently (or you tend to read your messages on the phone as soon as they arrive), - otherwise you'd be seeing more messages disappearing.

By the way, - do you have some other e-mail client accessing e-mail via EWS (e.g. your laptop's Outlook). I haven't worked much with the Exchange server, but I wouldn't be surprised that there is a scenario where something similar to what I described with POP3 client/server might be happening due to the EWS client/server, and how the interaction between EWS server and IMAP is configured internally.

PS. While I was writing this, Kostya has responded to you. I post my suggestions anyway in case they will help you understanding possible interference effects.

[Kostya Vasilyev]

Re: Basically, Aquamail (and any IMAP mail client) is a "window" into what is on the e-mail server

This 10000%

Re: do you have any other e-mail client that might be connecting via POP3 (cell phone, tablet, laptop/desktop, webmail, etc.)? If you do, - once it downloads e-mails, they may be deleted on the server

And this too

It's a good idea to check what's there in Office 365 web mail.

Re: While I was writing this, Kostya has responded to you. I post my suggestions anyway in case they will help you understanding possible interference effects

Thank you StR, your comments are always spot on and insightful.

[Ralf Landgraf]

Thank you for the suggestions. Based on the responses it seems like this problem has not come up so far.
To clarify in response to your questions.

This is an institutional office365 account

There are no POP interactions setup on my end

I am accessing on three computers (work, home, laptop, all using Thunderbird and IMAP for consistency  [no issues])

I am also using Aquamail on my Android via IMAP. This is the only case I have come across the issue.

I checked all thunderbird data as well as the direct web interface with the server. The email has vanished in all of them. The latter is declared impossible by IT. So I must be dreaming as do the senders who confirm sending.

Also, I am aware of the right swipe option (very nice feature)but the bright red trashcan is hard to miss and the items still end up in the deleted folder in that case

The problem I am referring to is EXTREMELY infrequent and does not follow any logic. I had 4 instances of individual mails that I noticed because I was expecting the answers and are interacting directly with the sender. This is 4 / month, i.e. approx  4 out of 2000-3000 emails. So if you say that you checked the IMAP connection with Office365 and it worked, that is I would have said until now. I have used aquamail with the work outlook server for five years and the last year has seen the transition to office365.
Given the extremely low frequency, running a log on my email exchange for a month would not be very helpful unless the mode of interaction for a small subset of messages, even without the problem, would be helpful.
It takes an accidental observation or a really important contact who follows up with a second mail or phone call to even notice the rare problem. For important mail that may not get a follow up, that is too big a risk. If the same issue should occur with owa, I let you know. If a short sample log of normal server communication would be helpful, please let me know.

[Kostya Vasilyev]

A short log showing just this specific incident would be enough.

Question is, how to go about making one, given how infrequent the issue is? I really don't know.

One idea is to keep the log enabled and wait for the issue to happen, then save the log, and trim it really well.

When you open a message, the log will have things tagged with MessageDisplayShard.

And then the actual IMAP commands (e.g. fetch the message to completion, mark as read) would follow that, and they'll be all tagged with IMAP or IMAP_RAW.

Example:

          [IMAP.203011]  I  Sending: kman6 UID STORE 1692 +FLAGS (\Flagged)
      [IMAP_RAW.203011]  I  Data is <111>:
                         I  * 793 FETCH (FLAGS (\Flagged \Seen $X-ME-Annot-2 $HasAttachment) UID 1692 MODSEQ (86394))
                         I  kman6 OK Completed
                         I  Line: * 793 FETCH (FLAGS (\Flagged \Seen $X-ME-Annot-2 $HasAttachment) UID 1692 MODSEQ (8639
                            4))
          [IMAP.203011]  I  New flags: 0x0003, UID: 1692, MODSEQ: 86394
      [IMAP_RAW.203011]  I  Line: kman6 OK Completed
          [IMAP.203011]  I  Result for kman6: 0 Completed, traffic: 111 read, 40 write
          [DATA.203011]  I  Clearning bits 0x00000002 for message with _id = 2333
                         I  Updating MESSAGE with _id = 2333

[Kostya Vasilyev]

Oh I see, you don't even open the message.

l I see a message, even the summary content when I go over it (without opening it). When I get back later to try and read the message it has vanished

Please try this then: long press on the account on the app's main screen -> options and folders -> IMAP options -> turn off "use optimizations".

Just in case Office 365 IMAP advertises "CONDSTORE" and "QRESYNC" but their implementation is buggy (I assume this would be a bug on their side, since I've had no reports of anything strange about it until now).

Neither K9 nor Thunderbird support/use these IMAP extensions, to my knowledge (AFAIK, Thunderbird use to use CONDSTORE, but it got removed fairly recently).

[Ralf Landgraf]

Thank you. That is very helpful. I will disable optimization. Given that Thunderbird does not use this option this looks like a very good candidate for trouble shooting.
I will make myself a calendar entry to get back in a month to report back if I experienced it afterwards.

[Ralf Landgraf]

OK, here is the feedback. It seems like not using optimization did indeed fix the problem. I have now used this setting for approx a month, and  I had no problems since. Thank you again for your help. I am happy that I can continue to use my favorite email app. Keep up the great work

[Kostya Vasilyev]

OK, here is the feedback. It seems like not using optimization did indeed fix the problem. I have now used this setting for approx a month, and  I had no problems since. Thank you again for your help. I am happy that I can continue to use my favorite email app. Keep up the great work

Wouldn't be the first time that an IMAP server advertises support for CONDSTORE / QRESYNC (optional capabilities that make it faster to re-sync from an earlier known state), but there are issues with them.

I'm glad the workaround worked, thanks for the follow up!