user suddenly lost signature, language-settings, all added folders, nearly all messages

fbartels

@hodor27 hmm there have only been reports about webapp settings going missing in 8.7.0, but not in 8.6.9.

If folders and mail are deleted (is it seemingly has been the case, since only some mail remained) then its usually a client going riot.

@hodor27 said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

What could be the easiest way to restore this users store? kopano-backup?

Yes, kopano-backup can be utilised to restore single items, folders or complete mailboxes.

hodor27

@fbartels said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

If folders and mail are deleted (is it seemingly has been the case, since only some mail remained) then its usually a client going riot.

Is there any logging that i can enable, so i can see when a “client goes crazy”? the user in question used a mobile client (zpush), deskapp, webapp concurrently.
Can a zpush-client “force” it’s state onto the server, so folders,settings,etc on the serverside in the store could be lost? is this even possible? (I’d like to try and recreate this)

thx for your help!

fbartels

@hodor27 no, there is no central user logging. instead you’d need to check the logging of the individual services (z-push can have a per user logging, but it needs to be enable, all other services just have a general logging).

Z-Push actually has a setting for conflict handling, but here the server wins by default:
https://stash.z-hub.io/projects/ZP/repos/z-push/browse/src/config.php#151-156

hodor27

@fbartels said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

Z-Push actually has a setting for conflict handling, but here the server wins by default:
https://stash.z-hub.io/projects/ZP/repos/z-push/browse/src/config.php#151-156

and that is what i have on the server in question:
define('SYNC_CONFLICT_DEFAULT', SYNC_CONFLICT_OVERWRITE_PIM);

So how could this data loss have happened?

fbartels

@hodor27 with the current data one possible conclusion is “the user did it”.

If you need help in analysing this I’d recommend to get in direct contact with our support. They can help you identify log files. But without the proper logging, they also may not be able to identify the actual cause.

hodor27

I found the reason why everything was gone:
the user was autodeleted by kopano. How can this happen?
And since a new store was created, where did the old store go? why don’t i have an orphaned store? wtf?

Shouldn’t kopano keep the old store around?
how can i prevent this in the future, while keeping the ability to at least create users via ldap?

...
Sun Apr  7 00:03:08 2019: [info   ] Auto-deleting user 15 from external source
Sun Apr  7 00:03:08 2019: [info   ] Command "/etc/kopano/userscripts/deleteuser" ran successfully
Sun Apr  7 00:03:08 2019: [info   ] Auto-deleting user 15 done.
Sun Apr  7 00:03:08 2019: [error  ] User "user@domain.com" authenticated, but failed to create session: Unknown error code (0x80000002)
Sun Apr  7 00:03:08 2019: [info   ] Auto-creating user from external source
Sun Apr  7 00:03:08 2019: [info   ] Started to create store (userid=44, type=0)
Sun Apr  7 00:03:08 2019: [info   ] Finished create store (userid=44, storeid=447239, type=0)
Sun Apr  7 00:03:08 2019: [info   ] Command "/etc/kopano/userscripts/createuser" ran successfully
Sun Apr  7 00:03:09 2019: [warning] Authentication by plugin failed for user "user@domain.com": Trying to authenticate failed: user@domain.com not found in LDAP; username = user@domain.com
...

fbartels

@hodor27 said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

How can this happen?

this can only happen if the value of the attribute configured as ldap_user_unique_attribute has changed

@hodor27 said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

And since a new store was created, where did the old store go? why don’t i have an orphaned store?

this action ~~should~~ will have left an orphan store. only reason for it no not have done so if there is a script to automatically remove the store or (i am always a bit fuzzy if this is actually the case) if the store gets cleaned up during soft delete.

@hodor27 said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

how can i prevent this in the future, while keeping the ability to at least create users via ldap?

make sure that identifying values for the users do not change. alternatively you could enable the user safe mode, but this will prevent any create/delete actions.

hodor27

@fbartels
that’s odd.

the ldap_user_unique_attribute is uidNumber and this stayed the same.
Also I have no scripts in /etc/kopano/userscripts/deleteuser.d
also: i restored a server-snapshot from the first day the user was gone. There is no orphaned store present. The user who lost everything just got a new (different ID) store.

Can you point me to the code where kopano decides to “kill” users or not ?

thx in advance.

fbartels

@hodor27 said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

Can you point me to the code where kopano decides to “kill” users or not ?

You can find this at https://github.com/Kopano-dev/kopano-core/search?utf8=✓&q=Auto-deleting&type=

hodor27

well unfortunately I can more or less reliably reproduce the issue.

system:

ubuntu 16.04
kopano-server: 8.6.9.0-0+24.1
ldap-server:  389-ds-base 1.3.4.9-1

how to reproduce:

create test user, send ~50 testmails to testuser.
#for debugging only: set log_level to log_level = 0x0020006 in /etc/kopano/server.cfg
in /etc/kopano/ldap.cfg set ldap_network_timeout to 1 and restart kopano-server
run this loop: while true; do kopano-cli --sync; sleep 0.5; done
I then login as a valid user into the webapp. and keep reloading(shift+ctrl+r) the page every 5-10 seconds (I’m unsure if this step is needed)
set back and wait for user/stores to be auto-deleted/recreated (with contents missing!): watch -n 1 'zgrep -iE "deleting|creating" /var/log/kopano/server.log*'

note: after a store/user is autodeleted & recreated the contents that were present in the store before deletion are gone
Now I know that one is probably not supposed to sync kopano-users every 0.5 seconds, but that’s just my test case to put load on kopano-server and the ldap-server - I don’t see any reason why kopano-cli --sync should ever delete stores only because the ldap-server is temporarily unreachable etc.

In the server log I see a ldap query prior to the auto-deletion that returns 0 results(see below). but if I run the same query in a loop I never get 0 results. I suspect that somewhere in the ldapuserplugin / searchfunction a network timeout is interpreted as a query returning 0 results. could that be the case?

Please let me know on what more info you’d need for a bug report.

server.log showing ldapquery returning 0 results prior to auto-deleting:

Thu May  2 12:08:14 2019: [  20006] plugin: ldaptiming [00000.00] ("dc=MYDOOMAIN,dc=TLD" "(|(&(&(&(kopanoAccount=1))(|(objectClass=posixAccount)))(|(uidNumber=9004))))" objectClass uidNumber text cn uid userPassword mail kopanoAliases kopanoAdmin kopanoSharedStoreOnly kopanoResourceType kopanoResourceCapacity userCertificate;binary kopanoSendAsPrivilege gidNumber text cn kopanoSecurityGroup ou text ou kopanoSystemAdmin text cn text cn cn text cn kopanoHidden kopanoUserServer kopanoCompanyServer givenName telephoneNumber homePhone initials preferredLanguage employeeNumber sn postalAddress company title department physicalDeliveryOfficeName otherTelephone mobile pager facsimileTelephoneNumber otherFacsimileTelephoneNumber co l st streetAddress postalCode postOfficeBox otherHomePhone assistant wWWHomePage o  kopanoHidden kopanoEnabledFeatures kopanoDisabledFeatures kopanoUserArchiveServers kopanoUserArchiveCouplings url userCertificate;binary EntryUUID jpegPhoto ), results: 0
            Thu May  2 12:08:14 2019: [info   ] Auto-deleting user 56 from external source
...

my “counter check”:

root@cloud:~# while true; do ldapsearch -x -b "dc=MYDOMAIN,dc=TLD" -H ldap://172.17.0.1 "(|(&(&(&(kopanoAccount=1))(|(objectClass=posixAccount)))(|(uidNumber=9004))))" | grep "numEntries: 1" || break; done

^this should exit if the query ever returns 0 results, which it never does.

fbartels

@hodor27 said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

In the server log I see a ldap query prior to the auto-deletion that returns 0 results(see below).

that sounds then more like a bug in your ldap server to me. But I have asked our support to get in contact with you to gather some more details. If they have not reached out to you until tomorrow, please open up a support case on your own.

hodor27

@fbartels said in user suddenly lost signature, language-settings, all added folders, nearly all messages:

that sounds then more like a bug in your ldap server to me.

I thought so too for the last 2 days, but imho the correlation between network_timeout=1 and kopano suddenly starting to happily kill stores indicates against this - especially while running the same ldap-query for hours against the same ldapserver never returning 0 results.

But I have asked our support to get in contact with you to gather some more details. If they have not reached out to you until tomorrow, please open up a support case on your own.

thx, will do!