K-1537: Unable to move object user

bhuisman

Hi @thctlo - is the first one of your systems perhaps a multi-tenant setup?

thctlo

Hi, yes correct, the one having this is a Multi-tentant setup with mysql (db) plugin.

thctlo

Bob ( or Felix),

Any suggestions here because i cant find where this is coming from.
Its a repeative interval of 15 min .

i’ve looked at kopano-dbadmin, i dont see or can find anything about this when searching with google, except the site with the K-XXXX log entries.

i saw in : Kopano-dev/kopano-core/blob/master/provider/libserver/ECUserManagement.cpp
lines 2434 - 2447 the part what generates the error.
ps. line 2437 * the the … ( small typo )

How should in interper this message?

bhuisman

@thctlo we managed to reproduce the issue on our lab environment - the problem occurs when in a multitenant setup the sync (kopano-admin --sync) tries to determine whether a user (-mailbox) has been recreated in a different tentant or that a mailbox has simply been moved (trying to avoid delete + create and do a move instead).

The development team is looking into it - you can track progress in https://jira.kopano.io/browse/KC-1450.

It does not look like this error message affects normal operation, but it could cause trouble when mailboxes are actually being moved between tenants.

thctlo

Hello Bob,

Thank you for reply back.
I’ll keep an eye on jira bug report.

And i see on my server it does not effect the users also, only logs are growing fast.
i’ll handle that.

Greetz,

Louis

kg

I have to add something on that.
It is occuring for us as well (Multi-Tenant Setup; LDAP Backend, mysql). But for us it’s actually causing issues, because it’s deleting the users and companies and the re-creating them (attaching the existing stores). If a user tries to login or sync via z-push or is receiving an email during this, it fails (user is unable to login or sync and mails are rejected).

So this is a very severe issue for us forcing us to use older versions of Kopano until it’s fixed.

thctlo

Hai, any progress on this?

Because of you try to cleanup or reattach users then NO users can be resolved by kopano.
while the tests of the email adresses resolves fine with postmap.

Update, after running , in this order:

kopano-dbadm index-tags
kopano-dbadm kc-1375
kopano-dbadm k-1216
kopano-dbadm usmp

mail is delivered again in the user mailboxes, but this was a very strange thing here.

How did this happen.?

I did run : kopano-storeadm -O and removed some old stores and domain names. ( companies ).

kopano-storeadm -O
Stores without users:
Store GUID                              Guessed username        Last login      Store size      Store type
--------------------------------------------------------------------------------------------------------------
934df4xxxxxxxxxxx09379ad3af0       vdom2.tld         <unknown>       0.00 MB         public
bdcf0axxxxxxx6c8b58693c6f68e        vdom1.tld          <unknown>       0.00 MB         public

Users without stores:
Username
--------------------------------
# Multple username ( for me companies) are shown here as: 
vdom1.tld
vdom2.tld

but also 
Everybody

I dont touch this again untill this bug is fixed.

Rolleps

Hey,

I want to push this problem too. We have a multi tenant setup and are also experiencing the problem like @kg.
Please fix this bug as soon as possible or tell us how to work around :)

fbartels

@Rolleps said in K-1537: Unable to move object user:

We have a multi tenant setup and are also experiencing the problem like @kg.

Since you both share the same email domain, I would be surprised if that wouldn’t be the case ;-)

The workaround would be to use our official releases, especially if you are providing Kopano in a hosted environment to your customers I would strongly recommend to not rely on untested nightly builds.

Chris2

Hi all,

i am also faceing this issue, being suspicious about those log entries.

Also the error came up with the upgrade to latest nightly version.

If there is any possibility to help speeding up solving this problem, please let me know.

thctlo

If your kopano is crashing, you can use this as workaround.

systemctl edit kopano-server and add:

[Service]
Restart=always
RestartSec=3

Chris2

Thank you very much for your reply, but in this case the server is not crashing, it keeps running functional, but those mentioned log entries are flooding the log all 15 minutes.

There are no further informations in the log, also in log mode = debug.

If there is anything i can do, help fixing this bug, please let ne know.

thctlo

@Chris2 said in K-1537: Unable to move object user:

informations in the log, also in log mode = debug.

disable kopano-monitor and kopano-statsd

That did help me in the log flood.
I got my kopano stable running again, but i disabled everything i didnt need.

Chris2

Thanks, sounds like a good plan to get the log clean again as workaround, until the bug is fixed.

But in long term i would like to fix the problem itself, so if i can do anything to support this, please let me know.

thctlo

Just to keep this thread complete on what im doing in my DB.

this database also suffert from the problem.
SQL Failed: Duplicate key name ‘tmptag’, Query: “ALTER TABLE properties ADD INDEX tmptag (tag)”

after a look at git : f43e00622a9624e31b58c280058c2bb6580e6be2
im gambled that this would help me to fix this problem.

so my steps,
While everything is running.

mysql -u root

use kopano;
ALTER TABLE properties drop INDEX tmptag;
ALTER TABLE tproperties drop INDEX tmptag;
ALTER TABLE mvproperties drop INDEX tmptag;
ALTER TABLE indexedproperties drop INDEX tmptag;
ALTER TABLE singleinstances drop INDEX tmptag;
ALTER TABLE lob drop INDEX tmptag;
\q;

systemctl restart kopano-server
kopano-dbadm k-1216

which cleared the INDEX, removed the error, make kopano-server do it thing, and re-added the INDEX with kopano-dbadm.

Can someone confirm these steps if i did not miss anything here.

At least, i happy to see these messages, are gone.

Fri May  3 12:02:04 2019: [error  ] 0x7fbc8365c890: SQL Failed: Duplicate key name 'tmptag', Query: "ALTER TABLE properties ADD INDEX tmptag (tag)"
Fri May  3 12:02:04 2019: [error  ] KDatabase::I_Update() query failed: "Duplicate key name 'tmptag'", query: ALTER TABLE properties ADD INDEX tmptag (tag)

thctlo

hm, i’ve tried to update today also .

From 8.7.80.931 to 8.7.80.936.529d71bca-0+193.1

kopano-server still crashes if you check the users/group/company list.
if this is still enabled in server.cfg
user_safe_mode = yes

set to no, kopano-server does not crash.

im still not seeing my last added company and user, disapeared again.
and im sure thats related to :

Fri May  3 14:03:56 2019: [info   ] Auto-deleting company 1353 from external source

Any suggestion to remove that manual from the DB? or how can i find that.

gduran

Hi everyone,
It took me more than a day but I’ve found a workaround to this problem while setting up a brand new install to avoid corrupting real data.
Anyway, I also stumbled on the “K-1537: Unable to move object user” issue. Moreover I also got “K-1537: Unable to move object security group” on top of that after a while. After investigating closely all the log entries I all of a sudden got an idea. Eureka!!!
The workaround is as follows and I bet this will also will help the dev-team:
I all the time noticed that the username in the log looked like joe@example.comExample Ltd. So the company name was added too.
I renamed all the company names in my ldap tree to the FQDN. i.e. renamed “Example Ltd” into “example.com”.
Changed loginname_format into just loginname_format = %u in the server.cfg.
Stopped kopano-server service and dropped all the tables and started the kopano-server service. (Could do this as I was starting from scratch. Next I will try to apply it to a life database using the kopano-dbadm tools although I experienced also odd behaviour as it broke the relations between groups and companies several times while figuring out what the problem was but that maybe was caused by the fact of the K-1537 error.)
executed kopano-admin --list-companies, kopano-admin --list-groups, kopano-admin --list-users and checked the users table which looked OK.
Note: Another problem I ran into is that kopano-cli --list-companies works but kopano-cli --list-groups, kopano-cli --list-users don’t!!! Another bug?
After the above I’m able to login using the email address as username and works like charm. Obviously this is not a permanent solution but I hope that the dev-team understands what to look for. Please let me know when this bug is ironed out…

Next day edit:
Maybe cheered too early! Although I was (and still am) able to login after the above exercise nasty log entrees keep on coming:
1) K-1515: Object not found unknown user “example.com”: example.com not found in LDAP:
This is repeated for all 6 defined companies in the LDAP tree. (example.com, example2.com, example3.com, example4.com, example5.com, example6.com). Though these are defined as objectClass: kopano-company. Does Kopano thinks these are users?
2) K-1537: Unable to move object user “joe@example.com” (id=19):
Only for this user as only one user is defined.

Chris2

Just wanted to mention that the latest nightly build solves the problem, error log is clean now.

Thanks to all suggestions and the dev-team for fixing this bug.

gduran

I did install version core-8.7.81.48-Ubuntu_18.04-amd64 today, the previous errors disappeared indeed but instead now “[warning] K-1515: Object not found unknown user “example.com”: example.com not found in LDAP” while this entry is not a user but a company. Same for the other companies in the LDAP tree. Any ideas?

I just noticed that after the upgrade kopano-cli no longer exists in /usr/sbin/

A day later: same with core-8.7.81.88-Ubuntu_18.04-amd64, does not contain kopano-cli (???).

fbartels

@gduran said in K-1537: Unable to move object user:

A day later: same with core-8.7.81.88-Ubuntu_18.04-amd64, does not contain kopano-cli (???).

yes, that was already mentioned in https://forum.kopano.io/post/13627. it will return, no worries.

@gduran said in K-1537: Unable to move object user:

while this entry is not a user but a company. Same for the other companies in the LDAP tree.

yes, that is an old behaviour and can safely be ignored. I thought we had a ticket for that already, but could not find it with a quick search.