"To-Do List" stops displaying new items, searching for items still works

con356

Hi forum,
i am not sure how to explain this one correct…

If i flag emails or enter new entries in “Tasks” no new entry is shown in the “To-Do List”. But if i search for those newly made or flagged items, then they are shown as a single search-result.

The “To-Do List” shows older entries correct. The List just stops working…

As far as i remember: A few days ago i changed language settings in /etc/default/kopano

#KOPANO_LOCALE="C"
KOPANO_LOCALE="de_DE.UTF-8"

#KOPANO_USERSCRIPT_LOCALE="C"
KOPANO_USERSCRIPT_LOCALE="de_DE.UTF-8"

Changing this back did not help. May be this has nothing to do with the problem.

This problem is effective i every user account.

Could somebody “push” me in the right direction?

Server: Debian_9 64bit
core-8.6.80.3_0+79-Debian_9.0-amd64
webapp-3.4.7.1275+733-Debian_9.0-all

Regards
Enno

Coffee_is_life

Hello @con356,

this issue seems related to system folders-bug.
try this script on a testuser and systemfolder ‘task’
https://stash.kopano.io/projects/KSC/repos/core-tools/browse/recreate-systemfolders

//EDIT: its a bit strange that all users are effected, thats why try with a testuser

btw if you want to rename your stores to german locale, you need to run
https://stash.kopano.io/projects/KSC/repos/core-tools/browse/localize-folders
in order to rename existing stores

coffee_is_life

con356

Hi coffee_is_life,

thank’s for help!
I made a full clone of the whole debian-system, so any test is without risk. :-)
“recreate-systemfolders” dosen’t seem to made a change to nothing … !?

python recreate-systemfolders.py --user test --create Tasks --systemfolder task
User('test')
create tmp folder Tasks
Promote new folder to be a system folder

But there is no difference in Systemfolder ‘Tasks’ nor in Folder ‘To-Do List’

I give it a try with an other name:

recreate-systemfolders.py --user test --create Tasks_new --systemfolder task
User('test')
create tmp folder Tasks_new
Promote new folder to be a system folder

The new folder ‘Tasks_new’ was created as an normal empty(!) folder. The old folder Tasks is still there with the same tasks inside as before. No difference ether in ‘To-Do List’.

By the way: if I flag an eMail in webapp --> in a z-push device the new task rise up as expected.

And thank you for the hint to localize-folders! It works as it should.

Any idea?

con356

Coffee_is_life

@con356,

i got a similar error with some random mails in random users.
they are synced to z-push devices, vanishes without a trace at some point.
if i mark them in webapp or even just click them, they are synced again to the device

i had a ticket for that, but its still unresolved due to the randomness.
Kopano-support said, that this might be related to the exporter, triggered by kopano-server, so that z-push gets the info which items it should sync… but its not confirmed…

since im the only one answered here i dont think it can be solved that easily… if you have a subscription i advice to create a ticket

but we can try here aswell :)
try to backup the user, unhook the store, create empty store for this one and restore the items

kopano-backup -u test
kopano-admin --unhook-store test
kopano-admin --create-store test 
kopano-backup --restore -u test

this is just guessing but might help :)

coffee_is_life

con356

Hi coffee_is_life,

yes you’re right. This is not easy without the subscription. But your answer makes me feel not so lost and alone. ;-)
I’ve got the same idea to restore a backup, while i walk around in this forum reading posts. I’ve created an new user and restore the backup of an existing one:

kopano-backup --restore user_old -u user_new --recursive

This is a solution. The new marked (flagged) eMails were now displayed immediately in ‘To-Do List’. But then i have a lot of work for all users:

1.: Reset every profile on mobile devices
2.: Reset every permission on calendars etc. on every user
… Reset public folders
etc.

Hmm, wait…

--unhook-store
--create-store

OK, this makes sense, not to have all the work. Great! :-)

I will try few things first, with your named scipts in second post. Maybe switching languages in combination with z-push / public folders will cause the problem.

Other solutions will be listed here.

con356

Coffee_is_life

hello @con356,

in case you got tons of users a simple scripting solution would do it:

for e in $(kopano-admin -l |awk -F" " '{print $1}' | sed -e 1,4d); do 
  echo "kopano-backup -u $e" 
  echo "kopano-admin --unhook-store $e" 
  echo "kopano-admin --create-store $e"
  echo "kopano-backup --restore -u $e"
done

i’ve put everything in echos, so you can test the output of this little script.
its untested, so i dont know if the command will wait for the backup etc…

you can test this by changing

for e in $(kopano-admin -l |awk -F" " '{print $1}' | sed -e 1,4d)

to:

for e in $(kopano-admin -l |awk -F" " '{print $1}' | sed -e 1,4d |grep test)  #  for a testing user named test

if this solves your problems you can safely remove all stores listed under

kopano-admin --list-orphans

by simpyl using:

for e in $(kopano-admin --list-orphans |grep private |awk -F" " '{print $1}');  do 
  echo "kopano-admin -remove-store $e"
done

same here with the echos
:thumbsup:

coffee_is_life

con356

hi @coffee_is_life,

i checked it again: yes, every user is effected. But there are only less then 10 of them. Puh!

Will try your scipts this weekend.
:sunglasses: :thumbsup:

con356

con356

Finaly everything is all right now. But i have to rebuild every user with kopano-backup. Step by step i did the following:

1.: Be sure to have a valid backup (mysqldump / rsync attachments)

2.: Create a script for backup webapp-settings

/usr/local/bin/webapp_settings.py

#!/usr/bin/env python
#encoding: utf-8
# https://stash.kopano.io/projects/KSC/repos/webapp-tools/browse/webapp_settings.py

from MAPI import *
from MAPI.Util import *
import sys
try:
    import kopano
except ImportError:
    import zarafa as kopano
try:
    import json
except ImportError:
    import simplejson as json

def opt_args():
    parser = kopano.parser('skpcfm')
    parser.add_option("--user", dest="user", action="store", help="Run script for user")
    parser.add_option("--backup", dest="backup", action="store_true", help="Backup webapp setting ")
    parser.add_option("--restore", dest="restore", action="store_true", help="Restore webapp settings")
    parser.add_option("--remove", dest="remove", action="store_true", help="Remove webapp settings")

    return parser.parse_args()


def main():
    options, args = opt_args()

    if not options.user or (not options.backup and not options.restore and not options.remove):
        print 'Please use:\n %s --user <username> (--backup or --restore)  ' % (sys.argv[0])
        sys.exit()

    user = kopano.Server(options).user(options.user)
    if options.backup:
        webapp = json.loads(user.store.prop(PR_EC_WEBACCESS_SETTINGS_JSON).value)
        f = open('%s.json' % user.name,'w')

        f.write(json.dumps(webapp, sort_keys=True,
                     indent=4, separators=(',', ': ')))
        f.close()
    if options.restore:
        with open('%s.json' % user.name) as data_file:
            data = json.load(data_file)

        # print data
        user.store.create_prop(PR_EC_WEBACCESS_SETTINGS_JSON, json.dumps(data))
        print 'restore done'
    if options.remove:
        user.store.delete(user.store.prop('PR_EC_WEBACCESS_SETTINGS_JSON'))

if __name__ == "__main__":
    main()

Then i use the script coffee_is_life posted:

/usr/local/bin/kopano-recreate-user.sh

#! /bin/bash
BACKUPPATH=/backup/kopano/full/
for e in $(kopano-admin -l |awk -F" " '{print $1}' | sed -e 1,4d); do
  kopano-backup -u $e --output-dir $BACKUPPATH
  cd $BACKUPPATH$e
  python /usr/local/bin/webapp_settings.py --backup --user=$e
  kopano-admin --unhook-store $e
  kopano-admin --create-store $e
  kopano-backup --restore $BACKUPPATH$e -u $e --recursive 
  python /usr/local/bin/webapp_settings.py --restore --user=$e
done

Now almost everything is in place again. Manualy the following has to be done:

3: Reset colors of calendars (own / imported) in webapp of every user

4.: If there were public-folders like calender/tasks/notes shared over z-push with the script soap-folders.php

<?php

$server = "http://localhost";
$user = "user1";
$pass = "user1";

$devid = "n2lmve8dm51a94esgor1gav65l";

define("SYNC_FOLDER_TYPE_USER_MAIL", 12);
define("SYNC_FOLDER_TYPE_USER_APPOINTMENT", 13);
define("SYNC_FOLDER_TYPE_USER_CONTACT", 14);
define("SYNC_FOLDER_TYPE_USER_TASK", 15);
define("SYNC_FOLDER_TYPE_USER_JOURNAL", 16);
define("SYNC_FOLDER_TYPE_USER_NOTE", 17);
define("FLD_FLAGS_REPLYASUSER", 1);

$client = new SoapClient(null, array(
                'location' => "$server/Microsoft-Server-ActiveSync?Cmd=WebserviceDevice&DeviceId=webservice&DeviceType=webservice&User=$user",
                'uri' => "http://localhost/soap",
                'trace' => 1,
                'login' => $user,
                'password' => $pass
                ));

try {
    $add_user = "user2";
    $add_id = "5a37a3f4faa340e49f5c0dc09cf6cb043a0000000000";
    $add_name = "User2 - Inbox";
    $add_type = SYNC_FOLDER_TYPE_USER_MAIL;
    $add_flags = FLD_FLAGS_REPLYASUSER;
	
//    $r = $client->AdditionalFolderAdd($devid, $add_user, $add_id, $add_name, $add_type, $add_flags);
//    echo "Add:\n". print_r($r, true) ."\n";

//    $r = $client->AdditionalFolderEdit($devid, $add_id, "! Different name", $add_flags);
//    echo "Edit:\n". print_r($r, true) ."\n";


//    $r = $client->AdditionalFolderRemove($devid, $add_id);
//    echo "Remove: ". print_r($r, true) ."\n";

    $r = $client->AdditionalFolderList($devid);
    echo "Current folders:\n". print_r($r, true);
}
catch (SoapFault $sf) {
    echo "soap fault: ". $sf->getMessage() . "\n";
    exit(1);
}

4a.: List the old IDs

z-push-admin -a list -u <user>

4b.: Remove the old IDs with soap-folders.php
4c: List the new IDs

php /usr/share/z-push/backend/kopano/listfolders.php -l <user>

4d.: Add the new IDs with soap-folders.php
4e.: Reset settings in mobile devices for shared folders

5.: Remove the old stores

kopano-admin --list-orphans  # List Store GUID
kopano-admin --remove-store <Store GUID>

/@coffee_is_life: Thanks for help!!

fbartels

For everyone else reading this: this is/was a known bug in Kopano 8.5.x (and therefore the master builds as well). Yesterday we merged a fix which still needs to pass final checks, afterwards we are going to release an updated 8.5 release.

@con356 said in "To-Do List" stops displaying new items, searching for items still works:

kopano-backup --restore $BACKUPPATH$e -u $e --recursive
python /usr/local/bin/webapp_settings.py --restore --user=$e

btw. kopano-backup also backs up and restores webapp settings, so this is not necessary

Coffee_is_life

@con356

glad i could help and everything is fine now

coffee_is_life