"To-Do List" stops displaying new items, searching for items still works



  • Hi forum,
    i am not sure how to explain this one correct…

    If i flag emails or enter new entries in “Tasks” no new entry is shown in the “To-Do List”. But if i search for those newly made or flagged items, then they are shown as a single search-result.

    The “To-Do List” shows older entries correct. The List just stops working…

    As far as i remember: A few days ago i changed language settings in /etc/default/kopano

    #KOPANO_LOCALE="C"
    KOPANO_LOCALE="de_DE.UTF-8"
    
    #KOPANO_USERSCRIPT_LOCALE="C"
    KOPANO_USERSCRIPT_LOCALE="de_DE.UTF-8"
    

    Changing this back did not help. May be this has nothing to do with the problem.

    This problem is effective i every user account.

    Could somebody “push” me in the right direction?

    Server: Debian_9 64bit
    core-8.6.80.3_0+79-Debian_9.0-amd64
    webapp-3.4.7.1275+733-Debian_9.0-all

    Regards
    Enno



  • Hello @con356,

    this issue seems related to system folders-bug.
    try this script on a testuser and systemfolder ‘task’
    https://stash.kopano.io/projects/KSC/repos/core-tools/browse/recreate-systemfolders

    //EDIT: its a bit strange that all users are effected, thats why try with a testuser

    btw if you want to rename your stores to german locale, you need to run
    https://stash.kopano.io/projects/KSC/repos/core-tools/browse/localize-folders
    in order to rename existing stores

    coffee_is_life



  • Hi coffee_is_life,

    thank’s for help!
    I made a full clone of the whole debian-system, so any test is without risk. :-)
    “recreate-systemfolders” dosen’t seem to made a change to nothing … !?

    python recreate-systemfolders.py --user test --create Tasks --systemfolder task
    User('test')
    create tmp folder Tasks
    Promote new folder to be a system folder
    

    But there is no difference in Systemfolder ‘Tasks’ nor in Folder ‘To-Do List’

    I give it a try with an other name:

    recreate-systemfolders.py --user test --create Tasks_new --systemfolder task
    User('test')
    create tmp folder Tasks_new
    Promote new folder to be a system folder
    

    The new folder ‘Tasks_new’ was created as an normal empty(!) folder. The old folder Tasks is still there with the same tasks inside as before. No difference ether in ‘To-Do List’.

    By the way: if I flag an eMail in webapp --> in a z-push device the new task rise up as expected.

    And thank you for the hint to localize-folders! It works as it should.

    Any idea?

    con356



  • @con356,

    i got a similar error with some random mails in random users.
    they are synced to z-push devices, vanishes without a trace at some point.
    if i mark them in webapp or even just click them, they are synced again to the device

    i had a ticket for that, but its still unresolved due to the randomness.
    Kopano-support said, that this might be related to the exporter, triggered by kopano-server, so that z-push gets the info which items it should sync… but its not confirmed…

    since im the only one answered here i dont think it can be solved that easily… if you have a subscription i advice to create a ticket

    but we can try here aswell :)
    try to backup the user, unhook the store, create empty store for this one and restore the items

    kopano-backup -u test
    kopano-admin --unhook-store test
    kopano-admin --create-store test 
    kopano-backup --restore -u test
    

    this is just guessing but might help :)

    coffee_is_life



  • Hi coffee_is_life,

    yes you’re right. This is not easy without the subscription. But your answer makes me feel not so lost and alone. ;-)
    I’ve got the same idea to restore a backup, while i walk around in this forum reading posts. I’ve created an new user and restore the backup of an existing one:

    kopano-backup --restore user_old -u user_new --recursive
    

    This is a solution. The new marked (flagged) eMails were now displayed immediately in ‘To-Do List’. But then i have a lot of work for all users:

    • 1.: Reset every profile on mobile devices
    • 2.: Reset every permission on calendars etc. on every user
    • … Reset public folders
    • etc.

    Hmm, wait…

    --unhook-store
    --create-store
    

    OK, this makes sense, not to have all the work. Great! :-)

    I will try few things first, with your named scipts in second post. Maybe switching languages in combination with z-push / public folders will cause the problem.

    Other solutions will be listed here.

    con356



  • hello @con356,

    in case you got tons of users a simple scripting solution would do it:

    for e in $(kopano-admin -l |awk -F" " '{print $1}' | sed -e 1,4d); do 
      echo "kopano-backup -u $e" 
      echo "kopano-admin --unhook-store $e" 
      echo "kopano-admin --create-store $e"
      echo "kopano-backup --restore -u $e"
    done
    

    i’ve put everything in echos, so you can test the output of this little script.
    its untested, so i dont know if the command will wait for the backup etc…

    you can test this by changing

    for e in $(kopano-admin -l |awk -F" " '{print $1}' | sed -e 1,4d)
    

    to:

    for e in $(kopano-admin -l |awk -F" " '{print $1}' | sed -e 1,4d |grep test)  #  for a testing user named test
    

    if this solves your problems you can safely remove all stores listed under

    kopano-admin --list-orphans
    

    by simpyl using:

    for e in $(kopano-admin --list-orphans |grep private |awk -F" " '{print $1}');  do 
      echo "kopano-admin -remove-store $e"
    done
    

    same here with the echos
    :thumbsup:

    coffee_is_life



  • hi @coffee_is_life,

    i checked it again: yes, every user is effected. But there are only less then 10 of them. Puh!

    Will try your scipts this weekend.
    :sunglasses: :thumbsup:

    con356



  • Finaly everything is all right now. But i have to rebuild every user with kopano-backup. Step by step i did the following:

    1.: Be sure to have a valid backup (mysqldump / rsync attachments)

    2.: Create a script for backup webapp-settings

    /usr/local/bin/webapp_settings.py
    
    #!/usr/bin/env python
    #encoding: utf-8
    # https://stash.kopano.io/projects/KSC/repos/webapp-tools/browse/webapp_settings.py
    
    from MAPI import *
    from MAPI.Util import *
    import sys
    try:
        import kopano
    except ImportError:
        import zarafa as kopano
    try:
        import json
    except ImportError:
        import simplejson as json
    
    def opt_args():
        parser = kopano.parser('skpcfm')
        parser.add_option("--user", dest="user", action="store", help="Run script for user")
        parser.add_option("--backup", dest="backup", action="store_true", help="Backup webapp setting ")
        parser.add_option("--restore", dest="restore", action="store_true", help="Restore webapp settings")
        parser.add_option("--remove", dest="remove", action="store_true", help="Remove webapp settings")
    
        return parser.parse_args()
    
    
    def main():
        options, args = opt_args()
    
        if not options.user or (not options.backup and not options.restore and not options.remove):
            print 'Please use:\n %s --user <username> (--backup or --restore)  ' % (sys.argv[0])
            sys.exit()
    
        user = kopano.Server(options).user(options.user)
        if options.backup:
            webapp = json.loads(user.store.prop(PR_EC_WEBACCESS_SETTINGS_JSON).value)
            f = open('%s.json' % user.name,'w')
    
            f.write(json.dumps(webapp, sort_keys=True,
                         indent=4, separators=(',', ': ')))
            f.close()
        if options.restore:
            with open('%s.json' % user.name) as data_file:
                data = json.load(data_file)
    
            # print data
            user.store.create_prop(PR_EC_WEBACCESS_SETTINGS_JSON, json.dumps(data))
            print 'restore done'
        if options.remove:
            user.store.delete(user.store.prop('PR_EC_WEBACCESS_SETTINGS_JSON'))
    
    if __name__ == "__main__":
        main()
    

    Then i use the script coffee_is_life posted:

    /usr/local/bin/kopano-recreate-user.sh
    
    #! /bin/bash
    BACKUPPATH=/backup/kopano/full/
    for e in $(kopano-admin -l |awk -F" " '{print $1}' | sed -e 1,4d); do
      kopano-backup -u $e --output-dir $BACKUPPATH
      cd $BACKUPPATH$e
      python /usr/local/bin/webapp_settings.py --backup --user=$e
      kopano-admin --unhook-store $e
      kopano-admin --create-store $e
      kopano-backup --restore $BACKUPPATH$e -u $e --recursive 
      python /usr/local/bin/webapp_settings.py --restore --user=$e
    done
    

    Now almost everything is in place again. Manualy the following has to be done:

    3: Reset colors of calendars (own / imported) in webapp of every user

    4.: If there were public-folders like calender/tasks/notes shared over z-push with the script soap-folders.php

    <?php
    
    $server = "http://localhost";
    $user = "user1";
    $pass = "user1";
    
    $devid = "n2lmve8dm51a94esgor1gav65l";
    
    define("SYNC_FOLDER_TYPE_USER_MAIL", 12);
    define("SYNC_FOLDER_TYPE_USER_APPOINTMENT", 13);
    define("SYNC_FOLDER_TYPE_USER_CONTACT", 14);
    define("SYNC_FOLDER_TYPE_USER_TASK", 15);
    define("SYNC_FOLDER_TYPE_USER_JOURNAL", 16);
    define("SYNC_FOLDER_TYPE_USER_NOTE", 17);
    define("FLD_FLAGS_REPLYASUSER", 1);
    
    $client = new SoapClient(null, array(
                    'location' => "$server/Microsoft-Server-ActiveSync?Cmd=WebserviceDevice&DeviceId=webservice&DeviceType=webservice&User=$user",
                    'uri' => "http://localhost/soap",
                    'trace' => 1,
                    'login' => $user,
                    'password' => $pass
                    ));
    
    try {
        $add_user = "user2";
        $add_id = "5a37a3f4faa340e49f5c0dc09cf6cb043a0000000000";
        $add_name = "User2 - Inbox";
        $add_type = SYNC_FOLDER_TYPE_USER_MAIL;
        $add_flags = FLD_FLAGS_REPLYASUSER;
    	
    //    $r = $client->AdditionalFolderAdd($devid, $add_user, $add_id, $add_name, $add_type, $add_flags);
    //    echo "Add:\n". print_r($r, true) ."\n";
    
    //    $r = $client->AdditionalFolderEdit($devid, $add_id, "! Different name", $add_flags);
    //    echo "Edit:\n". print_r($r, true) ."\n";
    
    
    //    $r = $client->AdditionalFolderRemove($devid, $add_id);
    //    echo "Remove: ". print_r($r, true) ."\n";
    
        $r = $client->AdditionalFolderList($devid);
        echo "Current folders:\n". print_r($r, true);
    }
    catch (SoapFault $sf) {
        echo "soap fault: ". $sf->getMessage() . "\n";
        exit(1);
    }
    

    4a.: List the old IDs

    z-push-admin -a list -u <user>
    

    4b.: Remove the old IDs with soap-folders.php
    4c: List the new IDs

    php /usr/share/z-push/backend/kopano/listfolders.php -l <user>
    

    4d.: Add the new IDs with soap-folders.php
    4e.: Reset settings in mobile devices for shared folders

    5.: Remove the old stores

    kopano-admin --list-orphans  # List Store GUID
    kopano-admin --remove-store <Store GUID>
    

    /@coffee_is_life: Thanks for help!!


  • Kopano

    For everyone else reading this: this is/was a known bug in Kopano 8.5.x (and therefore the master builds as well). Yesterday we merged a fix which still needs to pass final checks, afterwards we are going to release an updated 8.5 release.

    @con356 said in "To-Do List" stops displaying new items, searching for items still works:

    kopano-backup --restore $BACKUPPATH$e -u $e --recursive
    python /usr/local/bin/webapp_settings.py --restore --user=$e

    btw. kopano-backup also backs up and restores webapp settings, so this is not necessary



  • @con356

    glad i could help and everything is fine now

    coffee_is_life