Navigation

    Kopano
    • Register
    • Login
    • Search
    • Categories
    • Get Official Kopano Support
    • Recent
    Statement regarding the closure of the Kopano community forum and the end of the community edition

    Kopano search improvement

    Kopano Groupware Core
    3
    8
    426
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • mcostan
      mcostan last edited by

      Good afternoon,

      I have noticed a behaviour of Kopano search which, in my case, causes issues during the initial indexing.

      As far as I understand Kopano search goes through the folders one by one and keeps the state so that if it is restarted it can carry on from there.

      EXCEPT

      That if the indexing of a specific folder isn’t completed, when kopano-search is restarted, that folder will be restarted from scratch.

      For me personally this is a problem because I keep all emails I sent from say 2010 in my the Deleted Items or Sent folders and this means that if the server is restarted or kopano-search restarts, this is all lost and it starts from scratch.

      Is there any way to improve this so that if kopano-search is restarted during the initial search, the individual folders indexing isn’t restarted from scratch but it continues from where it was.

      1 Reply Last reply Reply Quote 0
      • thctlo
        thctlo last edited by

        @mcostan said in Kopano search improvement:

        Deleted Items or Sent folders

        Why are you using an recycle bin for your archive.
        I hope that your not doing this in hous also. …

        Depending on your outlook version, a max limit on items in the outlook folders applies also.
        If you notice strange things in outlook, subfolder you items.

        you might want to read this :
        https://danielkharman.wordpress.com/2016/09/10/revisiting-outlook-maximum-folder-limitations/

        1 Reply Last reply Reply Quote 0
        • mcostan
          mcostan last edited by

          no… this is purely in Webapp.

          I just have a lot of stuff in “sent items” or “Deleted Items” and that’s why kopano search takes forever in the initial search.

          The problem is that if kopano-search is restarted for whatever reason before a folder is finished, it restarts from scratch hence wasting all the indexing that was done to that folder.

          The result is that this may never happen unless your server stays up for 3 weeks at the time to complete that folder…

          fbartels 1 Reply Last reply Reply Quote 0
          • fbartels
            fbartels Kopano @mcostan last edited by fbartels

            Hi @mcostan ,

            I had also seen your post in https://forum.kopano.io/topic/2160/kopano-search-initial-sync/7 but did not have further time to think about it. In the linked post you say that your total database is 30gb. Which yes, will/could take some hours to complete indexing, but weeks sounds really strange so I would say something else is going on in your system.

            Normally a single folder should also not take too long, so some checkmarking could be implemented, but the question is then always: is it worth the effort?

            How many items do you have in these folders? how big are these folders (this is afaik a number webapp could tell you)?

            Regards Felix

            Resources:
            https://kopano.com/blog/how-to-get-kopano/
            https://documentation.kopano.io/
            https://kb.kopano.io/

            Support overview:
            https://kopano.com/support/

            1 Reply Last reply Reply Quote 0
            • mcostan
              mcostan last edited by

              Felix,

              This is the size of the deleted item which I think may be the problem:

              Items:
              133819
              Unread:
              4
              Size:
              12.4 GB

              Note that I don’t know exactly what folder it is because from the kopano-search log it says folder id but how do I know which folder.

              Interestingly when I look at the kopano-search folder I can see the following:

              key 9995A8356BA047CD800EDEB80B480023428E4C000000, docid 7117089                                                                                                                                                                             
2019-02-15 12:54:30,896 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B373548010000000500000045FBE83CBEFD450CB0E2B5B60F4B46CF00000000, source
key 9995A8356BA047CD800EDEB80B480023418E4C000000, docid 7117087                                                                                                                                                                             
2019-02-15 12:54:30,933 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B373548010000000500000086A9416611714A178D4FC401032103FC00000000, source
key 9995A8356BA047CD800EDEB80B480023408E4C000000, docid 7117086

              Which suggests that folder 770701 has 7 million items? But which items are these?

              How do I find out?

              fbartels 1 Reply Last reply Reply Quote 0
              • fbartels
                fbartels Kopano @mcostan last edited by

                @mcostan taking a probably quite conservative number of indexing 10 items per second (and just from the short log you’ve posted your system seems to index one item in 0,04 seconds) such a folder should be indexed in 3,7 hours and not weeks.

                @mcostan said in Kopano search improvement:

                docid 7117086

                the id mentioned here is the hierarchy id. this is a per server index and means that this was the 7.117.086th mapi object (folder, mail, etc) created on this system.

                A longer log would be interesting.

                But of course the easy recommendation is to finally clear your trash ;-)

                Regards Felix

                Resources:
                https://kopano.com/blog/how-to-get-kopano/
                https://documentation.kopano.io/
                https://kb.kopano.io/

                Support overview:
                https://kopano.com/support/

                mcostan 1 Reply Last reply Reply Quote 0
                • mcostan
                  mcostan last edited by

                  Felix,

                  Indeed housekeeping and clearing the trash would be useful… But hey… This is the strength of Kopano!

                  There is no limit to the storage you can have as long as you have disks!

                  I think by the looks of it the indexing will take a lot longer than 37 hours… it has been going on for a week and it went from item 8,000,000 to item 6.000,000 before the server was restarted and hence it went back to 8,000,000 and start again…

                  The trouble as I mentioned is that without any state of the indexing process, there is the danger that this may actually never finish…

                  Longer log below:

                  2019-02-15 12:54:28,849 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000E66DEED3FF054943B66FC9ED941632E900000000, source
key 9995A8356BA047CD800EDEB80B480023598E4C000000, docid 7117112                                                                                                                                                                             
2019-02-15 12:54:28,949 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B373548010000000500000092167F46138B4D0893C0C36B38D9E7BD00000000, source
key 9995A8356BA047CD800EDEB80B480023588E4C000000, docid 7117111                                                                                                                                                                             
2019-02-15 12:54:29,001 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B373548010000000500000058BB19CFDEBC4044872746987F003A6B00000000, source
key 9995A8356BA047CD800EDEB80B480023578E4C000000, docid 7117110                                                                                                                                                                             
2019-02-15 12:54:29,125 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000F9A2261C6E924DAE9E754E7DF504795400000000, source
key 9995A8356BA047CD800EDEB80B480023568E4C000000, docid 7117109                                                                                                                                                                             
2019-02-15 12:54:29,212 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000657B8837F01A4712A1882E8F648262AC00000000, source
key 9995A8356BA047CD800EDEB80B480023558E4C000000, docid 7117108                                                                                                                                                                             
2019-02-15 12:54:29,273 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000D527D70CB46040FC81C0D746A4B6D7D000000000, source
key 9995A8356BA047CD800EDEB80B480023548E4C000000, docid 7117107                                                                                                                                                                             
2019-02-15 12:54:29,390 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000003303E1C501A2419E871AB9A936A2E3D900000000, source
key 9995A8356BA047CD800EDEB80B480023538E4C000000, docid 7117106                                                                                                                                                                             
2019-02-15 12:54:29,450 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000CB9DDE33FB964D7E9AA9A4E96FDA163D00000000, source
key 9995A8356BA047CD800EDEB80B480023528E4C000000, docid 7117105                                                                                                                                                                             
2019-02-15 12:54:29,485 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B373548010000000500000027B7FD26212B4109AC596697EDF61FC400000000, source
key 9995A8356BA047CD800EDEB80B480023518E4C000000, docid 7117104                                                                                                                                                                             
2019-02-15 12:54:29,566 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000009589EA063150483D8A3A0291776EDDBB00000000, source
key 9995A8356BA047CD800EDEB80B480023508E4C000000, docid 7117103                                                                                                                                                                             
2019-02-15 12:54:29,691 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B373548010000000500000091E42B2C2AB84D1BA581B4212390FC2400000000, source
key 9995A8356BA047CD800EDEB80B4800234F8E4C000000, docid 7117102                                                                                                                                                                             
2019-02-15 12:54:29,813 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000563D1E5069AA4BA9A92148D3C634A5FF00000000, source
key 9995A8356BA047CD800EDEB80B4800234E8E4C000000, docid 7117101                                                                                                                                                                             
2019-02-15 12:54:29,969 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000000E54862DD3464205B376160D2390B4E500000000, source
key 9995A8356BA047CD800EDEB80B4800234D8E4C000000, docid 7117100                                                                                                                                                                             
2019-02-15 12:54:30,071 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000E2649343F5104C11978E151BFFB7119600000000, source
key 9995A8356BA047CD800EDEB80B4800234C8E4C000000, docid 7117099                                                                                                                                                                             
2019-02-15 12:54:30,167 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000001A931A1558ED43BB88F3EE8F10E24C7000000000, source
key 9995A8356BA047CD800EDEB80B4800234B8E4C000000, docid 7117098                                                                                                                                                                             
2019-02-15 12:54:30,295 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000009405BADDBE114CA6B89A594AC342C95B00000000, source
key 9995A8356BA047CD800EDEB80B4800234A8E4C000000, docid 7117097                                                                                                                                                                             
2019-02-15 12:54:30,391 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000D636782C4F0A44FC8FEA9D42B01E758D00000000, source
key 9995A8356BA047CD800EDEB80B480023498E4C000000, docid 7117096                                                                                                                                                                             
2019-02-15 12:54:30,442 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000007AD697B3826B4E6298413CFD166FD02C00000000, source
key 9995A8356BA047CD800EDEB80B480023488E4C000000, docid 7117095                                                                                                                                                                             
2019-02-15 12:54:30,526 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000A0078513439648F4AB3B598D032C0FDB00000000, source
key 9995A8356BA047CD800EDEB80B480023478E4C000000, docid 7117094                                                                                                                                                                             
2019-02-15 12:54:30,586 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000006393720B19A6412DAADD466E566A778A00000000, source
key 9995A8356BA047CD800EDEB80B480023468E4C000000, docid 7117093                                                                                                                                                                             
2019-02-15 12:54:30,621 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000001B443AD59657416A88C49A8D1B8812D900000000, source
key 9995A8356BA047CD800EDEB80B480023458E4C000000, docid 7117092                                                                                                                                                                             
2019-02-15 12:54:30,657 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000004753FF560CE045DDB7D8BE0A5D78330C00000000, source
key 9995A8356BA047CD800EDEB80B480023448E4C000000, docid 7117091                                                                                                                                                                             
2019-02-15 12:54:30,707 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B37354801000000050000007D4C38789FF34B27BFD837534969C0BF00000000, source
key 9995A8356BA047CD800EDEB80B480023438E4C000000, docid 7117090                                                                                                                                                                             
2019-02-15 12:54:30,757 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B3735480100000005000000EE04FC324D6445989B9BDE169CCD1B3800000000, source
key 9995A8356BA047CD800EDEB80B480023428E4C000000, docid 7117089                                                                                                                                                                             
2019-02-15 12:54:30,896 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B373548010000000500000045FBE83CBEFD450CB0E2B5B60F4B46CF00000000, source
key 9995A8356BA047CD800EDEB80B480023418E4C000000, docid 7117087                                                                                                                                                                             
2019-02-15 12:54:30,933 - index3 - DEBUG - store 79C88C2154BB4BF08133E79E3B373548, folder 770701: new/updated document with entryid 0000000079C88C2154BB4BF08133E79E3B373548010000000500000086A9416611714A178D4FC401032103FC00000000, source
key 9995A8356BA047CD800EDEB80B480023408E4C000000, docid 7117086
                  1 Reply Last reply Reply Quote 0
                  • mcostan
                    mcostan @fbartels last edited by

                    @fbartels

                    Good morning,

                    So after 3 weeks the initial search is supposed to be finished.

                    I however now have another issue, which I can’t work out.

                    In essence everything works, except the search in the “Deleted Items” folder is not triggered!

                    This is the log when I move an item from the sent items folder to the deleted items folder. The same happens for items that appear in Inbox.

                    Those folders are never triggered…

                    What do I need to do? What went wrong? Do I need to start the search all over again from scratch?

                    019-02-27 10:35:12,973 - index1 - INFO - syncing folder: "mrc" "Sent Items"                                                                               
2019-02-27 10:35:12,974 - index1 - INFO - found previous folder sync state: 027C3D0073D19600                                                               
2019-02-27 10:35:12,982 - index1 - DEBUG - store A5096D7C3F404F6798579D650EE13FC1: deleted document with sourcekey 9995A8356BA047CD800EDEB80B480023D5CC5A00
0000                                                                                                                                                       
2019-02-27 10:35:14,252 - search - DEBUG - commit took 1.26 seconds (0 items)                                                                              
2019-02-27 10:35:14,339 - index1 - INFO - saved folder sync state: 027C3D0075D19600                                                                        
2019-02-27 10:35:14,340 - index1 - INFO - syncing folder "Sent Items" took 1.37 seconds (1 changes, 0 attachments)                                         
2019-02-27 10:35:14,372 - index0 - INFO - syncing folder: "mrc" "Sent Items"                                                                               
2019-02-27 10:35:14,374 - index0 - INFO - found previous folder sync state: 027C3D0075D19600                                                               
2019-02-27 10:35:14,420 - search - INFO - queue processed in 1.50 seconds (1 changes, ~0.66/sec)                                                           
2019-02-27 10:35:14,454 - search - INFO - saved server sync state = F77B3D000FB896009000000067B99600160000009995A83
                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post