Kopano-search initial sync

mcostan

Good morning,

I recently started an initial sync of the indices and this is taking ages, which is I suppose to be expected.

I was however wondering if there is any way to find out at what stage the progress is and when it will complete.

Thank you.

itserv

Any output in the search log file or the server log file (check both!)?

kopano-search can take quite a long time for large mailboxes. It might collapse in some cases, retrying over and over again, but again: the log files tell you more.

You might also enable a more verbose log level for kopano search.

Peter

mcostan

Thank you Peter,

Indeed I am looking at the log file, but I was wondering if there was any way to know an estimate of how much left there is to do…

It has been 2 weeks and still going…

itserv

@mcostan said in Kopano-search initial sync:

e, but I was wondering if there was any way to know an estimate of how much left there is to do…
It has been 2 weeks and still going…

Again - it depends on the size of the DB.

Mine (backed up/gipped/including attachments is 12GB, 10 Users, multiple years) took about one or two days for a complete reindex. Two weeks seems to me way too long except if you have a really large installation.

Do your index files, located in the directory named by the index_path variable of /etc/kopano/search.cfg change in size?

I don’t think there is a way to estimate the expected time left.

As far as I know, you can cancel the process of indexing and start it up again, and it will pick up where it left. Doing so, you could increase the number of worker processes (index_processes in config file) to get more power.

And as a Unix admin, I would certainly look at the current processes using strace. This gives you the system calls, and so you would clearly see if the indexer is still working or stuck.

mcostan

Thanks

yes the file of the indices is increasing, so hopefully it will come to an end.

I may move the index files to the SSD disk, perhaps it will make It faster.

Thanks,

itserv

@mcostan said in Kopano-search initial sync:

s the file of the indices is increasing, so hopefully it will come to an end.
I may move the index files to the SSD disk, perhaps it will make It faster.
Thanks,

How big is your DB?
Do you have your attachments in the DB or in the filesystem?

mcostan

I checked and it’s about 30GB of data. The attachments are on the file system but I have disabled the attachments indexing.

Monitoring the log file it may be that kopano-search keeps hold of the existing state between restarts but perhaps not of the same folder. So overnight I reboot the server and perhaps it keeps going over and over the same folder which, I suspect, it’s either the Sent Items or the Deleted Items.

I will have to monitor the situation carefully and perhaps disable the overnight reboot until the sync process has finished.

itserv

Good luck. Moving to an SSD will probably make it faster, but the overall performance will surely benefit if you move your DB to an SSD too.

mcostan

the DB is indeed on SSD already, but the attachments aren’t nor the index files themselves.

Just because the SSD raid array is very small… hence I have to check very carefully what’s on there to avoid running out of space…