Mapping from LDAP attributes to Kopano Address Book

bhuisman

@Pax all the field mappings are listed and described in the ldap.propmap.cfg

genz

Hey Guys,
Thanks it worked for me but the correct location of that file is: /usr/share/kopano/ldap.propmap.cfg

Anyway once I added all my 45k records in the ldap with all the attributes populated every time I do the total search in the kopano web app this is what happen:

The mysql service starts to use a lot of CPU time:

0_1534866025613_4e58c9e4-b321-4f1d-bb34-cd49d787cb14-image.png

The search does not retrive any record and I get an error saying: 500 server error.

Instead when I do a search for a specific users it works

Do you have any ideas? @bhuisman

bhuisman

@genz do you have sync_gab_realtime set to yes? In that case, please set it to no and try again.

genz

sync_gab_realtime it’s set to NO.
Indeed the openldap deamon does not do anything the problem is mysql that seems stuck on the search.

bhuisman

@genz we will set up a testing system with a large amount of GAB contacts - it probably requires tweaking indeed. We will keep you posted.

bhuisman

@genz We see slower responses when loading the GAB on our testing systems, but it does not break or crash. So it can be a tuning issue (database tuning) or something else that needs to be looked into.

When you open the GAB dialog, WebApp will try to load all the items from the GAB, but this can get too large to handle with such an amount of users (or complete within the allowed timeframe). Can you disable the loading of the full GAB through the WebApp configuration:

// Set addressbook for GAB not to show any users unless searching for a specific user
define("DISABLE_FULL_GAB", true);

You can then search the GAB by typing (part of) the name of a user - this will only load the relevant results. Through this we can eliminate (some) potential performance issues.

robing

@genz We created some more ldap contacts and where able to get the internal server error now.

On our test system this was caused due the memory limit of php that was hit.
Raising it to 256Mb fixed this.

/etc/php/7.0/apache2/php.ini (debian 9)

memory_limit = 256M

genz

Hey Guys,
Indeed with this new setup, everything is ok. But When I search for a single user using name and surname mysql starts to use a lot of CPU time and I get the result in about 40-50 seconds.

These are the technical details of our server where we run the kopano-server and the mysql database :

It’s a virtual machine running CENTOS 7
4 CPUs (virtual CPUs)
8 GB of RAM
40 GB of Disk

Maybe there is something wrong in my configuration. But I checked the queries executed when I search something in the GAB and for every new query in the GAB the database has to execute something like 80.000 sql commands when I search for name and surname. Instead when I try to get the entire GAB it has to execute something like 180.000 queries.

Our Global Address book has 45.000 contacts.

Is it something we can fix with the currect setup? Or we need to optimize and tune the database?

Thanks Riccardo

bhuisman

Hi Riccardo, yes - it is very likely that there is some tuning required, on both the database server side as well as the Kopano side.

It is recommended to run kopano-cachestat.py only on a system that has been running for a few hours at least so every cache has been filled (warmed up), this should give some insight in how the caches are utilized. Judging by what you see (80k to 180k queries) I expect that at least the userid cache shows a lot of misses.

Based on the outcome, it might be that you need to add more memory to the system as well.

genz

This my outcome:

[root@m4-kopano-mailbox-node-02 ~]# kopano-cachestat
Kopano Cache Statistics
  Server start time: Fri Aug 31 15:04:20 2018
  Current time     : Mon Sep  3 15:46:08 2018

     Cache                Hit ratio                  Mem usage ratio
    uquota (           0/           0) (N/A)   (           0/     1048576) ( 0%)
       obj (     1892470/     1899564) (99%)   (     1145232/     5242880) (21%)
    abinfo (      727341/      773486) (94%)   (     8490128/    26214400) (32%)
    userid (      245374/      859595) (28%)   (     1031576/     1048576) (98%)
     quota (        8545/       14351) (59%)   (         104/     1048576) ( 0%)
    server (           0/           0) (N/A)   (           0/     1048576) ( 0%)
      cell (     1231586/     1266124) (97%)   (     5703488/  3196059648) ( 0%)
    extern (      125209/      142344) (87%)   (       14544/     1048576) ( 1%)
    index2 (      771779/      842346) (91%)   (     2028904/    10485760) (19%)
    index1 (      327611/      355117) (92%)   (     2152704/    10485760) (20%)
     store (      841243/      849215) (99%)   (      765024/     1048576) (72%)
       acl (          83/          97) (85%)   (        1224/     1048576) ( 0%)

Could you help me to understand where is the problem?
I think the problem is related with the userId as you told me. Now I tried to increase the cache_user_size from 1M to 10M parameter in the /etc/kopano/server.cfg.

Do you think that if I have a good hit ratio (between 70% to 99%) the GAB should work better?

genz

I solved it,

The problem was the userid cache.
I add the configuration parameter cache_user_size=20M in the /etc/kopano/server.cfg and I restarted the kopano-server. Now it works perfectly.

Thanks!

bhuisman

Great news, thanks for sharing!

-Bob