KC 8.7.80 upgrade issues



  • Dear ladies and gentlemen

    I’m probably not the first one reporting things like that. And I have now tried for several hours to fix the issues found. But no luck yet.
    First of all - I was a bit stupid as I didn’t prepare myself for the upgrade of Kopano and what all needs to be done. Secondly it happened by chance, basically I wanted to update to new apache2 version which resulted in installing libcurl4 and a non working Kopano environment.

    I was running KC 8.4 for several time without any bigger issues. But the upgrade to KC 8.7 gives me headache.
    System: Debian 10
    Apache: 2.4
    PHP: 7.0 and 7.2 installed

    Kopano packages were installed using dpkg -i
    The upcoming issues for libmariadbclien18 and libcurl3 have been resolved by putting manual symlinks.
    So kopano-server -gateway -spooler -dagent are running.
    I used to authenticate against LDAP where I also struggeld getting it working.
    What I currently see in the gateway.log is following:

    Sun May 12 18:40:11 2019: [notice ] Client [::ffff:80.219.29.45]:51832 thread exiting
    Sun May 12 18:40:12 2019: [info   ] Accepted connection from [::ffff:80.219.29.45]:51840
    Sun May 12 18:40:12 2019: [notice ] Starting worker process for IMAPs request
    Sun May 12 18:40:12 2019: [debug  ] > * OK [CAPABILITY IMAP4rev1 LITERAL+ AUTH=PLAIN] IMAP gateway ready
    Sun May 12 18:40:13 2019: [debug  ] < 1.54 AUTHENTICATE PLAIN
    Sun May 12 18:40:13 2019: [debug  ] > + 
    Sun May 12 18:40:13 2019: [info   ] Invalid authentication data received, expected 3 items, have 0 items.
    Sun May 12 18:40:13 2019: [debug  ] > 1.54 NO AUTHENTICATE PLAIN incomplete data received
    Sun May 12 18:40:13 2019: [debug  ] < 2.54 AUTHENTICATE PLAIN
    Sun May 12 18:40:13 2019: [debug  ] > + 
    Sun May 12 18:40:13 2019: [info   ] Invalid authentication data received, expected 3 items, have 0 items.
    Sun May 12 18:40:13 2019: [debug  ] > 2.54 NO AUTHENTICATE PLAIN incomplete data received
    Sun May 12 18:40:13 2019: [debug  ] < 3.54 LOGOUT
    Sun May 12 18:40:13 2019: [debug  ] > * BYE server logging out
    Sun May 12 18:40:13 2019: [debug  ] > 3.54 OK LOGOUT completed
    Sun May 12 18:40:13 2019: [notice ] Disconnecting client.
    

    The session is initiated from a Mac Mail client. I don’t understand why it states plain. Shouldn’t be to my understnading. However none of my mail clients are working by now.

    kopano-search fails with:

    May 12 17:45:45 oklahoma.haller-net.ch kopano-search[27580]:     from MAPICore import *
    May 12 17:45:45 oklahoma.haller-net.ch kopano-search[27580]:   File "/usr/lib/python3/dist-packages/MAPICore.py", line 21, in <module>
    May 12 17:45:45 oklahoma.haller-net.ch kopano-search[27580]:     _MAPICore = swig_import_helper()
    May 12 17:45:45 oklahoma.haller-net.ch kopano-search[27580]:   File "/usr/lib/python3/dist-packages/MAPICore.py", line 20, in swig_import_helper
    May 12 17:45:45 oklahoma.haller-net.ch kopano-search[27580]:     return importlib.import_module('_MAPICore')
    May 12 17:45:45 oklahoma.haller-net.ch kopano-search[27580]:   File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    May 12 17:45:45 oklahoma.haller-net.ch kopano-search[27580]:     return _bootstrap._gcd_import(name[level:], package, level)
    May 12 17:45:45 oklahoma.haller-net.ch kopano-search[27580]: ModuleNotFoundError: No module named '_MAPICore'
    May 12 17:45:45 oklahoma.haller-net.ch systemd[1]: kopano-search.service: Main process exited, code=exited, status=1/FAILURE
    

    I’m currently really out of ideas where to continue and would appreciate your help.

    Cheers
    Stefan


  • Kopano

    @smhaller said in KC 8.7.80 upgrade issues:

    kopano-search fails with:

    Missing python3-kopano or matching python3-kopano?

    @smhaller said in KC 8.7.80 upgrade issues:

    System: Debian 10

    Isn’t buster having PHP 7.3? Buster isn’t even properly release yet. Seems you messed up your system by mixing up repositories from different Debian releases.



  • Yes PHP 7.3 would be available and I probably will update to that too. I’m also aware that Buster isn’t fully released yet. But up to now I had no issues with it.

    This one is not my main problem yet. The main problem is, that the whole mail chain is broken. So I’m not able to receive / send mails at all.
    I’m also not clear if I need Kconnectd setup or not?


  • Kopano

    @smhaller said in KC 8.7.80 upgrade issues:

    I’m also not clear if I need Kconnectd setup or not?

    No, that one is strictly optional for the old stack.

    Ps: I’d recommend to go back to Debian 9. Your issues are probably related to dependencies not being able to resolve cleanly since there are no public Debian 10 packages for Kopano, yet.



  • Hi Felix

    No - a downgrade is even more horrable than failing forward.
    So as of now:
    Sending mails works
    Retrieving mails not

    Mon May 13 09:59:19 2019: [notice ] Client [::ffff:153.46.253.213]:43409 thread exiting
    Mon May 13 10:00:27 2019: [info   ] Accepted connection from [::ffff:153.46.253.213]:43413
    Mon May 13 10:00:27 2019: [notice ] Starting worker process for IMAP request
    Mon May 13 10:00:27 2019: [debug  ] > * OK [CAPABILITY IMAP4rev1 LITERAL+ STARTTLS AUTH=PLAIN] IMAP gateway ready
    Mon May 13 10:00:27 2019: [debug  ] < kman1 CAPABILITY
    Mon May 13 10:00:27 2019: [debug  ] > * CAPABILITY IMAP4rev1 LITERAL+ STARTTLS AUTH=PLAIN CHILDREN XAOL-OPTION NAMESPACE QUOTA IDLE
    Mon May 13 10:00:27 2019: [debug  ] > kman1 OK CAPABILITY Completed
    Mon May 13 10:00:27 2019: [debug  ] < kman2 STARTTLS
    Mon May 13 10:00:27 2019: [debug  ] > kman2 OK Begin TLS negotiation now
    Mon May 13 10:00:27 2019: [info   ] Using SSL now
    Mon May 13 10:00:27 2019: [debug  ] < kman3 CAPABILITY
    Mon May 13 10:00:27 2019: [debug  ] > * CAPABILITY IMAP4rev1 LITERAL+ AUTH=PLAIN CHILDREN XAOL-OPTION NAMESPACE QUOTA IDLE
    Mon May 13 10:00:27 2019: [debug  ] > kman3 OK CAPABILITY Completed
    Mon May 13 10:00:27 2019: [debug  ] < kman4 AUTHENTICATE PLAIN
    Mon May 13 10:00:27 2019: [debug  ] > + 
    Mon May 13 10:00:27 2019: [debug  ] Initializing provider "Kopano Directory Service"
    Mon May 13 10:00:27 2019: [debug  ] Initializing provider "Private Folders"
    Mon May 13 10:00:27 2019: [error  ] HrGetStore failed: No store present.
    Mon May 13 10:00:27 2019: [warning] Failed to login from [[::ffff:153.46.253.213]:43413] with invalid username "shaller" or wrong password: not found (8004010f)
    Mon May 13 10:00:27 2019: [debug  ] > kman4 BAD Internal error: OpenECSession failed
    Mon May 13 10:00:27 2019: [debug  ] < kman5 LOGOUT
    Mon May 13 10:00:27 2019: [debug  ] > * BYE server logging out
    Mon May 13 10:00:27 2019: [debug  ] > kman5 OK LOGOUT completed
    Mon May 13 10:00:27 2019: [notice ] Disconnecting client.
    

    The server connection to LDAP is now working as following confirms.

    kopano-admin -l
    User list for Default(4):
    	Username	Fullname	Homeserver	
    	-------------------------------------------------
    	SYSTEM		SYSTEM		Kopano	
    	shaller		Stefan Haller		
    	ehaller		Elena Haller		
    	challer		Cornelia Haller
    

    But it seems that Kopano doesn’t know about the user store

    kopano-admin --details shaller
    User object id:         00000000AC21A95040D3EE48B319FBA7533044250100000006000000150000005533526C5A6D4675494568686247786C63673D3D00000000
    WARNING: Unable to get user store entry id. User possibly has no store.
    

    I think to remember that I once had such an issue in the past as well. But don’t remember how to solve it.


  • Kopano

    @smhaller said in KC 8.7.80 upgrade issues:

    No - a downgrade is even more horrable than failing forward.

    Then I’ll wish you good look on your journey.



  • @smhaller

    How about you just setup a Debian 9 Virtual Machine, setup a very basic Kopano installation out of the box, then connect to the database and see if that works?

    If it does, you know your database works and the issue is just with the Debian 10 setup. And the answer there would be just to roll back to Debian 9.

    if instead it doesn’t work, then I would go back to your last good database backup…



  • @mcostan said in KC 8.7.80 upgrade issues:

    @smhaller

    How about you just setup a Debian 9 Virtual Machine, setup a very basic Kopano installation out of the box, then connect to the database and see if that works

    If it does, you know your database works and the issue is just with the Debian 10 setup. And the answer there would be just to roll back to Debian 9.

    if instead it doesn’t work, then I would go back to your last good database backup…

    I could for sure do this, but the access to the database works, I don’t need to proove this.
    The issue is somewhere in the MAPI -> python3 relation, at least to my understanding.
    As reported in the thread:
    https://forum.kopano.io/topic/2014/kopano-spamd-not-starting-because-of-no-module-named-_mapicore-no-module-named-_mapicore/2

    I also see the same issue. He somehow downgraded the system. most probably by a fresh install, which is for me not the way to go.

    As per Debian specification a downgrade is not forseen. Even if I put the repos back to stretch, which did meanwhile, It is not supportive in going back with the software installed.



  • @smhaller

    My advise on this is never to run on installations directly on physical servers.

    I learned the hard way a few years back that although I run on physical hardware, I have nothing on it.

    All it does is to run VirtualBox and everything runs in Virtual Machines.

    Before I upgrade anything including Kopano, I take a physical copy of the virtual machine, perform the upgrade, if it works great, otherwise I just roll back to the previous version of the virtual machine.

    This way I literally have nothing to do, roll back files whatever, I just copy the disk of the virtual machine back and I am done with the rollback.

    I would strongly advise doing this in the future. At least this is the way I do it.

    In addition once a week automatically the virtual machines are backed up, even without an upgrade performed.


  • Kopano

    @smhaller said in KC 8.7.80 upgrade issues:

    The issue is somewhere in the MAPI -> python3 relation, at least to my understanding.

    But neither kopano-dagent nor kopano-gateway have a direct relation to python. while the dagent has this plugin framework to to processing of message before the final delivery, the delivery is not related to python at all.



  • @smhaller

    Also this:

    But it seems that Kopano doesn’t know about the user store
    kopano-admin --details shaller
    User object id: 00000000AC21A95040D3EE48B319FBA7533044250100000006000000150000005533526C5A6D4675494568686247786C63673D3D00000000
    WARNING: Unable to get user store entry id. User possibly has no store.

    I don’t think it is related to python? It appears something is now wrong in the database or am I wrong?



  • Well you have hit multple bugs.

    Some can be fixed easy, some there are workarounds for.

    run :
    kopano-admin --create-store shaller-c shaller@domain.tld -p PassWher -e email@domain.tld
    kopano-storeadm -h default: -Cn shaller

    and when the password isnt working run : kopano-admin -P -u shaller

    try that,



  • Thanks mate
    that was a good starting point.

    @thctlo said in KC 8.7.80 upgrade issues:

    run :
    kopano-admin --create-store shaller-c shaller@domain.tld -p PassWher -e email@domain.tld
    kopano-storeadm -h default: -Cn shaller

    Meanwhile the username to store is hooked again.

    kopano-admin -u shaller --hook-store 524848af999946e492b5fbdf4aa4afe1
    

    Now I can see that even login timestamp is updated when checking with kopano-admin --detail shaller

    But the server doesn’t really let me in. On my mobile I get reported that “NO LOGIN imap feature is disabled”. Which in fact matches with the settings given.

    Mapped properties:
    	PR_EC_ENABLED_FEATURES	mobile; outlook; webapp	
    	PR_EC_DISABLED_FEATURES	imap; pop3
    

    Why are my client settings not working anymore?



  • IMAP is indeed disabled.

    Try this:

    kopano-admin -u username --enable-feature imap



  • Hmm yes I thought about that. But the settings from kopano should be as they were before

    kopano-admin -v -u shaller --enable-feature imap
    Unable to update user information: action not supported by server (0x80040102)
    Using the -v option (possibly multiple times) may give more hints.

    where server.log tells me:

    Mon May 13 12:30:15 2019: [warning] K-1505: Unable to set details for external user 21: Change object is not supported when using the LDAP user plugin.
    

    Meanwhile also incoming email is correctly delivered to the user stores. The only residing issue now is: to retrieve the mails with client.
    In fact Kopano Webapp is not yet working, as the mapi.so cannot be found.

    May 13 12:39:02 oklahoma sessionclean[6769]: PHP Warning:  PHP Startup: mapi: Unable to initialize module
    May 13 12:39:02 oklahoma sessionclean[6769]: Module compiled with module API=20151012
    May 13 12:39:02 oklahoma sessionclean[6769]: PHP    compiled with module API=20170718
    May 13 12:39:02 oklahoma sessionclean[6769]: These options need to match
    May 13 12:39:02 oklahoma sessionclean[6769]:  in Unknown on line 0
    


  • run dpkg -l | grep mapi
    whats the output?

    enable the kopano mapi.
    phpenmod kopano

    the stop and start your webserver, (no restart, stop/start) .



  • @thctlo said in KC 8.7.80 upgrade issues:

    dpkg -l | grep mapi

    ii  libkcicalmapi0                            8.7.80.936.529d71bca-0+193.1                                       amd64        iCal interface for MAPI
    ii  libkcicalmapi0-dbgsym                     8.7.80.936.529d71bca-0+193.1                                       amd64        Debug symbols for libkcicalmapi0
    ii  libkcinetmapi0                            8.7.80.936.529d71bca-0+193.1                                       amd64        Interface between internet e-mail and MAPI
    ii  libkcinetmapi0-dbgsym                     8.7.80.936.529d71bca-0+193.1                                       amd64        Debug symbols for libkcinetmapi0
    ii  libmapi1                                  8.7.80.936.529d71bca-0+193.1                                       amd64        Kopano's implementation of the Messaging API
    ii  libmapi1-dbgsym                           8.7.80.936.529d71bca-0+193.1                                       amd64        Debug symbols for libmapi1
    rc  php-mapi                                  8.7.0-3                                                            amd64        Complete and feature rich groupware solution - PHP MAPI bindings
    ii  php7-mapi                                 8.7.80.936.529d71bca-0+193.1                                       amd64        PHP bindings for MAPI
    ii  php7-mapi-dbgsym                          8.7.80.936.529d71bca-0+193.1                                       amd64        Debug symbols for php7-mapi
    ii  python3-mapi                              8.7.80.936.529d71bca-0+193.1                                       amd64        Python 3 bindings for MAPI
    ii  python3-mapi-dbgsym                       8.7.80.936.529d71bca-0+193.1                                       amd64        Debug symbols for python3-mapi
    

    As you can see from this, the mapi packages are installed.



  • Issue with php-mapi is now fixed.
    I’m able to login through Kopano WebApp and read my email.

    Remaining issue, login with outlook, mobile client not possible.

    UPDATE: got connection from mobile phone and Mac Mail working.
    Had to change the parameter in server.cfg
    from: disabled_features = imap pop
    to: disabled_features = pop

    Honestly I have no clue why, because with 8.4 this setting was working. users in LDAP have all the setting enabledKopanoFeatures: imap

    I also notified that gateway isn’t using MD5 as I thought it should be.

    Cheers
    Stefan



  • @smhaller said in KC 8.7.80 upgrade issues:

    rc php-mapi

    remove that one php-mapi…

    and apt-get remove *-dbgsym



  • @thctlo said in KC 8.7.80 upgrade issues:

    @smhaller said in KC 8.7.80 upgrade issues:

    rc php-mapi

    remove that one php-mapi…

    ok - I thought it is still needed …

    @thctlo said in KC 8.7.80 upgrade issues:

    and apt-get remove *-dbgsym

    yes that makes sense.


Log in to reply