SerializeObject failed with error code 0x80000004 for object

martimcfly

Hey Felix!

Thanks for calling me up for the wrong link and taking care of my issue :) I’ve fixed the link.

All errors have been there simultaneously. ECSearchFolders, K-1573 and K-1571 appeared right after the start of kopano-server. By deleting all sarchfolders I could make it go away.

After the start of kopano-backup the kopano-server shows some workload and logs the SerializeObject exception. kopano-backup is dead from then on, but does not exit. Kopano-server continous to run.

kopano    2007  0.0  1.1 198912 23108 ?        Ssl  19:19   0:06 /usr/sbin/kopano-server -F
root      2963  0.8  1.3  93492 28220 pts/0    Sl+  21:40   0:00 /usr/bin/python2 /usr/bin/kopano-backup -u user
root      2966 24.4  0.0      0     0 pts/0    Z+   21:40   0:11 [kopano-backup] <defunct>

Same with kopano-search. Each SerializeObject exception causes the corresponding indexing processes to die (<defunct>).

I thought about data corruption, so I figured out some of the logged mails and opened them in the Webapp. But they showed well without any error messages.

To be sure this isn’t a dependency problem, I checked each version of gsoap starting with 2.8.66 down to 2.8.44 (I use 2.8.65). Problem persists. After a quick check I figured out that gsoap 2.8.60 is used in the Fedore release.

This are my compile flags.

CPPFLAGS=-I/usr/include/python2.7 ./configure \
    --prefix=/ \
    --localstatedir=/var \
    --sysconfdir=/etc \
    --exec-prefix=/usr \
    --sbindir=/usr/bin \
    --datarootdir=/usr/share \
    --includedir=/usr/include \
    --enable-release \
    --enable-epoll \
    --enable-python \
    --disable-debug \
    --disable-static \
    --with-userscript-prefix=/etc/kopano/userscripts \
    --with-quotatemplate-prefix=/etc/kopano/quotamails \
    --with-searchscripts-prefix=/etc/kopano/searchscripts \
    --with-php=7 \
    PYTHON=/usr/bin/python2

make

A check with Zarafa 7.2.6 has shown the same issues. ECSearchFolders exception right after the start of zarafa-server and SerializationMessage exceptions after the start of zarafa-search.

With an empty database everything works great.

A friend and I tried debugging the code but, we had no luck compiling the debug information for ECStreamSerializer. CXXFLAGS=" -ggdb3" wasn’t enough.

How can we find the cause and fix it?

Marti

jengelh

Since the error code is 80000004 (KCERR_NETWORK_ERROR), it suggests that the client closed the connection, possibly because it received what it considers garbage as part of the server’s serialization. Theoretically, it could also indicate the mysql server going away. Unfortunately, that area of code does not help establishing very well where an error comes from. A gdb session would be most helpful here (or a database dump, so that gdb can be run on message 647968).

martimcfly

Hej jengelh!

only the client dies (search, backup… never tried archive or others). kopano-server, mysql, webapp do well.

We can do gdb through Teamviewer. With some advice about the needed build flags I can prepare the server.

Due to personal data and size I’m not able to handout the database dump. But I extracted the properties for a failed message. Its only a newsletter but still containing e-mail-adresses. I can show that to you by teamviewer :)

Would teamviewer be an option or would you preferre other options?

Thanks so much for your effort so far!

Marti

jengelh

TV will do. Given that TeamViewer Linux client has had its integrated chat functionality gutted from v13, I will point out that irc.freenode.net/#kopano is a possible alternative for textual communication during the session.

martimcfly

@jengelh Can you tell me the debug build command in advanced please? I build this for Armv7 and it takes a while to finish. Or don’t you need all symbols for gdb?

jengelh

(assuming Debian 9/10) apt-get install autoconf automake autotools-dev binutils debhelper devscripts fakeroot g++ gettext gsoap libcurl4-openssl-dev libical-dev libicu-dev libjsoncpp-dev libkrb5-dev libldap2-dev libmariadbclient-dev libncurses5-dev libncursesw5-dev libpam0g-dev libpopt-dev libssl-dev libtool libtool-bin libvmime-dev libxml2-dev m4 openssl php-dev pkg-config python3-dev python3-setuptools python-dev python-setuptools swig uuid-dev zlib1g-dev git-core gdb python-nose python-vobject joe valgrind python-requests

git clone https://github.com/kopano-dev/kopano-core

./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var CXXFLAGS="-O0 -ggdb3"

martimcfly

Well, I compiled it with those flags. It went all through and the package have gotten bigger. But it appears as if the symbols haven’t been installed with “make install”.

Following is giving empty output.

readelf -s /usr/lib/libkcserver.so | grep Serialize


Install prefix	:	/
Userscript dir	:	/etc/kopano/userscripts
Quota dir	:	/etc/kopano/quotamails
Search scripts dir	:	/etc/kopano/searchscripts

Build host	:	armv7l-unknown-linux-gnueabihf
Compile flags	:	 -DLINUX 	-I${top_builddir}/common/include 	-I${top_srcdir}/common/include 	-I${top_srcdir}/m4lcommon/include 	-I${top_srcdir}/mapi4linux/include 	-I${top_srcdir}/provider/common/include 	-I${top_srcdir}/provider/libserver/include 	-I${top_srcdir}/ECtools/archiver/include 	-I${top_srcdir}/inetmapi/include 	-I${top_srcdir}/swig/include 	-I${top_srcdir}/swig/python/include 	-I${top_srcdir} -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DNDEBUG -DUNICODE  -DUSE_EPOLL   -I/usr/include/python2.7 -Wall -Wmissing-declarations -std=c++11 -Wno-variadic-macros -O2 -fno-strict-aliasing -fvisibility=hidden -fvisibility-inlines-hidden -O0 -ggdb3
Debug flags	:	
PHP version	:	7.2.3
PHP ext dir	:	/usr/lib/php/modules
PHP config dir	:	/etc/php/conf.d
PHP flags	:	-I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib
PHP libs	:	
gSOAP flags	:	-DWITH_OPENSSL -DWITH_GZIP -DHAVE_OPENSSL_SSL_H -DHAVE_ZLIB_H
gSOAP libs	:	-lgsoapssl++
gSOAP soapcpp2	:	
OpenSSL libs	:	-lssl
MySQL flags	:	-I/usr/include/mysql -I/usr/include/mysql/..
MySQL libs	:	-L/usr/lib  -lmysqlclient -lpthread -lz -lm -ldl -lssl -lcrypto
VMime flags	:	-I/usr/include/vmime/ -I/usr/include/p11-kit-1
VMime libs	:	-lvmime
XML2 flags	:	-I/usr/include/libxml2
XML2 libs	:	-lxml2
ICal flags	:	
ICal libs	:	-lical -licalss -licalvcal -lpthread /lib/libicuuc.so /lib/libicui18n.so
ncurses libs	:	-lncursesw
Clients libs	:	-luuid   -lz
LDAP support	:	yes
LDAP flags	:	-DLDAP_DEPRECATED=1
LDAP libs	:	-lldap_r
Curl libs	:	-lcurl
Pam libs	:	-lpam
Krb5 libs	:	-lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err
TCMalloc lib	:	-ltcmalloc_minimal
S3 lib		:	
KCOIDC lib	:	
ICU libs	:	-licuuc -licudata -licui18n -licuuc -licudata
Python flags	:	-I/usr/include/python2.7
Python libs	:	-lpython2.7

...

libtool: install: /usr/bin/install -c .libs/libkcsoap.so.0.0.0T /build/kopano-core/pkg/kopano-core/usr/lib/libkcsoap.so.0.0.0
libtool: install: (cd /build/kopano-core/pkg/kopano-core/usr/lib && { ln -s -f libkcsoap.so.0.0.0 libkcsoap.so.0 || { rm -f libkcsoap.so.0 && ln -s libkcsoap.so.0.0.0 libkcsoap.so.0; }; })
libtool: install: (cd /build/kopano-core/pkg/kopano-core/usr/lib && { ln -s -f libkcsoap.so.0.0.0 libkcsoap.so || { rm -f libkcsoap.so && ln -s libkcsoap.so.0.0.0 libkcsoap.so; }; })
libtool: install: /usr/bin/install -c .libs/libkcsoap.lai /build/kopano-core/pkg/kopano-core/usr/lib/libkcsoap.la
libtool: warning: relinking 'libkcserver.la'
libtool: install: (cd /build/kopano-core/src/kopanocore; /bin/sh "/build/kopano-core/src/kopanocore/libtool"  --silent --tag CXX --mode=relink g++ -Wall -Wmissing-declarations -std=c++11 -Wno-variadic-macros -O2 -fno-strict-aliasing -fvisibility=hidden -fvisibility-inlines-hidden -Wno-sign-compare -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O2 -pipe -fstack-protector-strong -fno-plt -Wl,--version-script=provider/libkcserver.sym -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -o libkcserver.la -rpath /usr/lib common/database.lo provider/libserver/ECABObjectTable.lo provider/libserver/ECAttachmentStorage.lo provider/libserver/ECCacheManager.lo provider/libserver/ECConvenientDepthObjectTable.lo provider/libserver/ECDatabaseFactory.lo provider/libserver/ECDatabaseMySQL.lo provider/libserver/ECDatabaseUpdate.lo provider/libserver/ECDatabaseUtils.lo provider/libserver/ECGenProps.lo provider/libserver/ECGenericObjectTable.lo provider/libserver/ECICS.lo provider/libserver/ECICSHelpers.lo provider/libserver/ECIndexer.lo provider/libserver/ECKrbAuth.lo provider/libserver/ECLockManager.lo provider/libserver/ECMailBoxTable.lo provider/libserver/ECMultiStoreTable.lo provider/libserver/ECNotificationManager.lo provider/libserver/ECPluginFactory.lo provider/libserver/ECPluginSharedData.lo provider/libserver/ECS3Attachment.lo provider/libserver/ECSearchFolders.lo provider/libserver/ECSearchObjectTable.lo provider/libserver/ECSecurity.lo provider/libserver/ECServerEntrypoint.lo provider/libserver/ECSession.lo provider/libserver/ECSessionGroup.lo provider/libserver/ECSessionManager.lo provider/libserver/ECStatsCollector.lo provider/libserver/ECStatsTables.lo provider/libserver/ECStoreObjectTable.lo provider/libserver/ECStringCompat.lo provider/libserver/ECSubRestriction.lo provider/libserver/ECTPropsPurge.lo provider/libserver/ECTableManager.lo provider/libserver/ECTestProtocol.lo provider/libserver/ECUserManagement.lo provider/libserver/ECUserStoreTable.lo provider/libserver/StorageUtil.lo provider/libserver/StreamUtil.lo provider/libserver/cmd.lo provider/libserver/cmdutil.lo provider/libserver/logontime.lo provider/common/ECSearchClient.lo provider/common/SOAPDebug.lo libkcutil.la libkcmapi.la libkcssl.la libkcsoap.la -lpthread -licui18n -licuuc -licudata -licuuc -licudata -lgsoapssl++ -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -L/usr/lib -lmysqlclient -lpthread -lz -lm -ldl -lssl -lcrypto -lpam -lssl -inst-prefix-dir /build/kopano-core/pkg/kopano-core)
libtool: install: /usr/bin/install -c .libs/libkcserver.so.0.0.0T /build/kopano-core/pkg/kopano-core/usr/lib/libkcserver.so.0.0.0
libtool: install: (cd /build/kopano-core/pkg/kopano-core/usr/lib && { ln -s -f libkcserver.so.0.0.0 libkcserver.so.0 || { rm -f libkcserver.so.0 && ln -s libkcserver.so.0.0.0 libkcserver.so.0; }; })
libtool: install: (cd /build/kopano-core/pkg/kopano-core/usr/lib && { ln -s -f libkcserver.so.0.0.0 libkcserver.so || { rm -f libkcserver.so && ln -s libkcserver.so.0.0.0 libkcserver.so; }; })
libtool: install: /usr/bin/install -c .libs/libkcserver.lai /build/kopano-core/pkg/kopano-core/usr/lib/libkcserver.la

Marti

jengelh

that’s because readelf for some stupid reason truncates the names by default, even if they’re in the last column (one explicitly needs -W), so a trivial grep would not find it. nm is better suited for that.

martimcfly

Badly not…

[root@server lib]# nm libkcserver.so
nm: libkcserver.so: no symbols

[root@serverlib lib]# readelf -w libkcserver.so
Contents of the .eh_frame section:

00000000 ZERO terminator


[root@server lib]#

It didn’t matter if I did this…

./bootstrap.sh
CPPFLAGS="-I/usr/include/python2.7" CXXFLAGS="-O0 -ggdb3" ./configure \
  --prefix=/ \
  --localstatedir=/var \
  --sysconfdir=/etc \
  --exec-prefix=/usr \
  --sbindir=/usr/bin \
  --datarootdir=/usr/share \
  --includedir=/usr/include \
  --enable-release \
  --enable-epoll \
  --enable-python \
  --disable-debug \
  --disable-static \
  --with-userscript-prefix=/etc/kopano/userscripts \
  --with-quotatemplate-prefix=/etc/kopano/quotamails \
  --with-searchscripts-prefix=/etc/kopano/searchscripts \
  --with-php=7 \
  PYTHON=/usr/bin/python2

make
make install DESTDIR="$(realpath ${pkgdir})"

or this

./bootstrap.sh
CPPFLAGS="-I/usr/include/python2.7" ./configure \
 CXXFLAGS="-O0 -ggdb3" \
  --prefix=/ \
  --localstatedir=/var \
  --sysconfdir=/etc \
  --exec-prefix=/usr \
  --sbindir=/usr/bin \
  --datarootdir=/usr/share \
  --includedir=/usr/include \
  --enable-release \
  --enable-epoll \
  --enable-python \
  --disable-debug \
  --disable-static \
  --with-userscript-prefix=/etc/kopano/userscripts \
  --with-quotatemplate-prefix=/etc/kopano/quotamails \
  --with-searchscripts-prefix=/etc/kopano/searchscripts \
  --with-php=7 \
  PYTHON=/usr/bin/python2

make
make install DESTDIR="$(realpath ${pkgdir})"

Here is the log for the last build command…

https://gist.githubusercontent.com/mpietruschka/79ef730d2fa2defeac80a0301d59e3ec/raw/a911d8dfe68fa8a477de54dd681d1b39804a5f9e/build_kopano-core.log

This is the packaging script…

https://gist.githubusercontent.com/mpietruschka/8c8293b1898e9de53bbe77f0f50607f1/raw/ee256e7a9d26c7d70a0e39d1483dcb024e1df46c/PKGBUILD

Any ideas?

jengelh

==> Tidying install...
  -> Removing libtool files...
  -> Purging unwanted files...
  -> Removing static library files...
  -> Stripping unneeded symbols from binaries and libraries...

This strip may be it, and it’s not coming from KC. But worry not, kopano-server can be started right from the build directory, it does not need make install or extra packaging for a debug session.

nm .libs/libkcserver.so

martimcfly

You’re awesome! This has been it!

I had to change the packager defaults to not strip symbols or remove libtool .la files and debug informations. I disabled it specifically for this package.

/etc/makepkg.conf

#########################################################################
# GLOBAL PACKAGE OPTIONS
#   These are default values for the options=() settings
#########################################################################
#
# Default: OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge !optipng !upx !debug)
#  A negated option will do the opposite of the comments below.
#
#-- strip:      Strip symbols from binaries/libraries
#-- docs:       Save doc directories specified by DOC_DIRS
#-- libtool:    Leave libtool (.la) files in packages
#-- staticlibs: Leave static library (.a) files in packages
#-- emptydirs:  Leave empty directories in packages
#-- zipman:     Compress manual (man and info) pages in MAN_DIRS with gzip
#-- purge:      Remove files specified by PURGE_TARGETS
#-- upx:        Compress binary executable files using UPX
#-- optipng:    Optimize PNG images with optipng
#-- debug:      Add debugging flags as specified in DEBUG_* variables
#
OPTIONS=(!strip docs libtool staticlibs emptydirs zipman purge !optipng !upx debug)

The method finally got visible!

[root@server alarm]# nm /usr/lib/libkcserver.so | grep SerializeM
0018da7c t _ZN2KC16SerializeMessageEPNS_9ECSessionEPNS_10ECDatabaseEPNS_19ECAttachmentStorageEPKNS_10StreamCapsEjjjP4GUIDjPNS_12ECSerializerEb

Now we can go on with gdb :D

Marti

martimcfly

@jengelh I couldn’t start kopano from build directory because its build in a chroot environment on a different device :)

jengelh

Let me put up an obligatory questioning of the motive here ;-)

thought you wanted a native build because “I build this for Armv7 and it takes a while to finish”
if you have a fast machine to compile with, the DB could be transferred

I built KC in reasonable time even with an AMD AXP from 2003, which is probably a reasonable assumption for what a today’s raspberry pi ARM would be capable of.

martimcfly

I build for odroid on another ordroid :) both are armv7 and way faster as a RPI. The setup for the chroot environment and compilation take about 30 minutes together. A transfer of the database takes for ever. I don‘t think this is an option ;)

I could speed up compilation with distributed compilation. But I had no time to update my workers with gcc 7.

jengelh

It is because libtidy crashes, and that kills kopano-backup’s worker subprocess, which the main process also completely fails to act on.

The server side then reports 80000004 because the client disappeared in the middle of a send(2) call.

martimcfly

Hey jengelh,

I’d like to say thank you for all the tremendous help you gave going through the code finding the problem in the last place one would expect!

Marti

jengelh

https://github.com/htacg/tidy-html5/issues/739