Request for Feedback: New CalDAV and CardDAV implementation for Kopano

MichaelAnders2

Hi,

First of all I am very happy that Kopano has decided to implement CardDAV! Now I do not need MS Exchange anymore which I cannot get to work on my Android cell anymore since Oreo, so this is great.

However, I have two points of feedback / bugs which need to be ironed out before I can use it productively:

International characters like german umlaut ä, ö, ß or others like ć are not working for names, addresses, it’s a total mess. They can be stored like that in the webapp, but once transfered to a cellphone it’s not working.

Using export2csv.php file shows that everything is correct, all the umlaute are there. But they appear not to be UTF encoded - my Ubuntu terminal shows a slight difference in how a character is displayed in terminal in case of ASCII or UTF, the later being more “drawn” style and a bit bolder. However, this download works.

To test kdav I used on my Android cellphone:

a) Davdroid - cannot communicate with kdav (but works fine for my Nextcloud Server)

b) contactsync (https://play.google.com/store/apps/details?id=com.vcard.android.free) - works for downloads of contacts. Uploads do do not work well. Looking at the kdav logs I can see that contactsync is sending the VCF data with a “characterset UTF-8” in the HTTP header, but the result is that the kopano database gets data which is not double byte but two single bytes.
Consequences:

The characters are not “ä” anymore but are two digits which are the two single bytes of the unicode character - export2csv.php also shows the same result
The contact in the webapp is a new contact if the name was changed causing a different ID (with the old contact remaining)
Upon a new sync to contactsync the “ä” is now also replaces with the two single bytes
In turn, upon sycing with kdav the two single bytes which contactsync sends are now each replaced with two new single bytes - so in total taking it from 1 character to 2 characters and now to 4 characters. You can imagine what happens upon the new download and upload I guess?

To me it looks like kopano does not interpret the codepage in which the client sent the data (I can clearly see the double byte character in my terminal as it’s a different visual style as normal) - kdav always converts the input in some way which is wrong / not required causing serious problems.

Birthdays
These are stored as UTC timestamps in Kopano. Nothing wrong with that.
What is wrong however is that these birthdays are not sent in an expected format “YYYY-mm-dd” - they also contain a timezone… This may be OK based on e.g. rfc6474 for “deathdate”. But the problem is that using a UTC timestamp usually triggers code to do a conversion into the timezone of the client - this is the correct behaviour. But this requires that the client and server both “understand” what each one does, if one does it differently then there will be problems.
For me, a contact has the bday “1990-09-05”. This is how I see it in the webapp. When I export this contact it is sent with a UTC timestamp of 4th september 11:00pm. This makes the client set the bday as 4th September. In other cases kdav sends it as 0:00am - no idea why, maybe due to DST?
Now one can start arguing “kdav is doing it right, the client should fix it”. But the trivial approach is just to always STORE the bday in UTC+0 and send the bday as YYYY-mm-dd - without ANY conversion at all to the client. Then there can never be any mixup as no client will do any timezone adjustments as it doesn’t know the timezone.
And another argument - Nextcloud, which also uses sabreDav, works perfectly fine also for the birthdays of my contacts.

This should be trivial to reproduce. TIll these issues are solved, I’m using my Nextcloud Server for my contacts.

Fux

I’ve tried but always get:

Fatal error: Uncaught PDOException: SQLSTATE[HY000] [14] unable to open database file in /var/www/kdav/lib/KopanoSyncState.php:45 Stack trace: #0 /var/www/kdav/lib/KopanoSyncState.php(45): PDO->__construct(‘sqlite:/var/lib…’) #1 /var/www/kdav/lib/KopanoDavBackend.php(48): Kopano\DAV\KopanoSyncState->__construct(Object(Kopano\DAV\KLogger), ‘sqlite:/var/lib…’) #2 /var/www/kdav/server.php(54): Kopano\DAV\KopanoDavBackend->__construct(Object(Kopano\DAV\KLogger)) #3 {main} thrown in /var/www/kdav/lib/KopanoSyncState.php on line 45

/var/lib/kopano/kdav is owned by www-data:

drwxr-xr-x 2 www-data www-data 4096 Dez 11 17:28 kdav

php-sqlite3 is installed. System is devuan ascii. Kopano 8.6.81, php7.2

Can anyone help? Thanks in advance!

fbartels

@Fux how are the permissions of the file itself? so /var/lib/kopano/kdav/syncstate.db?

fbartels

@fbartels said in Request for Feedback: New CalDAV and CardDAV implementation for Kopano:

“ln -s /usr/share/kopano/php/mapi /var/www/kdav”: mapi directory doesn’t exist on my server, main kDAV directory is selectable in the start of the installation guide

just a node on that. since we recently removed the mapi classes from php-mapi they are now part of the repo itself and no symlinking is required anymore.

Fux

The file /var/lib/kopano/kdav/syncstate.db doesn’t exist yet. I tried to create (with touch) and set the right permissions - but no success.
I thought it would be created at first time kdav runs?

fbartels

@Fux said in Request for Feedback: New CalDAV and CardDAV implementation for Kopano:

I thought it would be created at first time kdav runs?

Yes, that is indeed what should be the case. The remaining conclusion is that you are running apparmor or something similar and that prevents php from writing into this folder.

Solution: adapt your apparmor config or change the location of the file into /var/www in the config of kdav

Fux

After changing the path to /tmp only for testing) it works and file is created. Must be something wrong with my /var/lib/kopano/kdav.
Although I have no apparmor installed (selinux either) and the permissions look good.
OK, but this seem to bbe a special problem with my system.

Thanks for your help!

smilbert

Hi,
thank you for working on a complete dav implementation for kopano. It is appreciated!
In my test installation it works fine for the most part. However I do encounter similar problems with special characters (ä/ö/ü/…) as described by MichaelAnders2.
Entering a name with “ä” in kopano webapp and syncing it to android ==> OK, correctly displayed in android.
Entering a name with “ä” in android an syncing it to kopano server/webapp ==> Not OK, Character is displayed like “Ã¤” in webapp.
I am using Kopano Core: 8.6.82 and WebApp 3.4.3.1137+674.1, running on nginx 1.10.
My smartphone uses android 8.1 and DavDroid (I also tried CardDav with the same result).
Thanks for your good work!

Fux

When I try to access contacts via kdav I get the following error at a few contacts - the others work:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:sabredav-version>3.2.2</s:sabredav-version>
  <s:exception>Sabre\VObject\ParseException</s:exception>
  <s:message>Invalid Mimedir file. Line starting at 19 did not follow iCalendar/vCard conventions</s:message>
</d:error>

22.02.2019 14:50:13 [29577] [DEBUG] main httpcode='500' memory='1.25 MiB/2.00 MiB' time='0.06s'
22.02.2019 14:50:13 [29577] [DEBUG] main ------------------ End

Does anyone know where to search for a solution?

SchweidJ

@Fux said in Request for Feedback: New CalDAV and CardDAV implementation for Kopano:

Invalid Mimedir file

Hi @Fux

I get exactly the same error, did you find an answer for the problem?

Regards,
Joachim

Fux

Unfortunatly not. Most of my contacts work. But some of them produce this errror. I cannot see why.

fbartels

@Fux @SchweidJ can you provide an export of your contacts. it seems there is something in one of your contacts that cannot be handled by libical.

To export you can use kopano-backup. The command to only backup contacts would be kopano-backup -u user1 -f Contacts -l debug. If would help if you could send your contacts (or even better only a subset of failing contacts) to feedback at kopano dot io.

Mornsgrans

Hello,
very amazing - the KDav-package is very helpful.

Unfortunatelly I found an issue, but I am not really sure, whether it is caused by emClient or Kdav or something else:
My szenario exists of my UCS-Kopano server, two client computers and an Android phone.

PC1: Windows 10, Outlook 2007 (IMAP) with CalDAV-Synchronizer as Addin
PC2: Windows 10, eClient (IMAP) with builtin CalDAV/CardDAV
UCS-Server with Kopmano, Caldav and KDav
Huawei P10 with latest Android

I can create Tasks and Calendar-entries on all devices and they become synchronized without problems.

I can create contacts on Android and PC1 with Outlook 2007 and they become synchronized to all devices - even to PC2.

If I create a contact on PC2 it appears for one second and becomes deleted, after synchronizing with KDav-server.
Strange: Deleting a contact works, so it seems not to be a permission-problem.

The setting of the contact-access is the same on PC1 and PC2:
https://dav.meinserver.de:8443/addressbooks/vw/Kontakte

Enclosure you will find the links to my kdav.log files, because my posting will be too large, if I post the logs.

PC1 Outlook 2007 with DAV-Synchronizer, where sync works:

https://www.dropbox.com/s/5ws1sbdjprdulzr/CalDAV-Synchronizer.txt?dl=0

and now the log of PC2-sync, where the added contact becomes deleted after sync:

https://www.dropbox.com/s/6fxrf83b478d6vl/CARDDAV-emClient_deletes_Vcards .txt?dl=0

I found an older thread in emClient-forum, where a similar problem appeared caused by bugs in horde (not on my UCS-server) and sabre:
https://forum.emclient.com/emclient/topics/carddav_server_contacts_not_showing_in_em_but_when_we_create_contact_on_em_it_is_getting_synced_on_carddav

I will be very appreciated, if you can help me.

Mornsgrans

Very helpful :(

Mornsgrans

Additional Info:
Contacts with phone numbers only, but without e-mail addresses become synchronized and stored and synchronized with KDav in the contacts folder.
Just if any email address was typed in, the contact becomes deleted just after sync-attempt.

If you do any changes in a contact, the changes become rejected (after sync the old state is back again).

Deleting contacts works.

Manfred

Hi Mornsgrans,

the emClient uses two different uids when creating a contact:

11.05.2019 12:47:34 [ 3199] [DEBUG] main PUT /addressbooks/vw/Kontakte/fce26428-fc67-49bf-929b-bc2a4f156af9.vcf

but in the vcard data:

BEGIN:VCARD
VERSION:4.0
PRODID:-//MailClient.VObject/7.2.35128.0
UID:urn:uuid:e70bbd69-7c81-4be3-8cb8-7bca4a5356fe
N;SORT-AS="Test, Otto":Test;Otto;;;
FN:Otto Test
EMAIL;PREF=1:otto@test.net
END:VCARD

KDAV uses the one in vcard data (e70bbd69-7c81-4be3-8cb8-7bca4a5356fe) to create a contact, but emClient uses the one from the PUT request (fce26428-fc67-49bf-929b-bc2a4f156af9). I guess that the contact gets deleted because KDAV response is 404 (not found). However the contact should still be in Kopano.

Manfred

Mornsgrans

Hello Manfred,
thank you for your reply. Unfortunatelly the contact does not remain in Kopano. But I think, that I now can contact emClient-Support using the hint in your answer.

Thank you very much.

Mornsgrans

Yesterday in the evening I got answer from emClient-support:

…
We have come to the conclusion that the issue is server-side announcing support for vCard 3 and vCard 4 versions (vcardVersion.PNG), but only supporting 3 or not fully supporting v4. We at eM Client try to use the latest possible version the server announced it supports and that seems to be the issue.
…
We suggest you either get in touch with Kopano support with this detailed screenshots/information or provide us a direct contact in Kopano to whom we could explain and/or test more. We would also welcome some kind of feedback on the Kopano forum as the analysis done by their team is incorrect, as explained and demonstrated earlier.
…

So my question is:
How can the emClient-support-team contact you?

P.S.: Changing calendar-entries also sometimes leads to an error.

Mornsgrans

Well, now I have the permission to post the full answer from emClient-support.

In addition emclient suppert will be very appreciated, if the Kopano team gets in contact with emClient-team to exchange informations to get emClient to sync with Kopano. The Kopano tam can use the case number- but I do not want to publish it in the internet.

Thank you for the test account. We have tried to simulate the same UID and URL, but as we were afraid of, this did not solve the issue.

We have come to the conclusion that the issue is server-side announcing support for vCard 3 and vCard 4 versions (vcardVersion.PNG), but only supporting 3 or not fully supporting v4. We at eM Client try to use the latest possible version the server announced it supports and that seems to be the issue.

Let us explain in more details but please note that if the explanation is too technical, Kopano should be able to fully understand it or if you can provide direct contact at Kopano, we can get in touch with them.

1, Confirmation that UID and URL can be different:
We have created very simple basic contact information containing a single word name “TestName” - no space nor any other information. This contact is correctly uploaded to Kopano and stays in server. Please see the screenshot from eM Client,
raw communication log (TestNameUpload.PNG) and confirmation that the URL is correctly found in SabreDAV/Kopano (SabreDavConfirmation.PNG).

Creating contact with full name and surname, i.e. “Test Name” will fail here

2, Creating complex name with same UID and URL
Like the above case, we have forced eM Client to keep the fields same. This does not help as contact is not created (sameUidURLNotCreated.PNG).

3, Manually creating a contact in v3 format
It can be seen from screenshot v3DiffUidURLSaved.PNG that this is correctly processed by Kopano and saved. The URL and UID are different, the version is v3 and name/surname/phone and email are filled in.

Unfortunately, the vCard version v3 and v4 are not fully backward compatible and there is no configuration option to manually switch the version in eM Client as we fully trust the information server sends us. We try to use the latest supported format to be able to provide our users as much functionality as possible.

We suggest you either get in touch with Kopano support with this detailed screenshots/information or provide us a direct contact in Kopano to whom we could explain and/or test more. We would also welcome some kind of feedback on the Kopano forum as the analysis done by their team is incorrect, as explained and demonstrated earlier.

With regards

Attached I received 5 screenshots:
vcardVersion.PNG

v3DiffUidURLSaved.PNG

TestNameUpload.PNG

sameUidURLNotCreated.PNG

SabreDavConfirmation.PNG

Manfred

Hi Mornsgrans,

we’ll investigate why it works with vCard version 3., but not with version 4.

Regarding the vCard 3 and vCard 4 - it’s coming from sabredav which KDAV uses. The current quick workaround this issue would be to edit /usr/share/kdav/vendor/sabre/dav/lib/CardDAV/Xml/Property/SupportedAddressData.php around line 39, so that it looks like this:

        if (is_null($supportedData)) {
            $supportedData = [
                ['contentType' => 'text/vcard', 'version' => '3.0'],
                //['contentType' => 'text/vcard', 'version' => '4.0'],
                //['contentType' => 'application/vcard+json', 'version' => '4.0'],
            ];
        }

This will force using vCard version 3.

Manfred