Request for Feedback: New CalDAV and CardDAV implementation for Kopano

freyuh

I think my problem ist the following:

15.08.2018 18:00:55 [23963] [DEBUG] dav KopanoDavBackend->Sync(): Exporter found 0 changes, buffer size for mapi_exportchanges_synchronize 1000
15.08.2018 18:00:55 [23963] [DEBUG] dav KopanoDavBackend->Sync(): Exported 0 changes, pending 0

Ulrich.

markVnl

Those hostnames I referenced have northing to do with it, they are just xml namespaces (it’s so embarrassing almost edited the post)

however here a cross check of the creation of a contact (Test User test.user@example.com )

eM client (windows)

17.08.2018 08:51:58 [ 2084] [DEBUG] main ------------------ Start
17.08.2018 08:51:58 [ 2084] [DEBUG] main PUT /kdav/addressbooks/kopano_user/Contacts/5d877935-c577-4337-9eb6-d0797142fd5d.vcf
17.08.2018 08:51:58 [ 2084] [DEBUG] main KDAV version GIT
17.08.2018 08:51:58 [ 2084] [DEBUG] main SabreDAV version 3.2.2
17.08.2018 08:51:58 [ 2084] [DEBUG] main KLogger->LogIncoming(): INPUT
PUT /kdav/addressbooks/kopano_user/Contacts/5d877935-c577-4337-9eb6-d0797142fd5d.vcf HTTP/1.1
Content-Length: 217
Host: my.kopanohost.lan
If-None-Match: *
Content-Type: text/vcard; charset=utf-8
Accept-Encoding: gzip, deflate
User-Agent: eM Client/7.1.33101.0
Authorization: Basic REDACTED

BEGIN:VCARD
VERSION:4.0
PRODID:-//MailClient.VObject/7.1.33101.0
UID:urn:uuid:49cb6acf-4237-42cb-88ef-ed67208acdcc
N;SORT-AS="User, Test":User;Test;;;
FN:Test User
EMAIL;PREF=1:test.user@example.com
END:VCARD

17.08.2018 08:51:58 [ 2084] [DEBUG]  dav KopanoDavBackend->Logon(): Auth: OK - user kopano_user - session Resource id #129
17.08.2018 08:51:58 [ 2084] [DEBUG]  dav KopanoDavBackend->GetMapiMessageForId(): Nothing found for 5d877935-c577-4337-9eb6-d0797142fd5d
17.08.2018 08:51:58 [ 2084] [DEBUG] card KopanoCardDavBackend->getCard(): Object NOT FOUND
17.08.2018 08:51:58 [ 2084] [DEBUG]  dav KopanoDavBackend->GetMapiMessageForId(): Nothing found for 5d877935-c577-4337-9eb6-d0797142fd5d
17.08.2018 08:51:58 [ 2084] [DEBUG] card KopanoCardDavBackend->getCard(): Object NOT FOUND
17.08.2018 08:51:58 [ 2084] [DEBUG]  dav KopanoDavBackend->GetMapiMessageForId(): Nothing found for 5d877935-c577-4337-9eb6-d0797142fd5d
17.08.2018 08:51:58 [ 2084] [DEBUG] card KopanoCardDavBackend->getCard(): Object NOT FOUND
17.08.2018 08:51:58 [ 2084] [DEBUG] main KLogger->LogOutgoing(): OUTPUT:
HTTP/1.1 201 Created
X-KDAV-Version: GIT
X-Sabre-Version: 3.2.2
Content-Length: 0


17.08.2018 08:51:58 [ 2084] [DEBUG] main httpcode='201' memory='3.75 MiB/2.00 MiB' time='0.24s'
17.08.2018 08:51:58 [ 2084] [DEBUG] main ------------------ End

TB linux

17.08.2018 08:55:07 [ 2081] [DEBUG] main ------------------ Start
17.08.2018 08:55:07 [ 2081] [DEBUG] main PUT /kdav/addressbooks/kopano_user/Contacts/e6822467-8d73-40eb-80d2-d15468a2b2e3.vcf
17.08.2018 08:55:07 [ 2081] [DEBUG] main KDAV version GIT
17.08.2018 08:55:07 [ 2081] [DEBUG] main SabreDAV version 3.2.2
17.08.2018 08:55:07 [ 2081] [DEBUG] main KLogger->LogIncoming(): INPUT
PUT /kdav/addressbooks/kopano_user/Contacts/e6822467-8d73-40eb-80d2-d15468a2b2e3.vcf HTTP/1.1
Connection: keep-alive
Content-Length: 226
If-None-Match: *
Content-Type: text/vcard; charset=utf-8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.5
Accept: */*
User-Agent: Thunderbird CardBook/30.5 Lightning/5.4.9.1
Host: my.kopanohost.lan
Authorization: Basic REDACTED

BEGIN:VCARD
VERSION:4.0
PRODID:-//Thunderbird.net/NONSGML Thunderbird CardBook V30.5//EN
UID:e6822467-8d73-40eb-80d2-d15468a2b2e3
FN:Test User
N:User;Test;;;
REV:20180817T105507Z
EMAIL:test.user@example.com
END:VCARD

17.08.2018 08:55:07 [ 2081] [DEBUG]  dav KopanoDavBackend->Logon(): Auth: OK - user kopano_user - session Resource id #129
17.08.2018 08:55:07 [ 2081] [DEBUG]  dav KopanoDavBackend->GetMapiMessageForId(): Nothing found for e6822467-8d73-40eb-80d2-d15468a2b2e3
17.08.2018 08:55:07 [ 2081] [DEBUG] card KopanoCardDavBackend->getCard(): Object NOT FOUND
17.08.2018 08:55:07 [ 2081] [DEBUG]  dav KopanoDavBackend->GetMapiMessageForId(): Nothing found for e6822467-8d73-40eb-80d2-d15468a2b2e3
17.08.2018 08:55:07 [ 2081] [DEBUG] card KopanoCardDavBackend->getCard(): Object NOT FOUND
17.08.2018 08:55:07 [ 2081] [DEBUG]  dav KopanoDavBackend->GetMapiMessageForId(): Nothing found for e6822467-8d73-40eb-80d2-d15468a2b2e3
17.08.2018 08:55:07 [ 2081] [DEBUG] card KopanoCardDavBackend->getCard(): Object NOT FOUND
17.08.2018 08:55:07 [ 2081] [DEBUG] main KLogger->LogOutgoing(): OUTPUT:
HTTP/1.1 201 Created
X-KDAV-Version: GIT
X-Sabre-Version: 3.2.2
Content-Length: 0
ETag: "1534496107"

With eM client the UID of the VCard is parsed differently :

UID:urn:uuid:49cb6acf-4237-42cb-88ef-ed67208acdcc

results in:

dav KopanoDavBackend->GetMapiMessageForId(): Nothing found for 5d877935-c577-4337-9eb6-d0797142fd5d

Do not know if it’s client- or server-side the mismatch is made
(EDIT) log of eM Client:

10:50:53.364|013|   SynchronizeItemsCommand finished on ( /32ed1819-7849-4187-91e3-c0f6a2330f45/addressbooks/kopano_user/Contacts)
10:51:54.663|013|   C: PutCommand https://my.kopanohost.lan/kdav/addressbooks/kopano_user/Contacts/5d877935-c577-4337-9eb6-d0797142fd5d.vcf
10:51:54.663|013|   BEGIN:VCARD
10:51:54.663|013|   VERSION:4.0
10:51:54.663|013|   PRODID:-//MailClient.VObject/7.1.33101.0
10:51:54.663|013|   UID:urn:uuid:49cb6acf-4237-42cb-88ef-ed67208acdcc
10:51:54.663|013|   N;SORT-AS="User, Test":User;Test;;;
10:51:54.663|013|   FN:Test User
10:51:54.664|013|   EMAIL;PREF=1:test.user@example.com
10:51:54.664|013|   END:VCARD
10:51:54.664|013|
10:51:54.967|013|   S: PUT https://my.kopanohost.lan/kdav/addressbooks/kopano_user/Contacts/5d877935-c577-4337-9eb6-d0797142fd5d.vcf
10:51:54.967|013|   Created Created
10:51:54.972|013|   C: GetCommand https://my.kopanohost.lan/kdav/addressbooks/kopano_user/Contacts/5d877935-c577-4337-9eb6-d0797142fd5d.vcf
10:51:55.190|013|   S: GET https://my.kopanohost.lan/kdav/addressbooks/kopano_user/Contacts/5d877935-c577-4337-9eb6-d0797142fd5d.vcf
10:51:55.190|013|   NotFound Not Found
10:51:55.201|013|   Response, stream length:

markVnl

Just to confirm Thunderbird CardBook card book works for me on windows to, with the same exception as on linux:

“File as” is not synced / populated.

freyuh

Very strange.

I get a ‘created’ in the kdav.log:

17.08.2018 13:38:16 [31065] [TRACE] dav KopanoDavBackend->GetMapiFolder(): Id: principals/u.frey:0a26138985c745caa857320d0479f69f72000
0000000
17.08.2018 13:38:16 [31065] [TRACE] dav KopanoDavBackend->GetStore(): storename u.frey
17.08.2018 13:38:16 [31065] [TRACE] dav KopanoDavBackend->GetStore(): storename u.frey
17.08.2018 13:38:16 [31065] [TRACE] card KopanoCardDavBackend->setData(): mapimessage: Resource id #29 - vcf: BEGIN:VCARD
VERSION:4.0
PRODID:-//Thunderbird.net/NONSGML Thunderbird CardBook V30.5//EN
UID:798ad0b5-b844-f84c-9f2d-a90378acc04d
FN:Test User
N:User;Test;;;
GENDER:M
REV:20180817T133816Z
EMAIL;TYPE=HOME:test@user.com
END:VCARD

17.08.2018 13:38:16 [31065] [TRACE] dav KopanoDavBackend->GetStore(): storename u.frey
17.08.2018 13:38:16 [31065] [DEBUG] main KLogger->LogOutgoing(): OUTPUT:
HTTP/1.1 201 Created
X-KDAV-Version: GIT
X-Sabre-Version: 3.2.2
Content-Length: 0
ETag: “1534505896”

But I can’t see it inside of the DeskApp …

Ulrich.

markVnl

same trace from TB windows

17.08.2018 16:00:25 [ 7448] [TRACE]  dav KopanoDavBackend->GetStore(): storename kopano_user
17.08.2018 16:00:25 [ 7448] [TRACE]  dav KopanoDavBackend->GetStore(): storename kopano_user
17.08.2018 16:00:25 [ 7448] [TRACE] card KopanoCardDavBackend->setData(): mapimessage: Resource id #24 - vcf: BEGIN:VCARD
VERSION:4.0
PRODID:-//Thunderbird.net/NONSGML Thunderbird CardBook V30.5//EN
UID:7f62d94d-4b65-411c-991b-a31455714d99
FN:test2 user
N:user;test2;;;
GENDER:M
REV:20180817T160021Z
EMAIL;TYPE=HOME:test2.user@example.com
END:VCARD

17.08.2018 16:00:25 [ 7448] [TRACE]  dav KopanoDavBackend->GetStore(): storename kopano_user
17.08.2018 16:00:25 [ 7448] [DEBUG] main KLogger->LogOutgoing(): OUTPUT:
HTTP/1.1 201 Created
X-KDAV-Version: GIT
X-Sabre-Version: 3.2.2
Content-Length: 0
ETag: "1534514425"

The only difference that catches my eyes is you have a ". " (dot) in your username. And i am not sure if that can cause problems, it is part of the URI (even part of the url to the dav collection).
just guessing here!

freyuh

Will continue next week with deeper searching …

freyuh

I have another error:

23.08.2018 14:40:42 [ 1474] [TRACE] dav KopanoDavBackend->GetIdOfMapiMessage(): Finding ID of Resource id #4575
23.08.2018 14:40:42 [ 1474] [DEBUG] dav KopanoDavBackend->GetIdOfMapiMessage(): Found PROP_APPTTSREF: fc13f44b-29cf-4beb-903b-011b24cdec1f
23.08.2018 14:40:42 [ 1474] [TRACE] cal KopanoCalDavBackend->getCalendarObject(): returned data id: fc13f44b-29cf-4beb-903b-011b24cdec1f - size: 1822 - etag: “1453797237”
23.08.2018 14:40:42 [ 1474] [DEBUG] main KLogger->LogOutgoing(): OUTPUT:
HTTP/1.1 500 Internal Server Error
X-KDAV-Version: GIT
X-Sabre-Version: 3.2.2
Content-Type: application/xml; charset=utf-8

<?xml version=“1.0” encoding=“utf-8”?>
<d:error xmlns:d=“DAV:” xmlns:s=“http://sabredav.org/ns”>
<s:sabredav-version>3.2.2</s:sabredav-version>
<s:exception>InvalidArgumentException</s:exception>
<s:message>The objectData argument must contain an ‘uri’ property</s:message>
</d:error>

23.08.2018 14:40:42 [ 1474] [DEBUG] main httpcode=‘500’ memory=‘6.47 MiB/8.00 MiB’ time=‘40.49s’
23.08.2018 14:40:42 [ 1474] [DEBUG] main ------------------ End

Ulrich.

markVnl

Just a follow up on my findings with eM Client :)

Strange enough eM client offers fruux as a pre configured service, and this is actually the the company (original?) behind the sabre dav / main developer. Meaning you can expect em Client works oke Sabre (Card)dav.

Reading old and resolved issues @sabre dav it seems to boil down to specific behavior of em Client:

It “PUT”(s) a vCard and immediately after confirmation of creation tries to “GET” it, if this “GET” fails it removes the new contact (vCard).

...
13.08.2018 13:38:17 [ 4640] [DEBUG] main KLogger->LogIncoming(): INPUT
PUT /kdav/addressbooks/kopano_user/Contacts/ede4362c-9a8d-4e49-8f5f-27cc3c7296ca.vcf HTTP/1.1
...
13.08.2018 13:38:17 [ 4640] [TRACE] card KopanoCardDavBackend->getCard(): addressBookId: principals/kopano_user:d3b2f10bf8604f25be6070fe96590b342b0000000000 - cardUri: ede4362c-9a8d-4e49-8f5f-27cc3c7296ca.vcf
...
13.08.2018 13:38:17 [ 4640] [TRACE] card KopanoCardDavBackend->setData(): mapimessage: Resource id #172 - vcf: BEGIN:VCARD
VERSION:4.0
PRODID:-//MailClient.VObject/7.1.33101.0
UID:urn:uuid:081044be-a742-4f4b-a5ba-f6011679d9bb
N;SORT-AS="User, Test":User;Test;;;
FN:Test User
EMAIL;TYPE=WORK;PREF=1:test.user@example.com
END:VCARD

13.08.2018 13:38:17 [ 4640] [TRACE]  dav KopanoDavBackend->GetStore(): storename kopano_user
13.08.2018 13:38:17 [ 4640] [DEBUG] main KLogger->LogOutgoing(): OUTPUT:
HTTP/1.1 201 Created
X-KDAV-Version: GIT
X-Sabre-Version: 3.2.2
Content-Length: 0


13.08.2018 13:38:17 [ 4640] [DEBUG] main httpcode='201' memory='3.95 MiB/4.00 MiB' time='0.21s'
13.08.2018 13:38:17 [ 4640] [DEBUG] main ------------------ End



13.08.2018 13:38:17 [ 4641] [DEBUG] main ------------------ Start
13.08.2018 13:38:17 [ 4641] [DEBUG] main GET /kdav/addressbooks/kopano_user/Contacts/ede4362c-9a8d-4e49-8f5f-27cc3c7296ca.vcf
13.08.2018 13:38:17 [ 4641] [DEBUG] main KDAV version GIT
13.08.2018 13:38:17 [ 4641] [DEBUG] main SabreDAV version 3.2.2
13.08.2018 13:38:17 [ 4641] [TRACE]  dav KopanoSyncState->__construct(): Using db sqlite:/var/lib/kopano/kdav/syncstate.db
13.08.2018 13:38:17 [ 4641] [TRACE]  dav KopanoSyncState->__construct(): well db sqlite:/var/lib/kopano/kdav/syncstate.db
13.08.2018 13:38:17 [ 4641] [DEBUG] main KLogger->LogIncoming(): INPUT
GET /kdav/addressbooks/kopano_user/Contacts/ede4362c-9a8d-4e49-8f5f-27cc3c7296ca.vcf HTTP/1.1
Host: my.kopanohost.lan
Accept: text/calendar
Accept-Encoding: gzip, deflate
User-Agent: eM Client/7.1.33101.0
Authorization: Basic REDACTED
...

This GET fails, can not pin point it though (full log of event)

From a far past there where timing issues not able to serve a newly created vCards although this only seemed to happen with mass synchronization.
and the already mentioned differences in UID:urn:uuid ( 081044be-a742-4f4b-a5ba-f6011679d9bb) and cardUri: ede4362c-9a8d-4e49-8f5f-27cc3c7296ca.vcf. My knowledge of this technolgy comes to short to judge if this is a issue

Playing with it i really grown like kdav, what can we do to bring it closer to a test/alpha/… release?

fbartels

@markvnl said in Request for Feedback: New CalDAV and CardDAV implementation for Kopano:

Playing with it i really grown like kdav, what can we do to bring it closer to a test/alpha/… release?

for us the next step is to complete packaging for it (maybe that has already happened, just catching up on holiday emails). After that we have to check what needs to be adapted. One word of advice though, as stated above our focus client for this are the bultin MacOS clients and these are also the only clients that we are actively testing. PRs for other clients are welcome, though.

markVnl

Thanx for the heads up

@fbartels said in Request for Feedback: New CalDAV and CardDAV implementation for Kopano:

One word of advice though, as stated above our focus client for this are the bultin MacOS clients and these are also the only clients that we are actively testing. PRs for other clients are welcome, though.

Understandable,
although I think the open-source community would appreciate if TB {lightning, cardbook} support can be improved in the future.

norman

Thanks for the installation info (https://stash.kopano.io/projects/KC/repos/kdav/browse)!

Here are some suggestions for improvement:

Info: Installation of SQLite / PDO
Info: Installation of mod_rewrite
Info: Usage chusr/chmod on “main kDAV directory”
Info: Web Server has to listen on port 8123 “Listen 8123” (in the Virtual Host example)
“ln -s /usr/share/kopano/php/mapi /var/www/kdav”: mapi directory doesn’t exist on my server, main kDAV directory is selectable in the start of the installation guide
“In your CalDAV client, set the server URL to http://develop.local/calendars/<user>/Calendar/”: No port 8123 in the url
“The DAV_ROOT_URI parameter must match your webserver configuration”: relative path to the DocumentRoot!

fbartels

Hi @norman,

thanks for your feeback. I have adopted the readme a bit.

@norman said in Request for Feedback: New CalDAV and CardDAV implementation for Kopano:

“ln -s /usr/share/kopano/php/mapi /var/www/kdav”: mapi directory doesn’t exist on my server, main kDAV directory is selectable in the start of the installation guide

I think this is probably because of your old Kopano installation, for me this path exists.

norman

One thing I noticed:

In my main calender there ist a folder “Abfall”. In “principals/norman” “/calendars/norman/Abfall/” is the “schedule-default-calendar-URL”.

d:owner	                                /principals/norman/
cal:schedule-outbox-URL	                /calendars/norman/outbox/
cal:schedule-inbox-URL             	/calendars/norman/inbox/
cal:schedule-default-calendar-URL	/calendars/norman/Abfall/

Norman

MichaelAnders2

Hi,

First of all I am very happy that Kopano has decided to implement CardDAV! Now I do not need MS Exchange anymore which I cannot get to work on my Android cell anymore since Oreo, so this is great.

However, I have two points of feedback / bugs which need to be ironed out before I can use it productively:

International characters like german umlaut ä, ö, ß or others like ć are not working for names, addresses, it’s a total mess. They can be stored like that in the webapp, but once transfered to a cellphone it’s not working.

Using export2csv.php file shows that everything is correct, all the umlaute are there. But they appear not to be UTF encoded - my Ubuntu terminal shows a slight difference in how a character is displayed in terminal in case of ASCII or UTF, the later being more “drawn” style and a bit bolder. However, this download works.

To test kdav I used on my Android cellphone:

a) Davdroid - cannot communicate with kdav (but works fine for my Nextcloud Server)

b) contactsync (https://play.google.com/store/apps/details?id=com.vcard.android.free) - works for downloads of contacts. Uploads do do not work well. Looking at the kdav logs I can see that contactsync is sending the VCF data with a “characterset UTF-8” in the HTTP header, but the result is that the kopano database gets data which is not double byte but two single bytes.
Consequences:

The characters are not “ä” anymore but are two digits which are the two single bytes of the unicode character - export2csv.php also shows the same result
The contact in the webapp is a new contact if the name was changed causing a different ID (with the old contact remaining)
Upon a new sync to contactsync the “ä” is now also replaces with the two single bytes
In turn, upon sycing with kdav the two single bytes which contactsync sends are now each replaced with two new single bytes - so in total taking it from 1 character to 2 characters and now to 4 characters. You can imagine what happens upon the new download and upload I guess?

To me it looks like kopano does not interpret the codepage in which the client sent the data (I can clearly see the double byte character in my terminal as it’s a different visual style as normal) - kdav always converts the input in some way which is wrong / not required causing serious problems.

Birthdays
These are stored as UTC timestamps in Kopano. Nothing wrong with that.
What is wrong however is that these birthdays are not sent in an expected format “YYYY-mm-dd” - they also contain a timezone… This may be OK based on e.g. rfc6474 for “deathdate”. But the problem is that using a UTC timestamp usually triggers code to do a conversion into the timezone of the client - this is the correct behaviour. But this requires that the client and server both “understand” what each one does, if one does it differently then there will be problems.
For me, a contact has the bday “1990-09-05”. This is how I see it in the webapp. When I export this contact it is sent with a UTC timestamp of 4th september 11:00pm. This makes the client set the bday as 4th September. In other cases kdav sends it as 0:00am - no idea why, maybe due to DST?
Now one can start arguing “kdav is doing it right, the client should fix it”. But the trivial approach is just to always STORE the bday in UTC+0 and send the bday as YYYY-mm-dd - without ANY conversion at all to the client. Then there can never be any mixup as no client will do any timezone adjustments as it doesn’t know the timezone.
And another argument - Nextcloud, which also uses sabreDav, works perfectly fine also for the birthdays of my contacts.

This should be trivial to reproduce. TIll these issues are solved, I’m using my Nextcloud Server for my contacts.

Fux

I’ve tried but always get:

Fatal error: Uncaught PDOException: SQLSTATE[HY000] [14] unable to open database file in /var/www/kdav/lib/KopanoSyncState.php:45 Stack trace: #0 /var/www/kdav/lib/KopanoSyncState.php(45): PDO->__construct(‘sqlite:/var/lib…’) #1 /var/www/kdav/lib/KopanoDavBackend.php(48): Kopano\DAV\KopanoSyncState->__construct(Object(Kopano\DAV\KLogger), ‘sqlite:/var/lib…’) #2 /var/www/kdav/server.php(54): Kopano\DAV\KopanoDavBackend->__construct(Object(Kopano\DAV\KLogger)) #3 {main} thrown in /var/www/kdav/lib/KopanoSyncState.php on line 45

/var/lib/kopano/kdav is owned by www-data:

drwxr-xr-x 2 www-data www-data 4096 Dez 11 17:28 kdav

php-sqlite3 is installed. System is devuan ascii. Kopano 8.6.81, php7.2

Can anyone help? Thanks in advance!

fbartels

@Fux how are the permissions of the file itself? so /var/lib/kopano/kdav/syncstate.db?

fbartels

@fbartels said in Request for Feedback: New CalDAV and CardDAV implementation for Kopano:

“ln -s /usr/share/kopano/php/mapi /var/www/kdav”: mapi directory doesn’t exist on my server, main kDAV directory is selectable in the start of the installation guide

just a node on that. since we recently removed the mapi classes from php-mapi they are now part of the repo itself and no symlinking is required anymore.

Fux

The file /var/lib/kopano/kdav/syncstate.db doesn’t exist yet. I tried to create (with touch) and set the right permissions - but no success.
I thought it would be created at first time kdav runs?

fbartels

@Fux said in Request for Feedback: New CalDAV and CardDAV implementation for Kopano:

I thought it would be created at first time kdav runs?

Yes, that is indeed what should be the case. The remaining conclusion is that you are running apparmor or something similar and that prevents php from writing into this folder.

Solution: adapt your apparmor config or change the location of the file into /var/www in the config of kdav

Fux

After changing the path to /tmp only for testing) it works and file is created. Must be something wrong with my /var/lib/kopano/kdav.
Although I have no apparmor installed (selinux either) and the permissions look good.
OK, but this seem to bbe a special problem with my system.

Thanks for your help!