Info/steps for create a custom backend
-
Hi all !
Is there somewhere a guide/basic examples to help the creation of a custom backend ?
I have searched in this forum but i’m not lucky with this topic…I want to implement a backend to handle contacts and Calendar events sync.
Thanks
Denis -
Hi Denis,
there aren’t any guides on how-to develop an own backend, just some general suggestions available at http://z-push.org/developers/ and https://wiki.z-hub.io/display/ZP/Development.
If caldav and carddav backends are not suitable for your needs, they at least could be helpful to get you an idea what you need.
Manfred
-
Hi Manfred,
thanks for your feedback.
I have already checked the 2 backends mentioned and implements our own methods.
Calendar seems good but i have issues with the contacts part.
I have tried with some account and the sync on Android/iPhone get stuck on a contact.
Into the log I see:29/05/2018 15:10:24 [ 956] [WARN] [user] Mobile loop detected! Messages sent to the mobile will be restricted to 2 items in order to identify the conflict 29/05/2018 15:10:24 [ 956] [ERROR] [user] Ignored broken message (SyncContact). Reason: '2' Folderid: 'contacts' message id '574'
What should I check ?
Denis
-
Hi Denis,
set the loglevel to WBXML and look if there’s something wrong with the data of the contact. The mobile device seems not to be able to process some of it.
Manfred
-
Hi Manfred,
here WBXML log:29/05/2018 15:56:50 [16427] [DEBUG] -------- Start 29/05/2018 15:56:50 [16427] [DEBUG] cmd='Sync' devType='Android' devId='androidc1390261315' getUser='user' from='<IP>' version='2.4.1-1' method='POST' 29/05/2018 15:56:50 [16427] [DEBUG] Used timezone 'Europe/Rome' 29/05/2018 15:56:50 [16427] [DEBUG] Including backend file: '/var/www/zpush/backend/custombackend/custombackend.php' 29/05/2018 15:56:50 [16427] [DEBUG] Request::ProcessHeaders() ASVersion: 14.1 29/05/2018 15:56:50 [16427] [DEBUG] ZPush::CommandNeedsProvisioning(0): true 29/05/2018 15:56:50 [16427] [DEBUG] FileStateMachine->GetState() read '3598' bytes from file: '/var/www/zpush/state/5/1/androidc1390261315-devicedata' 29/05/2018 15:56:50 [16427] [DEBUG] ASDevice data loaded for user: 'user' 29/05/2018 15:56:50 [16427] [DEBUG] TopCollector(): Initialized mutexid Resource id #46 and memid Resource id #47. 29/05/2018 15:56:50 [16427] [DEBUG] TopCollector initialised with IPC provider 'IpcSharedMemoryProvider' with type '20' 29/05/2018 15:56:50 [16427] [DEBUG] LoopDetection(): Initialized mutexid Resource id #49 and memid Resource id #50. 29/05/2018 15:56:50 [16427] [DEBUG] LoopDetection initialised with IPC provider 'IpcSharedMemoryProvider' with type '1337' 29/05/2018 15:56:50 [16427] [DEBUG] ZPush::HierarchyCommand(0): false 29/05/2018 15:56:50 [16427] [DEBUG] DeviceManager->ProvisioningRequired('943537602') saved device key '943537602': false 29/05/2018 15:56:50 [16427] [DEBUG] DeviceManager->getPolicyName(): determined policy name: 'default' 29/05/2018 15:56:50 [16427] [DEBUG] DeviceManager->getProvisioningPolicies(): loaded 'default' policy. 29/05/2018 15:56:50 [16427] [DEBUG] ZPush::CommandNeedsAuthentication(0): true 29/05/2018 15:56:50 [16427] [DEBUG] custombackend::Logon(user,,password) 29/05/2018 15:56:50 [16427] [DEBUG] custombackend::Logon -> Succesfull ! 29/05/2018 15:56:50 [16427] [DEBUG] ZPush::CommandNeedsPlainInput(0): false 29/05/2018 15:56:50 [16427] [WBXML] I <Synchronize> 29/05/2018 15:56:50 [16427] [WBXML] I <Folders> 29/05/2018 15:56:50 [16427] [WBXML] I <Folder> 29/05/2018 15:56:50 [16427] [WBXML] I <SyncKey> 29/05/2018 15:56:50 [16427] [WBXML] I {b7442e21-2abc-4d70-b2c0-2f9e87c4bd46}6 29/05/2018 15:56:50 [16427] [WBXML] I </SyncKey> 29/05/2018 15:56:50 [16427] [WBXML] I <FolderId> 29/05/2018 15:56:50 [16427] [WBXML] I contacts 29/05/2018 15:56:50 [16427] [WBXML] I </FolderId> 29/05/2018 15:56:50 [16427] [DEBUG] FileStateMachine->GetState() read '942' bytes from file: '/var/www/zpush/state/5/1/androidc1390261315-b7442e21-2abc-4d70-b2c0-2f9e87c4bd46-fd' 29/05/2018 15:56:50 [16427] [DEBUG] SyncParameters->UseCPO('DEFAULT') 29/05/2018 15:56:50 [16427] [DEBUG] DeviceManager->GetBackendIdForFolderId(): no backend-folderid available for 'contacts', returning as is. 29/05/2018 15:56:50 [16427] [DEBUG] SyncCollections->AddCollection(): Folder id 'contacts' : ref. PolicyKey '943537602', ref. Lifetime '480', last sync at '1527602141' 29/05/2018 15:56:50 [16427] [DEBUG] SyncCollections->AddCollection(): Updated reference PolicyKey '943537602', reference Lifetime '480', Last sync at '1527602141' 29/05/2018 15:56:50 [16427] [WBXML] I <DeletesAsMoves/> 29/05/2018 15:56:50 [16427] [WBXML] I <GetChanges/> 29/05/2018 15:56:50 [16427] [WBXML] I <WindowSize> 29/05/2018 15:56:50 [16427] [WBXML] I 10 29/05/2018 15:56:50 [16427] [WBXML] I </WindowSize> 29/05/2018 15:56:50 [16427] [WBXML] I <Options> 29/05/2018 15:56:50 [16427] [WBXML] I <AirSyncBase:BodyPreference> 29/05/2018 15:56:50 [16427] [DEBUG] SyncParameters->UseCPO('DEFAULT') 29/05/2018 15:56:50 [16427] [WBXML] I <AirSyncBase:Type> 29/05/2018 15:56:50 [16427] [WBXML] I 1 29/05/2018 15:56:50 [16427] [WBXML] I </AirSyncBase:Type> 29/05/2018 15:56:50 [16427] [WBXML] I <AirSyncBase:TruncationSize> 29/05/2018 15:56:50 [16427] [WBXML] I 200000 29/05/2018 15:56:50 [16427] [WBXML] I </AirSyncBase:TruncationSize> 29/05/2018 15:56:50 [16427] [WBXML] I </AirSyncBase:BodyPreference> 29/05/2018 15:56:50 [16427] [WBXML] I </Options> 29/05/2018 15:56:50 [16427] [WBXML] I </Folder> 29/05/2018 15:56:50 [16427] [WBXML] I </Folders> 29/05/2018 15:56:50 [16427] [WBXML] I </Synchronize> 29/05/2018 15:56:50 [16427] [DEBUG] HandleSync(): Start Output 29/05/2018 15:56:50 [16427] [DEBUG] Sync->loadStates(): loading states for folder 'contacts' 29/05/2018 15:56:50 [16427] [DEBUG] FileStateMachine->GetState() read '334' bytes from file: '/var/www/zpush/state/5/1/androidc1390261315-b7442e21-2abc-4d70-b2c0-2f9e87c4bd46-6' 29/05/2018 15:56:50 [16427] [DEBUG] ZPush::GetAdditionalSyncFolderStore('contacts'): 'false' 29/05/2018 15:56:50 [16427] [DEBUG] ZPush::GetAdditionalSyncFolderStore('contacts'): 'false' 29/05/2018 15:56:50 [16427] [DEBUG] ExportChangesDiff->InitializeExporter(): Initializing message diff engine. '4' messages in state 29/05/2018 15:56:50 [16427] [DEBUG] custombackend::GetMessageList(contacts,) 29/05/2018 15:58:10 [16427] [DEBUG] custombackend::record count(10071) 29/05/2018 15:58:10 [16427] [ INFO] ExportChangesDiff->InitializeExporter(): Found '10067' changes for 'contacts' 29/05/2018 15:58:10 [16427] [DEBUG] LoopDetection->ProcessLoopDetectionAddStatus: 'contacts' with status 1 29/05/2018 15:58:10 [16427] [DEBUG] WBXMLEncoder->startWBXML() type: vnd.ms-sync.wbxml 29/05/2018 15:58:10 [16427] [DEBUG] Folder type: Contacts 29/05/2018 15:58:10 [16427] [WBXML] O <Synchronize> 29/05/2018 15:58:10 [16427] [WBXML] O <Folders> 29/05/2018 15:58:10 [16427] [WBXML] O <Folder> 29/05/2018 15:58:10 [16427] [WBXML] O <SyncKey> 29/05/2018 15:58:10 [16427] [WBXML] O {b7442e21-2abc-4d70-b2c0-2f9e87c4bd46}7 29/05/2018 15:58:10 [16427] [WBXML] O </SyncKey> 29/05/2018 15:58:10 [16427] [WBXML] O <FolderId> 29/05/2018 15:58:10 [16427] [WBXML] O contacts 29/05/2018 15:58:10 [16427] [WBXML] O </FolderId> 29/05/2018 15:58:10 [16427] [WBXML] O <Status> 29/05/2018 15:58:10 [16427] [WBXML] O 1 29/05/2018 15:58:10 [16427] [WBXML] O </Status> 29/05/2018 15:58:10 [16427] [DEBUG] LoopDetection->Detect(): folderid:'contacts' uuid:'b7442e21-2abc-4d70-b2c0-2f9e87c4bd46' counter:'6' max:'10' queued:'10067' 29/05/2018 15:58:10 [16427] [DEBUG] LoopDetection->Detect(): case 3.3 detected - in loop mode, increase loop counter 29/05/2018 15:58:10 [16427] [DEBUG] LoopDetection->Detect(): case 3.3.1 detected - broken item should be next, attempt to ignore it - id '574' 29/05/2018 15:58:10 [16427] [DEBUG] LoopDetection->Detect(): loop data: loopcount(131), maxCount(16), queued(10066), ignored(574) 29/05/2018 15:58:10 [16427] [WARN] Mobile loop detected! Messages sent to the mobile will be restricted to 2 items in order to identify the conflict 29/05/2018 15:58:10 [16427] [WBXML] O <MoreAvailable/> 29/05/2018 15:58:10 [16427] [DEBUG] custombackend::StatMessage(contacts, 574) 29/05/2018 15:58:10 [16427] [DEBUG] custombackend::StatMessage: 2017-02-08 11:52:10 29/05/2018 15:58:10 [16427] [DEBUG] custombackend::GetMessage(contacts, 574) 29/05/2018 15:58:10 [16427] [ERROR] Ignored broken message (SyncContact). Reason: '2' Folderid: 'contacts' message id '574' 29/05/2018 15:58:10 [16427] [DEBUG] ImportChangesStream->ImportMessageChange('574'): message ignored and requested to be removed from mobile 29/05/2018 15:58:10 [16427] [WBXML] O <Perform> 29/05/2018 15:58:10 [16427] [WBXML] O <Remove> 29/05/2018 15:58:10 [16427] [WBXML] O <ServerEntryId> 29/05/2018 15:58:10 [16427] [WBXML] O 574 29/05/2018 15:58:10 [16427] [WBXML] O </ServerEntryId> 29/05/2018 15:58:10 [16427] [WBXML] O </Remove> 29/05/2018 15:58:10 [16427] [DEBUG] HandleSync(): Exported maxItems of messages: 1 / 10067 29/05/2018 15:58:10 [16427] [WBXML] O </Perform> 29/05/2018 15:58:10 [16427] [DEBUG] HandleSync(): Stopping export as limits of request timeout or available memory are almost reached! 29/05/2018 15:58:10 [16427] [WBXML] O </Folder> 29/05/2018 15:58:10 [16427] [DEBUG] FileStateMachine->SetState() written 416 bytes on file: '/var/www/zpush/state/5/1/androidc1390261315-b7442e21-2abc-4d70-b2c0-2f9e87c4bd46-7' 29/05/2018 15:58:10 [16427] [DEBUG] SyncCollections->SaveCollection(): Data of folder 'contacts' changed 29/05/2018 15:58:10 [16427] [DEBUG] FileStateMachine->SetState() written 942 bytes on file: '/var/www/zpush/state/5/1/androidc1390261315-b7442e21-2abc-4d70-b2c0-2f9e87c4bd46-fd' 29/05/2018 15:58:10 [16427] [WBXML] O </Folders> 29/05/2018 15:58:10 [16427] [WBXML] O </Synchronize> 29/05/2018 15:58:10 [16427] [DEBUG] WBXMLEncoder->endTag() WBXML output completed 29/05/2018 15:58:10 [16427] [WBXML] WBXML-OUT: AwFqAEVcT0sDe2I3NDQyZTIxLTJhYmMtNGQ3MC1iMmMwLTJmOWU4N2M0YmQ0Nn03AAFSA2NvbnRhY3RzAAFOAzEAARRWSU0DNTc0AAEBAQEBAQ== 29/05/2018 15:58:10 [16427] [WBXML] WBXML-IN : AwFqAEVcT0sDe2I3NDQyZTIxLTJhYmMtNGQ3MC1iMmMwLTJmOWU4N2M0YmQ0Nn02AAFSA2NvbnRhY3RzAAEeE1UDMTAAAVcAEUVGAzEAAUcDMjAwMDAwAAEBAQEBAQ== 29/05/2018 15:58:10 [16427] [DEBUG] DeviceManager->Save(): Device data changed 29/05/2018 15:58:10 [16427] [DEBUG] FileStateMachine->SetState() written 3598 bytes on file: '/var/www/zpush/state/5/1/androidc1390261315-devicedata' 29/05/2018 15:58:10 [16427] [DEBUG] DeviceManager->Save(): Device data saved 29/05/2018 15:58:10 [16427] [DEBUG] LoopDetection->ProcessLoopDetectionTerminate() 29/05/2018 15:58:10 [16427] [ INFO] cmd='Sync' memory='27.74 MiB/28.00 MiB' time='82.08s' devType='Android' devId='androidc1390261315' getUser='user' from='<IP>' idle='0s' version='2.4.1-1' method='POST' httpcode='200' 29/05/2018 15:58:10 [16427] [DEBUG] -------- End
PS: PHP version 5.5.9
-
Hi Denis,
the log part you’ve posted is useless. There’s no data for the contact with the id 574, just a request to remove it. Or your backend doesn’t return a proper SyncContact object.
Manfred
-
Here a print_r of SyncContact Object (I have changes some confidential data)
SyncContact Object ( [anniversary] => [assistantname] => [assistnamephonenumber] => [birthday] => [body] => [bodysize] => [bodytruncated] => [business2phonenumber] => [businesscity] => [businesscountry] => [businesspostalcode] => [businessstate] => [businessstreet] => [businessfaxnumber] => [businessphonenumber] => [carphonenumber] => [children] => [companyname] => My Company Name [department] => [email1address] => foo@company.com [email2address] => [email3address] => [fileas] => [firstname] => Denis [home2phonenumber] => [homecity] => [homecountry] => [homepostalcode] => [homestate] => [homestreet] => [homefaxnumber] => [homephonenumber] => [jobtitle] => [lastname] => Lastname [middlename] => [mobilephonenumber] => 3333333333 [officelocation] => [othercity] => [othercountry] => [otherpostalcode] => [otherstate] => [otherstreet] => [pagernumber] => [radiophonenumber] => [spouse] => [suffix] => [title] => [webpage] => [yomicompanyname] => [yomifirstname] => [yomilastname] => [rtf] => [picture] => [categories] => [customerid] => [governmentid] => [imaddress] => [imaddress2] => [imaddress3] => [managername] => [companymainphone] => [accountname] => [nickname] => [mms] => [asbody] => [unsetVars:protected] => Array ( ) [supportsPrivateStripping:protected] => [mapping:protected] => Array ( [POOMCONTACTS:Anniversary] => Array ( [1] => anniversary [3] => 3 [5] => 1 ) [POOMCONTACTS:AssistantName] => Array ( [1] => assistantname [5] => 1 ) [POOMCONTACTS:AssistnamePhoneNumber] => Array ( [1] => assistnamephonenumber [5] => 1 ) [POOMCONTACTS:Birthday] => Array ( [1] => birthday [3] => 3 [5] => 1 ) [POOMCONTACTS:BodySize] => Array ( [1] => bodysize ) [POOMCONTACTS:Business2PhoneNumber] => Array ( [1] => business2phonenumber [5] => 1 ) [POOMCONTACTS:BusinessCity] => Array ( [1] => businesscity [5] => 1 ) [POOMCONTACTS:BusinessCountry] => Array ( [1] => businesscountry [5] => 1 ) [POOMCONTACTS:BusinessPostalCode] => Array ( [1] => businesspostalcode [5] => 1 ) [POOMCONTACTS:BusinessState] => Array ( [1] => businessstate [5] => 1 ) [POOMCONTACTS:BusinessStreet] => Array ( [1] => businessstreet [5] => 1 ) [POOMCONTACTS:BusinessFaxNumber] => Array ( [1] => businessfaxnumber [5] => 1 ) [POOMCONTACTS:BusinessPhoneNumber] => Array ( [1] => businessphonenumber [5] => 1 ) [POOMCONTACTS:CarPhoneNumber] => Array ( [1] => carphonenumber [5] => 1 ) [POOMCONTACTS:Children] => Array ( [1] => children [2] => POOMCONTACTS:Child [5] => 1 ) [POOMCONTACTS:CompanyName] => Array ( [1] => companyname [5] => 1 ) [POOMCONTACTS:Department] => Array ( [1] => department [5] => 1 ) [POOMCONTACTS:Email1Address] => Array ( [1] => email1address [5] => 1 ) [POOMCONTACTS:Email2Address] => Array ( [1] => email2address [5] => 1 ) [POOMCONTACTS:Email3Address] => Array ( [1] => email3address [5] => 1 ) [POOMCONTACTS:FileAs] => Array ( [1] => fileas [5] => 1 ) [POOMCONTACTS:FirstName] => Array ( [1] => firstname [5] => 1 ) [POOMCONTACTS:Home2PhoneNumber] => Array ( [1] => home2phonenumber [5] => 1 ) [POOMCONTACTS:HomeCity] => Array ( [1] => homecity [5] => 1 ) [POOMCONTACTS:HomeCountry] => Array ( [1] => homecountry [5] => 1 ) [POOMCONTACTS:HomePostalCode] => Array ( [1] => homepostalcode [5] => 1 ) [POOMCONTACTS:HomeState] => Array ( [1] => homestate [5] => 1 ) [POOMCONTACTS:HomeStreet] => Array ( [1] => homestreet [5] => 1 ) [POOMCONTACTS:HomeFaxNumber] => Array ( [1] => homefaxnumber [5] => 1 ) [POOMCONTACTS:HomePhoneNumber] => Array ( [1] => homephonenumber [5] => 1 ) [POOMCONTACTS:JobTitle] => Array ( [1] => jobtitle [5] => 1 ) [POOMCONTACTS:LastName] => Array ( [1] => lastname [5] => 1 ) [POOMCONTACTS:MiddleName] => Array ( [1] => middlename [5] => 1 ) [POOMCONTACTS:MobilePhoneNumber] => Array ( [1] => mobilephonenumber [5] => 1 ) [POOMCONTACTS:OfficeLocation] => Array ( [1] => officelocation [5] => 1 ) [POOMCONTACTS:OtherCity] => Array ( [1] => othercity [5] => 1 ) [POOMCONTACTS:OtherCountry] => Array ( [1] => othercountry [5] => 1 ) [POOMCONTACTS:OtherPostalCode] => Array ( [1] => otherpostalcode [5] => 1 ) [POOMCONTACTS:OtherState] => Array ( [1] => otherstate [5] => 1 ) [POOMCONTACTS:OtherStreet] => Array ( [1] => otherstreet [5] => 1 ) [POOMCONTACTS:PagerNumber] => Array ( [1] => pagernumber [5] => 1 ) [POOMCONTACTS:RadioPhoneNumber] => Array ( [1] => radiophonenumber [5] => 1 ) [POOMCONTACTS:Spouse] => Array ( [1] => spouse [5] => 1 ) [POOMCONTACTS:Suffix] => Array ( [1] => suffix [5] => 1 ) [POOMCONTACTS:Title] => Array ( [1] => title [5] => 1 ) [POOMCONTACTS:WebPage] => Array ( [1] => webpage [5] => 1 ) [POOMCONTACTS:YomiCompanyName] => Array ( [1] => yomicompanyname [5] => 1 ) [POOMCONTACTS:YomiFirstName] => Array ( [1] => yomifirstname [5] => 1 ) [POOMCONTACTS:YomiLastName] => Array ( [1] => yomilastname [5] => 1 ) [POOMCONTACTS:Rtf] => Array ( [1] => rtf ) [POOMCONTACTS:Picture] => Array ( [1] => picture [6] => Array ( [15] => 5242880 ) [5] => 1 ) [POOMCONTACTS:Categories] => Array ( [1] => categories [2] => POOMCONTACTS:Category [5] => 1 ) [POOMCONTACTS2:CustomerId] => Array ( [1] => customerid [5] => 1 ) [POOMCONTACTS2:GovernmentId] => Array ( [1] => governmentid [5] => 1 ) [POOMCONTACTS2:IMAddress] => Array ( [1] => imaddress [5] => 1 ) [POOMCONTACTS2:IMAddress2] => Array ( [1] => imaddress2 [5] => 1 ) [POOMCONTACTS2:IMAddress3] => Array ( [1] => imaddress3 [5] => 1 ) [POOMCONTACTS2:ManagerName] => Array ( [1] => managername [5] => 1 ) [POOMCONTACTS2:CompanyMainPhone] => Array ( [1] => companymainphone [5] => 1 ) [POOMCONTACTS2:AccountName] => Array ( [1] => accountname [5] => 1 ) [POOMCONTACTS2:NickName] => Array ( [1] => nickname [5] => 1 ) [POOMCONTACTS2:MMS] => Array ( [1] => mms [5] => 1 ) [AirSyncBase:Body] => Array ( [1] => asbody [3] => SyncBaseBody [5] => 1 ) ) [flags] => [content] => )
Sorry for long code block :(
-
Hi Denis,
it would be more helpful also to see how the SyncObject was converted by the WBXMLEncoder and what data is being sent to the mobile.
The data you’ve posted looks ok, but the device might consider that some property is missing or invalid and not process it correctly (e.g. fileas). Does this contact have a picture?
It might even be something trivial that the device doesn’t accept integer only value for the ServerEntryid. You could tryz-push-admin -a clearloop
and start over to check if that helps. But that all are only guesses.
You could try debuging the device wit logcat (assuming it’s an android device).
Manfred
-
Hi Manfred,
i finally figured out what’s going on.
The back-end are slow in returning the contacts so -> “HandleSync(): Stopping export as limits of request timeout or available memory are almost reached!”
Fixing on our side, now the contacts are synced !Thanks for your support