Problem with moved Attachments from Local Store to s3 Storage

jengelh

You can enable log_level=6 in kopano-server.cfg to get more S3 messages.

jimismith

Hi buddy, you are performing via which logs?

Regards,
Jimi Smith

NeGrusti

looking at this code:
https://github.com/Kopano-dev/kopano-core/blob/master/provider/libserver/ECS3Attachment.cpp
do I understand correct that s3 storage adapter does not support gzip compressed attachments at all? If not, why not?

jengelh

The answer is quite boring: No one asked for it yet.

NeGrusti

@jengelh
it’s quite crazy to hear, as people are obviously moving from file storage to s3 storage and have to unpack all their gzips, and considering the usual compression rate of the attachments this probably increases their S3 bills tenfold… In my case from 1TB to ~10TB

jengelh

S3 is terrible. More specifically,

AWS-S3 is terrible: The more remote you go, the higher the access latency, and this happens to add up quickly. When I last dealt with an AWS case in '18, I observed access latencies in the order of 100 ms per attachment.
I see similar latency when using e.g. sshfs with a private dedicated server, so the AWS observation is not a far-fetched phantasm. But, sshfs has a cache, so on second access, it is just 30ms (of which 22 are for the Internet line itself). To the best of my rememberance, AWS exhibited the big latency on every access, which is why this was so annoying for the customers (of course, someone sent 20 attachments in one mail, so that’s that).
S3-the-API is terrible: it is lacking the atomic rename operation, so it cannot support concurrent access by multiple kopano-servers and you cannot get the benefits of that (cross-server deduplication). Many other distributed filesystems honor it, based on what their documentation suggests, some of which can be used with just the files_v2 driver.
Deduplication (even within just one server) is a bigger win on the long run than compression alone. Ideally both are combined—perhaps in a future files_v3.

NeGrusti

In my case the Kopano server itself is on AWS, latency is acceptable for our customers. Only a single server so concurrent access is not an issue. Deduplication as in files_v2 can be easily added to S3 backend as well.

NeGrusti

@jengelh - S3 latencies and throughput from our Kopano server instance attached, tested with
https://github.com/dvassallo/s3-benchmark

jengelh

@NeGrusti said in Problem with moved Attachments from Local Store to s3 Storage:

Deduplication as in files_v2 can be easily added to S3 backend as well.

In a way that is race-free with respect to, and does not DoS other kopano-server in a multiserver? Try me.

In my case the Kopano server itself is on AWS

Well ok, that’s not a case of “very remote”, so it will pan out. Though I presume it will not be much different, one should measure at the ECS3Attachment.cpp location rather than with s3-benchmark.

scheini72

Hi @negrusti
Do you have an an anser to your question reagarding Gzip?
I’m tryiing to move as well to an S3 compatible storage (wasabi) and I’m wondeing, if I have to unzip first all attachments, How to doit?