Hey there,

I was hoping someone may have had experience compressing WCF (using http/s bindings).

What would everyone recommend? (I'm assuming net.tcp can't be used on server/client).

SharpZipLib?

XML?

Serializing EntityCollection?

Performance?

I hope to try it out tomorrow. We are backing up DB (>500M of data) using LLBL.

Cheers Andrew

May I first ask why you're using WCF to backup a lot of data? XML serialization isn't the fastest. As it's backup, both sides are .net and know llblgen pro entities, so I'd use remoting and fast serialization. You can then add a zip component in the chain so data is zipped on the fly as well for even smaller datablocks.

Otis wrote:

May I first ask why you're using WCF to backup a lot of data? XML serialization isn't the fastest. As it's backup, both sides are .net and know llblgen pro entities, so I'd use remoting and fast serialization. You can then add a zip component in the chain so data is zipped on the fly as well for even smaller datablocks.

Well from my knowledge, the WCF services work with Binary/MTOM/Text and can be compressed...

http://blogs.msdn.com/yassers/archive/2006/01/21/515887.aspx http://msdn2.microsoft.com/en-us/library/ms751458.aspx http://weblogs.asp.net/cibrax/archive/2007/08/29/sending-attachments-with-wcf.aspx

There is a chunking channel and a compression example in the windows sdk that look like they would do a job as good as remoting....

the net.tcp binding looks like it supports binary serialization out of the box, was thinking this was the way forward (long term)?

It looks like Walaa recommends WCF? http://www.llblgen.com/tinyforum/Messages.aspx?ThreadID=12652 http://geekswithblogs.net/danielcarbajal/archive/2006/07/05/84160.aspx http://jeffbarnes.net/portal/blogs/jeff_barnes/archive/2007/02/22/wcf-enable-binary-encoding-over-http.aspx

I thought the best way would be to serialize llblgen entity collection (fastest way possible), then zip it, then pass it as object to WCF. In WCF, I can then choose to use binary/MTOM with chunks given links above?!

How would you recommend doing it specifically/differently?

dioptre wrote:

I thought the best way would be to serialize llblgen entity collection (fastest way possible), then zip it, then pass it as object to WCF. In WCF, I can then choose to use binary/MTOM with chunks given links above?!

How would you recommend doing it specifically/differently?

I would do it the same way.

dioptre wrote:

Otis wrote:

May I first ask why you're using WCF to backup a lot of data? XML serialization isn't the fastest. As it's backup, both sides are .net and know llblgen pro entities, so I'd use remoting and fast serialization. You can then add a zip component in the chain so data is zipped on the fly as well for even smaller datablocks.

Well from my knowledge, the WCF services work with Binary/MTOM/Text and can be compressed...

Sure, but the data to compress/send as binary/ is ALWAYS first serialized to XML, so entities are always first serialized to XML, and then binary encoded.

There is a chunking channel and a compression example in the windows sdk that look like they would do a job as good as remoting....

Though our remoting serialization is a tad faster and way more compact than standard

the net.tcp binding looks like it supports binary serialization out of the box, was thinking this was the way forward (long term)?

No, it supports binary encoding.

I thought the best way would be to serialize llblgen entity collection (fastest way possible), then zip it, then pass it as object to WCF. In WCF, I can then choose to use binary/MTOM with chunks given links above?!

How would you recommend doing it specifically/differently?

Use fast serialization (plain remoting with our fast serialization enabled) and add a zip component to the serializer helper (see manual) to compress it even further.

You're looking at a lot of data, a lot of entities, and with WCF these are all going to be serialized to XML and then the XML is post-processed into binary data, etc. but is that really what you NEED for this particular problem? I don't think it's the best solution.

Just try to serialize a lot of entities through WCF with binary encoding and then try to serialize a lot of entities through remoting with fast serialization enabled (see manual). 10 to 1 the fast serialization is faster and more compact.

dioptre wrote:

I thought the best way would be to serialize llblgen entity collection (fastest way possible), then zip it, then pass it as object to WCF

So it sonds like we are agreeing?

I've gone for serializing the entitycollection to a stream using a binaryformatter (utilising optimisation and compression), and sent that through the WCF methods. Looks like it works heaps faster!

Anything better than this? If I find anything I'll let everyone know.

Cheers Andrew

PS I have done some initial tests and this method is about 4 times faster than using XML de/serialization. (without compression)

The biggest bottleneck in the whole pipeline is serialization/deserialization. So if that's done with the fastest way possible, I think the biggest hurdle is taken