Comments inline: On 8/27/2010 10:56 AM, Mike Miller wrote: > On Fri, 27 Aug 2010, Mr. MailingLists wrote: > >> Also, in my experience, compression destroys file transfer rates over >> OpenSSH (on my LAN). >> >>> $ scp chickenclucker:~/175MBfile . >>> 175MBfile 100% 175MB 29.1MB/s 00:06 >>> >>> $ scp -C chickenclucker:~/175MBfile . >>> 175MBfile 100% 175MB 4.6MB/s 00:38 > > > The first issue is that your network is super fast. You are sending about > 230 Mbps uncompressed. You have to be able to compress and uncompress at > a faster rate than that. You don't want to use compression when the > network is unbelievably fast, but you may want to use it for slower > networks, though only for certain file types. Trying to compress something that cannot be compressed also uses CPU (a lot) time even though you get nothing in return (but possibly a larger file). > > The next issue is the file type. What kind of file were you transferring? dd if=/dev/random of=175MBFile bs=1024k count=175 > The thing is, many common files are already compressed. This is true of > most image, audio and video formats. If you try to compress a file that > is already compressed, it will get larger, not smaller. This is because > there is some overhead in compression, and random binary streams > (essentially what you get in an already-compressed file) cannot be > compressed. Exactly, it really depends on how compressible the data is. > > Try making a big text file with a lot of redundancy in it, something that > will compress a lot. For example: > > mkdir junk ; cd junk > for i in $(seq 1000); do echo "0000000000000000000000000000000000000000000" >> foo.txt ; done > for i in $(seq 1000); do cat foo.txt >> bar.txt ; done > for i in $(seq 10); do cat bar.txt >> baz.txt ; done > > rm foo.txt bar.txt > > $ wc -c baz.txt > 440000000 baz.txt > > $ gzip -c baz.txt | wc -c > 1279822 > > > In other words, baz.txt will compresses from 440 MB down to 1.3 MB. Try > making that baz.txt file and see what its transfer speed looks like with > and without compression. On my computer, to compress the file takes about > 3.2 seconds. If your CPUs on both ends are at least that fast, then > compression probably will be faster for that file than uncompressed > transfer. If compression doesn't help you there, it will never help on > your LAN, but it still will help for compressable files across a slower > network, especially when both machines have fast CPUs and are available > (not overloaded with other jobs). My test case is being performed on the same machine transmitting to a jail. Here are my findings: dd if=/dev/random of=175MBFile bs=1024k count=175 scp 175MBFile chickenclucker:~/ 175MBFile 100% 175MB 35.0MB/s 00:05 scp -C 175MBFile chickenclucker:~/ 175MBFile 100% 175MB 9.7MB/s 00:18 time gzip -c 175MBFile > 175MBFile.gz 12.513s 183500800 Aug 27 11:58 175MBFile 183556808 Aug 27 11:58 175MBFile.gz dd if=/dev/zero of=175MBFile bs=1024k count=175 scp 175MBFile chickenclucker:~/ 175MBFile 100% 175MB 35.0MB/s 00:05 scp -C 175MBFile chickenclucker:~/ 175MBFile 100% 175MB 35.0MB/s 00:05 time gzip -c 175MBFile > 175MBFile.gz 0m2.552s 183500800 Aug 27 12:03 175MBFile 178393 Aug 27 12:03 175MBFile.gz Interesting results and I learned something new today (I friggan love when that happens!). > > A few days ago I was transferring some files from the U to Scripps in San > Diego. The files were highly compressable, but transfer was very slow. I > think it was because the CPU on the other end was dealing with a lot of > jobs, so it wasn't readily available for uncompressing the incoming data > stream, and that slowed it way down. > > Mike >