Forum Index  ViceVersa HOME         FAQ and Knowledge Base

 FAQForum FAQ   SearchSearch Forum  RegisterRegister 
 ProfileProfile   Log inLog in 

Methods to speed up high file count transfers

 
Post new topic   Reply to topic     Forum Index -> Tips, Tricks and More
Author Message
KCHughesIT



Joined: 23 Apr 2013
Posts: 1

PostPosted: Tue Apr 23, 2013 6:20 pm    Post subject: Methods to speed up high file count transfers Reply with quote

I've just started a project to copy a massive amount of data between NAS/SANs and I need to come up with any tips/tricks to increase object transfer rates. I'm dealing with over 400 million files, about 100 million folders and over 70 TB of data. I understand that the shear volume of files will definitely slow this down. I have it set to copy across the /copy:dat level permissions, but not NTFS.

When transferring the the tiny files I'm getting sometimes only around 300-400 KB/sec which equates to about 25 objects/sec, whereas when I hit the occasional large files I can hit 50 MB/sec. Accross my data pool I seem to average a little over 2 MB/sec and 20 objects a second.

Any thoughts/suggestions on how to speed up the data transfer/copy?

Oh, I have the buffer set at 2GB. The machine I'm running it on only has 8GB total and I've seen the memory usage creep up and use nearly 6 (I'm guessing from the stored error/skipped log), so I'm hesitant to double that to the 4GB option.
Back to top
TGRMN Software
Site Admin


Joined: 10 Jan 2005
Posts: 8671

PostPosted: Tue Apr 23, 2013 11:16 pm    Post subject: Reply with quote

Try the option "Use Windows Native Copy" in profile settings -> performance. You could also split the job into multiple profiles and run them concurrently, see: http://www.tgrmn.com/web/kb/item104.htm
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com
Back to top
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Tips, Tricks and More All times are GMT
Page 1 of 1

 
Jump to:  
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group
Copyright © TGRMN Software. TGRMN Software products: