Forum Index  ViceVersa HOME         FAQ and Knowledge Base

 FAQForum FAQ   SearchSearch Forum  RegisterRegister 
 ProfileProfile   Log inLog in 

ViceVersa 3 copy speed to AWS vs S3Express

 
Post new topic   Reply to topic     Forum Index -> Comments / Feedback
Author Message
Alice



Joined: 29 Jan 2015
Posts: 205

PostPosted: Sun Mar 15, 2020 7:53 pm    Post subject: ViceVersa 3 copy speed to AWS vs S3Express Reply with quote

We are new customers to AWS and the first task I have is to copy quite a few million files there from a local IBM Storwize V7000 Unified device.

The initial data set to be 'archived' (ie copied to AWS then deleted at source) is 22,000,000 files (11TB), this is a mix of 'user generated' files so .docs / .pdfs / .msgs all quite small in size.

I've set up lots of VV Profiles to copy sections of the data at approx. 250,000 files per Profile to AWS, but as you would expect its taking ages. So measure this process in weeks not days.

For some other unrelated Enterprise Vault data I was able to use .zip (Store option) so no compression for speed at Source, then I VV'ed the very large files across to AWS, then unzipped them. But I can't do that with the 22,000,000 as I don't really have the disk space needed locally to .zip them or the time !

Would your S3Express product 'push' the data more quickly from Source to AWS than VV 3.0 ? If so by a factor ofÖ..roughly ?
Back to top
TGRMN Software
Site Admin


Joined: 10 Jan 2005
Posts: 8229

PostPosted: Mon Mar 16, 2020 12:26 am    Post subject: Reply with quote

Hi, do you have a SMB connection to AWS which you are using ViceVersa PRO on? Is it to S3 strorage or a server on AWS? thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com
Back to top
Alice



Joined: 29 Jan 2015
Posts: 205

PostPosted: Mon Mar 16, 2020 4:35 pm    Post subject: Reply with quote

We have an S3 Storage Gateway VM Appliance with 1TB cache and two Buckets. I am using VV to copy data to one of its Shares... For medium/large files itís ok speed, for smaller files itís dire in performance terms. Only 3,000,000 files copied in an entire week !

I know we can use a local Snowball, but that's not available at the mo.

S3express could improve the transfer speeds compared to VV ?
Back to top
TGRMN Software
Site Admin


Joined: 10 Jan 2005
Posts: 8229

PostPosted: Tue Mar 17, 2020 12:18 am    Post subject: Reply with quote

S3Express can upload very very fast (max speed achievable on S3 with multiple threads and part/restartable uploads) ... but it uses only the S3 protocol over https, no shares or SMB.

Instead ViceVersa only works over network shares and SMB.
If you have not done this already, try increasing the copy buffer to max size in the ViceVersa profile settings -> performance. It should help a bit.

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com
Back to top
Alice



Joined: 29 Jan 2015
Posts: 205

PostPosted: Sat Apr 04, 2020 8:34 am    Post subject: Reply with quote

Can S3Express be used to push data to a local Snowball device ?
Back to top
TGRMN Software
Site Admin


Joined: 10 Jan 2005
Posts: 8229

PostPosted: Sun Apr 05, 2020 11:35 pm    Post subject: Reply with quote

If it supports the S3 protocol, then yes... never tried sorry Sad
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com
Back to top
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Comments / Feedback All times are GMT
Page 1 of 1

 
Jump to:  
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © phpBB Group
Copyright © TGRMN Software. TGRMN Software products: