Solution: Preserve timestamp while migrating data from S3 to GCS

[rohankalhans]

Note: I tried finding the correct place to post this fix, but unfortunately couldn't find it, therefore here with Tips and Tricks and General Discussion labels. If someone knows please let me know, if this is not the correct place.

Statement

Referring to my own questions "Preserve timestamp while migrating data from S3 to GCS"

We have found the solution to achieve this.

Note: This solution was found by one of the team members we have. Sharing here to let everyone know that's it's possible.

Solution

It's pretty easy to achieve by enabling versioning and the number of copies you want to have in the destination bucket which was the GCS Bucket in our case. Upon starting the job in GCP Storage Transfer Service the original or last modified timestamp of the objects of the AWS S3 bucket will be intact.

Would also require setting the lifecycle on the GCS bucket to check for custom time and if the date is beyond 60 days move those object versions into the archive from standard. Once a transfer job has been created using the available agent pool and the batch-specific manifest file to initiate the transfer with the destination (GCS bucket). Make sure there are no new versions for the objects transferred in S3 and then initiate a new job with active objects in the manifest. Ensure there are no errors in STS jobs

I hope this helps!!!

[Roderick · Accepted solution]

Hey @rohankalhans, yup, this is the perfect area to share this solution. Thanks for sharing!