So I have a windows computer that is continuously collecting data for a physics experiment. The data is saved in a text file for which each new event is appended in a new line.
Every-time I need to re-analyse the data I transfer the file to a ubuntu computer by ssh, this is fine at the beginning but when the file passes the 1GB it starts to be really time consuming to transfer the hole file again and again, when only the last lines are different.
To give you an idea the experiments runs for around 3h, transfer takes ~5min per GB, files have typically a max of 3GB and files have a line number in the order of the tens of million.
In case you have a solution that works only for a linux to linux transfer I am also interested, maybe I can try it with cygwin.
2 Answers
rsync is what you need, and it should be available for Windows.
rsync is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the difference between the source files and the existing files in the destination. (taken from man rsync).
Install rsync on your windows system, then from the Linux system do:
rsync --progress ip.of.windows.server:/path/to/file ./It will transfer only the parts that changed. If the file evolves as you say, then each transfer will be very small and fast.
Note that rsync connects using ssh, so if you have ssh already set up and working, rsync should also work fine.
If only the last few lines of the file change each time, transferring the whole file again and again is not the best solution.
There are tools out there that calculate the differences of the file and copy over only the new bits, like rsync (using delta encoding, more info here: )
