Trying to stand up a local mirror for my company, issues with rsync
by BeepCluster8765 from LinuxQuestions.org on (#53RGE)
I am roughly basing what I am trying to do around this article: https://blog.programster.org/centos-...entos-7-mirror
tldr:
Step 1. rsync with a CentOs mirror and a EPEL mirror
Step 2. stand up company internal CentOs mirror with nginx
Step 3. Profit
I have been going through and trying to pick mirrors near me to rsync with, but it seems all of them at one point or another hang on a file and die. I checked with the networking team, the corporate firewall isn't known to be blocking my rsync, and I am able to get a lot of files before it chokes, so that's not it. Sometimes switching mirrors fixes the problem, sometimes it doesn't.
Here is an example of the command I am running to sync EPEL:
sudo rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64 ./CentOSEpelMirror/7/x86_64
which yields the following error:
x86_64/Packages/o/openarena-0.8.8-7.el7.noarch.rpm
335,753,632 83% 26.23MB/s 0:00:02
rsync: connection unexpectedly closed (3910004 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.2]
rsync: connection unexpectedly closed (522056 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.2]
The only thing I can think of is that it's an issue with WSL, since that is how I am rsyncing, but I was wondering if anyone had any additional input for me. I was about to write a python script to parse the mirror list and get me all the rsync urls, and then just write a bash script that loops over all of them whenever rsync chokes and dies.
tldr:
Step 1. rsync with a CentOs mirror and a EPEL mirror
Step 2. stand up company internal CentOs mirror with nginx
Step 3. Profit
I have been going through and trying to pick mirrors near me to rsync with, but it seems all of them at one point or another hang on a file and die. I checked with the networking team, the corporate firewall isn't known to be blocking my rsync, and I am able to get a lot of files before it chokes, so that's not it. Sometimes switching mirrors fixes the problem, sometimes it doesn't.
Here is an example of the command I am running to sync EPEL:
sudo rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64 ./CentOSEpelMirror/7/x86_64
which yields the following error:
x86_64/Packages/o/openarena-0.8.8-7.el7.noarch.rpm
335,753,632 83% 26.23MB/s 0:00:02
rsync: connection unexpectedly closed (3910004 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.2]
rsync: connection unexpectedly closed (522056 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.2]
The only thing I can think of is that it's an issue with WSL, since that is how I am rsyncing, but I was wondering if anyone had any additional input for me. I was about to write a python script to parse the mirror list and get me all the rsync urls, and then just write a bash script that loops over all of them whenever rsync chokes and dies.