Re: GLUG: transfering lots of files

[Wayne Pascoe <wayne@xxxxxxxxxxxxxxxxxxxxx>]

On Thu, 15 Jun 2000, Marc Peiser wrote:

> Hi there,
> 
> I have a staging server with html and grahics that I need to push to 3
> live load balanced web servers about twice a day. A utility like rsync can
> do the job if you only have a few files, but I have 250 000 images and
> html files, and rsync just can't cope. It really takes too long...
> 
> One script which works well is mirror.pl (on freshmeat) but that is run
> from the live web servers and pulls the files from the staging server via
> ftp. I need something similar but must be run from the staging server for
> security reasons. The staging server is on a inside network and the 3
> web servers are on the DMZ. Content must be pushed from the Hidden staging
> server to the live web servers.
> 
> Does anyone have any suggestions on how I can do this. 
> 
> Does anyone do anything similar? I'm keen to hear how:-)
> 
> Regards,
> Marc
> 
> ---
> To unsubscribe from this list: send the line "unsubscribe glug" in
> the body of a message to majordomo@xxxxxxxxxxxx
> 

Write a perl wrapper to rsync. Have the wrapper build a list of files
that have been modified since the last access time. Then feed rsync
that list. Should DRAMATICALLY cut down on the amount of changed files
to sync...

-- 
Wayne Pascoe <wayne@xxxxxxxxxxxxxxxxxxxxx>

Visit http://www.penguinpowered.org.uk for the answer to 
the meaning of life :-)