bbcp

Category: Program Dev-Data Movement

Description

Allows the user to securely and quickly copy a file from source to destination. It is ideally suite to transfer large files with its multistreaming transfer abilites. bbcp is capable of breaking up your transfer into multiple simultaneously transferring streams, thereby transferring data much faster than single-streaming utilities such as scp and sftp.

We are using a modified version of bbcp that incorporates the -r (recursive) option to support tranfer of entire directory trees. Until SLAC incorporates this update into its source code and binaries, be aware that binaries downloaded from SLAC will not support recursive directory-tree transfers.

More information on bbcp can be found by typing bbcp -h on the NICS systems as well as on the bbcp site.

Use

Before you can use the bbcp utility, it must be installed on both the local and remote systems. It is currently available on Nautilus as part of each user's default environment. Several precompiled binaries as well as the source can be downloaded from the Stanford Linear Accelerator Center (SLAC) bbcp download page.

There is a man page at bbcp for detailed documentation.

To transfer the local file /local/path/largefile.tar to the remote system remotesystem as /remote/path/largefile.tar, use the following:

> bbcp -P 2 -V -w 8m -s 16 /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

where

• -P 2 produces progress messages every 2 seconds,
• -V produces verbose output, including detailed transfer-speed statistics,
• -w 8m sets the size of the disk input/output (I/O) buffers, and
• -s 16 sets the number of parallel network streams to 16.

bbcp assumes the remote system's noninteractive environment contains the path to the bbcp utility. This can be determined with the following command:

> ssh remotesystem which bbcp

If this is not the case, the -T option can be used to specify how to start bbcp on the remote system. For example, you could use the following:

> bbcp -P 2 -V -w 8m -s 16 -T 'ssh -x -a -oFallBackToRsh=no %I -l %U %H /remote/path/to/bbcp' /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

Often, during large transfers the connection between the transferring systems is lost. The -a options gives bbcp the ability to pick up where it left off. For example, you could use the following:

> bbcp -k -a /remotesystem/homedir/.bbcp/ -P 2 -V -w 8m -s 16 /local/path/largefile.tar remotesystem:/remote/path/largefile.tar

To transfer an entire directory tree, use the following:

> bbcp -r -P 2 -V -w 8m -s 16 /local/path/* remotesystem:/remote/path

Licensing

Legal Notice
Copyright © 2002, Board of Trustees of the Leland Stanford, Jr. University.
Produced under contract DE-AC03-76-SF00515 with the US Department of Energy.
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

a. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

b. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

c. Neither the name of the Leland Stanford, Jr. University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Support

This package has the following support level : Supported

Available Versions

Version	Available Builds
	intel	pgi	gnu	Other
10.07.26.00.0			sles11_gnu4.3.4 ?
07.11.09.01.0			sles11_gnu4.3.4 ?