<?xml version="1.0"?>
<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook V4.1.2//EN"
  "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">

<!-- $Id: gfprep.1.docbook 9754 2015-06-24 00:48:54Z takuya-i $ -->

<refentry id="gfprep.1">

<refentryinfo><date>24 Jun 2015</date></refentryinfo>

<refmeta>
<refentrytitle>gfprep</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo>Gfarm</refmiscinfo>
</refmeta>

<refnamediv id="name">
<refname>gfprep</refname>
<refpurpose>replicate Gfarm files in parallel</refpurpose>
</refnamediv>

<refsynopsisdiv id="synopsis">
<cmdsynopsis sepchar=" ">
  <command moreinfo="none">gfprep</command>
  <arg choice="opt" rep="norepeat">-nqvdpPU</arg>
  <arg choice="opt" rep="norepeat">-mx</arg>
  <arg choice="opt" rep="norepeat">-S <replaceable>source-domainname</replaceable></arg>
  <arg choice="opt" rep="norepeat">-h <replaceable>source-hostfile</replaceable></arg>
  <arg choice="opt" rep="norepeat">-D <replaceable>destination-domainname</replaceable></arg>
  <arg choice="opt" rep="norepeat">-H <replaceable>destination-hostfile</replaceable></arg>
  <arg choice="opt" rep="norepeat">-j <replaceable>num-of-processes</replaceable></arg>
  <arg choice="opt" rep="norepeat">-J <replaceable>num-of-processes</replaceable></arg>
  <arg choice="opt" rep="norepeat">-M <replaceable>total-byte</replaceable></arg>
  <arg choice="opt" rep="norepeat">-z <replaceable>minimum-byte</replaceable></arg>
  <arg choice="opt" rep="norepeat">-Z <replaceable>maximum-byte</replaceable></arg>
  <arg choice="opt" rep="norepeat">-w <replaceable>way-of-scheduling</replaceable></arg>
  <arg choice="opt" rep="norepeat">-W <replaceable>kilobytes-for-threshold</replaceable></arg>
  <arg choice="opt" rep="norepeat">-s <replaceable>KB/s-to-simulate</replaceable></arg>
  <arg choice="opt" rep="norepeat">-F <replaceable>num-for-readahead</replaceable></arg>
  <arg choice="opt" rep="norepeat">-N <replaceable>num-of-replicas</replaceable></arg>
  <arg choice="plain" rep="norepeat"><replaceable>path</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>

<!-- body begins here -->

<refsect1 id="description"><title>DESCRIPTION</title>

<para>
<command moreinfo="none">gfprep</command> creates replicas of Gfarm
files in parallel.
</para>
<para>
If the <parameter moreinfo="none">path</parameter> parameter is a
directory, files under the directory will be replicated recursively.
</para>
<para>
<parameter moreinfo="none">path</parameter> is a Gfarm URL or a path
on a mountpoint of gfarm2fs.
</para>
<para>
A set of source/destination hosts can be specified by a domain name
and/or a hostlist file.  When both a domain name and a hostlist file
are specified, a set of hosts is determined by both conditions.  When
a set of source hosts is specified, only files stored on the source
hosts are replicated.
</para>
<para>
Idler filesystem nodes will be selected for source or destination.
The maximum number of parallel to send or receive per host is the
number of CPUs.  Please see the manual page
of <command moreinfo="none">gfhost</command> for information about the
number of CPUs.
</para>
<para>
<command moreinfo="none">gfprep</command> can also read the directory and get
information of the entries in parallel.
</para>
</refsect1>

<refsect1 id="options"><title>GFPREP OPTIONS</title>
<para>
These are the options only for <command moreinfo="none">gfprep</command>.
</para>
<variablelist>

<varlistentry>
<term><option>-N</option> <parameter moreinfo="none">num-of-replicas</parameter></term>
<listitem>
<para>
Specifies the required number of file replicas.  If there are a
sufficient number of replicas in the destination hosts, neither any
copying nor any removal will be done.
</para>
<para>
If this option is not specified, -N 1 is assumed.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-x</option></term>
<listitem>
<para>
Removes extra file replicas when a set of destination nodes has more
file replicas than specified by -N option.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-m</option></term>
<listitem>
<para>
Migrates file replicas within a set of source nodes to destination
nodes.
</para>
<para>
This option requires -S or -h or -D or -H options.
</para>
<para>
This option and -N option are mutually exclusive.
</para>
</listitem>
</varlistentry>

</variablelist>
</refsect1>

<refsect1 id="common-options"><title>COMMON OPTIONS</title>
<para>
<command moreinfo="none">gfprep</command> has the same options
with <command moreinfo="none">gfpcopy</command>.
</para>

<variablelist>
<varlistentry>
<term><option>-S</option> <parameter moreinfo="none">source-domainname</parameter></term>
<listitem>
<para>
Specifies a domain name to select files.
Only files which have the real file on the domain will be sended.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-h</option> <parameter moreinfo="none">source-hostfile</parameter></term>
<listitem>
<para>
Specifies a file of which describes hostnames of file system nodes to
replicate or copy files.  Only files which have the real file on the
host names will be selected.
The <parameter moreinfo="none">source-hostfile</parameter> consists of
a filesystem node name on each line.
</para>
<para>
If ``-'' is specified, standard input is used to read the host list.</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-L</option></term>
<listitem>
<para>
Filesystem nodes selected by -S or -h are only used to send for the
source.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-D</option> <parameter moreinfo="none">destination-domainname</parameter></term>
<listitem>
<para>
Specifies the domain name for destination.
</para>
<para>
If neither this nor the <option>-H</option> option is specified,
replicas may be copied to any available host.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-H</option> <parameter moreinfo="none">destination-hostfile</parameter></term>
<listitem>
<para>

Specifies a file which describes hostnames for destination.
The <parameter moreinfo="none">destination-hostfile</parameter>
consists of a filesystem node name on each line.
</para>
<para>
If ``-'' is specified, standard input is used to read the host list.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-j</option> <parameter moreinfo="none">num-of-processes</parameter></term>
<listitem>
<para>
Specifies the maximum number of processes to create file replicas (or
copy files) simultaneously.
</para>
<para>
The default value is the parameter of client_parallel_copy in gfarm2.conf.
(see man gfarm2.conf)
</para>
<para>
The maximum number of process per filesystem node for source or destination
is the number of CPUs (see man <command moreinfo="none">gfhost</command>).
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-J</option> <parameter moreinfo="none">num-of-processes</parameter></term>
<listitem>
<para>
Specifies the number of processes to read directories in parallel.
</para>
<para>
The default value is 8.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-M</option> <parameter moreinfo="none">total-byte</parameter></term>
<listitem>
<para>
Specifies the bytes to limit total copied size.
</para>
<para>
The default value is unlimited.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-z</option> <parameter moreinfo="none">minimum-byte</parameter></term>
<listitem>
<para>
Specifies the bytes to limit minimum size of a file.
</para>
<para>
The default value is unlimited.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-Z</option> <parameter moreinfo="none">maximum-byte</parameter></term>
<listitem>
<para>
Specifies the bytes to limit maximum size of a file.
</para>
<para>
The default value is unlimited.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-w</option> <parameter moreinfo="none">way-of-scheduling</parameter></term>
<listitem>
<para>
Specifies the name of scheduling way.
``noplan'' replicates/copies while files are found.
``greedy'' schedules greedily the order of replication/copy beforehand.
</para>
<para>
The default behavior is ``noplan''.
</para>
<para>
``greedy'' scheduling cannot use the -N option or -m option.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-W</option> <parameter moreinfo="none">kibibytes</parameter></term>
<listitem>
<para>
Specifies a threshold size/cost(KiB) to flat costs of Connections.
A Connection means a scheduling information (assgined files) per a
child-process.
</para>
<para>
This option is effective with -w greedy.
</para>
<para>
The default value is 50*1024 KiB (50MiB).
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-U</option></term>
<listitem>
<para>
Disables checking the available disk space of the selected node every
time.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-F</option> <parameter moreinfo="none">num-of-dirents</parameter></term>
<listitem>
<para>
Specifies the number of entries to perform the directory readahead.
</para>
<para>
The default value is 10000.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-s</option> <parameter moreinfo="none">kilobytes-per-sec.</parameter></term>
<listitem>
<para>
Specifies a throughput(KB/s) to simulate the replication/copy, and
does nothing (gets file information only).
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-n</option></term>
<listitem>
<para>
Does nothing.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-p</option></term>
<listitem>
<para>
Reports the total performance and various results.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-P</option></term>
<listitem>
<para>
Reports the each and total performance and various results.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-q</option></term>
<listitem>
<para>
Suppresses non-error messages.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-v</option></term>
<listitem>
<para>Displays verbose output.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-d</option></term>
<listitem>
<para>Displays debug output.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-?</option></term>
<listitem>
<para>Displays a list of command options.</para>
</listitem>
</varlistentry>

</variablelist>
</refsect1>

<refsect1 id="examples"><title>EXAMPLES</title>
<para>
To replicate files under the directory recursively.
</para>
<literallayout format="linespecific" class="normal">
$ gfprep -N 3 gfarm:///dir
</literallayout>
<literallayout format="linespecific" class="normal">
$ cd /mnt/gfarm2fs
$ gfprep -N 3 dir
</literallayout>

<para>
To replicate a file or reduce surplus replicas of the file.
</para>
<literallayout format="linespecific" class="normal">
$ gfprep -N 3 -x gfarm:///dir/file
</literallayout>

<para>
To migrate replicas from gfsd1.example.com to other nodes.
</para>
<literallayout format="linespecific" class="normal">
$ gfprep -m -S gfsd1.example.com gfarm:///
</literallayout>
</refsect1>

<refsect1 id="notes"><title>NOTES</title>
<para>
When the -L or -m option is not specified, a source node is not always
selected within a set of specified source filesystem nodes.
</para>
<para>
<command moreinfo="none">gfprep</command> command should be executed
at the same host of gfmd.  If the gfmd
and <command moreinfo="none">gfprep</command> are far apart,
the <command moreinfo="none">gfprep</command> operation is slow.
Specifying a large value by -j or -J options may be effective in such
a case.
</para>
</refsect1>

<refsect1 id="see-also"><title>SEE ALSO</title>
<para>
  <citerefentry>
  <refentrytitle>gfrep</refentrytitle><manvolnum>1</manvolnum>
  </citerefentry>,
  <citerefentry>
  <refentrytitle>gfpcopy</refentrytitle><manvolnum>1</manvolnum>
  </citerefentry>,
  <citerefentry>
  <refentrytitle>gfhost</refentrytitle><manvolnum>1</manvolnum>
  </citerefentry>,
  <citerefentry>
  <refentrytitle>gfarm2.conf</refentrytitle><manvolnum>5</manvolnum>
  </citerefentry>
</para>
</refsect1>

</refentry>
