LLNL

HTAR Manual

HTAR Manual

Introduction

HTAR's Role

HTAR ("HPSS Tape Archiver") is an LC-designed TAR-like utility program that makes TAR-compatible archive (library) files but with High Performance Storage System (HPSS) support and enhanced archive-management features. HTAR's enhancements include its ability to:

The TAR and HTAR Compared section compares traditional UNIX TAR with LC's enhanced HTAR feature by feature to reveal the value of this added tool, but in general HTAR maintains full output compatibility with the POSIX 1003.1 standard TAR format while successfully archiving hundreds or even thousands of incoming files and handling files of greatly mixed sizes or types. In most cases, creating a stored archive directly using HTAR will be much faster than either creating a local TAR file and then copying it to storage with HSI or piping TAR output into an FTP connection to HPSS or into HSI.

HTAR can store archive-member files as large as 68 GB. There is no maximum size for a whole HTAR archive other than site-imposed restrictions (about 10.7 TB) or amount of space available. HTAR makes two copies of each stored archive by default only for files up to 256 MB; you can request dual-copy storage (for extra safety) of a mission critical archive of any size by using HTAR's -Y dualcopy option. (The HTAR -Y dualcopy option can also be specified to pick the class of service (COS) for either the archive file, the index file, or both, or to specify automatic COS selection.) NFT's DIR -h command-with-option combination reveals the COS value of stored files (in output column 3). FTP and HSI can reveal COS, too.

Because HTAR combines two features usually separate (file bundling and file storage), the How HTAR Works section explains the relationship among the three files (the archive file, the index file, and the consistency file) that HTAR uses. See How to Run HTAR for information about how to run HTAR, common error conditions, known limitations (with work-arounds), and HTAR environment variables. The HTAR Options section describes the function of each HTAR option (distinguishing the required action options from the control options). HTAR Examples gives annotated step-by-step examples of how to use HTAR to handle common file-archiving tasks and problems.

HTAR users may also benefit from familiarity with another LC-developed specialty tool that provides nonstandard file-handling and file-transfer features linked to file storage, namely NFT (see the NFT Reference Manual for details). HTAR itself does not, however, use NFT's "persistence" mechanisms to manage file-storage delays. For a general introduction to LC storage tools and techniques, see EZSTORAGE. Hopper, LC's graphical file-handling interface, can also serve as a front end for HTAR in situations where Hopper's scalability limits are not too severe. Also of interest to the HTAR user is the HSI utility, which provides a user-friendly UNIX-style interface to storage. HSI can recursively store, retrieve, and list entire trees with a single command. Consult the HSI manual for details.

For help, contact the LC Hotline at 925-422-4531 (OCF e-mail: lc-hotline@llnl.gov, SCF e-mail: lc-hotline@pop.llnl.gov).


How HTAR Works

HTAR makes an archive (or library) file in the standard POSIX 1003.1 TAR format, which allows TAR to open any HTAR archive file. Because HTAR offers more services than ordinary TAR, it needs extra internal machinery to support those services, some of which reveals itself in HTAR status messages or command responses. This section briefly explains how HTAR makes an archive file and the role that several support files play in that process.


TAR and HTAR Compared

HTAR is specifically designed to efficiently store a set of files together in HPSS or get them back (not merely to make an archive file and leave it).The table below compares the TAR features and effects with those of HTAR.

Feature TAR HTAR
Can create an archive file without storing it?Yes (the default) With -E
Can create an archive file without using local disk space?NoYes (the default)
Can store an archive file while creating it?No, needs FTPYes (the default)
Can read any TAR archive file?Yes (the default)Yes, if -X first
Can read any HTAR archive file?YesYes (the default)
Can extract just one file from a stored archive?NoYes (the default)
Can add file(s) to an existing archive?YesNo
Default target if no archive specified?Yes (tape)No, -f required
Treats input directories recursively?YesYes
Preserves original permissions on files?No (uses UMASK)Yes, with -p
Depends on HPSS availability to work?NoYes
Archive duplicated automatically in storage?NoOnly with -Y dualcopy
Builds and needs an external index file?NoYes
Builds and needs a consistency check file?NoYes
Overwrites existing files without warning?YesYes (-w disables)
Can use standard input or output?Yes (with -f -)Yes (with -L, -O)
Order of options important?SomewhatSomewhat
Table of contents (-t) reveals what?File names onlyFile names and properties
Can create and verify CRC checksums of member files?NoYes
Can verify contents of a newly created archive as part of creation operation?NoYes

How to Use HTAR

HTAR Execute Line

To run HTAR you must log on to an LC production machine where HTAR has been installed at a time when the storage system (HPSS) is up and available to users. The HTAR execute line has the general form

htar action archive [options] [filelist]

and the specific form

htar -c|t|x|D|K|U|X -f archivename [-BdEhHILmMoOpPSTvVwY] [flist]

where exactly one action and the archivename are always required, while the control options and (except when using -c) the filelist (or flist) can be omitted (and the options can share a hyphen flag with the action for convenience). See the HTAR Options section for details.

Syntax Issues

Traditional TAR is such an old utility that syntax differences have evolved under different versions of the UNIX operating system. Linux at LC offers some different TAR options and uses some of the same options (such as -L) for different purposes. (Refer to the comparison of TAR and HTAR features.) Generally, HTAR syntax follows the more restrictive implementations of TAR. Thus, with HTAR:

Defaults

Directories. By default, HTAR creates an archive by copying files from the online directory where you run it into a file in your storage (HPSS) home directory, and it extracts files by reversing that process. You must always specify the name of the archive file on which HTAR operates (there is never a default archive). In its reports, HTAR appends slash (/) to each directory name listed.

File Names. Once you name the archive, HTAR calls the corresponding external index file archivename.idx by default and stores it in the same HPSS directory as the archive (by default). HTAR's -I option lets you specify a nondefault name or location for the index file. The HTAR consistency file's name begins with with /var/tmp/uname/HTAR, where uname is your login name on the machine where you run HTAR.

Class of Service (COS). By default, HTAR stores two copies of each archive in HPSS for files up to 256 MB; you can request dual-copy storage of any mission critical HTAR archive, regardless of its size, by using the -Y dualcopy option on HTAR's execute line. NFT's command DIR -h reports the COS for stored files (in output column 3). FTP and HSI also report COS.


HTAR Error Conditions

HTAR prefixes all ordinary messages with the string 'HTAR:', but it prefixes nonfatal errors with 'INFO:' and fatal errors with 'ERROR:'. Unexpected situations are usually flagged with a '###WARNING' prefix. The most common error conditions and HTAR's responses to them are summarized here to help you troubleshoot:

Storage (HPSS) is down. When HPSS is unavailable to users, no stored archive can be read or written. HTAR returns a message of this form and ends. (There is no persistence as with NFT.)

hpssex_OpenConnection: unable to obtain remote site info
result = -5000, errno = 0
Unable to setup communication to HPSS. Exiting...

Specified archive directory does not exist. If -f specifies a child directory (of your storage home directory) that you have not previously created (with FTP's or HSI's mkdir option), when you attempt to create an archive in a nonexistent (sub)directory, HTAR responds:

***Error -2 on hpss_Open (create) for archivename

When you attempt to extract files from an archive in a nonexistent (sub)directory, HTAR replaces the first line of this error message with:

***Fatal error opening index file archivename.idx

Specified archive file does not exist. If -f specifies an archive file that does not exist (perhaps because you deleted it or mistyped its name), HTAR responds:

[FATAL] no such HPSS archive file: archivename

Specified index file does not exist. If you try to list (-t) or extract (-x) files from an actual HTAR archive whose corresponding external index file (archivename.idx) has been deleted or moved, HTAR pinpoints the problem only by reporting the missing index name:

No such file: archivename.idx

You can work around the missing index by using HTAR's -X option to rebuild the index while the archive remains stored, or you can retrieve the whole archive from storage with FTP or HSI and then open it with TAR.

HTAR's filelist omitted. If you try to create (-c) an archive without specifying a filelist (or without using a filelist replacement such as -L), HTAR connects to HPSS but quickly ends with the message

Refusing to create empty archive.

If you try to list (-t) or extract (-x) without specifying a filelist, HTAR defaults to processing all files in the archive.

HTAR run with no options. Because HTAR requires one action (-c|t|x|X|K) and a specified archive file (-f) to run, executing the program with nothing else on the execute line yields a terse syntax summary. There is no prompt for input, and HTAR terminates.

Command line too long for shell. The easy way to build an HTAR archive of very many like-named files is to specify them indirectly by using a UNIX metacharacter (filter, wild card) such as * (to match any string) or ? (to match any single character). But if the selected file set has thousands of members, the list of input names that the UNIX shell generates by expanding such an "ambiguous file reference" may grow too long to handle. See the Limitations and Restrictions section below for several ways to work around such excessively long command lines when building large archives with HTAR.

Wild cards (metacharacters) used for retrieval. HTAR allows * only to create an archive, not to retrieve files from one ("no match" is the usual, but not the only possible, error message). See the Retrieving Files


HTAR Limitations and Restrictions

The current version of HTAR has the following known limitations or usage restrictions.

I/O

You can redirect any HTAR output into a file (with >) for separate postprocessing (see the Retrieving Files section for one helpful application of this), but HTAR normally does not read from or write to UNIX pipes (standard input, standard output). Two HTAR control options, however, let you enable the use of pipes if you need them:

Metacharacters

HTAR leaves all processing of metacharacters (filters or wild cards, such as *) to the shell. This means that when you create an HTAR archive you can use * to select from among your local files to store, but when you retrieve specific files from within an already stored archive you cannot use * to select from among the stored files to retrieve. See the Retrieving Files example for details on this limitation and a few suggested ways to work around it. Another side effect of this approach to metacharacters is that C shell (csh) users must type the three-character string -\? (instead of -?) to display HTAR's help message.

Updates

No options exist to update (replace), remove (delete), or append to individual files that HTAR has already archived. You must replace (create again) an entire archive to alter the member files within it.

Name Length

To comply with POSIX 1003.1 standards regarding TAR-file input names, the longest input file name of the form prefix/name that HTAR can accept has 154 characters in the prefix and 99 characters in the name. Link names likewise cannot exceed 99 characters.

File Size

The maximum size of a single member file within an HTAR archive is 68 GB. HTAR's site-imposed maximum size for an archive file is about 10.7 TB, and local disk space (when using -E) or storage space might externally limit an archive's size. Users can specify a maximum number of member files per archive with HTAR's -M option.

Passwordless FTP

Because HTAR (unlike FTP) does not support user dialog with a server and has no password-passing option, you can only manipulate HTAR archives on machines with preauthenticated (passwordless) FTP notrun the PFTP client.

Too Many Names

For users who make HTAR archives containing thousands of files, limitation is of the UNIX shell rather than of the HTAR program itself. One would normally select multiple files for archiving by using a UNIX "ambiguous file reference," a partial file name adjacent to one or more shell metacharacters (or "wild card" filters, such as the asterisk). Your current shell automatically expands the metacharacter(s) to generate a (long) alphabetical list of matching file names, which it inserts into the execute line as if you had typed them. Thus,

htar -cf test.tar a*

might become equivalent to a command line with dozens of a-named files on the end. Each shell has a maximum length for execute lines, however, and if your specified metacharacter filter matches thousands of file names, HTAR's execute line may grow too long for the shell to accept, which would prevent building your intended many-file archive.

The most effective, least resource-intensive way to work around the problem of having a (virtual) HTAR execute line too long for the shell to handle is to plan ahead and keep (or generate) in a single directory all and only the files that you want to archive. HTAR processes directory names recursively by default, so, if you specify only the relevant directory name on HTAR's execute line, HTAR will (internally) archive every file within the directory without any filter-induced length problems. For example,

htar -cf test.tar projdir

will successfully archive any number of files within the "projdir" directory (and use no shell-mediated file-name generation to do it).

The UNIX find utility is designed to produce lists of files (that meet specified criteria) to feed into other programs for further processing and so offers a second way for HTAR to archive very large numbers of files without having a very long execute line. Indirection is required for success, however. Enable find to pipe standard input directly into HTAR by invoking HTAR's -L option with a hyphen (-) argument instead of a file name. The correct sequence is:

find . -name 'a*' -print | htar -cf test.tar -L -

The find -name option generates the list of matching names internally without expanding find's execute line, and the use of the metacharacter * in the execute line does not pose the same too-long problem as it did originally in HTAR's execute line because the surrounding quotes shelter the filter from shell processing. If you need to keep the list of input names (for verification or audit purposes, for example), you could break this single line into two equivalent steps mediated by a helper file (here called "alist") that you preserve.

find . -name 'a*' -print > alist
htar -cf test.tar -L alist


Executing HTAR Using Hopper

Hopper is a graphical front end for several file-transfer tools (FTP, NFT, HTAR, HSI) that is installed on all LC production machines. With Hopper you can create HTAR archives by graphically dragging files and directories to the Hopper storage window, and you can just as easily extract contents of HTAR archives. For more details on Hopper, type "man hopper" on an LC host, use Hopper's built-in help package, or visit the Hopper Web pages.

HTAR Options

Action Options

One of these action options is required every time that you run HTAR.

-c
(create) opens a connection to storage, creates an archive file at the storage location (not online) and with the name specified by -f, and transfers (copies) into the archive each file specified by filelist (required whenever you use -c). If archivename already exists, HTAR overwrites it without warning. To create a local archive file instead (the way TAR does), also use -E. If filelist specifies any directories, HTAR includes them and all of their children recursively. Use -P with -c to automatically create all needed subdirectories along the archive path name.
-D
(soft delete) opens a connection to storage and reads the existing index file, creating a new temporary index file in the local file system and marking each of the specifiied member files as deleted in the new index file. It then replaces the existing index file with the new temporary copy.
-K
(verify) opens a connection to storage, verifies the index file for the archive that you specify with -f, then uses the index file to verify every entry in (member of) the archive file itself. The default responses from -K appear very quickly and overwrite, so you may only be able to read the last one ("HTAR successful," if it is). If the index file is missing for an archive, -K reports the error message "no such file archivename.idx." If you combine -K with -v, HTAR lists the name of each file that it finds in the specified archive in alphabetical order, one per line, along with the size of each in bytes and in blocks (excluding the consistency file), then gives a total file count.
-t
(table of contents) opens a connection to storage, then lists the files currently within the stored archive file specified by -f, along with their owner, size, permissions, and modification date (the list includes HTAR's own consistency file). Here filelist defaults to * (all files in the archive), but you can specify a more restrictive subset (usually by making filelist a filter).
-U
(undelete) undeletes the specified member files from the archive that were previously softdeleted by -D by removing the deleted flag in their index file entires.
-x
(extract) opens a connection to storage, then transfers (extracts, copies) from the stored (remote) archive file specified by -f each internal file specified by filelist (or all files in the archive if you omit filelist). If filelist specifies any directories, HTAR extracts them and all their children recursively. If any file already exists locally, HTAR overwrites it without warning, and it creates all new files with the same owner and group IDs (and if you use -p, with the same UNIX permissions) as they had when stored in the archive. (If you lack needed permissions, extracted files get your own user and group IDs and the local UMASK permissions; if you lack write permission then -x creates no files at all.) Note that -x works directly on the remote archive file; you never retrieve the whole archive from storage just to extract a few specified files from within it.
-X
(index) opens a connection to storage, then creates an (external) index file for the existing archive file specified by -f (a stored TAR format file by default or a local TAR-format file if you also use -E). Using -X rescues an HTAR archive whose (stored) index file was lost, and it enables HTAR to manage an archive originally created by traditional TAR. The resulting external index file is stored if the corresponding archive is stored, but local if the archive is local (with -E). See the How HTAR Works section for an explanation of HTAR index files.

Archive Option

This option is required every time you run HTAR unless you only use the -? option.

-f archivename
(required option) specifies the archive file on which HTAR performs the action options -c|t|x|X|K. HTAR has no default for -f (whose argument must appear immediately after the option name). Because HTAR operates on stored archive files, archivename also locates the archive file relative to your HPSS home directory: a simple file name here (e.g., abc.tar) resides in your storage home directory, while a relative pathname (e.g., xyz/abc.tar) specifies a subdirectory of your storage home directory (i.e., /users/unn/username/xyz/abc.tar). Never use tilde (~) in archivename. HTAR's -f makes no subdirectories; you must have created them in advance.

Control Options

Control options change how HTAR behaves, but they are not required. Default values are indicated when they exist.

-?
displays a short syntax summary of the HTAR execute line and a one-line description of each option. Users running HTAR under some shells may need to protect the question mark by using the three-character string -\? to display this help message.
-B
adds block numbers to the listing (-t) output.
-d debuglevel
(default is 0) sets to an integer from 0 through 5 the level of debug output from HTAR, where 0 disables debug information for normal use and 1 to 5 enable progressively more elaborate debug output.
-E
emulates TAR by forcing the archive file to reside on the local machine (where you run HTAR) rather than in HPSS (storage), where it resides by default (-f always specifies the archive pathname, which -E interprets as local rather than remote). The HTAR index file goes into the same (local) directory as the archive. Option -P works with -E.
-h
(used only with -c; has no effect otherwise) for each symbolic link that it encounters, causes HTAR to replace the link with the actual contents of the linked-to file (stored under the link name, not under the file's original name). Later use of -t or -x treats the linked-to file as if it had always been present as an actual file with the link name. Without -h, HTAR records, reports, and restores every symbolic link overtly, but it does not replace the link with the linked-to contents.
-H subopt[:subopt...]
specifies a colon-delimited list of HTAR suboptions to control program execution. Possible subopt values include:
acct=id/acctname
specifies the numeric account ID or alphabetic account name to use for the current HTAR run. This option is only meaningful for HPSS-resident archives.
cix
used with the extract (-x) operation with HPSS-resident archives. If specified, precopies the index file to a temporary local file before reading the archive file. This option is normally not needed, but was added to avoid problems that were encountered with multithreaded I/O on some hardware platforms.
crc
enables generation of Cyclic Redundancy Checksums (CRCs) when copying member files into the archive and when verifying the contents of the archive (-K command line option, or -Hverify option for creates). Enabling checksums usually degrades HTAR's I/O performance and increases its CPU utilization.
exfile=path
specifies a path name to an "exceptions" file, which contains a list of failed member files and an explanation of the failure. Note: This option is currently implemented only for the GPFS/HPSS Interface (GHI).
family=id[,index_id]
specifies tape file family ID to use when creating HPSS-resident archive files, and, optionally, the family ID to use when creating the index file. This option is useful at sites which make use of the HPSS "file family" capability. Family ID 0, which is the default, uses the default pool of tapes. Contact your HPSS administrator to determine the file families that are available at your site.
nocfchk
causes HTAR to disable the verification of the index file and the consistency file. Use of this option can avoid extra tape mounts if the consistency file lives on a different tape cartridge than the specified member file(s). Currently, this option is only effective for the -D (softdelete) action.
nocrc
(the default) disables generation of CRCs when creating files and when extracting files from or verifying existing archive files.
nostage
avoids prestaging tape-resident (stored) archive files when HTAR performs -x or -X actions.
port=x
specifies the TCP port number to use when HTAR connects to the remote HPSS server. This parameter is only used in conjunction with the - -Hserver parameter.
relpaths
used with the verify (-K) action. When comparing member files in the archive file with local files, forces relative local file paths to be used by removing any leading "/" from the member file path name before attempting to read it in the local file system.
rmlocal
removes local member files after HTAR has successfully written both the archive file and the index file (used with -c).
server=host
specifies the hostname or TCP/IP address of the HPSS server. The HPSS administrator defines the default server host or IP address when HTAR is built. The -Hport parameter (see above) can be used in conjunction with this option to completely specify the connection address to be used.
tss=stack_size
specifies the thread stack size to be used when HTAR creates threads to read local files during a create (-c) operation. In most cases, the system default value can be used, but situations such as the case where the default thread stack size is set very large, for example, on machines that are tuned for compute-type problems, can cause HTAR thread creations to fail. stack_size can be specified in bytes, kilobytes, or megabytes by appending a case-insensitive suffix (k, kb, m, or mb).
umask=octal_mask
used with the -c option. This specifies the HPSS umask value to be set during HTAR startup. This impacts the permissions that are set on the resulting archive and index files that HTAR creates in the same manner as the Unix umask command.
verify=option[,option,..]
specifies one or more verification options that should be performed following successful creation of the archive (-c), or for the verify (- K) command. Multiple options can be specified by separating them with a comma, with no whitespace. Options are processed from left to right, and, in the case of conflicting options, the last one encountered is used without comment. The options can be either individual items or the keyword "all" or a numeric value of 0, 1, or 2. Each numeric level includes all of the checks for lower-valued levels and adds additional checks. The verifications options are:
all
enables all possible verification options except paranoid.
info
reads and verifies the tar-format headers that precede each member file in the archive.
crc|noncrc
enables or disables recalculation of the cyclic redundancy checksum (CRC) and verification that it matches the value that is stored in the index file. Note that this option only applies if the -Hcrc option was specified, which causes a CRC to be generated for each member file as it is copied into the archive file.
compare|nocompare
enables or disables byte-by-byte comparison of the local member files with the corresponding archive files. If -Hrelpaths is not specified, then absolute paths for member files in the archive will also be treated as absolute local paths.
paranoid|noparanoid
enables or disables (the default) extreme efforts to detect problems (such as discovering whether local files were modified during archive creation before deleting them if authorized by RMLOCAL).
0|1|2
0 enables the "info" verification. 1 enables level 0 and "crc" (i.e., info,crc). 2 enables level 1 and "compare" (i.e., info,crc,compare). It is also possible to specify a verification option such as "all" or a numeric level such as 0, 1, or 2, and then selectively disable one or more options.
-I indexname
specifies a nondefault name for the HTAR external index file that supports the archive specified by -f.
WARNING: if you use -I to make any nondefault index name (3 cases, below) when you create (-c) an archive, then you MUST also use -I with the same argument every time you extract (-x) files from that archive (else HTAR will look for the default index, not find it, and end with an error).
There are three cases based on the first character of indexname:
. (dot)
If indexname begins with a period (dot), HTAR treats it as a suffix to append to the current archive name.
Example: -I .xnd yields an index file called archivename.xnd
/
If indexname begins with a / (slash), HTAR treats it as an absolute path name (you must create all the subdirectories ahead of time with FTP's or HSI's mkdir option).
Example: -I /users/unn/yourname/projects/text.idx uses that absolute path name in storage (HPSS) or the local file system (-E) or remote file system (-F) for the index file.
other
If indexname begins with any other character, HTAR treats it as a relative pathname (relative to the storage directory where the archive file resides, which might be different than your storage home directory).
Example: -I projects/first.index locates first.index at storagehome/ projects/first.index if the archive file is in your storagehome (the default), but tries to locate first.index at storagehome/projects/projects/ first.index if the archive was specified as -f projects/aname in the first place. (All such subdirectories must be created in advance or the -P command line option must be specified to create any missing intermediate subdirectories.)
-L inputfile
(used with -c) writes the files and directories specified by their literal names (in the inputfile, which contains file names one per line) into the archive specified by -f. Directories are treated recursively; a directory entry and its subdirectories or subfiles are all written to the archive. Normal metacharacters (tilde, asterisk, question mark) are treated literally, not expanded as filters. Replace inputfile with a hyphen (-L -) for HTAR to read the list of file names from standard input; the HTAR Limitations section shows how to use this technique.
(used with -x) retrieves the files and directories specified by their literal names. See the Retrieving Files example below for how to use -L instead of wild cards to retrieve only specified files from a stored archive.
WARNING: HTAR's -L differs from both AIX TAR's -L (which handles directories nonrecursively) and Linux TAR's -L (which changes tapes).
-m
(used only with -x; applies only to files) makes the time of extraction the last-modified time for each member file (the default preserves each file's original time of last modification). For directories, HTAR itself always preserves the original modification time for top-level directories that it copies from an archive, even if you invoke -m. However, subsequently creating subdirectories or files within a directory may cause the operating system to change the modification time on one or more directories (so that it too appears to be the time of extraction).
-M maxfiles
(default is 10,000,000 at LLNL) specifies the maximum number of member files allowed when you use -c to create an HTAR archive. Internal limits are set when HTAR is compiled at each site; at LLNL, you can increase maxfiles as high as 50,000,000.
-n timeinterval
(used only with -c; has no effect otherwise) includes in a new archive only those files (that meet your other naming criteria and) that were either created or modified between now and the start of timeinterval. Option -n is intended mostly to simplify the creation of incremental backup archives. Here timeinterval can have the form:
d
an integer that specifies days (e.g., 5 for 5 days), or
:h
an integer that specifies hours (e.g., :12 for :12 hours), or
d:h
a pair of integers that specify days and then hours (e.g., 1:6 for 1 day and 6 hours).
-o
(lowercase, used only with -x) (default for all nonroot users) causes the extracted files to take on the user and group ID (UID, GID) of the person running HTAR, not those of the original archive. This makes a difference for root users but not for ordinary HTAR users.
-O
(uppercase, used only with -x, mimics the Linux TAR --to-stdout option) writes the file(s) extracted from an archive (with -x) to standard output (and hence to a UNIX pipe for postprocessing, if you wish). The HTAR Limitations section shows how to use this technique. Because HTAR does not separate files in the output stream, -O is usually useful only when you extract a single file.
-p
preserves all UNIX permission fields (on extracted files) in their original modes, ignoring the present UMASK (the default changes the permissions to the local UMASK where HTAR extracts the files). Root users can also preserve the setuid, setgid, and sticky bit permissions with this option.
-P
(used only with -c, has no effect otherwise) automatically creates all intermediate subdirectories specified on the archive file's pathname if they do not already exist. HTAR's -P thus works the same as MKDIR's -P option. You can use -P with archives created in HPSS (storage, the default) or on your local machine (with -E).
-q
(quiet mode) supresses most HTAR informational messages, such as its usual interactive progress reports as it creates an archive file.
-S bufsize
(default is 16 Mbyte) specifies the buffer size to use when HTAR reads from or writes to an HPSS archive file. Here bufsize can be a plain integer (interpreted as bytes), an integer suffixed by k, K, kb, or KB for kilobytes, or an integer suffixed by m, M, mb, or MB for megabytes (e.g., 16mb). -S is intended mostly for LC staff, not ordinary HTAR users.
-T maxthreads
specifies the maximum number of threads that HTAR will use to copy member files to or from the archive file (default varies from 5 to 20 threads). This value is ignored when extracting member files from an archive (-x). HTAR reports the actual number of threads used on each run if you invoke -v or -V. HTAR creates a maxthreads pool of threads and then uses buffer size (see -S), average member file size, and HPSS network transfer rates to estimate how many threads to actually deploy. Normally, the smaller the member file size, the more threads can be active when creating files. For small files, setting -T to a larger number (up to 100 has been tested) can dramatically improve the transfer rates if the operating system is able to support the load.
-U
undeletes soft-deleted member files (see -D above) by copying the existing index file to a temporary local file, removing the deleted flag in the specified index entries along the way, and then rewriting the temporary index to the same location.
-V
requests "slightly verbose" reporting of file-transfer progress (often very brief, overwritten messages to the terminal). Do not use with -v.
-v
requests "very verbose" reporting of file-transfer progress. For each member file transferred to an archive, HTAR prints A (added) and its name on one line; for each member file extracted from an archive, HTAR prints X, its name, and its size on a line, along with a summary of the whole transfer at the end. For each file added during a build index (-X) operation, HTAR prints i and its name. For each file verified during a verify operation (-K), HTAR prints v (or V if comparing archive and local file contents), its name, and a trailing / if this is a directory. For each file that is soft-deleted during a delete (-D) operation, HTAR prints d; similarly, for an undelete (-U) operation, HTAR prints u. Do not use with -V.
-w
(works only with -x, -D, -U, not with -c) lists (one by one) each member file to be extracted from the archive and prompts you for your choice of confirmatory action, where possible responses are:
y[es]
extracts the named file.
n[o]
skips the named file.
a[ll]
extracts the named file and all remaining (not yet processed) selected files too.
q[uit]
skips the named file and stops prompting. HTAR ends.
-Y auto | [archiveCOS][:indexCOS]
specifies the HPSS class of service (COS) for each stored archive and its corresponding index file. The default is AUTO, which causes HTAR to use a site-specific COS chosen for archive suitability (at LC, the default COS for HTAR files is 160, which automatically stores a single copy of each archive, regardless of its size). You can specify a nondefault COS for the archive, the index, or both (e.g., -Y 120:110), but this is usually undesirable except when testing new HPSS features or devices (if your archive size grows to exceed that allowed by a nondefault COS, HPSS will stop the transfer and HTAR will end with an error). Use -Y dualcopy (i.e., COS 200) to request dual-copy storage of any mission critical archive of any size for extra safety. Using -Y overrides the HTAR_COS environment variable. NFT's DIR command with the -h option reports the COS for stored files (in output column 3), while NFT's SETCOS command offers a different way to specify the storage class of service.

HTAR Examples

Creating an HTAR Archive File

Goal

To create an HTAR archive file in a subdirectory of your storage home directory and use a filter to install several files within that stored archive.

Strategy

  1. One HTAR execute line can perform all of the desired tasks quickly and in parallel:
    • The -cvf options create (c) an archive, verbosely (v) report the incoming files, and (f) name the envelope file.

    • The relative pathname case3/myproject.tar locates the archive (myproject.tar) in preexisting subdirectory case3 of your storage home directory (omitting case3/ leaves the archive at the top level of your storage home directory). HTAR will not create case3 by default, however; you must either have previously used FTP's or HSI's mkdir command or else you must invoke -P to explicitly request creation of all needed subdirectories along the archive pathname.

    • File filter tim* selects all and only the files whose names begin with TIM (in the directory where you run HTAR) to be stored in the archive.

  2. HTAR opens a preauthenticated connection to your storage (HPSS) home directory and reports its housekeeping activities (very quickly, in lines that overwrite, so you may not notice all of these status reports on your screen).

  3. HTAR creates your requested archive and uses parallel connections (but not the PFTP client) to move your requested files directly into it. Directories are handled recursively and directory names (if any) appear with a slash (/) appended to identify them.

  4. The last incoming file that HTAR reports is always the 256-byte consistency file by which HTAR coordinates your archive with its external index file.

  5. HTAR summarizes the work done (time, rate, amount, thread count), then copies into storage the index file that it made, destroys the local version, and ends.

htar -cvf case3/myproject.tar tim* ---(1)

  HTAR: Opening HPSS server connection ---(2)
  HTAR: Getting HPSS site info
  HTAR: Writing temp index file to /usr/tmp/aaamva09A

  HTAR: creating HPSS Archive file case3/myproject.tar ---(3)
  HTAR: a tim1.txt
  HTAR: a tim2.txt
  HTAR: a tim2a.txt
  HTAR: a tim3.a
  HTAR: a time.txt
  HTAR: a time2.gif

  HTAR: a /tmp/HTAR_CF_CHK_13805_997722535 ---(4)

  HTAR: Create complete for case3/myproject.tar. 399,360 ---(5)
      bytes written for 6 member files, max threads: 8
      Transfer time: 0.555 seconds (7.257 MB/s)

  HTAR: Copying Index File to HPSS...Creating file


Retrieving Files from within an Archive

Goal

To retrieve several files from within an existing stored HTAR archive file (without retrieving the whole archive first).

Strategy

HTAR does not process metacharacters (file filters such as *) itself, but leaves them for the shell to expand and compare with file names in your local directory. Hence, you CANNOT use * to select a subset of already archived files to retrieve. For example, "natural" execute lines

htar -xvf case3/myproject.tar time* [WRONG]
htar -xvf case3/myproject.tar 'time*' [WRONG]

both FAIL to select (and hence to retrieve) any stored files from the MYPROJECTS.TAR stored archive (each yields its own set of error messages). These lines work only accidentally, if you happen to have files with the same name in both your local directory and your stored archive (unlikely except when you are just testing HTAR).

Work-arounds

  1. Type the name of each file that you want to retrieve (at the end of the HTAR execute line).

  2. If you have a long list of files to retrieve, or if you plan to reuse the same retrieval list often, put the list of sought files into a file and use HTAR's -L option to invoke that list. You can use HTAR's -t (reporting) option to help generate that retrieval list by reporting all the files you have archived and then editing that report to include only the relevant file names to retrieve. For instance,

    htar -tf case3/myproject.tar > hout
    grep 'time' hout | cut -c 50-80 > tlist

    captures the list of all your stored files in the local file HOUT and then selects just the file names that contain the string TIME for use with HTAR's -L option (here, in local file TLIST). Note that HTAR automatically appends slash (/) at the end of every directory name that -t reports.

  3. Use Hopper to run HTAR as a controllee, then select visually the files that you want to retrieve.
  1. Once you have laid the groundwork above, a single HTAR execute line can retrieve your specified files quickly and in parallel from within your stored archive:
    • The -xvf options request retrieval/extraction (x), verbosely (v) report the retrieved files, and (f) name the target archive.

    • The relative pathname case3/myproject.tar locates the archive (myproject.tar) in preexisting subdirectory case3 of your storage home directory.

    • The explicit file list (A) or name-containing file (B) selects all and only the files that you want (here, those whose names begin with TIME, a subset of all files stored in this archive in the previous example).
  2. HTAR opens a preauthenticated connection to your storage (HPSS) home directory and reports its housekeeping activities (very quickly, in lines that overwrite, so you may not notice all of these status reports on your screen).

  3. HTAR uses its external index to locate in the archive the (two) specific files that you requested and then it transfers them by using parallel connections (but not the PFTP client) to your local machine without retrieving the whole archive file.

  4. HTAR summarizes the work done (time, rate, amount) and then ends.

htar -xvf case3/myproject.tar time.txt time2.gif ---(A)
   OR
htar -xvf case3/myproject.tar -L tlist ---(B)

  HTAR: Opening HPSS server connection ---(2)
  HTAR: Reading index file
  HTAR: Opening archive file

  HTAR: Reading archive file ---(3)
  HTAR: x time.txt, 1085 bytes, 4 media blocks
  HTAR: x time2.gif, 3452 bytes, 8 media blocks

  HTAR: Extract complete for case3/myproject.tar, ---(4)
        2 files. total bytes read 116,736 in 0.070 seconds (1.669 MB/s)
  HTAR: HTAR SUCCESSFUL


Rebuilding a Missing Index

Goal

To rebuild the missing index file for a stored HTAR archive file and thereby (re)enable blocked access to the files within it (and extract some).

Strategy

  1. You try to retrieve all files (-xvf) from the HTAR archive myproject.tar in the case3 subdirectory of your storage home directory.

  2. But HTAR cannot find the external index file (here, called myproject.tar.idx) for this archive, and it returns a somewhat cryptic error message, retrieves no requested files, and ends. (File myproject.tar.idx may have been moved, renamed, or accidentally deleted from storage.)

  3. So you execute HTAR again with the special action -X (uppercase, not lowercase, eks) to request rebuilding the external index for the (same) disabled archive.

  4. HTAR opens a preauthenticated connection to your storage (HPSS) home directory, locates the archive in subdirectory case3, scans (but does not retrieve) its contents, and thereby creates a new myproject.tar.idx file (temporarily on local disk, then moved to the same storage directory as the archive file that it supports). HTAR ends.

  5. Now you again try your original (1) file-retrieval request.

  6. HTAR opens a preauthenticated connection to your storage (HPSS) home directory and reports its housekeeping activities (very quickly, in lines that overwrite, so you may not notice all of these status reports on your screen).

  7. HTAR uses its (newly rebuilt) external index to locate the files within the archive and transfers them by parallel connections to your local machine (it transfers all of them because there is no filelist on the execute line).

  8. HTAR summarizes the work done (time, rate, amount) and then ends.

htar -xvf case3/myproject.tar ---(1)

  HTAR: Opening HPSS server connection
  HTAR: Getting HPSS site info
  ERROR: Received unexpected reply from server: 550 ---(2)
  ERROR: Error -1 getting Index File attributes...
  HTAR: HTAR FAILED
  ###WARNING htar returned non-zero exit status.
             72 = /usr/local/bin/htar.exe...

htar -Xf case3/myproject.tar ---(3)

  HTAR: Opening HPSS server connection ---(4)
  HTAR: Reading archive
  HTAR: Copying Index File to HPSS... creating file
  HTAR: HTAR SUCCESSFUL

htar -xvf case3/myproject.tar ---(5)

  HTAR: Opening HPSS server connection ---(6)
  HTAR: Reading index file
  HTAR: Opening archive file

  HTAR: Reading archive file ---(7)
  HTAR: x tim1.txt, 3503 bytes, 8 media blocks
  HTAR: x tim2.txt, 4310 bytes, 10 media blocks
  HTAR: x tim2a.txt, 5221 bytes, 12 media blocks
  HTAR: x tim3a., 5851 bytes, 13 media blocks
  HTAR: x time.txt, 1085 bytes, 4 media blocks
  HTAR: x time2.gif, 3452 bytes, 8 media blocks

  HTAR: Extract complete. total bytes read: ---(8)
        28,160 in 0.141 seconds (0.200 MB/s)
  HTAR: HTAR SUCCESSFUL