.\" =()<.ds a @@>()= .ds a /var/news .\" =()<.ds b @@>()= .ds b /usr/local/libexec/cnews .\" =()<.ds c @@>()= .ds c /var/lib/news .\" .\" .\" .TH NEWSBATCH 8 "10 March 1995" .BY "C News" .SH NAME sendbatches, batchsplit \- news batching to other sites .br batcher, batchih, batchsm, batchra \- news-batch preparation .br compcun \- news-batch compression .br c7encode, bencode \- compressed-news-batch encoding .br viauux \- news-batch transmission via uucp .br viamail \- news-batch transmission via mail .br viainews, viarsh \- news-batch transmission by misc. means .SH SYNOPSIS .B \*b/batch/sendbatches [ .B \-p ] [ .B \-c class ] [ site ... ] .PP .B \&.../batchsplit batchsize maxsize count .br .B \&.../batcher [ listfile ] .br .B \&.../batchih [ listfile ] .br .B \&.../batchsm [ listfile ] .br .B \&.../batchra [ listfile ] .br .B \&.../compcun [ compressoptions ] .br .B \&.../c7encode .br .B \&.../bencode [ file ] .br .B \&.../viauux [ .BI \-g grade ] [ .B \-z ] [ .B \-n ] [ .B \-d decompressor ] [ .B \-c ] [ site ] .br .B \&.../viamail [ .B \-e ] [ \B -@ ] [ site ] .br .B \&.../viainews .br .B \&.../viarsh [ site ] .SH DESCRIPTION .I Sendbatches administers batched transmission of news to other sites. It should be run periodically, as the owner of the news database, by \fIcron\fR(8) or similar means. It prepares and sends batches of news, subject to restrictions on available space and length of outbound queues. .PP Normally, .I sendbatches does locking to ensure that only one .I sendbatches is running at a time. The .B \-p option suppresses the locking and permits parallel .I sendbatches runs, although lower-level locking is done to ensure that only one is trying to prepare batches for any particular site at a given time. Parallel .I sendbatches runs impose very heavy system loads but may be useful to systems with extensive hardware parallelism and many outbound news feeds. .PP Each site that can have batches sent to it needs a \fIsite\fR directory under \fI\*a/out.going\fR. If \fIsendbatches\fR is invoked with specific \fIsite\fRs given, it considers batching for those sites, only, in that order. By default, \fIsendbatches\fR consults the \fIbatchparms\fR file (see below) to determine what to do: If there is a \fB/default/\fR entry in \fIbatchparms\fR, \fIsendbatches\fR will consider batching for all sites that have directories in \fI\*a/out.going\fR, in oldest-first order by modification time of the directory. If there is no \fB/default/\fR entry, \fIsendbatches\fR considers batching for those sites named in \fIbatchparms\fR, in the order named. .PP To use the batcher, names of files to be sent to a specific site should be appended to a \fItogo\fR file in its \fIsite\fR directory. The batcher expects the lines in \fItogo\fR to have two fields, a filename (as a full pathname, or relative to \fI\*a\fR) of an article and its size in bytes. A missing size field is arbitrarily assumed to be a default average. .PP \fISendbatches\fR uses a number of auxiliary programs to do the real work. The search path it uses to find them includes, in order, the \fIsite\fR directory for the site in question, \fI\*c/bin\fR, and \fI\*b/batch\fR. This permits per-site and per-news-database overrides of the default behaviors. \fISendbatches\fR provides all these programs with environment variables \fBNEWSSITE\fR, containing the name of the site that batches are being prepared for, and \fBNEWSSITEDIR\fR, containing the full pathname of the \fIsite\fR directory, in case these are useful in customization. .PP The names of most of the auxiliary programs, and some other parameters, are taken from the file \fI\*c/batchparms\fR, an ASCII text file. Empty lines and lines starting with `#' are ignored. Other lines specify the behavior for sites, one line per site. A site line is four fields, separated by white space. .PP The first field is the site name. A line whose site name is \fB/default/\fR specifies what parameters should be used for sites not explicitly mentioned. (The presence or absence of such a line also influences the behavior of \fIsendbatches\fR when invoked without arguments; see above.) .PP The second field is the .I class of the site. If .I sendbatches is invoked with the .B \-c option, it attempts batching only for sites of the specified .IR class . A class is a single letter, by convention `u' for UUCP feeds and `n' for NNTP feeds; user-defined classes should be uppercase letters. .PP If the character `!' appears anywhere in the class field of a .I batchparms line, that .I disables the line. Any site whose batching would have been controlled by that line will have no batching attempted for it under any circumstances. This provides a way to turn a site off temporarily. .PP The third field is the size of batches to be prepared (before compression), in bytes. It may optionally be two sizes separated by `\-', in which case the first is the nominal size and the second is the absolute maximum. If only one size is given, that is the nominal size, and the absolute maximum is three times that. .PP The fourth field is the maximum length of the output queue for transmission to that site. If it is `\-', no queue-length limiting is done. .PP The fifth field, which may contain white space, is the command line (normally a pipeline of three-four programs, possibly with options) to be used to build, compress, and transmit batches to that site. It receives the contents of the .I togo file on standard input. It may not contain any single quotes ('). .PP For each site being considered for batches, \fIsendbatches\fR first determines whether there are in fact any articles to be batched. Assuming there are, \fIsendbatches\fR then finds the \fIbatchparms\fR line for that site and (if queue-length limiting is done for that site) invokes \fIqueuelen\fR (see \fInewsaux\fR(8CN)) to find out the size of the outbound queue for the site. \fISendbatches\fR limits the number of batches prepared to the minimum of the limits implied by queue lengths and available space. .PP \fISendbatches\fR uses \fIbatchsplit\fR as necessary to slice chunks out of the \fItogo\fR file, each chunk containing the \fItogo\fR lines for a batch limited to the nominal size. Exception: a single article bigger than the nominal size will still go out as one batch, provided it does not exceed the absolute-maximum size. .PP Each chunk is then processed through the command line, which usually consists of a batch preparer (typically \fIbatcher\fR), which assembles the articles into a batch, a batch compressor (typically \fIcompcun\fR), which performs compression, possibly a batch encoder, which encodes the compressed batch against the vagaries of ill-designed transmission channels, and a batch transmitter (typically \fIviauux\fR), which sends the batch on its way (e.g. enqueues it for transmission). All are run with \fI\*a\fR as the current directory, so non-absolute pathnames in the chunk are valid filenames; NOTE that this represents a change from earlier releases, in which only the preparer ran there. .PP If there is anything left in the file .I togo.leftover after a chunk is processed, .I sendbatches assumes this is the portion of the chunk which could not be processed at this time. If/when this happens, .I sendbatches replaces the chunk file with the contents of .I togo.leftover and attempts no further batching to that site on this run. .PP Batch preparers in the standard distribution are: .RS .IP batcher 9 Normal batching. .IP batchih \fIIhave\fR-sending part of uucp ihave/sendme (not to be confused with NNTP). .IP batchsm \fISendme\fR-sending part of ihave/sendme. .IP batchra Requested-article-sending part of ihave/sendme. .RE .PP .IR Batchih , .IR batchsm , and .I batchra have to map from the phony ``site name'' given in their \fIbatchparms\fR line to the name of the site they should actually send to; they do this by stripping off the last `.' and everything that follows (usually `.ihave' or `.sendme' respectively, but on machines which limit the size of filenames these may have to be shortened). .PP Caution: .I batchih and .I batchsm do their work by constructing an article and feeding it to .IR inews , which means that the batch size must be within what .I inews will accept, if it imposes limits (see .IR inews (1CN)). .PP Batch compressors can include ordinary programs like .I compress and .IR gzip , but one special compressor is provided by the distribution: .RS .IP compcun 9 Compression with .I compress plus the silly B-news-compatible `#!\ cunbatch' header. Options, if any, are passed to .IR compress . .RE .PP Batch encoders supplied with the distribution are: .RS .IP bencode 9 Encodes 8-bit data using only the ASCII characters ``A'' \- ``Z'', ``a'' \- ``z'', ``0'' \- ``9'', ``+'', and ``-''. The ASCII characters blank, newline, and ``/'' also appear in the encoded file, but do not represent encoded bits. The encoded data is terminated with a byte count and cyclic redundancy check for detecting corruption. This ought to suffice to get data through almost any network. .IP c7encode Encodes 8-bit data into a 7-bit form optimized for transmission by uucp `f' protocol. The encoding is complex and bizarre. Obsolete; use .I bencode instead. .IP nencode Feebly attempts to protect text (not compressed!) data by prepending a blank line to the input and an `N' to each line. Obsolete and not recommended; use .I bencode instead. .RE .PP Most transmitters take an optional .I site argument, using the value of the .B NEWSSITE variable if no argument is supplied. Batch transmitters in the standard distribution are: .RS .IP viauux 10 Normal transmission via UUCP. .B \-z or .B \-n may be used to feed the corresponding option to .IR uux . A .B \-g option may be used to feed the .I grade to .I uux (the default is grade `d'). A .B \-d option may be used to specify a .I decompressor program (e.g., .IR gunzip ) to the site's .IR rnews ; beware that old .IR rnews es may not recognize this. .B \-c may be used to request execution of the site's .I cunbatch program rather than .IR rnews . .IP viamail Mail the batch to `\fIsite\fB!rnews\fR'. .B \-e specifies mailing to `\fIsite\fB!enews\fR' instead. .B \-@ specifies use of @ syntax, e.g. `\fBrnews@\fIsite\fR', instead of ! syntax. .IP viainews Feed the batch back to \fIinews\fR (arguments are ignored) (normally useful only for ihave/sendme). .IP viarsh Use \fIrsh\fR to run \fIrnews\fR on the \fIsite\fR via Ethernet, Internet, etc. (the directory containing \fIrnews\fR must be in the default PATH on \fIsite\fR). .RE .PP \fISendbatches\fR logs some information about sites with backlogs in \fI\*c/batchlog\fR (see also \fInewsdaily\fR(8CN)). This is intended to help detection and diagnosis of flow problems. .SH FILES .ta \w'\*a/out.going/*/LOCK'u+2n .nf \*c/LOCKbatch lock for \fIsendbatches\fR \*c/LOCKexplode overall batch-file lock (used by \fIbatchsplit\fR) \*c/L.* lock temporaries \*a/out.going/* batch directories \*a/out.going/*/LOCK per-site locks \*a/out.going/*/L.* per-site lock temporaries \*c/batchparms parameter file also see text .SH SEE ALSO inews(1CN), compress(1), uux(1), relaynews(8CN), rnews(8CN) .SH DIAGNOSTICS Complaints, if any, from \fIsendbatches\fR and its auxiliaries are mailed via .IR report . .SH HISTORY Written at University of Toronto as part of the C News project. A number of the ideas came from Chris Lewis, who was then with Motorola. .I Bencode written at University of Waterloo by Reg Quinton and Ken Lalonde. .SH BUGS \fISendbatches\fR estimates available space without considering the effects of compression; this is usually too conservative. .PP .I Viarsh does not incorporate a spooling subsystem, so a slow site stalls the entire batching system and a non-responding site loses news. It is not recommended for bulk transmission or where high reliability is essential. .PP There ought to be an encoder using MIME's base64 encoding and suitable descriptive headers. .PP The logging is overly simplistic and doesn't work well when selective batching (controlled by .I site arguments, classes, or disabling) is being done.