From ffedec1d6f237df71efd912f926daa60e00149d2 Mon Sep 17 00:00:00 2001 From: espie Date: Wed, 19 Apr 2000 01:39:58 +0000 Subject: [PATCH] Document the way distfiles are mirrored. Document a few things that changed in the ports tree. --- share/man/man7/Makefile | 3 +- share/man/man7/mirroring-ports.7 | 178 +++++++++++++++++++++++++++++++ share/man/man7/ports.7 | 82 ++++++++++---- 3 files changed, 239 insertions(+), 24 deletions(-) create mode 100644 share/man/man7/mirroring-ports.7 diff --git a/share/man/man7/Makefile b/share/man/man7/Makefile index 329f6c31b05..0f3627cf58e 100644 --- a/share/man/man7/Makefile +++ b/share/man/man7/Makefile @@ -4,7 +4,8 @@ # missing: eqnchar.7 man.7 ms.7 term.7 MAN= ascii.7 environ.7 hier.7 hostname.7 intro.7 mailaddr.7 \ - mdoc.7 mdoc.samples.7 operator.7 ports.7 securelevel.7 + mdoc.7 mdoc.samples.7 operator.7 securelevel.7 \ + ports.7 mirroring-ports.7 MLINKS= mdoc.7 mandoc.7 mdoc.samples.7 mandoc.samples.7 .include diff --git a/share/man/man7/mirroring-ports.7 b/share/man/man7/mirroring-ports.7 new file mode 100644 index 00000000000..19902a31898 --- /dev/null +++ b/share/man/man7/mirroring-ports.7 @@ -0,0 +1,178 @@ +.\" $OpenBSD: mirroring-ports.7,v 1.1 2000/04/19 01:39:58 espie Exp $ +.\" +.\" Copyright (c) 2000 Marc Espie +.\" +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR +.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, +.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT +.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF +.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.\" +.Dd February 19, 2000 +.Dt MIRRORING-PORTS 7 +.Os +.Sh NAME +.Nm mirroring-ports +.Nd how to build a mirror for ports distfiles +.Sh DESCRIPTION +The +.Nm OpenBSD Ports Collection +offers some powerful tools to mirror software sources. +The ports infrastructure provides a +.Ar mirror-maker +target that can be used to build a Makefile to facilitate mirroring +distfiles. +This target builds +.Ic ${DISTDIR}/Makefile , +which can be used on almost any Unix-like machine to mirror +.Ox +distfiles. +.Pp +A sample Makefile entry is formatted like this: +.Bd -literal +all:: audio/tracker/tracker-5.3 +.PHONY: audio/tracker/tracker-5.3 +audio/tracker/tracker-5.3:: tracker-5.3.tgz + +tracker-5.3.tgz: $F + @MAINTAINER="espie@openbsd.org" \\ + SITES="ftp://ftp.uni-trier.de/pub/unix/audio/tracker/ " \\ + CIPHER="sha1" CKSUM="b0973d6a9c363caebd3a71547412f42b0681f323" \\ + ${FETCH} "$@" + + +.Ed +This Makefile is usually invoked by the user from the directory where +they wish to mirror distfiles, with variables +.Ev FETCH +and +.Ev F +set on the command line, e.g., +.Bd -literal -offset indent + cd mirror && make -j 5 -f path_to_makefile FETCH=fetch_script +.Ed + +Targets are set up so that each port is referenced by its full name, and +retrieves all its distfiles. +The default +.Ar all +target is used to retrieve all distfiles. +Targets +.Ar ftp +and +.Ar cdrom +can be used to retrieve all distfiles necessary to build ftp and cdrom +packages, respectively. +Actual fetching is usually invoked with a parallel-make option, so that +several retrievals through ftp can proceed simultaneously. +.Pp +The +.Ev F +variable can be set to a dummy name, or a recent filename, to force +re-fetching of anything which is older than the filename. +Its intended use is to force re-fetching existing files, +or to checksum all files. +.Pp +The +.Ev ${FETCH} +script should be supplied by the user, and will download and verify the +archive file. +It must obey the following variables: +.Bl -tag -width DIST_SUBDIR +.It Ev MAINTAINER +Port maintainer, used to report errors, +.It Ev ERROR +Some ports problems can be detected while building the Makefile, in which +case this variable will be set to a proper error message. +.It Ev DIST_SUBDIR +See +.Xr ports 7 +for more details. +The +.Ev ${FETCH} +script is responsible for creating this subdirectory and cd'ing to it +before performing the actual fetch. +.It Ev SITES +A list of sites to try for fetching the distribution file. +.It Ev CIPHER +The checksumming utility to use for verifying the distribution file. +It will normally be set to +.Xr sha1 1 +unless you tinker with +.Ev PREFERRED_CIPHER +while building the mirroring Makefile. +.It Ev CKSUM +The corresponding checksum. +If neither +.Ev CIPHER +nor +.Ev CKSUM +nor +.Ev ERROR +are set, the distribution file needs not be checked. +.El +.Pp +A standard fetch strategy is to try all sites in order: whenever the +distribution file is found, download it; verify the checksum; erase the +file and try the next site if it doesn't match. +.Pp +Mirroring sites should update their master Makefile fairly often. +Activities a proper mirror should offer (in order of decreasing importance): +.Bl -tag -width XXXXXXXXX +.It Mirror new files +Use a proper fetch script to download missing files, +.It Verify all checksums +All checksums should be verified from time to time, and maintainers +notified of persistant discrepancies, +.It Check mastersites liveliness +Use a tool such as +.Sq mirror +to check that the master sites haven't fallen +off the Earth. +Even though the first site in the site list is the +most important site, good mirrors will scan all sites and report all +problems, +.It Remove old files +To gain room this, the mirror should maintain a list of +.Sq active +files (easy enough, just provide a fetch script that just lists the +file names), and remove files that are no longer active. +Since +.Ox +releases happen every six months, this delay should be longer than that. +.El +Sample scripts are provided in the +.Pa ${PORTSDIR}/infrastructure/fetch +directory. +.Sh FILES +.Bl -tag -width XXXXXXXXX -compact +.It Pa ${DISTDIR}/Makefile +Main mirroring Makefile +.It Pa ${PORTSDIR}/infrastructure/fetch/fetch-all +Sample script usable to retrieve distfiles. +.It Pa ${PORTSDIR}/infrastructure/fetch/check-all +Sample script to check all distfiles checksums. +.El +.Sh HISTORY +This infrastructure was introduced for +.Ox 2.7 +by Marc Espie, with feedback from Bob Beck, Todd Fries, Camiel Doobelbar, +and a few other people. +.Sh SEE ALSO +.Xr ports 7 diff --git a/share/man/man7/ports.7 b/share/man/man7/ports.7 index 7174c3aea5f..38fdb1dbccd 100644 --- a/share/man/man7/ports.7 +++ b/share/man/man7/ports.7 @@ -23,7 +23,7 @@ .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .\" -.\" $OpenBSD: ports.7,v 1.9 2000/03/19 19:25:37 aaron Exp $ +.\" $OpenBSD: ports.7,v 1.10 2000/04/19 01:39:58 espie Exp $ .\" $FreeBSD: ports.7,v 1.7 1998/06/23 04:38:50 hoek Exp $ .\" .Dd January 25, 1998 @@ -67,8 +67,8 @@ Package System documentation .Pp .Sh PORTS MASTER MAKEFILE The ports master Makefile, normally located in -.Pa /usr/ports/Makefile -(but see +.Pa /usr/ports/Makefile +(but see .Ev PORTSDIR below) offers a few useful targets. @@ -76,23 +76,59 @@ offers a few useful targets. .It Ar index rebuild the ports complete index, .Pa /usr/ports/INDEX +.It Ar mirror-maker +see +.Xr mirroring-distfiles 7 , .It Ar print-index -display the contents of the index in a user-friendly way. +display the contents of the index in a user-friendly way, .It Ar search -invoked with a key, e.g., +invoked with a key, e.g., .Ic make search key=foo , -retrieve information relevant to a given port. +retrieve information relevant to a given port (obsolescent). +.El +.Sh SELECTING A SET OF PORTS +If +.Pa /usr/ports/INDEX +is up to date, it is possible to select subsets by setting the following +variables on the command line: +.Bl -tag -width category +.It Va key +package name matching the given key, +.It Va category +port belonging to category, +.It Va maintainer +port maintained by a given person. +.El + +For instance, to invoke clean on all ports in the x11 category, one can say: +.Bd -literal -offset indent + make category=x11 clean +.Ed + +The index search is done by a perl script, so all regular expressions from +.Xr perlre 1 +apply. .Sh TARGETS Individual ports are controlled through a few documented targets. Some of these targets work recursively through subdirectories, so that someone can, for examples, install all of the net ports. +.Pp +In case of failure in a subdirectory, the shell fragment held in +.Ev REPORT_PROBLEM +is executed. +Default behavior is to call exit, but this can be overriden on the command +line, e.g., to avoid stopping after each problem. +.Bd -literal -offset indent + make REPORT_PROBLEM=true +.Ed + The targets that do this are -.Ar build , checksum , clean , configure , extract , fetch , fetch-list , +.Ar build , checksum , clean , configure , extract , fetch , fetch-list , .Ar install , distclean , deinstall, reinstall, mirror-distfiles, obj , -.Ar list-distfiles +.Ar list-distfiles , package , cdrom-packages and -.Ar package . +.Ar ftp-packages . .Pp Target names starting with _ are private to the ports infrastructure, should not be invoked directly, and are liable to change without notice. @@ -107,12 +143,12 @@ by .Ar install , and so on all the way to .Ar fetch . -Typical use only runs +Typical use only runs .Ar install -explicitly (if root), or +explicitly (if root), or .Ar build (as user), then -.Ar install +.Ar install (as root). .Bl -tag -width configure .It Ar fetch @@ -144,7 +180,7 @@ targets, this is run in scattered pieces as .Ar fetch-depends , lib-depends , build-depends , run-depends , misc-depends . Defining .Ev NO_DEPENDS -to +to .Dv Yes will skip this step. .It Ar extract @@ -170,7 +206,7 @@ Install the port and register it with the package system. The following targets are not run during the normal install process. .Bl -tag -width fetch-list .It Ar fetch-list -Build a script that lists the files needing to be fetched in order to +Build a script that lists the files needing to be fetched in order to build the port. .It Ar depends-list package-depends Print an ordered list of all the compile and run dependencies. @@ -203,13 +239,13 @@ and .Ev PKGFILE . .El .Sh PORT VARIABLES -These can be changed in the environment, or in +These can be changed in the environment, or in .Pa /etc/mk.conf for persistence. They can also be set on make's command line, e.g., .Ic make VAR_FOO=foo .Pp -Boolean variables should be set to +Boolean variables should be set to .Dv Yes instead of simply being defined, for uniformity and future compatibility. .Pp @@ -275,25 +311,25 @@ Primary location(s) for distribution patch files if not found locally. .It Ev MASTER_SITE_OPENBSD If set to -.Dv Yes , +.Dv Yes , go to the master OpenBSD site for all files. .It Ev MASTER_SITE_FREEBSD If set to -.Dv Yes , +.Dv Yes , go to the master FreeBSD site for all files. .It Ev MASTER_SITE_OVERRIDE Try going to this site for all files and patches, first. .It Ev CLEANDEPENDS -If set to +If set to .Dv Yes , -let -.Sq clean +let +.Sq clean recurse to dependencies. .It Ev NOCLEANDEPENDS If defined, don't let .Sq clean -recurse to dependencies (deprecated, use -.Ev CLEANDEPENDS +recurse to dependencies (deprecated, use +.Ev CLEANDEPENDS instead). .It Ev FETCH_CMD Command to use to fetch files. -- 2.20.1