Getting started/XML Toolchain and Man Pages

From OpenGL.org
< Getting started
Revision as of 19:48, 1 September 2009 by Alfonse (Talk | contribs) (Removing tinyurl and dead link.)

Jump to: navigation, search

<< Back to Getting started

OpenGL Man Pages in Subversion

The Docbook OpenGL man pages are now in Subversion. We are using the modular stylesheets from the Docbook project, which simplifies the toolchain installation.

The ARB Ecosystem TSG maintains an up-to-date version of the OpenGL man pages processed into XHTML+MathML in the xhtml/ directory under the man pages. These pages can also be viewed in the OpenGL SDK on opengl.org.

Licenses

There are a variety of licenses from SGI, 3Dlabs, Sams Publishing, and the Khronos Group applied to different parts of the man pages. We think they all boil down to "BSD style" licenses that should allow you to modify and redistribute the man pages. However, this is not an official legal opinion or committment from Khronos. You must make your own judgements based on reading the licenses involved.

Downloading the man pages

For Khronos Members

Khronos members can access the man pages using their normal Subversion login.

For non-Members

Non-members can download the man pages read-only, using a command like:

svn co --username anonymous --password anonymous https://cvs.khronos.org/svn/repos/ogl/trunk/ecosystem/public/sdk/docs/man/ man 

Please do not attempt to access parts of the Khronos Subversion server that you do not have permission for (e.g. parts of Subversion above the specified path above). We are opening up this part of the server with some trepidation, and if it results in attacks on khronos.org, we'll have to rethink keeping it open.

Reporting Problems and Suggesting Changes

For Khronos Members

Please report problems using the internal Khronos Bugzilla with project "OpenGL", component "Man Pages & Other Documentation".

For non-Members

Non-members can use the public Bugzilla. Go to the public Bugzilla and report a bug against project "OpenGL", component "Man Pages & Other Documentation". You must first create an account on the public Bugzilla.

Khronos will attempt to fix bugs you identify, and will consider incorporating changes you suggest, but we cannot provide much in the way of technical support. If you choose to work with the man pages, you should first educate yourself on Subversion, Docbook, XSLT, and the other software described below.

Toolchain

To build the man pages, you need some support tools, described in detail below:

  • Khronos build tools
  • GNU make
  • xsltproc
  • Docbook 4.3, MathML 2.0, and Docbook MathML Module V1.1b1 DTDs
  • Docbook Modular Stylesheets (versions 1.69 and later should be fine)
  • Updates to the system XML Catalog for the DTDs and stylesheets

Khronos build tools

For Khronos Members

The Khronos build tools are a collection of scripts, LaTeX macros, GNUmake include files, and other material used across several projects in Khronos Subversion. To install them, check out the top-level util repository in /repos/util/, then follow the installation instructions for 'tools/' in the README under /repos/util/trunk/README.txt.

For non-Members

A subset of the build tools - just enough to build the man pages - has been copied into the man page directories under usr/include/make. To use them, you must set the environment variable ROOT to point to the man page directory; e.g. if you checked the Subversion tree out into a path like /home/fred/man , then

setenv ROOT /home/fred/man

should be done before running 'make' in that directory.

Other support tools (Linux oriented, but a useful base for Cygwin / OS X discussions below)

xsltproc is a tool that applies XSL transformation rules (defined in the modular stylesheets) to XML source (the man page sources, and supporting files that get pulled in by directives in those files).

  • For Ubuntu 8.04, install the xsltproc package
  • For Fedora Core 5 (FC5), install the libxslt RPM

The XML Catalog (which can probably be found in /etc/xml/catalog) maps references to the external PUBLIC and SYSTEM names of a DTD (Document Type Definition, e.g. a set of files defining a particular SGML or XML schema - set of tags, syntax/semantics, etc.) into locally cached copies. Most of the docbook DTDs referenced below make use of the XML catalog. If the catalog is not updated correctly, then every time you process a file using a DTD not in the catalog, it will be retrieved over the net, which is very slow. You can determine if this is happening by invoking xsltproc with the --nonet argument.

The Docbook 4.3 DTD is the basic schema used in the OpenGL man pages. There are later versions of the DTD, but they do not work as well with MathML yet. Eventually we should update to Docbook 5 as 4.3 is getting rather out of date, but it still works just fine. The Docbook website is the canonical reference source for all things Docbook.

  • For Ubuntu 8.04, install the docbook package
  • For FC5, install the docbook-dtds RPM

The MathML 2.0 DTD defines the schema for MathML, a system for marking up equations.

  • For Ubuntu 8.04, install the w3-dtd-mathml package. Then, as root (e.g. using sudo or in a root shell), execute the following commands to put the DTDs into the system XML catalog:
xmlcatalog --noout --add public "-//W3C//DTD MathML 2.0//EN" "file:///usr/share/xml/schema/w3c/mathml/dtd/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add public "-//W3C//DTD MathML//EN" "file:///usr/share/xml/schema/w3c/mathml/dtd/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.w3.org/TR/MathML2/dtd/mathml2.dtd" "file:///usr/share/xml/schema/w3c/mathml/dtd/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add public "-//W3C//DTD XHTML 1.1 plus MathML 2.0//EN" "file:///usr/share/xml/schema/w3c/mathml/dtd/xhtml-math11-f.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.w3.org/TR/MathML2/dtd/xhtml-math11-f.dtd" "file:///usr/share/xml/schema/w3c/mathml/dtd/xhtml-math11-f.dtd" /etc/xml/catalog
  • For FC5 no Fedora-specific RPM is known, but the Mandriva Linux docbook-dtd-mathml2 RPM works; this RPM can be found on RPMfind or Pbone, or a locally cached copy is in Subversion.

The Docbook MathML Module V1.1b1 DTD adds MathML 2.0 to Docbook 4.3. This is not available in packages for Ubuntu 8.04 or FC5, but is small and easily installed:

  • Installing from the web - the original dbmathml.dtd file can be found at http://www.docbook.org/xml/mathml/1.1CR1/. Download this file, and as root, execute the following commands to save it and add references to the XML catalog:
mkdir -p /usr/share/xml/docbook-mathml-dtd-1.1CR1
cp -f dbmathml.dtd /usr/share/xml/docbook-mathml-dtd-1.1CR1
xmlcatalog --noout --add public "-//OASIS//DTD DocBook MathML Module V1.1b1//EN" "file:///usr/share/xml/docbook-mathml-dtd-1.1CR1/dbmathml.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.oasis-open.org/docbook/xml/mathml/1.1CR1/dbmathml.dtd" "file:///usr/share/xml/docbook-mathml-dtd-1.1CR1/dbmathml.dtd" /etc/xml/catalog
  • Installing from Khronos internal copy - a cached copy is located in Subversion. Running 'make' in this directory, as root, should install the DTD and update the XML catalog which maps references to the external PUBLIC and SYSTEM names of the DTD into the locally cached copy. If the catalog is not updated correctly, then every time you run xsltproc on a file using this DTD, it will be retrieved over the net. This is very slow.

The Docbook Modular Stylesheets are used to transform Docbook XML documents into many other formats.

  • For Ubuntu 8.04, install the docbook-xsl package
  • For FC5, install the docbook-style-xsl RPM

Note that the XSLT stylesheets are incorporated from xhtml/opengl-man.xsl using the URI http://docbook.sourceforge.net/release/xsl/current/xhtml/docbook.xsl . If your XML catalog does not resolve this to a local reference to the installed stylesheets, processing will be very, very slow.

Windows toolchain details

Here are the steps for setting up the toolchain in Windows:

Cygwin

Cygwin is a Linux-like environment for Windows. It includes many of the necessary components of the toolchain. You can grab the latest here: Cygwin home

When selecting packages, make sure to select these:

  • Devel/make: the GNU version of the 'make' utility
  • Doc/docbook-xml43: DocBook XML DTD 4.3
  • Doc/docbook-xsl: XSL stylesheets for the DocBook XML DTD
  • Doc/libxml2: XML C parser and toolkit (runtime and applications)
  • Doc/libxslt: The GNOME XSLT C library (runtime)
  • Interpreters/perl: Larry Wall's Practical Extracting and Report Language

Because cygwin installs its packages in slightly different locations than FC5 above, you'll need to edit the Makefile to change XSLSS, or better yet just set up a symbolic link:

mkdir -p /usr/share/sgml/docbook/
ln -s /usr/share/docbook-xsl/ /usr/share/sgml/docbook/xsl-stylesheets

Khronos build tools

You'll need to follow the instructions above in order to get the required Makefile includes like commondefs and commonrules.

MathML

You need the MathML 2.0 DTD and the DocBook MathML Module, both described above.

The MathML 2.0 DTD package can be downloaded from W3C here. I recommend unpacking it to /usr/share/xml/mathml2/. Then issue these two commands to add it to your catalog:

xmlcatalog --noout --add public "-//W3C//DTD MathML 2.0//EN" \
   "file:///usr/share/xml/mathml2/mathml2.dtd" \
   /etc/xml/catalog
xmlcatalog --noout --add system \ 
   "http://www.w3.org/TR/MathML2/dtd/mathml2.dtd" \
   "file:///usr/share/xml/mathml2/mathml2.dtd" \
   /etc/xml/catalog

The DocBook MathML Module DTD can be downloaded from DockBook.org here. I recommend placing it in /usr/share/xml/docbook-mathml-dtd-1.1CR1/. Then issue these two commands to add it to your catalog:

xmlcatalog --noout --add public \
   "-//OASIS//DTD DocBook MathML Module V1.1b1//EN" \
   "file:///usr/share/xml/docbook-mathml-dtd-1.1CR1/dbmathml.dtd" \
   /etc/xml/catalog
xmlcatalog --noout --add system \
   "http://www.oasis-open.org/docbook/xml/mathml/1.1CR1/dbmathml.dtd" \
   "file:///usr/share/xml/docbook-mathml-dtd-1.1CR1/dbmathml.dtd" \
   /etc/xml/catalog

Now you can perform a make as described below (see Generating and Viewing Man Pages).

MiKTeX

MiKTeX is a TeX implementation for Windows, and includes pdflatex. You'll need this when we re-introduce PDF output into the toolchain. MiKTeX home

After installing the latest version of MiKTeX (2.4 worked for Benj), you'll need a couple additional packages. MiKTeX has a very handy package manager for automatically downloading and installing optional components. You need the "mdwtools" and "fancyhdr" packages.

OS X toolchain details

Unfortunately there aren't any handy packages for OS X. But the good news is that most of the tools you need are already present in OS X 10.4. The exception is subversion which you can obtain here.

In the process I used, I kept all the DTDs alongside the man page source and didn't install anything in /usr/. I reinstall my system fairly regularly (one of the perils of working on OSes) and didn't want everything to get removed all the time.

1. Create a directory to put all this stuff in

mkdir manpages
cd manpages

2. checkout man source from subversion

3. get docbook dtds

mkdir docbook-xml-4-3
cd docbook-xml-4-3
curl -O http://www.docbook.org/xml/4.3/docbook-xml-4.3.zip
unzip docbook-xml-4.3.zip

4. get docbook-xsl from sourceforge and expand

5. get mathml2 and expand

curl -O http://www.w3.org/Math/DTD/mathml2.tgz
tar xzvf mathml2.tgz

6. get dbmathml.dtd

curl -O http://www.docbook.org/xml/mathml/1.1CR1/dbmathml.dtd

7. create an xml catalog - I don't know what the technically correct description for this is - but - the xml catalog routes requests for documents that would otherwise go over the internet to the documents on your local machine

sudo xmlcatalog --create > /etc/xml/catalog

I only use this for the man source on my computer so I am using the default catalog to point to these local dtds. This happens to be just fine with me. If that doesn't work for you (for whatever reason) you could either create a custom catalog (and modify the make files) or you could install all the dtds somewhere that is accessible system wide.

8. add the local file locations to the xml catalog

xmlcatalog --noout --add public "-//W3C//DTD MathML 2.0//EN" "file:///path/to/my/dtds/mathml2/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.w3.org/TR/MathML2/dtd/mathml2.dtd" "file:///path/to/dtds/mathml2/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add public "-//OASIS//DTD DocBook MathML Module V1.1b1//EN" "file:///path/to/dtds/dbmathml.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.oasis-open.org/docbook/xml/mathml/1.1CR1/dbmathml.dtd" "file:///path/to/dtds/dbmathml.dtd" /etc/xml/catalog
xmlcatalog --noout --add public "-//OASIS//DTD DocBook XML V4.3//EN" "file:///path/to/dtds/docbook-xml-4-3/docbookx.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.oasis-open.org/docbook/xml/4.3b2/docbookx.dtd" "file:///path/to/dtds/docbook-xml-4-3/docbookx.dtd" /etc/xml/catalog

9. modify the man page's Makefile to point to the local dtds - for me this was:

XSLSS   = ../../docbook-xsl-1.71.1/

Troubleshooting

I got this working at first but it was taking forever because it was resolving the document requests over the internet. From Jon's suggestion I modified the XSLT command in the Makefile to include "--nonet" like this:

XSLT    = xsltproc --nonet

This prevented any net lookups and with that I was able to fiddle with /etc/xml/catalog until I got the local requests working. It is no speed demon, on my 2GHz x86 MacBook Pro the first file takes about 4 or 5 seconds to produce and subsequent ones take about 3 full seconds each. A full build takes 10 or 15 minutes.

XML Docbook Editors

There are a lot of XML editors out there. Jon evaluated XMLMind briefly. It is Java-based and should run on both Unix and Windows. It seems to work OK as a straight XML structure editor. There's also a WYSIWYG mode, but so far this only seems to come up by altering the <?xml header on the documents to refer to the canonical URL for the Docbook DTD (under http://www.docbook.org), instead of the locally cached copy. This definitely works. But every time you run XMLMind or xsltproc on the document, it appears to go fetch the entire DTD over the net, which makes using the toolchain remarkably, and painfully slow. We don't know yet how well it works in terms of the XML it generates for new pages.

I (Jon) am just using a straight text editor. Once you have a man page template, adding content is easy.

Generating and Viewing Man Pages

Once you have the toolchain installed, go to ogl/trunk/ecosystem/public/sdk/docs/man (for non-members, this is the 'man' directory you checked the pages out into) in your Subversion tree. Make sure the environment variable ROOT is correctly set as discussed above under Khronos Build Tools. Then just 'make'. xsltproc will transform each Docbook XML .xml document into an XHTML+MathML .xhtml document which can be viewed in modern browsers.

If you're getting errors, it's probably because one or more of the DTD and stylesheet packages described above are not properly installed. Please check the directions carefully to make sure everything required is installed.

If building is running extremely slowly, it's probably because the XML catalog is not properly updated. You can check this by going into the xhtml subdirectory and running

rm glAccum.xml
make XSLT="xsltproc --nonet" glAccum.xml

If the catalog is indeed the problem, this will generate errors instead of running extremely slowly. Again, please recheck the directions above to make sure all the steps involving updating the catalog have been performed.

Next Steps

  • Other output formats - PDF and Unix man. The modular stylesheets include a manpages stylesheet but nobody has tried it yet. PDF will probably require using XSL-FO (formatting objects) via a converter like Apache's FOP (Formatting Objects Processor). Correctly processing MathML is likely to be difficult; we may need to include alternative representations of equations along with the MathML.
  • Wiring up some of the editors we've mentioned previously to allow editing the man pages using the supplied DTDs and such.
  • Creating an xsl/docbook2help.xsl stylesheet for converting into Windows help-file format (CHM).
  • Japanese translations - requires Japanese speakers who are technically skilled and conversant with OpenGL; this would be a major project, but desirable since there's a significant developer base there that would benefit from more documentation in their own language.