Getting started/XML Toolchain and Man Pages

From OpenGL.org
Jump to: navigation, search

<< Back to Getting started

OpenGL, OpenGL ES, and EGL Man Pages in Subversion

The OpenGL, OpenGL ES, and EGL man page sources are in Khronos' Subversion server. The man pages linked from the respective SDK areas are XHTML or HTML5 generated starting from these sources.

The sources are in Docbook format and are processed using the modular stylesheets from the Docbook project, which simplifies the toolchain installation.

Licenses

There are a variety of licenses from SGI, 3Dlabs, Sams Publishing, and the Khronos Group applied to different parts of the man pages. We think they all boil down to "BSD style" licenses that should allow you to modify and redistribute the man pages. However, this is not an official legal opinion or commitment from Khronos. You must make your own judgements based on reading the licenses involved.

Downloading the man pages

For Khronos Members

Khronos members can access the man pages using their normal Subversion login.

For non-Members

The man page sources can be downloaded from Khronos' Subversion server read-only. Using the `svn' command-line client, check out the desired man pages sources using commands like:

svn co --username anonymous --password anonymous SVNURL LOCALDIR
Description SVNURL LOCALDIR
OpenGL 4.x (current) API and GLSL pages https://cvs.khronos.org/svn/repos/ogl/trunk/ecosystem/public/sdk/docs/man4/ man4
OpenGL 3.x API pages https://cvs.khronos.org/svn/repos/ogl/trunk/ecosystem/public/sdk/docs/man3/ man3
OpenGL 2.x API pages https://cvs.khronos.org/svn/repos/ogl/trunk/ecosystem/public/sdk/docs/man2/ man2
OpenGL Shading Language (GLSL) pages https://cvs.khronos.org/svn/repos/ogl/trunk/ecosystem/public/sdk/docs/manglsl/ manglsl
OpenGL ES pages (versions 3.1, 3.0, and 2.0 under man31/, man3/, and man/ + manglsl/ respectively) https://cvs.khronos.org/svn/repos/ogles/trunk/sdk/docs esdocs
EGL 1.4 API pages https://cvs.khronos.org/svn/repos/registry/trunk/public/egl/sdk/docs/man/ eglman

Please do not attempt to access parts of the Khronos Subversion server that you do not have permission for (e.g. parts of Subversion above the specified SVNURLs above).

Reporting Problems and Suggesting Changes

For Khronos Members

Please report problems using the internal Khronos Bugzilla with project "OpenGL", "OpenGL ES", or "EGL" and component "Man Pages & Other Documentation".

For non-Members

Non-members can use the public Bugzilla. Go to the public Bugzilla and report a bug against project "OpenGL", "OpenGL ES", or "EGL" and component "Man Pages & Other Documentation". You must first create an account on the public Bugzilla.

Khronos will attempt to fix bugs you identify, and will consider incorporating changes you suggest, but we cannot provide much in the way of technical support. If you choose to work with the man pages, you should first educate yourself on Subversion, Docbook, XSLT, and the other software described below.

Toolchain

To build the man pages, you need some support tools, described in detail below:

  • Khronos build tools
  • GNU make
  • xsltproc
  • Docbook Modular Stylesheets (versions 1.78.1 and later should be fine)
  • If you are building current man pages (OpenGL 4.x or OpenGL ES 3.0 / 3.1), you should not need the following. Older man pages sources (OpenGL ES 2.0, OpenGL 2.x / 3.x) do need these tools:
    • Docbook 4.3, MathML 2.0, and Docbook MathML Module V1.1b1 DTDs (for older man page sources)
    • Updates to the system XML Catalog for the DTDs and stylesheets

Khronos build tools

For Khronos Members

The Khronos build tools are a collection of scripts, LaTeX macros, GNUmake include files, and other material used across several projects in Khronos Subversion. To install them, check out the top-level util repository in /repos/util/, then follow the installation instructions for 'tools/' in the README under /repos/util/trunk/README.txt.

For non-Members

A subset of the build tools - just enough to build the man pages - has been copied into the man page directories under usr/include/make. To use them, you must set the environment variable ROOT to point to the man page directory; e.g. if you checked the Subversion tree out into a path like /home/fred/man , then

(if you use C shell)

setenv ROOT /home/fred/man

(if you use BASH)

export ROOT=/home/fred/man

should be done before running 'make' in that directory.

Other support tools (Linux oriented, but a useful base for Cygwin / OS X discussions below)

xsltproc

xsltproc is a tool that applies XSL transformation rules (defined in the modular stylesheets) to XML source (the man page sources, and supporting files that get pulled in by directives in those files).

  • For Ubuntu or Debian, install the xsltproc package.
  • Other possible package names: libxslt.

Docbook stylesheets

The Docbook Modular Stylesheets are XSL used to transform Docbook XML documents into many other formats.

  • For Ubuntu and Debian, install the docbook-xsl and docbook-xsl-ns packages
  • Other possible package names: docbook-style-xsl
  • If your XML catalog does not resolve the stylesheet URIs in the man page sources to a local reference to the installed stylesheets, processing will be very, very slow.

XML Catalog

The XML Catalog (which can probably be found in /etc/xml/catalog) maps references to the external PUBLIC and SYSTEM names of a DTD (Document Type Definition, e.g. a set of files defining a particular SGML or XML schema - set of tags, syntax/semantics, etc.) into locally cached copies. Most of the docbook DTDs referenced below make use of the XML catalog. If the catalog is not updated correctly, then every time you process a file using a DTD not in the catalog, it will be retrieved over the net, which is very slow. You can determine if this is happening by invoking xsltproc with the --nonet argument.

Docbook 4.3 DTD

The Docbook 4.3 DTD is the basic schema used in the older OpenGL man page sources. The Docbook website is the canonical reference source for all things Docbook.

  • For Ubuntu and Debian, install the docbook package.
  • Other possible package names: docbook-dtds.

MathML 2.0 DTD

The MathML 2.0 DTD defines the schema for MathML, a system for marking up equations.

  • For Ubuntu and Debian, install the w3-dtd-mathml package. This (should) put the DTDs into the system XML catalog. If not, you may need to do something like this (exact paths will vary depending on your installation):
xmlcatalog --noout --add public "-//W3C//DTD MathML 2.0//EN" "file:///usr/share/xml/schema/w3c/mathml/dtd/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add public "-//W3C//DTD MathML//EN" "file:///usr/share/xml/schema/w3c/mathml/dtd/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.w3.org/TR/MathML2/dtd/mathml2.dtd" "file:///usr/share/xml/schema/w3c/mathml/dtd/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add public "-//W3C//DTD XHTML 1.1 plus MathML 2.0//EN" "file:///usr/share/xml/schema/w3c/mathml/dtd/xhtml-math11-f.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.w3.org/TR/MathML2/dtd/xhtml-math11-f.dtd" "file:///usr/share/xml/schema/w3c/mathml/dtd/xhtml-math11-f.dtd" /etc/xml/catalog
  • Other possible package names: unknown

Docbook MathML Module 1.1

The Docbook MathML Module 1.1CR1-2 DTD adds MathML 2.0 to Docbook 4.3.

  • For Ubuntu or Debian, install the docbook-mathml package.
  • Other possible package names: unknown.
  • Installing from the web - the original dbmathml.dtd file can be found at http://www.docbook.org/xml/mathml/1.1CR1/dbmathml.dtd. Download this file, and as root, execute the following commands to save it and add references to the XML catalog:
mkdir -p /usr/share/xml/docbook-mathml-dtd-1.1CR1
cp -f dbmathml.dtd /usr/share/xml/docbook-mathml-dtd-1.1CR1
xmlcatalog --noout --add public "-//OASIS//DTD DocBook MathML Module V1.1b1//EN" "file:///usr/share/xml/docbook-mathml-dtd-1.1CR1/dbmathml.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.oasis-open.org/docbook/xml/mathml/1.1CR1/dbmathml.dtd" "file:///usr/share/xml/docbook-mathml-dtd-1.1CR1/dbmathml.dtd" /etc/xml/catalog
you run xsltproc on a file using this DTD, it will be retrieved over the net. This is very slow.

Windows toolchain details

Here are the steps for setting up the toolchain in Windows:

Note: this is long out of date and has not been updated for Docbook 5. Additions are welcome.

Cygwin

Cygwin is a Linux-like environment for Windows. It includes many of the necessary components of the toolchain. You can grab the latest here: Cygwin home

When selecting packages, make sure to select these:

  • Devel/make: the GNU version of the 'make' utility
  • Doc/docbook-xml43: DocBook XML DTD 4.3
  • Doc/docbook-xsl: XSL stylesheets for the DocBook XML DTD
  • Doc/libxml2: XML C parser and toolkit (runtime and applications)
  • Doc/libxslt: The GNOME XSLT C library (runtime)
  • Interpreters/perl: Larry Wall's Practical Extracting and Report Language

Because cygwin installs its packages in slightly different locations than FC5 above, you'll need to edit the Makefile to change XSLSS, or better yet just set up a symbolic link:

mkdir -p /usr/share/sgml/docbook/
ln -s /usr/share/docbook-xsl/ /usr/share/sgml/docbook/xsl-stylesheets

Khronos build tools

You'll need to follow the instructions above in order to get the required Makefile includes like commondefs and commonrules.

MathML

You need the MathML 2.0 DTD and the DocBook MathML Module, both described above.

The MathML 2.0 DTD package can be downloaded from W3C here. I recommend unpacking it to /usr/share/xml/mathml2/. Then issue these two commands to add it to your catalog:

xmlcatalog --noout --add public "-//W3C//DTD MathML 2.0//EN" \
   "file:///usr/share/xml/mathml2/mathml2.dtd" \
   /etc/xml/catalog
xmlcatalog --noout --add system \
   "http://www.w3.org/TR/MathML2/dtd/mathml2.dtd" \
   "file:///usr/share/xml/mathml2/mathml2.dtd" \
   /etc/xml/catalog

The DocBook MathML Module DTD can be downloaded from DockBook.org here. I recommend placing it in /usr/share/xml/docbook-mathml-dtd-1.1CR1/. Then issue these two commands to add it to your catalog:

xmlcatalog --noout --add public \
   "-//OASIS//DTD DocBook MathML Module V1.1b1//EN" \
   "file:///usr/share/xml/docbook-mathml-dtd-1.1CR1/dbmathml.dtd" \
   /etc/xml/catalog
xmlcatalog --noout --add system \
   "http://www.oasis-open.org/docbook/xml/mathml/1.1CR1/dbmathml.dtd" \
   "file:///usr/share/xml/docbook-mathml-dtd-1.1CR1/dbmathml.dtd" \
   /etc/xml/catalog

Now you can perform a make as described below (see Generating and Viewing Man Pages).

MiKTeX

MiKTeX is a TeX implementation for Windows, and includes pdflatex. You'll need this when we re-introduce PDF output into the toolchain. MiKTeX home

After installing the latest version of MiKTeX (2.4 worked for Benj), you'll need a couple additional packages. MiKTeX has a very handy package manager for automatically downloading and installing optional components. You need the "mdwtools" and "fancyhdr" packages.

OS X toolchain details

Note: this is long out of date and has not been updated for Docbook 5. Additions are welcome.

Unfortunately there aren't any handy packages for OS X. But the good news is that most of the tools you need are already present in OS X 10.4. The exception is subversion which you can obtain here.

In the process I used, I kept all the DTDs alongside the man page source and didn't install anything in /usr/. I reinstall my system fairly regularly (one of the perils of working on OSes) and didn't want everything to get removed all the time.

1. Create a directory to put all this stuff in

mkdir manpages
cd manpages

2. checkout man source from subversion

3. get docbook dtds

mkdir docbook-xml-4-3
cd docbook-xml-4-3
curl -O http://www.docbook.org/xml/4.3/docbook-xml-4.3.zip
unzip docbook-xml-4.3.zip

4. get docbook-xsl from sourceforge and expand

5. get mathml2 and expand

curl -O http://www.w3.org/Math/DTD/mathml2.tgz
tar xzvf mathml2.tgz

6. get dbmathml.dtd

curl -O http://www.docbook.org/xml/mathml/1.1CR1/dbmathml.dtd

7. create an xml catalog - I don't know what the technically correct description for this is - but - the xml catalog routes requests for documents that would otherwise go over the internet to the documents on your local machine

sudo xmlcatalog --create > /etc/xml/catalog

I only use this for the man source on my computer so I am using the default catalog to point to these local dtds. This happens to be just fine with me. If that doesn't work for you (for whatever reason) you could either create a custom catalog (and modify the make files) or you could install all the dtds somewhere that is accessible system wide.

8. add the local file locations to the xml catalog

xmlcatalog --noout --add public "-//W3C//DTD MathML 2.0//EN" "file:///path/to/my/dtds/mathml2/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.w3.org/TR/MathML2/dtd/mathml2.dtd" "file:///path/to/dtds/mathml2/mathml2.dtd" /etc/xml/catalog
xmlcatalog --noout --add public "-//OASIS//DTD DocBook MathML Module V1.1b1//EN" "file:///path/to/dtds/dbmathml.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.oasis-open.org/docbook/xml/mathml/1.1CR1/dbmathml.dtd" "file:///path/to/dtds/dbmathml.dtd" /etc/xml/catalog
xmlcatalog --noout --add public "-//OASIS//DTD DocBook XML V4.3//EN" "file:///path/to/dtds/docbook-xml-4-3/docbookx.dtd" /etc/xml/catalog
xmlcatalog --noout --add system "http://www.oasis-open.org/docbook/xml/4.3b2/docbookx.dtd" "file:///path/to/dtds/docbook-xml-4-3/docbookx.dtd" /etc/xml/catalog

9. modify the man page's Makefile to point to the local dtds - for me this was:

XSLSS   = ../../docbook-xsl-1.71.1/

Troubleshooting

I got this working at first but it was taking forever because it was resolving the document requests over the internet. From Jon's suggestion I modified the XSLT command in the Makefile to include "--nonet" like this:

XSLT    = xsltproc --nonet

This prevented any net lookups and with that I was able to fiddle with /etc/xml/catalog until I got the local requests working. It is no speed demon, on my 2GHz x86 MacBook Pro the first file takes about 4 or 5 seconds to produce and subsequent ones take about 3 full seconds each. A full build takes 10 or 15 minutes.

XML Docbook Editors

There are a lot of XML editors out there. Jon evaluated XMLMind briefly. It is Java-based and should run on both Unix and Windows. It seems to work OK as a straight XML structure editor. There's also a WYSIWYG mode, but so far this only seems to come up by altering the <?xml header on the documents to refer to the canonical URL for the Docbook DTD (under http://www.docbook.org), instead of the locally cached copy. This definitely works. But every time you run XMLMind or xsltproc on the document, it appears to go fetch the entire DTD over the net, which makes using the toolchain remarkably, and painfully slow. We don't know yet how well it works in terms of the XML it generates for new pages.

I (Jon) am just using a straight text editor. Once you have a man page template, adding content is easy.

Generating and Viewing Man Pages

Once you have the toolchain installed, go to the appropriate directory (one of the man* directories you checked out from Subversion). Make sure the environment variable ROOT is correctly set as discussed above under Khronos Build Tools. Then just 'make'. xsltproc will transform each Docbook XML .xml document into an HTML5 .html document (for the current man page sources) or an XHTML+MathML .xhtml document.

Unfortunately, only Firefox really supports MathML properly in contemporary (2014) browsers, so the older man pages are compromised on e.g. Chrome. The current man page sources are in HTML5 and use the MathJax Javascript plugin on other browsers, which does a fine job of MathML rendering.

If you're getting errors, it's probably because one or more of the DTD and stylesheet packages described above are not properly installed. Please check the directions carefully to make sure everything required is installed.

If building is running extremely slowly, it's probably because the XML catalog is not properly updated. You can check this by going into the html or xhtml subdirectory and running

rm (any page name).xml
make XSLT="xsltproc --nonet" (same page name).xml

If the catalog is indeed the problem, this will generate errors instead of running extremely slowly. Again, please recheck the directions above to make sure all the steps involving updating the catalog have been performed.

Output manual pages to man format

This require the same step as required before for the catalog. The only additional condition is about the XSL files used to perform the output.

Uncompress the archive and apply the xsl transform.

$ tar -jxvf docbook-xsl-1.76.1.tar.bz2
$ xsltproc --noout --nonet docbook-xsl-1.76.1/manpages/docbook.xsl glAccum.xml

The procedure is maybe to be completed, but it worked fine for me.

This procedure is noisy, because as is the manual page do not mention some information (as author). To reduce this noise you can transform the xml manual with:

<?xml version='1.0'?>

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output
        doctype-public="-//OASIS//DTD DocBook MathML Module V1.1b1//EN"
        doctype-system="http://www.oasis-open.org/docbook/xml/mathml/1.1CR1/dbmathml.dtd"
        cdata-section-elements="book"
        indent="yes"
        encoding="UTF-8"
    />
    <xsl:template match="@*|node()">
        <xsl:copy>
            <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
    </xsl:template>
    

    <xsl:template match="refentry/refmeta">
        <xsl:element name="info">
            <xsl:element name="orgname">
                <xsl:attribute name="class">consortium</xsl:attribute>
                <xsl:text>opengl.org</xsl:text>
            </xsl:element>
        </xsl:element>
        <xsl:element name="refmeta">
            <xsl:apply-templates select="@*|node()"/>
            <xsl:element name="refmiscinfo">
                <xsl:attribute name="class">manual</xsl:attribute>
                <xsl:text>OpenGL Manual</xsl:text>
            </xsl:element>
        </xsl:element>
    </xsl:template>

</xsl:stylesheet>

Next Steps

  • Update the makefile to process manual pages in man format.
  • Output to PDF format. It will probably require using XSL-FO (formatting objects) via a converter like Apache's FOP (Formatting Objects Processor). Correctly processing MathML is likely to be difficult; we may need to include alternative representations of equations along with the MathML.
  • Wiring up some of the editors we've mentioned previously to allow editing the man pages using the supplied DTDs and such.
  • Creating an xsl/docbook2help.xsl stylesheet for converting into Windows help-file format (CHM).
  • Japanese translations - requires Japanese speakers who are technically skilled and conversant with OpenGL; this would be a major project, but desirable since there's a significant developer base there that would benefit from more documentation in their own language.