If you're going to be doing development on Twisted itself, or if you want to take advantage of bleeding-edge features (or bug fixes) that are not yet available in a numbered release, you'll probably want to check out a tree from the Twisted Subversion repository. The Trunk is where all current development takes place.
This document lists some useful tips for working on this cutting edge.
Checkout
Subversion tutorials can be found elsewhere, see in particular the Subversion homepage. The relevant data you need to check out a copy of the Twisted tree is available on the pserver page, and is as follows:
$ svn co svn://svn.twistedmatrix.com/svn/Twisted/trunk Twisted
Alternate tree names
By using svn co svn://svn.twistedmatrix.com/svn/Twisted/trunk
otherdir, you can put the workspace tree in a directory other than
Twisted
. I do this (with a name like Twisted-Subversion
) to
remind myself that this tree comes from Subversion and not from a released
version (like Twisted-1.0.5
). This practice can cause a few problems,
because there are a few places in the Twisted tree that need to know where
the tree starts, so they can add it to sys.path without
requiring the user manually set their PYTHONPATH. These functions walk the
current directory up to the root, looking for a directory named
Twisted
(sometimes exactly that, sometimes with a
.startswith test). Generally these are test scripts or other
administrative tools which expect to be launched from somewhere inside the
tree (but not necessarily from the top).
If you rename the tree to something other than Twisted, these
tools may wind up trying to use Twisted source files from /usr/lib/python2.2
or elsewhere on the default sys.path. Normally this won't
matter, but it is good to be aware of the issue in case you run into
problems.
twisted/test/process_twisted.py is one of these programs.
Compiling C extensions
There are currently several C extension modules in Twisted: twisted.protocols._c_urlarg, twisted.internet.cfsupport, twisted.internet.iocpreactor._iocp, and twisted.python._epoll. These modules are optional, but you'll have to compile them if you want to experience their features, performance improvements, or bugs. There are two approaches.
The first is to do a regular distutils ./setup.py build, which
will create a directory under build/ to hold both the generated
.so files as well as a copy of the 600-odd .py files
that make up Twisted. If you do this, you will need to set your PYTHONPATH to
something like MyDir/Twisted/build/lib.linux-i686-2.2 in order to
run code against the Subversion twisted (as opposed to whatever's installed in
/usr/lib/python2.2 or wherever python usually looks). In
addition, you will need to re-run the build command every
time you change a .py file. The build/lib.foo
directory is a copy of the main tree, and that copy is only updated when you
re-run setup.py build. It is easy to forget this and then wonder
why your code changes aren't being expressed.
The second technique is to build the C modules in place, and point your
PYTHONPATH at the top of the tree, like MyDir/Twisted. This way
you're using the .py files in place too, removing the confusion a forgotten
rebuild could cause with the separate build/ directory above. To build the C
modules in place, do ./setup.py build_ext -i. You only need to
re-run this command when you change the C files. Note that
setup.py is not Make, it does not always get the dependencies
right (.h files in particular), so if you are hacking on the
cReactor you may need to manually delete the .o files before
doing a rebuild. Also note that doing a setup.py clean will
remove the .o files but not the final .so files,
they must be deleted by hand.
Running tests
To run the full unit-test suite, do:
./bin/trial -v twisted.test
To run a single test file (like twisted/test/test_defer.py),
do one of:
./bin/trial -v twisted.test.test_defer
or
./bin/trial -v twisted/test/test_defer.py
To run any tests that are related to a code file, like
twisted/protocols/imap4.py, do:
./bin/trial -v --testmodule twisted/protocols/imap4.py
This depends upon the .py file having an appropriate
test-case-name
tag that indicates which test cases provide coverage.
See the Test Standards document for
details about using test-case-name
. In this example, the
twisted.test.test_imap test will be run.
Many tests create temporary files in /tmp or ./_trial_temp, but
everything in /tmp should be deleted when the test finishes. Sometimes these
cleanup calls are commented out by mistake, so if you see a stray
/tmp/@12345.1 directory, it is probably from test_dirdbm or test_popsicle.
Look for an rmtree that has been commented out and complain to
the last developer who touched that file.
Admin scripts
The admin/ directory holds several administrative tools,
some of which are used when turning a Subversion checkout into a full numbered
release.
Building docs
Twisted documentation (not including the automatically-generated API docs)
is in Lore Format. These
.xhtml files are translated into .html files by the
bin/lore/lore
script, which can check the files for syntax problems
(hlint), process multiple files at once, insert the files into a template
before processing, and can also translate the files into LaTeX or PostScript
instead.
To generate the full documentation set, run the
admin/process-docs shell script. This will create processed
HTML, man pages, and 250-page book.pdf
file containing all the docs
rolled into a single nicely-formatted volume. This script needs several
helper tools to handle the images and the LaTeX conversion: debian packages
tetex-extra
, netpbm
, and gs-common
should be
sufficient. The docs-build process currently takes about 3 minutes on the
twistedmatrix.com build machine.
To build just the HTML form of the howto/ docs, do a subset of the work
done in admin/process-docs, such as the following. Note that
the index file will be placed in doc/howto/index.html.
./bin/lore/lore -p --config template=doc/howto/template.tpl doc/howto/*.xhtml
To run hlint over a single Lore document, such as
doc/development/policy/svn-dev.xhtml, do the following. This is
useful because the HTML conversion may bail without a useful explanation if
it sees mismatched tags.
./bin/lore/lore -n --output lint doc/development/policy/svn-dev.xhtml
To convert it to HTML (including markup, interpolation of examples,
footnote processing, etc), do the following. The results will be placed in
doc/development/policy/svn-dev.html:
./bin/lore/lore -p --config template=doc/howto/template.tpl \ doc/development/policy/svn-dev.xhtml
Note that hyperlinks to other documents may not be quite right unless you
include a -l
argument to bin/lore/lore. Links in the
.xhtml file are to .xhtml targets: when the .xhtml is turned into .html, the
link targets are supposed to be turned into .html also. In addition to this,
Lore markup of the form <code class="API"> is supposed to
turn into a link to the corresponding API reference page. These links will
probably be wrong unless the correct base URL is provided to Lore.
Emacs
Check out the TwistedEmacs module, which lives in the same Subversion
repository (TODO this is not true), just do cvs checkout TwistedEmacs
instead of cvs checkout Twisted
). twisted-dev.el has
several utility functions which make it easier to grep for methods, run test
cases, process Lore documents, etc.
Building Debian packages
Running debuild -uc -us
from the top of the Subversion tree will
(hopefully) result in a collection of .deb packages in the tree's parent
directory. This requires other tools to be installed (devscripts for one),
and requires that admin/process-docs
be run first. The .debs created
will have a version number based upon whatever is at the top of
debian/changelog, which is generally only updated when an
official release is made, so be careful that you don't create
confusingly-numbered package files.