diff options
Diffstat (limited to 'Doc/howto')
| -rw-r--r-- | Doc/howto/cporting.rst | 19 | ||||
| -rw-r--r-- | Doc/howto/descriptor.rst | 12 | ||||
| -rw-r--r-- | Doc/howto/functional.rst | 16 | ||||
| -rw-r--r-- | Doc/howto/index.rst | 1 | ||||
| -rw-r--r-- | Doc/howto/ipaddress.rst | 341 | ||||
| -rw-r--r-- | Doc/howto/logging-cookbook.rst | 29 | ||||
| -rw-r--r-- | Doc/howto/sockets.rst | 36 | ||||
| -rw-r--r-- | Doc/howto/unicode.rst | 5 | ||||
| -rw-r--r-- | Doc/howto/urllib2.rst | 7 |
9 files changed, 392 insertions, 74 deletions
diff --git a/Doc/howto/cporting.rst b/Doc/howto/cporting.rst index 9d8a1b0..1ad77d6 100644 --- a/Doc/howto/cporting.rst +++ b/Doc/howto/cporting.rst @@ -100,25 +100,6 @@ corresponds to Python 2's :func:`long` type--the :func:`int` type used in Python 2 was removed. In the C-API, ``PyInt_*`` functions are replaced by their ``PyLong_*`` equivalents. -The best course of action here is using the ``PyInt_*`` functions aliased to -``PyLong_*`` found in :file:`intobject.h`. The abstract ``PyNumber_*`` APIs -can also be used in some cases. :: - - #include "Python.h" - #include "intobject.h" - - static PyObject * - add_ints(PyObject *self, PyObject *args) { - int one, two; - PyObject *result; - - if (!PyArg_ParseTuple(args, "ii:add_ints", &one, &two)) - return NULL; - - return PyInt_FromLong(one + two); - } - - Module initialization and state =============================== diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst index 1616f67..0b513f9 100644 --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -36,9 +36,7 @@ continuing through the base classes of ``type(a)`` excluding metaclasses. If the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which descriptor methods -were defined. Note that descriptors are only invoked for new style objects or -classes (a class is new style if it inherits from :class:`object` or -:class:`type`). +were defined. Descriptors are a powerful, general purpose protocol. They are the mechanism behind properties, methods, static methods, class methods, and :func:`super()`. @@ -89,8 +87,6 @@ of ``obj``. If ``d`` defines the method :meth:`__get__`, then ``d.__get__(obj)` is invoked according to the precedence rules listed below. The details of invocation depend on whether ``obj`` is an object or a class. -Either way, descriptors only work for new style objects and classes. A class is -new style if it is a subclass of :class:`object`. For objects, the machinery is in :meth:`object.__getattribute__` which transforms ``b.x`` into ``type(b).__dict__['x'].__get__(b, type(b))``. The @@ -115,7 +111,6 @@ The important points to remember are: * descriptors are invoked by the :meth:`__getattribute__` method * overriding :meth:`__getattribute__` prevents automatic descriptor calls -* :meth:`__getattribute__` is only available with new style classes and objects * :meth:`object.__getattribute__` and :meth:`type.__getattribute__` make different calls to :meth:`__get__`. * data descriptors always override instance dictionaries. @@ -128,10 +123,7 @@ and then returns ``A.__dict__['m'].__get__(obj, A)``. If not a descriptor, ``m`` is returned unchanged. If not in the dictionary, ``m`` reverts to a search using :meth:`object.__getattribute__`. -Note, in Python 2.2, ``super(B, obj).m()`` would only invoke :meth:`__get__` if -``m`` was a data descriptor. In Python 2.3, non-data descriptors also get -invoked unless an old-style class is involved. The implementation details are -in :c:func:`super_getattro()` in +The implementation details are in :c:func:`super_getattro()` in `Objects/typeobject.c <http://svn.python.org/view/python/trunk/Objects/typeobject.c?view=markup>`_ and a pure Python equivalent can be found in `Guido's Tutorial`_. diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index ebbb229..d241f1a 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -292,13 +292,14 @@ ordering of the objects in the dictionary. Applying :func:`iter` to a dictionary always loops over the keys, but dictionaries have methods that return other iterators. If you want to iterate over values or key/value pairs, you can explicitly call the -:meth:`~dict.values` or :meth:`~dict.items` methods to get an appropriate iterator. +:meth:`~dict.values` or :meth:`~dict.items` methods to get an appropriate +iterator. The :func:`dict` constructor can accept an iterator that returns a finite stream of ``(key, value)`` tuples: >>> L = [('Italy', 'Rome'), ('France', 'Paris'), ('US', 'Washington DC')] - >>> dict(iter(L)) + >>> dict(iter(L)) #doctest: +SKIP {'Italy': 'Rome', 'US': 'Washington DC', 'France': 'Paris'} Files also support iteration by calling the :meth:`~io.TextIOBase.readline` @@ -478,13 +479,10 @@ Here's a sample usage of the ``generate_ints()`` generator: You could equally write ``for i in generate_ints(5)``, or ``a,b,c = generate_ints(3)``. -Inside a generator function, the ``return`` statement can only be used without a -value, and signals the end of the procession of values; after executing a -``return`` the generator cannot return any further values. ``return`` with a -value, such as ``return 5``, is a syntax error inside a generator function. The -end of the generator's results can also be indicated by raising -:exc:`StopIteration` manually, or by just letting the flow of execution fall off -the bottom of the function. +Inside a generator function, ``return value`` is semantically equivalent to +``raise StopIteration(value)``. If no value is returned or the bottom of the +function is reached, the procession of values ends and the generator cannot +return any further values. You could achieve the effect of generators manually by writing your own class and storing all the local variables of the generator as instance variables. For diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst index a11d3da..f44e8c0 100644 --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -28,4 +28,5 @@ Currently, the HOWTOs are: urllib2.rst webservers.rst argparse.rst + ipaddress.rst diff --git a/Doc/howto/ipaddress.rst b/Doc/howto/ipaddress.rst new file mode 100644 index 0000000..5e0ff3e --- /dev/null +++ b/Doc/howto/ipaddress.rst @@ -0,0 +1,341 @@ +.. _ipaddress-howto: + +*************************************** +An introduction to the ipaddress module +*************************************** + +:author: Peter Moody +:author: Nick Coghlan + +.. topic:: Overview + + This document aims to provide a gentle introduction to the + :mod:`ipaddress` module. It is aimed primarily at users that aren't + already familiar with IP networking terminology, but may also be useful + to network engineers wanting an overview of how :mod:`ipaddress` + represents IP network addressing concepts. + + +Creating Address/Network/Interface objects +========================================== + +Since :mod:`ipaddress` is a module for inspecting and manipulating IP addresses, +the first thing you'll want to do is create some objects. You can use +:mod:`ipaddress` to create objects from strings and integers. + + +A Note on IP Versions +--------------------- + +For readers that aren't particularly familiar with IP addressing, it's +important to know that the Internet Protocol is currently in the process +of moving from version 4 of the protocol to version 6. This transition is +occurring largely because version 4 of the protocol doesn't provide enough +addresses to handle the needs of the whole world, especially given the +increasing number of devices with direct connections to the internet. + +Explaining the details of the differences between the two versions of the +protocol is beyond the scope of this introduction, but readers need to at +least be aware that these two versions exist, and it will sometimes be +necessary to force the use of one version or the other. + + +IP Host Addresses +----------------- + +Addresses, often referred to as "host addresses" are the most basic unit +when working with IP addressing. The simplest way to create addresses is +to use the :func:`ipaddress.ip_address` factory function, which automatically +determines whether to create an IPv4 or IPv6 address based on the passed in +value: + +.. testsetup:: + >>> import ipaddress + +:: + + >>> ipaddress.ip_address('192.0.2.1') + IPv4Address('192.0.2.1') + >>> ipaddress.ip_address('2001:DB8::1') + IPv6Address('2001:db8::1') + +Addresses can also be created directly from integers. Values that will +fit within 32 bits are assumed to be IPv4 addresses:: + + >>> ipaddress.ip_address(3221225985) + IPv4Address('192.0.2.1') + >>> ipaddress.ip_address(42540766411282592856903984951653826561) + IPv6Address('2001:db8::1') + +To force the use of IPv4 or IPv6 addresses, the relevant classes can be +invoked directly. This is particularly useful to force creation of IPv6 +addresses for small integers:: + + >>> ipaddress.ip_address(1) + IPv4Address('0.0.0.1') + >>> ipaddress.IPv4Address(1) + IPv4Address('0.0.0.1') + >>> ipaddress.IPv6Address(1) + IPv6Address('::1') + + +Defining Networks +----------------- + +Host addresses are usually grouped together into IP networks, so +:mod:`ipaddress` provides a way to create, inspect and manipulate network +definitions. IP network objects are constructed from strings that define the +range of host addresses that are part of that network. The simplest form +for that information is a "network address/network prefix" pair, where the +prefix defines the number of leading bits that are compared to determine +whether or not an address is part of the network and the network address +defines the expected value of those bits. + +As for addresses, a factory function is provided that determines the correct +IP version automatically:: + + >>> ipaddress.ip_network('192.0.2.0/24') + IPv4Network('192.0.2.0/24') + >>> ipaddress.ip_network('2001:db8::0/96') + IPv6Network('2001:db8::/96') + +Network objects cannot have any host bits set. The practical effect of this +is that ``192.0.2.1/24`` does not describe a network. Such definitions are +referred to as interface objects since the ip-on-a-network notation is +commonly used to describe network interfaces of a computer on a given network +and are described further in the next section. + +By default, attempting to create a network object with host bits set will +result in :exc:`ValueError` being raised. To request that the +additional bits instead be coerced to zero, the flag ``strict=False`` can +be passed to the constructor:: + + >>> ipaddress.ip_network('192.0.2.1/24') + Traceback (most recent call last): + ... + ValueError: 192.0.2.1/24 has host bits set + >>> ipaddress.ip_network('192.0.2.1/24', strict=False) + IPv4Network('192.0.2.0/24') + +While the string form offers significantly more flexibility, networks can +also be defined with integers, just like host addresses. In this case, the +network is considered to contain only the single address identified by the +integer, so the network prefix includes the entire network address:: + + >>> ipaddress.ip_network(3221225984) + IPv4Network('192.0.2.0/32') + >>> ipaddress.ip_network(42540766411282592856903984951653826560) + IPv6Network('2001:db8::/128') + +As with addresses, creation of a particular kind of network can be forced +by calling the class constructor directly instead of using the factory +function. + + +Host Interfaces +--------------- + +As mentioned just above, if you need to describe an address on a particular +network, neither the address nor the network classes are sufficient. +Notation like ``192.0.2.1/24`` is commonly used by network engineers and the +people who write tools for firewalls and routers as shorthand for "the host +``192.0.2.1`` on the network ``192.0.2.0/24``", Accordingly, :mod:`ipaddress` +provides a set of hybrid classes that associate an address with a particular +network. The interface for creation is identical to that for defining network +objects, except that the address portion isn't constrained to being a network +address. + + >>> ipaddress.ip_interface('192.0.2.1/24') + IPv4Interface('192.0.2.1/24') + >>> ipaddress.ip_interface('2001:db8::1/96') + IPv6Interface('2001:db8::1/96') + +Integer inputs are accepted (as with networks), and use of a particular IP +version can be forced by calling the relevant constructor directly. + + +Inspecting Address/Network/Interface Objects +============================================ + +You've gone to the trouble of creating an IPv(4|6)(Address|Network|Interface) +object, so you probably want to get information about it. :mod:`ipaddress` +tries to make doing this easy and intuitive. + +Extracting the IP version:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> addr6 = ipaddress.ip_address('2001:db8::1') + >>> addr6.version + 6 + >>> addr4.version + 4 + +Obtaining the network from an interface:: + + >>> host4 = ipaddress.ip_interface('192.0.2.1/24') + >>> host4.network + IPv4Network('192.0.2.0/24') + >>> host6 = ipaddress.ip_interface('2001:db8::1/96') + >>> host6.network + IPv6Network('2001:db8::/96') + +Finding out how many individual addresses are in a network:: + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> net4.num_addresses + 256 + >>> net6 = ipaddress.ip_network('2001:db8::0/96') + >>> net6.num_addresses + 4294967296 + +Iterating through the "usable" addresses on a network:: + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> for x in net4.hosts(): + ... print(x) # doctest: +ELLIPSIS + 192.0.2.1 + 192.0.2.2 + 192.0.2.3 + 192.0.2.4 + ... + 192.0.2.252 + 192.0.2.253 + 192.0.2.254 + + +Obtaining the netmask (i.e. set bits corresponding to the network prefix) or +the hostmask (any bits that are not part of the netmask): + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> net4.netmask + IPv4Address('255.255.255.0') + >>> net4.hostmask + IPv4Address('0.0.0.255') + >>> net6 = ipaddress.ip_network('2001:db8::0/96') + >>> net6.netmask + IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff::') + >>> net6.hostmask + IPv6Address('::ffff:ffff') + + +Exploding or compressing the address:: + + >>> addr6.exploded + '2001:0db8:0000:0000:0000:0000:0000:0001' + >>> addr6.compressed + '2001:db8::1' + >>> net6.exploded + '2001:0db8:0000:0000:0000:0000:0000:0000/96' + >>> net6.compressed + '2001:db8::/96' + +While IPv4 doesn't support explosion or compression, the associated objects +still provide the relevant properties so that version neutral code can +easily ensure the most concise or most verbose form is used for IPv6 +addresses while still correctly handling IPv4 addresses. + + +Networks as lists of Addresses +============================== + +It's sometimes useful to treat networks as lists. This means it is possible +to index them like this:: + + >>> net4[1] + IPv4Address('192.0.2.1') + >>> net4[-1] + IPv4Address('192.0.2.255') + >>> net6[1] + IPv6Address('2001:db8::1') + >>> net6[-1] + IPv6Address('2001:db8::ffff:ffff') + + +It also means that network objects lend themselves to using the list +membership test syntax like this:: + + if address in network: + # do something + +Containment testing is done efficiently based on the network prefix:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> addr4 in ipaddress.ip_network('192.0.2.0/24') + True + >>> addr4 in ipaddress.ip_network('192.0.3.0/24') + False + + +Comparisons +=========== + +:mod:`ipaddress` provides some simple, hopefully intuitive ways to compare +objects, where it makes sense:: + + >>> ipaddress.ip_address('192.0.2.1') < ipaddress.ip_address('192.0.2.2') + True + +A :exc:`TypeError` exception is raised if you try to compare objects of +different versions or different types. + + +Using IP Addresses with other modules +===================================== + +Other modules that use IP addresses (such as :mod:`socket`) usually won't +accept objects from this module directly. Instead, they must be coerced to +an integer or string that the other module will accept:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> str(addr4) + '192.0.2.1' + >>> int(addr4) + 3221225985 + + +Getting more detail when instance creation fails +================================================ + +When creating address/network/interface objects using the version-agnostic +factory functions, any errors will be reported as :exc:`ValueError` with +a generic error message that simply says the passed in value was not +recognized as an object of that type. The lack of a specific error is +because it's necessary to know whether the value is *supposed* to be IPv4 +or IPv6 in order to provide more detail on why it has been rejected. + +To support use cases where it is useful to have access to this additional +detail, the individual class constructors actually raise the +:exc:`ValueError` subclasses :exc:`ipaddress.AddressValueError` and +:exc:`ipaddress.NetmaskValueError` to indicate exactly which part of +the definition failed to parse correctly. + +The error messages are significantly more detailed when using the +class constructors directly. For example:: + + >>> ipaddress.ip_address("192.168.0.256") + Traceback (most recent call last): + ... + ValueError: '192.168.0.256' does not appear to be an IPv4 or IPv6 address + >>> ipaddress.IPv4Address("192.168.0.256") + Traceback (most recent call last): + ... + ipaddress.AddressValueError: Octet 256 (> 255) not permitted in '192.168.0.256' + + >>> ipaddress.ip_network("192.168.0.1/64") + Traceback (most recent call last): + ... + ValueError: '192.168.0.1/64' does not appear to be an IPv4 or IPv6 network + >>> ipaddress.IPv4Network("192.168.0.1/64") + Traceback (most recent call last): + ... + ipaddress.NetmaskValueError: '64' is not a valid netmask + +However, both of the module specific exceptions have :exc:`ValueError` as their +parent class, so if you're not concerned with the particular type of error, +you can still write code like the following:: + + try: + network = ipaddress.IPv4Network(address) + except ValueError: + print('address/netmask is invalid for IPv4:', address) + diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index 370c757..92af0ec 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -1316,6 +1316,33 @@ For more information about this configuration, you can see the `relevant section <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_ of the Django documentation. +.. _cookbook-rotator-namer: + +Using a rotator and namer to customise log rotation processing +-------------------------------------------------------------- + +An example of how you can define a namer and rotator is given in the following +snippet, which shows zlib-based compression of the log file:: + + def namer(name): + return name + ".gz" + + def rotator(source, dest): + with open(source, "rb") as sf: + data = sf.read() + compressed = zlib.compress(data, 9) + with open(dest, "wb") as df: + df.write(compressed) + os.remove(source) + + rh = logging.handlers.RotatingFileHandler(...) + rh.rotator = rotator + rh.namer = namer + +These are not "true" .gz files, as they are bare compressed data, with no +"container" such as you’d find in an actual gzip file. This snippet is just +for illustration purposes. + A more elaborate multiprocessing example ---------------------------------------- @@ -1572,7 +1599,7 @@ UTF-8, then you need to do the following: 'ASCII section\ufeffUnicode section' - The Unicode code point ``'\feff```, when encoded using UTF-8, will be + The Unicode code point ``'\feff'``, when encoded using UTF-8, will be encoded as a UTF-8 BOM -- the byte-string ``b'\xef\xbb\xbf'``. #. Replace the ASCII section with whatever placeholders you like, but make sure diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst index 279bb3e..ca6528b 100644 --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -25,8 +25,8 @@ It's not really a tutorial - you'll still have work to do in getting things working. It doesn't cover the fine points (and there are a lot of them), but I hope it will give you enough background to begin using them decently. -I'm only going to talk about INET sockets, but they account for at least 99% of -the sockets in use. And I'll only talk about STREAM sockets - unless you really +I'm only going to talk about INET (i.e. IPv4) sockets, but they account for at least 99% of +the sockets in use. And I'll only talk about STREAM (i.e. TCP) sockets - unless you really know what you're doing (in which case this HOWTO isn't for you!), you'll get better behavior and performance from a STREAM socket than anything else. I will try to clear up the mystery of what a socket is, as well as some hints on how to @@ -208,10 +208,10 @@ length message:: totalsent = totalsent + sent def myreceive(self): - msg = '' + msg = b'' while len(msg) < MSGLEN: chunk = self.sock.recv(MSGLEN-len(msg)) - if chunk == '': + if chunk == b'': raise RuntimeError("socket connection broken") msg = msg + chunk return msg @@ -371,12 +371,6 @@ have created a new socket to ``connect`` to someone else, put it in the potential_writers list. If it shows up in the writable list, you have a decent chance that it has connected. -One very nasty problem with ``select``: if somewhere in those input lists of -sockets is one which has died a nasty death, the ``select`` will fail. You then -need to loop through every single damn socket in all those lists and do a -``select([sock],[],[],0)`` until you find the bad one. That timeout of 0 means -it won't take long, but it's ugly. - Actually, ``select`` can be handy even with blocking sockets. It's one way of determining whether you will block - the socket returns as readable when there's something in the buffers. However, this still doesn't help with the problem of @@ -386,26 +380,6 @@ determining whether the other end is done, or just busy with something else. files. Don't try this on Windows. On Windows, ``select`` works with sockets only. Also note that in C, many of the more advanced socket options are done differently on Windows. In fact, on Windows I usually use threads (which work -very, very well) with my sockets. Face it, if you want any kind of performance, -your code will look very different on Windows than on Unix. - - -Performance ------------ +very, very well) with my sockets. -There's no question that the fastest sockets code uses non-blocking sockets and -select to multiplex them. You can put together something that will saturate a -LAN connection without putting any strain on the CPU. - -The trouble is that an app written this way can't do much of anything else - -it needs to be ready to shuffle bytes around at all times. Assuming that your -app is actually supposed to do something more than that, threading is the -optimal solution, (and using non-blocking sockets will be faster than using -blocking sockets). - -Finally, remember that even though blocking sockets are somewhat slower than -non-blocking, in many cases they are the "right" solution. After all, if your -app is driven by the data it receives over a socket, there's not much sense in -complicating the logic just so your app can wait on ``select`` instead of -``recv``. diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst index b309f60..7500dce 100644 --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -422,7 +422,7 @@ References ---------- The :class:`str` type is described in the Python library reference at -:ref:`typesseq`. +:ref:`textseq`. The documentation for the :mod:`unicodedata` module. @@ -618,7 +618,6 @@ Marc-André Lemburg, Martin von Löwis, Chad Whitacre. and that the HOWTO only covers 2.x. .. comment Describe Python 3.x support (new section? new document?) -.. comment Additional topic: building Python w/ UCS2 or UCS4 support .. comment Describe use of codecs.StreamRecoder and StreamReaderWriter .. comment @@ -648,5 +647,3 @@ Marc-André Lemburg, Martin von Löwis, Chad Whitacre. - [ ] Writing Unicode programs - [ ] Do everything in Unicode - [ ] Declaring source code encodings (PEP 263) - - [ ] Other issues - - [ ] Building Python (UCS2, UCS4) diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst index 87f42ba..955e455 100644 --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -56,6 +56,13 @@ The simplest way to use urllib.request is as follows:: response = urllib.request.urlopen('http://python.org/') html = response.read() +If you wish to retrieve a resource via URL and store it in a temporary location, +you can do so via the :func:`urlretrieve` function:: + + import urllib.request + local_filename, headers = urllib.request.urlretrieve('http://python.org/') + html = open(local_filename) + Many uses of urllib will be that simple (note that instead of an 'http:' URL we could have used an URL starting with 'ftp:', 'file:', etc.). However, it's the purpose of this tutorial to explain the more complicated cases, concentrating on |