jupyter-notebooks/http/furl/readme.md

<h1 align="center">
  <img src="logo.svg" width="360px" height="123px" alt="furl">
</h1>

<p align="center">
  <a href="https://pypi.python.org/pypi/furl"><img src="https://badge.fury.io/py/furl.svg"></a>
  <a href="https://travis-ci.org/gruns/furl"><img src="https://img.shields.io/travis/gruns/furl.svg"></a>
  <a href="http://unlicense.org/"><img src="https://img.shields.io/pypi/l/furl.svg"></a>
  <a href="https://pypi.python.org/pypi/furl"><img src="https://img.shields.io/pypi/pyversions/furl.svg"></a>
</p>

## furl is a small Python library that makes parsing and<br>manipulating URLs easy.

Python's standard [urllib](https://docs.python.org/3/library/urllib.html) and
[urlparse](https://docs.python.org/3/library/urllib.parse.html) modules provide a
number of URL\
related functions, but using these functions to perform common
URL\
operations proves tedious. Furl makes parsing and manipulating URLs\
easy.

Furl is well tested, [Unlicensed](http://unlicense.org/) in the public domain,
and supports\
Python 2, Python 3, PyPy2, and PyPy3.

Code time: Paths and query arguments are easy. Really easy.

```python
>>> from furl import furl
>>> f = furl('http://www.google.com/?one=1&two=2')
>>> f /= 'path'
>>> del f.args['one']
>>> f.args['three'] = '3'
>>> f.url
'http://www.google.com/path?two=2&three=3'
```

Or use furl's inline modification methods.

```python
>>> furl('http://www.google.com/?one=1').add({'two':'2'}).url
'http://www.google.com/?one=1&two=2'

>>> furl('http://www.google.com/?one=1&two=2').set({'three':'3'}).url
'http://www.google.com/?three=3'

>>> furl('http://www.google.com/?one=1&two=2').remove(['one']).url
'http://www.google.com/?two=2'
```

Encoding is handled for you. Unicode, too.

```python
>>> f = furl('http://www.google.com/')
>>> f.path = 'some encoding here'
>>> f.args['and some encoding'] = 'here, too'
>>> f.url
'http://www.google.com/some%20encoding%20here?and+some+encoding=here,+too'
>>> f.set(host=u'ドメイン.テスト', path=u'джк', query=u'☃=☺')
>>> f.url
'http://xn--eckwd4c7c.xn--zckzah/%D0%B4%D0%B6%D0%BA?%E2%98%83=%E2%98%BA'
```

Fragments also have a path and a query.

```python
>>> f = furl('http://www.google.com/')
>>> f.fragment.path.segments = ['two', 'directories']
>>> f.fragment.args = {'one': 'argument'}
>>> f.url
'http://www.google.com/#two/directories?one=argument'
```

## Installation

Installing furl with pip is easy.

```
$ pip install furl
```


## API

  * [Basics](#basics)
  * [Scheme, Username, Password, Host, Port, Network Location, and Origin](#scheme-username-password-host-port-network-location-and-origin)
  * [Path](#path)
    * [Manipulation](#manipulation)
  * [Query](#query)
    * [Manipulation](#manipulation-1)
    * [Parameters](#parameters)
  * [Fragment](#fragment)
  * [Encoding](#encoding)
  * [Inline manipulation](#inline-manipulation)
  * [Miscellaneous](#miscellaneous)


### Basics

furl objects let you access and modify the various components of a URL.

```
scheme://username:password@host:port/path?query#fragment
```

 * __scheme__ is the scheme string (all lowercase) or None. None means no
   scheme. An empty string means a protocol relative URL, like
   `//www.google.com`.
 * __username__ is the username string for authentication.
 * __password__ is the password string for authentication with __username__.
 * __host__ is the domain name, IPv4, or IPv6 address as a string. Domain names
  are all lowercase.
 * __port__ is an integer or None. A value of None means no port specified and
  the default port for the given __scheme__ should be inferred, if possible
  (e.g. port 80 for the scheme `http`).
 * __path__ is a Path object comprised of path segments.
 * __query__ is a Query object comprised of key:value query arguments.
 * __fragment__ is a Fragment object comprised of a Path object and Query object
   separated by an optional `?` separator.



### Scheme, Username, Password, Host, Port, Network Location, and Origin

__scheme__, __username__, __password__, and __host__ are strings or
None. __port__ is an integer or None.

```python
>>> f = furl('http://user:pass@www.google.com:99/')
>>> f.scheme, f.username, f.password, f.host, f.port
('http', 'user', 'pass', 'www.google.com', 99)
```

furl infers the default port for common schemes.

```python
>>> f = furl('https://secure.google.com/')
>>> f.port
443

>>> f = furl('unknown://www.google.com/')
>>> print(f.port)
None
```

__netloc__ is the string combination of __username__, __password__, __host__,
and __port__, not including __port__ if it's None or the default port for the
provided __scheme__.

```python
>>> furl('http://www.google.com/').netloc
'www.google.com'

>>> furl('http://www.google.com:99/').netloc
'www.google.com:99'

>>> furl('http://user:pass@www.google.com:99/').netloc
'user:pass@www.google.com:99'
```

__origin__ is the string combination of __scheme__, __host__, and __port__, not
including __port__ if it's None or the default port for the provided __scheme__.

```python
>>> furl('http://www.google.com/').origin
'http://www.google.com'

>>> furl('http://www.google.com:99/').origin
'http://www.google.com:99'
```



### Path

URL paths in furl are Path objects that have __segments__, a list of zero or
more path segments that can be manipulated directly. Path segments in
__segments__ are percent-decoded and all interaction with __segments__ should
take place with percent-decoded strings.

```python
>>> f = furl('http://www.google.com/a/large%20ish/path')
>>> f.path
Path('/a/large ish/path')
>>> f.path.segments
['a', 'large ish', 'path']
>>> str(f.path)
'/a/large%20ish/path'
```

#### Manipulation

```python
>>> f.path.segments = ['a', 'new', 'path', '']
>>> str(f.path)
'/a/new/path/'

>>> f.path = 'o/hi/there/with%20some%20encoding/'
>>> f.path.segments
['o', 'hi', 'there', 'with some encoding', '']
>>> str(f.path)
'/o/hi/there/with%20some%20encoding/'

>>> f.url
'http://www.google.com/o/hi/there/with%20some%20encoding/'

>>> f.path.segments = ['segments', 'are', 'maintained', 'decoded', '^`<>[]"#/?']
>>> str(f.path)
'/segments/are/maintained/decoded/%5E%60%3C%3E%5B%5D%22%23%2F%3F'
```

A path that starts with `/` is considered absolute, and a Path can be absolute
or not as specified (or set) by the boolean attribute __isabsolute__. URL Paths
have a special restriction: they must be absolute if a __netloc__ (username,
password, host, and/or port) is present. This restriction exists because a URL
path must start with `/` to separate itself from the __netloc__, if
present. Fragment Paths have no such limitation and __isabsolute__ and can be
True or False without restriction.

Here's a URL Path example that illustrates how __isabsolute__ becomes True and
read-only in the presence of a __netloc__.

```python
>>> f = furl('/url/path')
>>> f.path.isabsolute
True
>>> f.path.isabsolute = False
>>> f.url
'url/path'
>>> f.host = 'blaps.ru'
>>> f.url
'blaps.ru/url/path'
>>> f.path.isabsolute
True
>>> f.path.isabsolute = False
Traceback (most recent call last):
  ...
AttributeError: Path.isabsolute is True and read-only for URLs with a netloc (a username, password, host, and/or port). URL paths must be absolute if a netloc exists.
>>> f.url
'blaps.ru/url/path'
```

Conversely, the __isabsolute__ attribute of Fragment Paths isn't bound by the
same read-only restriction. URL fragments are always prefixed by a `#` character
and don't need to be separated from the __netloc__.

```python
>>> f = furl('http://www.google.com/#/absolute/fragment/path/')
>>> f.fragment.path.isabsolute
True
>>> f.fragment.path.isabsolute = False
>>> f.url
'http://www.google.com/#absolute/fragment/path/'
>>> f.fragment.path.isabsolute = True
>>> f.url
'http://www.google.com/#/absolute/fragment/path/'
```

A path that ends with `/` is considered a directory, and otherwise considered a
file. The Path attribute __isdir__ returns True if the path is a directory,
False otherwise. Conversely, the attribute __isfile__ returns True if the path
is a file, False otherwise.

```python
>>> f = furl('http://www.google.com/a/directory/')
>>> f.path.isdir
True
>>> f.path.isfile
False

>>> f = furl('http://www.google.com/a/file')
>>> f.path.isdir
False
>>> f.path.isfile
True
```

A path can be normalized with __normalize()__, and __normalize()__ returns the
Path object for method chaining.

```python
>>> f = furl('http://www.google.com////a/./b/lolsup/../c/')
>>> f.path.normalize()
>>> f.url
'http://www.google.com/a/b/c/'
```

Path segments can also be appended with the slash operator, like with
[pathlib.Path](https://docs.python.org/3/library/pathlib.html#operators).

```python
>>> from __future__ import division  # For Python 2.x.
>>>
>>> f = furl('path')
>>> f.path /= 'with'
>>> f.path = f.path / 'more' / 'path segments/'
>>> f.url
'/path/with/more/path%20segments/'
```

For a dictionary representation of a path, use __asdict()__.

```python
>>> f = furl('http://www.google.com/some/enc%20oding')
>>> f.path.asdict()
{ 'encoded': '/some/enc%20oding',
  'isabsolute': True,
  'isdir': False,
  'isfile': True,
  'segments': ['some', 'enc oding'] }
```



### Query

URL queries in furl are Query objects that have __params__, a one dimensional
[ordered multivalue dictionary](https://github.com/gruns/orderedmultidict) of
query keys and values. Query keys and values in __params__ are percent-decoded
and all interaction with __params__ should take place with percent-decoded
strings.

```python
>>> f = furl('http://www.google.com/?one=1&two=2')
>>> f.query
Query('one=1&two=2')
>>> f.query.params
omdict1D([('one', '1'), ('two', '2')])
>>> str(f.query)
'one=1&two=2'
```

furl objects and Fragment objects (covered below) contain a Query object, and
__args__ is provided as a shortcut on these objects to access __query.params__.

```python
>>> f = furl('http://www.google.com/?one=1&two=2')
>>> f.query.params
omdict1D([('one', '1'), ('two', '2')])
>>> f.args
omdict1D([('one', '1'), ('two', '2')])
>>> f.args is f.query.params
True
```

#### Manipulation

__params__ is a one dimensional
[ordered multivalue dictionary](https://github.com/gruns/orderedmultidict) that
maintains method parity with Python's standard dictionary.

```python
>>> f.query = 'silicon=14&iron=26&inexorable%20progress=vae%20victus'
>>> f.query.params
omdict1D([('silicon', '14'), ('iron', '26'), ('inexorable progress', 'vae victus')])
>>> del f.args['inexorable progress']
>>> f.args['magnesium'] = '12'
>>> f.args
omdict1D([('silicon', '14'), ('iron', '26'), ('magnesium', '12')])
```

__params__ can also store multiple values for the same key because it's a
multivalue dictionary.

```python
>>> f = furl('http://www.google.com/?space=jams&space=slams')
>>> f.args['space']
'jams'
>>> f.args.getlist('space')
['jams', 'slams']
>>> f.args.addlist('repeated', ['1', '2', '3'])
>>> str(f.query)
'space=jams&space=slams&repeated=1&repeated=2&repeated=3'
>>> f.args.popvalue('space')
'slams'
>>> f.args.popvalue('repeated', '2')
'2'
>>> str(f.query)
'space=jams&repeated=1&repeated=3'
```

__params__ is one dimensional. If a list of values is provided as a query value,
that list is interpreted as multiple values.

```python
>>> f = furl()
>>> f.args['repeated'] = ['1', '2', '3']
>>> f.add(args={'space':['jams', 'slams']})
>>> str(f.query)
'repeated=1&repeated=2&repeated=3&space=jams&space=slams'
```

This makes sense: URL queries are inherently one dimensional -- query values
can't have native subvalues.

See the [orderedmultimdict](https://github.com/gruns/orderedmultidict)
documentation for more information on interacting with the ordered multivalue
dictionary __params__.

#### Parameters

To produce an empty query argument, like `http://sprop.su/?param=`, set the
argument's value to the empty string.

```python
>>> f = furl('http://sprop.su')
>>> f.args['param'] = ''
>>> f.url
'http://sprop.su/?param='
```

To produce an empty query argument without a trailing `=`, use `None` as the
parameter value.

```python
>>> f = furl('http://sprop.su')
>>> f.args['param'] = None
>>> f.url
'http://sprop.su/?param'
```

__encode(delimiter='&', quote_plus=True, dont_quote='')__ can be used to encode
query strings with delimiters like `;`, encode spaces as `+` instead of `%20`
(i.e. application/x-www-form-urlencoded encoded), or avoid percent-encoding
valid query characters entirely (valid query characters are
`/?:@-._~!$&'()*+,;=`).

```python
>>> f.query = 'space=jams&woofs=squeeze+dog'
>>> f.query.encode()
'space=jams&woofs=squeeze+dog'
>>> f.query.encode(';')
'space=jams;woofs=squeeze+dog'
>>> f.query.encode(quote_plus=False)
'space=jams&woofs=squeeze%20dog'
```

`dont_quote` accepts `True`, `False`, or a string of valid query characters to
not percent-enode. If `True`, all valid query characters `/?:@-._~!$&'()*+,;=`
aren't percent-encoded.

```python
>>> f.query = 'one,two/three'
>>> f.query.encode()
'one%2Ctwo%2Fthree'
>>> f.query.encode(dont_quote=True)
'one,two/three'
>>> f.query.encode(dont_quote=',')
'one,two%2Fthree'
```

For a dictionary representation of a query, use __asdict()__.

```python
>>> f = furl('http://www.google.com/?space=ja+ms&space=slams')
>>> f.query.asdict()
{ 'encoded': 'space=ja+ms&space=slams',
  'params': [('space', 'ja ms'),
             ('space', 'slams')] }
```



### Fragment

URL fragments in furl are Fragment objects that have a Path __path__ and Query
__query__ separated by an optional `?` __separator__.

```python
>>> f = furl('http://www.google.com/#/fragment/path?with=params')
>>> f.fragment
Fragment('/fragment/path?with=params')
>>> f.fragment.path
Path('/fragment/path')
>>> f.fragment.query
Query('with=params')
>>> f.fragment.separator
True
```

Manipulation of Fragments is done via the Fragment's Path and Query instances,
__path__ and __query__.

```python
>>> f = furl('http://www.google.com/#/fragment/path?with=params')
>>> str(f.fragment)
'/fragment/path?with=params'
>>> f.fragment.path.segments.append('file.ext')
>>> str(f.fragment)
'/fragment/path/file.ext?with=params'

>>> f = furl('http://www.google.com/#/fragment/path?with=params')
>>> str(f.fragment)
'/fragment/path?with=params'
>>> f.fragment.args['new'] = 'yep'
>>> str(f.fragment)
'/fragment/path?new=yep&with=params'
```

Creating hash-bang fragments with furl illustrates the use of Fragment's boolean
attribute __separator__. When __separator__ is False, the `?` that separates
__path__ and __query__ isn't included.

```python
>>> f = furl('http://www.google.com/')
>>> f.fragment.path = '!'
>>> f.fragment.args = {'a':'dict', 'of':'args'}
>>> f.fragment.separator
True
>>> str(f.fragment)
'!?a=dict&of=args'

>>> f.fragment.separator = False
>>> str(f.fragment)
'!a=dict&of=args'
>>> f.url
'http://www.google.com/#!a=dict&of=args'
```

For a dictionary representation of a fragment, use __asdict()__.

```python
>>> f = furl('http://www.google.com/#path?args=args')
>>> f.fragment.asdict()
{ 'encoded': 'path?args=args',
  'separator': True,
  'path': { 'encoded': 'path',
            'isabsolute': False,
            'isdir': False,
            'isfile': True,
            'segments': ['path']},
  'query': { 'encoded': 'args=args',
             'params': [('args', 'args')]} }
```



### Encoding

Furl handles encoding for you, and furl's philosophy on encoding is simple: raw
URL strings should always be percent-encoded.

```python
>>> f = furl()
>>> f.netloc = '%40user:%3Apass@google.com'
>>> f.username, f.password
'@user', ':pass'

>>> f = furl()
>>> f.path = 'supply%20percent%20encoded/path%20strings'
>>> f.path.segments
['supply percent encoded', 'path strings']

>>> f.set(query='supply+percent+encoded=query+strings,+too')
>>> f.query.params
omdict1D([('supply percent encoded', 'query strings, too')])

>>> f.set(fragment='percent%20encoded%20path?and+percent+encoded=query+too')
>>> f.fragment.path.segments
['percent encoded path']
>>> f.fragment.args
omdict1D([('and percent encoded', 'query too')])
```

Raw, non-URL strings should never be percent-encoded.

```python
>>> f = furl('http://google.com')
>>> f.set(username='@prap', password=':porps')
>>> f.url
'http://%40prap:%3Aporps@google.com'

>>> f = furl()
>>> f.set(path=['path segments are', 'decoded', '<>[]"#'])
>>> str(f.path)
'/path%20segments%20are/decoded/%3C%3E%5B%5D%22%23'

>>> f.set(args={'query parameters':'and values', 'are':'decoded, too'})
>>> str(f.query)
'query+parameters=and+values&are=decoded,+too'

>>> f.fragment.path.segments = ['decoded', 'path segments']
>>> f.fragment.args = {'and decoded':'query parameters and values'}
>>> str(f.fragment)
'decoded/path%20segments?and+decoded=query+parameters+and+values'
```

Python's
[urllib.quote()](http://docs.python.org/library/urllib.html#urllib.quote) and
[urllib.unquote()](http://docs.python.org/library/urllib.html#urllib.unquote)
can be used to percent-encode and percent-decode path strings. Similarly,
[urllib.quote_plus()](http://docs.python.org/library/urllib.html#urllib.quote_plus)
and
[urllib.unquote_plus()](http://docs.python.org/library/urllib.html#urllib.unquote_plus)
can be used to percent-encode and percent-decode query strings.



### Inline manipulation

For quick, single-line URL manipulation, the __add()__, __set()__, and
__remove()__ methods of furl objects manipulate various URL components and
return the furl object for method chaining.

```python
>>> url = 'http://www.google.com/#fragment' 
>>> furl(url).add(args={'example':'arg'}).set(port=99).remove(fragment=True).url
'http://www.google.com:99/?example=arg'
```

__add()__ adds items to a furl object with the optional arguments

 * __args__: Shortcut for __query_params__.
 * __path__: A list of path segments to add to the existing path segments, or a
   path string to join with the existing path string.
 * __query_params__: A dictionary of query keys and values to add to the query.
 * __fragment_path__: A list of path segments to add to the existing fragment
   path segments, or a path string to join with the existing fragment path
   string.
 * __fragment_args__: A dictionary of query keys and values to add to the
   fragment's query.

```python
>>> f = furl('http://www.google.com/').add(
...   path='/search', fragment_path='frag/path', fragment_args={'frag':'arg'})
>>> f.url
'http://www.google.com/search#frag/path?frag=args'
```

__set()__ sets items of a furl object with the optional arguments

 * __args__: Shortcut for __query_params__.
 * __path__: List of path segments or a path string to adopt.
 * __scheme__: Scheme string to adopt.
 * __netloc__: Network location string to adopt.
 * __origin__: Origin string to adopt.
 * __query__: Query string to adopt.
 * __query_params__: A dictionary of query keys and values to adopt.
 * __fragment__: Fragment string to adopt.
 * __fragment_path__: A list of path segments to adopt for the fragment's path
   or a path string to adopt as the fragment's path.
 * __fragment_args__: A dictionary of query keys and values for the fragment's
   query to adopt.
 * __fragment_separator__: Boolean whether or not there should be a `?`
   separator between the fragment path and the fragment query.
 * __host__: Host string to adopt.
 * __port__: Port number to adopt.
 * __username__: Username string to adopt.
 * __password__: password string to adopt.


```python
>>> f = furl().set(
...   scheme='https', host='secure.google.com', port=99, path='index.html',
...   args={'some':'args'}, fragment='great job')
>>> f.url
'https://secure.google.com:99/index.html?some=args#great%20job'
```

__remove()__ removes items from a furl object with the optional arguments

 * __args__: Shortcut for __query_params__.
 * __path__: A list of path segments to remove from the end of the existing path
       segments list, or a path string to remove from the end of the existing
       path string, or True to remove the entire path portion of the URL.
 * __query__: A list of query keys to remove from the query, if they exist, or
       True to remove the entire query portion of the URL.
 * __query_params__: A list of query keys to remove from the query, if they
       exist.
 * __fragment__: If True, remove the entire fragment portion of the URL.
 * __fragment_path__: A list of path segments to remove from the end of the
       fragment's path segments, or a path string to remove from the end of the
       fragment's path string, or True to remove the entire fragment path.
 * __fragment_args__: A list of query keys to remove from the fragment's query,
       if they exist.
 * __username__: If True, remove the username, if it exists.
 * __password__: If True, remove the password, if it exists.


```python
>>> url = 'https://secure.google.com:99/a/path/?some=args#great job'
>>> furl(url).remove(args=['some'], path='path/', fragment=True, port=True).url
'https://secure.google.com/a/'
```



### Miscellaneous

Like [pathlib.Path](https://docs.python.org/3/library/pathlib.html#operators),
path segments can be appended to a furl object's Path with the slash operator.

```python
>>> from __future__ import division  # For Python 2.x.
>>> f = furl('http://www.google.com/path?example=arg#frag')
>>> f /= 'add'
>>> f = f / 'seg ments/'
>>> f.url
'http://www.google.com/path/add/seg%20ments/?example=arg#frag'
```

__tostr(query_delimiter='&', query_quote_plus=True, query_dont_quote='')__
creates and returns a URL string. `query_delimiter`, `query_quote_plus`, and
`query_dont_quote` are passed unmodified to `Query.encode()` as `delimiter`,
`quote_plus`, and `dont_quote` respectively.

```python
>>> f = furl('http://spep.ru/?a+b=c+d&two%20tap=cat%20nap%24')
>>> f.tostr()
'http://spep.ru/?a+b=c+d&two+tap=cat+nap$'
>>> f.tostr(query_delimiter=';', query_quote_plus=False)
'http://spep.ru/?a%20b=c%20d;two%20tap=cat%20nap$'
>>> f.tostr(query_dont_quote='$')
'http://spep.ru/?a+b=c+d&two+tap=cat+nap$'
```

`furl.url` is a shortcut for `furl.tostr()`.

```python
>>> f.url
'http://spep.ru/?a+b=c+d&two+tap=cat+nap$'
>>> f.url == f.tostr() == str(f)
True
```

__copy()__ creates and returns a new furl object with an identical URL.

```python
>>> f = furl('http://www.google.com')
>>> f.copy().set(path='/new/path').url
'http://www.google.com/new/path'
>>> f.url
'http://www.google.com'
```

__join()__ joins the furl object's URL with the provided relative or absolute
URL and returns the furl object for method chaining. __join()__'s action is the
same as navigating to the provided URL from the current URL in a web browser.

```python
>>> f = furl('http://www.google.com')
>>> f.join('new/path').url
'http://www.google.com/new/path'
>>> f.join('replaced').url
'http://www.google.com/new/replaced'
>>> f.join('../parent').url
'http://www.google.com/parent'
>>> f.join('path?query=yes#fragment').url
'http://www.google.com/path?query=yes#fragment'
>>> f.join('unknown://www.yahoo.com/new/url/').url
'unknown://www.yahoo.com/new/url/'
```

For a dictionary representation of a URL, use __asdict()__.

```python
>>> f = furl('https://xn--eckwd4c7c.xn--zckzah/path?args=args#frag')
>>> f.asdict()
{ 'url': 'https://xn--eckwd4c7c.xn--zckzah/path?args=args#frag',
  'scheme': 'https',
  'username': None
  'password': None,
  'host': 'ドメイン.テスト',
  'host_encoded': 'xn--eckwd4c7c.xn--zckzah',
  'port': 443,
  'netloc': 'xn--eckwd4c7c.xn--zckzah',
  'origin': 'https://xn--eckwd4c7c.xn--zckzah',
  'path': { 'encoded': '/path',
            'isabsolute': True,
            'isdir': False,
            'isfile': True,
            'segments': ['path']},
  'query': { 'encoded': 'args=args',
             'params': [('args', 'args')]},
  'fragment': { 'encoded': 'frag',
                'path': { 'encoded': 'frag',
                          'isabsolute': False,
                          'isdir': False,
                          'isfile': True,
                          'segments': ['frag']},
                'query': { 'encoded': '',
                           'params': []},
                'separator': True} }
```