Python Requests: HTTP for Humans

Python requests module provides very interactive way to access web content via proper HTTP protocol.

Installation:
Here is the installation manual. You can also install using pip :
pip install requests

import requests

r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
print r.status_code
print r.headers['content-type']
print r.encoding
print r.text
print r.json()

Output:

200
'application/json; charset=utf8'
'utf-8'
u'{"type":"User"...'
{u'private_gists': 419, u'total_private_repos': 77, ...}

Python:Url Fetch

Python URL Request Response

Some times when we try to fetch url directly server does not return page content. For this reason we need to request the server first then read the page using this response. A sample code of python request response is given below.
We can also set some others additional parameter as timeout, max redirect so on

import urllib2
def get_url_content(site_url):
    rt=""
    try:
        request = urllib2.Request(site_url) 
        f=urllib2.urlopen(request)
        content=f.read()
        f.close()
    except urllib2.HTTPError, error:
        content=str(error.read())
    return content