Newspaper module of Python

In python, there is an awesome module to extract and parse newspaper arcticles named newspaper. This artcile is to get the initial taste of that module.

IMPORT NEWSPAPER Module

we assume newspaper is installed as a Python module (in my case I'm using Newspaper3k on Python3). Now we will import the module:

import newspaper

SET THE TARGET PAPER

For our testing purpose, we want to look at articles published in the Technology section of the Guardian. The first step was to build the newspaper object, like so:

techno = newspaper.build('https://www.theguardian.com/uk/technology')

ARTICLE EXTRACTION

We just wanted to extract a recent article, using the following code  :

target_article = techno.articles[1] 

target_article.download()

The first line stores the first article in a variable called target_article. The second line downloads the article stored in that variable. 

Printing the result with print(target_article.html) just spews out the entire HTML to the console, which isn't very helpful. But, the brilliant thing about newspaper is that it allows us to parse to article and then run some simple natural language processing against it. 

PARSING ARTICLE

Now that we've downloaded the article, we're in a position to parse it:

target_article.parse()

This in turn enables us to target specific sections of the article, like the body text, title or author. Here's how to scrape body text:

print(target_article.text)

This will print only the body text to the console. 

WRITING THE BODY TEXT TO A FILE

The body text isn't that helpful to us sitting there in the console output, so let's write the output of target_article.text to a file:

First off, import sys

import sys

Then,

f = open( 'article.txt', 'w')

f.write(target_article.text)

Python: Installation of newspaper3k module

The Newspaper library, developed and maintained by Lucas Ou-Yang, is specially designed for extracting information from the websites of newspapers and magazines. The objective of this library is to extract and curate the articles from the newspapers and similar websites. To install Newspaper, use the following commands.

Source
Source 2

• Install pip3 command needed to install newspaper3k package:

  $ sudo apt-get install python3-pip
  

• Python development version, needed for Python.h:

  $ sudo apt-get install python-dev
  

• lxml requirements:

  $ sudo apt-get install libxml2-dev libxslt-dev
  

• For PIL to recognize .jpg images:

  $ sudo apt-get install libjpeg-dev zlib1g-dev libpng12-dev
  

NOTE: If you find problem installing libpng12-dev, try installing libpng-dev.

• Download NLP related corpora:

  $ curl https://raw.githubusercontent.com/codelucas/newspaper/master/download_corpora.py | python3
  

• Install the distribution via pip:

  $ pip3 install newspaper3k

Python Libraries needed for scientific computations and data analysis

To import a library, we can do that in below ways:

import math as m

from math import *

In the first manner, we have defined an alias m to library math. We can now use various functions from math library (e.g. factorial) by referencing it using the alias m.factorial().

In the second manner, you have imported the entire name space in math i.e. you can directly use factorial() without referring to math.

Tip: Google recommends that you use first style of importing libraries, as you will know where the functions have come from.

NumPy stands for Numerical Python. The most powerful feature of NumPy is n-dimensional array. This library also contains basic linear algebra functions, Fourier transforms,  advanced random number capabilities and tools for integration with other low level languages like Fortran, C and C++

SciPy stands for Scientific Python. SciPy is built on NumPy. It is one of the most useful library for variety of high level science and engineering modules like discrete Fourier transform, Linear Algebra, Optimization and Sparse matrices.

Matplotlib for plotting vast variety of graphs, starting from histograms to line plots to heat plots.. You can use Pylab feature in ipython notebook (ipython notebook –pylab = inline) to use these plotting features inline. If you ignore the inline option, then pylab converts ipython environment to an environment, very similar to Matlab. You can also use Latex commands to add math to your plot.

Pandas for structured data operations and manipulations. It is extensively used for data munging and preparation. Pandas were added relatively recently to Python and have been instrumental in boosting Python’s usage in data scientist community.

Scikit Learn for machine learning. Built on NumPy, SciPy and matplotlib, this library contains a lot of effiecient tools for machine learning and statistical modeling including classification, regression, clustering and dimensionality reduction.

Statsmodels for statistical modeling. Statsmodels is a Python module that allows users to explore data, estimate statistical models, and perform statistical tests. An extensive list of descriptive statistics, statistical tests, plotting functions, and result statistics are available for different types of data and each estimator.

Seaborn for statistical data visualization. Seaborn is a library for making attractive and informative statistical graphics in Python. It is based on matplotlib. Seaborn aims to make visualization a central part of exploring and understanding data.

Bokeh for creating interactive plots, dashboards and data applications on modern web-browsers. It empowers the user to generate elegant and concise graphics in the style of D3.js. Moreover, it has the capability of high-performance interactivity over very large or streaming datasets.

Blaze for extending the capability of Numpy and Pandas to distributed and streaming datasets. It can be used to access data from a multitude of sources including Bcolz, MongoDB, SQLAlchemy, Apache Spark, PyTables, etc. Together with Bokeh, Blaze can act as a very powerful tool for creating effective visualizations and dashboards on huge chunks of data.

SymPy for symbolic computation. It has wide-ranging capabilities from basic symbolic arithmetic to calculus, algebra, discrete mathematics and quantum physics. Another useful feature is the capability of formatting the result of the computations as LaTeX code.

Scrapy for web crawling. It is a very useful framework for getting specific patterns of data. It has the capability to start at a website home url and then dig through web-pages within the website to gather information.

Requests for accessing the web. It works similar to the the standard python library urllib2 but is much easier to code. You will find subtle differences with urllib2 but for beginners, Requests might be more convenient.

Installing python cx_oracle on Ubuntu

cs_oracle is the python interface to oracle. To install this to your ubuntu environment, we can follow the below steps:

Step 1:

       sudo apt-get install build-essential unzip python-dev libaio-dev

Step 2:
Click here to download the appropriate zip files required for this. You'll need:

instantclient-basic-linux
instantclient-sdk-linux

Get the appropriate version for your system.. x86 vs 64 etc. Make sure you don't get version 12, since it's not supported by the cx_Oracle moduel yet.

Unzip the content in the same location, so you'll end up with a folder named: instantclient_11_2 which will contain a bunch of .so and jar files.

For my case I used,  $ORACLE_INST_CLIENT, which will basically point to the location where you unzipped your installclient folders.

export ORACLE_INST_CLIENT=$(pwd)/instantclient_11_2

Step 3:
create a symlink to your SO file.

cd $ORACLE_HOME
ln -s libclntsh.so.11.1   libclntsh.so  #the version number on your .so file might be different

Step 4: 
Update your /etc/profile or your ~/.bashrc

export ORACLE_INST_CLIENT=/location/of/your/files/instantclient_11_2
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_INST_CLIENT

Step 5: 
Edit /etc/ld.so.conf.d/oracle.conf

This is a new file, simple add the location of your .so files here, then update the ldpath using

sudo ldconfig

Step 6:
Finally just install cx_oracle module:

pip install cx_oracle



Knowledge Bank:♔
Mastering oracle python
Insert Crud using cx_oracle

Python: String Formats

ESCAPE	DESCRIPTION	EXAMPLE
%d	Decimal integers (not floating point).	"%d" % 45 == '45'
%i	Same as %d.	"%i" % 45 == '45'
%o	Octal number.	"%o" % 1000 == '1750'
%u	Unsigned decimal.	"%u" % -1000 == '-1000'
%x	Hexadecimal lowercase.	"%x" % 1000 == '3e8'
%X	Hexadecimal uppercase.	"%X" % 1000 == '3E8'
%e	Exponential notation, lowercase 'e'.	"%e" % 1000 == '1.000000e+03'
%E	Exponential notation, uppercase 'E'.	"%E" % 1000 == '1.000000E+03'
%f	Floating point real number.	"%f" % 10.34 == '10.340000'
%F	Same as %f.	"%F" % 10.34 == '10.340000'
%g	Either %f or %e, whichever is shorter.	"%g" % 10.34 == '10.34'
%G	Same as %g but uppercase.	"%G" % 10.34 == '10.34'
%c	Character format.	"%c" % 34 == '"'
%r	Repr format (debugging format).	"%r" % int == "'int'>"

%sString format."%s there" % 'hi' == 'hi there' %%A percent sign."%g%%" % 10.34 == '10.34%'

EScape

Python: Data Types

TYPE	DESCRIPTION	EXAMPLE
True	True boolean value.	True or False == True
False	False boolean value.	False and True == False
None	Represents "nothing" or "no value".	x = None
strings	Stores textual information.	x = "hello"
numbers	Stores integers.	i = 100
floats	Stores decimals.	i = 10.389
lists	Stores a list of things.	j = [1,2,3,4]
dicts	Stores a key=value mapping of things.	e = {'x': 1, 'y': 2}

Python: Some Keywords

KEYWORD	DESCRIPTION	EXAMPLE
and	Logical and.	True and False == False
as	Part of the with-as statement.	with X as Y: pass
assert	Assert (ensure) that something is true.	assert False, "Error!"
break	Stop this loop right now.	while True: break
class	Define a class.	class Person(object)
continue	Don't process more of the loop, do it again.	while True: continue
def	Define a function.	def X(): pass
del	Delete from dictionary.	del X[Y]
elif	Else if condition.	if: X; elif: Y; else: J
else	Else condition.	if: X; elif: Y; else: J
except	If an exception happens, do this.	except ValueError, e: print e
exec	Run a string as Python.	exec 'print "hello"'
finally	Exceptions or not, finally do this no matter what.	finally: pass
for	Loop over a collection of things.	for X in Y: pass
from	Importing specific parts of a module.	import X from Y
global	Declare that you want a global variable.	global X
if	If condition.	if: X; elif: Y; else: J
import	Import a module into this one to use.	import os
in	Part of for-loops. Also a test of X in Y.	for X in Y: pass also 1 in [1] == True
is	Like == to test equality.	1 is 1 == True
lambda	Create a short anonymous function.	s = lambda y: y ** y; s(3)
not	Logical not.	not True == False
or	Logical or.	True or False == True
pass	This block is empty.	def empty(): pass
print	Print this string.	print 'this string'
raise	Raise an exception when things go wrong.	raise ValueError("No")
return	Exit the function with a return value.	def X(): return Y
try	Try this block, and if exception, go to except.	try: pass
while	While loop.	while X: pass
with	With an expression as a variable do.	with X as Y: pass
yield	Pause here and return to caller.	def X(): yield Y; X().next()

Python

Python is a dynamic, strongly typed, object oriented, multipurpose programming language, designed to be quick (to learn, to use, and to understand), and to enforce a clean and uniform syntax.

1. Python is dynamically typed: it means that you don't declare a type (e.g. 'integer') for a variable name, and then assign something of that type (and only that type). Instead, you have variable names, and you bind them to entities whose type stays with the entity itself. a = 5 makes the variable name a to refer to the integer 5. Later, a = "hello" makes the variable name a to refer to a string containing "hello". Static typed languages would have you declare int a and then a = 5, but assigning a = "hello" would have been a compile time error. On one hand, this makes everything more unpredictable (you don't know what a refers to). On the other hand, it makes very easy to achieve some results a static typed languages makes very difficult.
2. Python is strongly typed. It means that if a = "5" (the string whose value is '5') will remain a string, and never coerced to a number if the context requires so. Every type conversion in python must be done explicitly. This is different from, for example, Perl or Javascript, where you have weak typing, and can write things like "hello" + 5 to get "hello5".
3. Python is object oriented, with class-based inheritance. Everything is an object (including classes, functions, modules, etc), in the sense that they can be passed around as arguments, have methods and attributes, and so on.
4. Python is multipurpose: it is not specialised to a specific target of users (like R for statistics, or PHP for web programming). It is extended through modules and libraries, that hook very easily into the C programming language.
5. Python enforces correct indentation of the code by making the indentation part of the syntax. There are no control braces in Python. Blocks of code are identified by the level of indentation. Although a big turn off for many programmers not used to this, it is precious as it gives a very uniform style and results in code that is visually pleasant to read.
6. The code is compiled into byte code and then executed in a virtual machine. This means that precompiled code is portable between platforms.