Liberating tabular data from the clutches of PDFs¶
By: Jayant Pahuja & Spongebob
Data Expert @OpenBudgetsIndia
Core Team Member @DataKindBlR
Budgets¶
- Why are budgets important ?
- What kind of data is published ?
- How are they published ?
150+ Budget Source Websites¶
In [246]:
Image('govt_budget_portals.png')
Out[246]:
Problems with Budget Documents¶
- Unstructured PDF documents
- Inconsistent Formats
- Budgets published in local languages
In [247]:
Image('whaat.jpg')
Out[247]:
Karnataka Budget¶
In [23]:
Image('kn_sample_image.png')
Out[23]:
West Bengal Budget¶
In [170]:
Image('west_bengal_sample.png')
Out[170]:
Gujarat Budget¶
In [171]:
Image('gj_sample_image.png')
Out[171]:
Extracting Tables From Documents with Table Boundaries¶
Concept: Detect Rows and Columns based on the lines detected and parse the infromation to extract tables in csv format.
- Detect Lines
- Vertical
- Horizontal
- Detect Table Boundaries
- Parse Tables To extract information in csv format.
Converting Pdf Page To Image - I¶
In [239]:
import cv2
import subprocess
# Converting a pdf page into an Image for processing
def get_page_image_from_pdf(page_num, image_file_name, pdf_file_path):
'''
Convert a pdf page into an image using imagemagick's convert utility
'''
command = "convert -density 300 '%s'[%s] '%s'" % (pdf_file_path,
page_num,
image_file_name)
subprocess.check_output(command, shell=True)
return cv2.imread(image_file_name, 0)
Converting Pdf Page To Image - II¶
In [99]:
kn_page_image = get_page_image_from_pdf(11, 'kn_sample_image.png', kn_pdf_file_path)
plt.figure(figsize=(30,20))
plt.imshow(kn_page_image, cmap='gray')
Out[99]:
Detecting Lines¶
To detect lines we use openCV's Hough Transform Implementation
cv2.HoughLines(), cv2.HoughLinesP()In [248]:
Image('lines.jpg')
Out[248]:
Extracting Edges¶
In [179]:
edges = cv2.Canny(kn_page_image, 120, 150, apertureSize=3)
plot_page(edges)
Detecting Lines From Edges - Hough Transform¶
In [187]:
lines = cv2.HoughLines(edges,1,np.pi/180,200)
lines[:5]
Out[187]:
In [184]:
HTML('<img src="http://opencv-python-tutroals.readthedocs.io/en/latest/_images/houghlinesdemo.gif">')
Out[184]:
Detecting Straight Lines¶
In [189]:
def get_straight_lines(img, aperture_size=3):
'''Extract long straight lines using Probabilistic Hough Transform
'''
edges = cv2.Canny(img, 120, 150, apertureSize=aperture_size)
min_line_length = 100
max_line_gap = 100
lines = cv2.HoughLinesP(edges, 1, np.pi/180, 80, min_line_length,
max_line_gap)
return lines
lines = get_straight_lines(kn_page_image)
plot_lines_on_image(lines, kn_page_image)
Extending Lines To Table Boundaries¶
In [192]:
table_limits = get_table_limits(kn_page_image, lines, False)
plot_page(extend_lines_for_table(kn_page_image, lines, False, table_limits))
Parsing PDF¶
In [193]:
print("tabula --pages 4 --area top,left,bottom,right --columns c1,c2,c3,c4 path/to/pdf")
Image("spongebob.jpg")
Out[193]:
Extracting Table Without Boundaries¶
In [218]:
Image('west_bengal_sample.png')
Out[218]:
Exercise: Sample 1¶
In [194]:
Image('subject2.jpg')
Out[194]:
Exercise: Sample 2¶
In [195]:
Image('subject1.jpg')
Out[195]:
Exercise: Sample 3¶
In [196]:
Image('subject3.jpg')
Out[196]:
Extracting Geometrical features¶
- Extract Blocks from Page Image.
- Extract geometrical features of each block.
Extract Blocks From Page Image¶
We use a modified version of the Run Length Smoothing Algorithm (RLSA)
RLSA convert a byte sequence `x` into `y` based on two rules : -
1. 0's in x are changed to 1 if their is 1 in the next C adjacent pixels.
2. 1's remain unchanged.
C here is a threshold
source: http://crblpocr.blogspot.in/2007/06/run-length-smoothing-algorithm-rlsa.html
RLSA Image¶
In [199]:
threshold = (25, 20)
wb_image_rlsa = rlsa_2d(wb_page_image, threshold)
plot_page(wb_image_rlsa)
Can we do better ?¶
In [201]:
def generate_blocks_dilation(img):
kernel = np.ones((5,10),np.uint8)
ret,thresh1 = cv2.threshold(img, 0, 1, cv2.THRESH_BINARY_INV)
return cv2.dilate(thresh1,kernel,iterations = 5)
plot_page(generate_blocks_dilation(wb_page_image))
Extracting Features Using OpenCV¶
Use cv2.connectedComponentsWithStats()In [225]:
import pandas as pd
n_comp, labels, stats, centroids = cv2.connectedComponentsWithStats(wb_image_rlsa)
block_stats = get_block_stats(stats, centroids)
block_stats['right'] = block_stats.left + block_stats.width
block_stats['bottom'] = block_stats.top + block_stats.height
block_stats.head(10)
Out[225]:
Layout Analysis¶
Identifying the layout of the table, where the elements of the table are :-
- Header
- Values
- Empty Cells
Reading Text From PDF¶
We use poppler's version of `pdftotext`In [226]:
def get_text(page_num, pdf_file_name, x, y, w, h):
command_text = 'pdftotext -enc UTF-8 -f {0} -l {0} -x {1} -y {2} -W {3} -H {4} "{5}" -'
command = command_text.format(page_num + 1, int(x),
int(y), int(w),
int(h), pdf_file_name)
return subprocess.check_output(command, shell=True)
Extracting Text based Features¶
In [230]:
block_stats = block_stats.apply(get_text_data, axis=1, args=[33, wb_pdf_file_path, horizontal_ratio, vertical_ratio])
block_stats[block_stats.text_length > 0][text_columns].sample(5)
Out[230]:
Rule Based Marking¶
- Mark Numbers
- Mark Headers
- Mark Titles
- Mark Groupings
In [233]:
block_stats[['text', 'label']].sample(10)
Out[233]:
In [244]:
Image('rules.jpg')
Out[244]:
In [236]:
block_stats.head()
Out[236]:
Opportunities¶
- Bayesian Models to predict labels for each block.
- Recurrent Neural Nets to mark blocks.
In [243]:
print('Source: https://twitter.com/fatiichi/status/466398802476548096')
Image('googly_eyes.jpg')
Out[243]:
Open By Default - Open Research¶
- Reproducable
- Understandable
- Collaborative
Open For Questions¶
Thank You¶
Github: https://github.com/cbgaindia¶
In [238]:
Image('static_qr_code_without_logo.jpg')
Out[238]:
