Header creates several extra columns for the rest of the table

[schoodic100]

Howdy,

Your tool is so good. For someone that has suffered through PDF extraction for a decade, I am so happy to have found your library.

This is most likely a newbie question, but I thought you might have a quick answer that would be hugely helpful. I am attaching a very long PDF along with the "display" from PDFPlumber for one of the pages (they all are the same, more or less). It appears that the majority of the table is getting thrown off by the column headings adding several "blank" columns to the output.

Is there a strategy for getting better behavior? I have played with table settings a bit but haven't found a combination that works too well.

Same output:
[['Prescription Drug Name', None, '', 'Drug', '', '', 'Coverage', ''], [None, None, None, 'Tier', None, None, 'Requirements/Limits', None], ['', 'ANTI-INFECTIVE AGENTS', None, None, None, None, None, ''], ['', 'ANTHELMINTICS', None, None, None, None, None, ''], ['albendazole tabs 200 mg', None, '1', None, None, '', None, None], ['BILTRICIDE TABS 600 MG [praziquantel]', None, '2', None, None, '', None, None], ['ivermectin tabs 3 mg', None, '1', None, None, '', None, None].....

Ideally the output would be (one drug example): ['ampicillin sodium solr 1 gm','1', 'MB']

What is being produced is TOTALLY usable, but I thought perhaps there was something I was missing to make it even better.

Thanks and have a great day,

Jeff

Kaiser_Commercial.pdf

[jsvine]

Thanks for the kind words, @schoodic100! Depending on how consistent your PDFs/tables are, it might help to:

• Manually figure out what attributes makes those trouble-causing "fake" lines/rects in the header different than the rest of the lines/rects on the page.
• Use page.filter(...) to filter them out
• Try extracting the table from the filtered page

Does that work? If not, I can take a closer look at the specific PDF. (Thank you for attaching it.)

[schoodic100]

Thanks @jsvine. Yes I was able to swipe some code from another discussion post to deal with the "fake" rectangles. It also gave me a better insight into the various tactics that one can employ. Impressive stuff.

[dimavologzh]

Hi @schoodic100 ! Can you, please, share what discussion post you used? Thank you!

[schoodic100]

`import pdfplumber
from   operator import itemgetter

def inside(self, other):
    return all((
        self['x0'] >= other['x0'],
        self['top'] >= other['top'],
        self['x1'] <= other['x1'],
        self['bottom'] <= other['bottom']
    ))

def largest_parent_rect(page, self):
    parent_rects = [other for other in page.rects if inside(self, other)]
    if parent_rects:
        parent_rect = max(parent_rects, key=itemgetter('width', 'height'))
        if self != parent_rect:
            return parent_rect

def remove_nested_rects(page, keep_largest=False):
    def filter_condition(other):
        if other['object_type'] == 'rect':
            return tuple(other['pts']) not in rects_to_remove
        return True

    rects_to_remove = set()

    for rect in page.rects:
        parent = largest_parent_rect(page, rect)
        if parent is not None:
            rects_to_remove.add(tuple(rect['pts']))
            if keep_largest is False:
                rects_to_remove.add(tuple(parent['pts']))

    return page.filter(filter_condition)

full_pdf = pdfplumber.open('xxxxCommercial.pdf')

for page in full_pdf.pages:
    filtered_page = remove_nested_rects(page)

    for table in filtered_page.find_tables():
        # fill in bottom and right lines
        table = filtered_page.crop(table.bbox).extract_table(dict(
            explicit_horizontal_lines = [table.bbox[3]],
            explicit_vertical_lines = [table.bbox[2]]
        ))
        print(table)`

@dimavologzh here is what I more or less used to exclude "fake" rectangles.