Parse HTML table with Python

[-Veteran007-]

Hey!

I get bowling results on my email. I have used Gmail API to get access to my email, find the emails and get to the result link. Now I am trying to parse the HTML table that is on the link, into a readable/logical table. I don't even know if I should use Pandas or just write to CSV file. I am using BeautifulSoup to parse through the HTML, but the table structure is so messy, I can't wrap my head around it. Here is a short example of the table code. Usually there are multiple tables on the page, tables sometimes with multiple player results, so it all makes that much harder.

 

<table class="scoresheet tenpin TenPin">
    <tr>
     <td>
      <table class="ss-data">
       <tr class="cls_frameheader">
        <td class="ss-name">
         <t>
          Players
         </t>
        </td>
        <td>
         1
        </td>
        <td>
         2
        </td>
        <td>
         3
        </td>
        <td>
         4
        </td>
        <td>
         5
        </td>
        <td>
         6
        </td>
        <td>
         7
        </td>
        <td>
         8
        </td>
        <td>
         9
        </td>
        <td class="cls_framehdr10">
         10
        </td>
        <td class="cls_frametotal">
         <t>
          Total
         </t>
        </td>
       </tr>
        <td class="cls_player">
       <tr class="notranslate">
         Anonymous
        </td>
        <td class="cls_frame">
         <table>
          <tr>
           <td>
            <table class="cls_tbl_balls">
             <tr class="cls_ball_row">
              <td class="cls_ball1">
               5
              </td>
              <td class="cls_ball2">
               /
              </td>
             </tr>
            </table>
           </td>
          </tr>
          <tr>
           <td class="cls_framescore">
            18
           </td>
          </tr>
         </table>
        </td>
        <td class="cls_frame">
         <table>
          <tr>
           <td>
            <table class="cls_tbl_balls">
             <tr class="cls_ball_row">
              <td class="cls_ball1">
               8
              </td>
              <td class="cls_ball2">
               1
              </td>
             </tr>
            </table>
           </td>
          </tr>
          <tr>
           <td class="cls_framescore">
            27
           </td>
          </tr>
         </table>
        </td>
        <td class="cls_frame">
         <table>
          <tr>
           <td>
            <table class="cls_tbl_balls">
             <tr class="cls_ball_row">
              <td class="cls_ball1">
               5
              </td>
              <td class="cls_ball2">
               -
              </td>
             </tr>
            </table>
           </td>
          </tr>
          <tr>
           <td class="cls_framescore">
            32
           </td>
          </tr>
         </table>
        </td>
        <td class="cls_frame">
         <table>
          <tr>
           <td>
            <table class="cls_tbl_balls">
             <tr class="cls_ball_row">
              <td class="cls_ball1">
               4
              </td>
              <td class="cls_ball2">
               /
              </td>
             </tr>
            </table>
           </td>
          </tr>
          <tr>
           <td class="cls_framescore">
            49
           </td>
          </tr>
         </table>
        </td>
        <td class="cls_frame">
         <table>
          <tr>
           <td>
            <table class="cls_tbl_balls">
             <tr class="cls_ball_row">
              <td class="cls_ball1">
               7
              </td>
              <td class="cls_ball2">
               1
              </td>
             </tr>
            </table>
           </td>
          </tr>
          <tr>
           <td class="cls_framescore">
            57
           </td>
          </tr>
         </table>
        </td>

That is the gist of the table. One full table is in the uploaded files. Could you guys give pointers on what to do, ideas on what other modules to use?

test1.html

[JohnDongus]

Use beautiful soup

[-Veteran007-]

I am trying with BeautifulSoup, but I am so overwhelmed, I don't know what kind of functions should I use, to get the best result out of this table. I have tried this:

records = []
for tr in table_soup.findAll("tr"):
	trs = tr.findAll("td")
    records.append(trs[0].text.replace('\n', ' '))

df = pd.DataFrame(data=records)

And the result is:

    Players 1 2 3 4 5 6 7 8 9 10 Total   Anonymous1       5 /      18          8 1      27          5 -      32          4 /      49          7 1      57          9 /      73          6 /      90          7 /      110          X       130          7 / 7      147       147      Anonymous2       7 2      9          6 3      18          - 2      20          - -      20          1 2      23          9 /      33          - 9      42          4 3      49          5 /      69          X - 1      80       80      Anonymous3       7 2      9          - 4      13          7 -      20          9 -      29          3 -      32          4 -      36          6 /      55          9 -      64          9 -      73          - / 1      84       84      Anonymous4       - 9      9          3 -      12          - 8      20          4 5      29          8 -      37          9 /      57          X       76          9 -      85          9 /      105          X 6 -      121       121       Game 1

So I can kinda get the result, but it also somewhy prints this same line all out on different rows before it gets to the second game results, that are in one row. Also I am so lost as to what to do next, I can get the frame numbers and total as column headers, but parsing results and totals as different variables/strings seems way too hard.

[Addvilz]

How about pandas.read_html?  But yes, structure seems kind of crazy...

[-Veteran007-]

I haven't found time to work on my script for some time now. But now I have and figured I'd update.

pandas.read_html really did work!! Don't know, how I missed that function. 

pd.read_html(link, header=0, index_col=0, attrs={'class': 'ss-data'})

That one line did it. Gets all the player names, frames and the results. Script is almost done, working now on pretty printing it to excel or csv. So I could make some statistics and stuff.