Scraping with requests-html on Raspberry Pi 4

Here’s what I try to achieve: I have a Raspberry Pi 4 with a Pimoroni Inky Frame. I want to use it for a sponsoring event (Tour for Life) to show the recent amount of gifted money.
This gifted amount is found on this page: https://www.tourforlife.nl/de-rollende-rijders

On my Macbook I’ve used Python code using requests-html to scrape the site and get the gifted-amount class. So scraping works.

Then I brought the code to my Raspberry Pi and then the problems began.

This is a snippet of my Python code:

import requests
from bs4 import BeautifulSoup
from requests_html import HTMLSession
from PIL import Image, ImageFont, ImageDraw

def get_donation_amount():
    """ Get the donation amount from the Tour for Life website

    Returns:
        float: The donation amount
    """
    url = "https://www.tourforlife.nl/de-rollende-rijders"

    session = HTMLSession()
    response = session.get(url)
    response.html.render()
    gift_amount = response.html.find('.gift-amount')[0].text
    # remove euro sign and convert to float
    gift_amount = float(gift_amount[2:].replace(",", "."))
    return gift_amount

And this was the output:

python3 pidonationtracker.py
[INFO] Starting Chromium download.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 183M/183M [00:17<00:00, 10.4Mb/s]
[INFO] Beginning extraction
[INFO] Chromium extracted to: /home/inky3/.local/share/pyppeteer/local-chromium/1181205
Traceback (most recent call last):
  File "/home/inky3/PythonProjects/pidonationtracker/src/pidonationtracker/pidonationtracker.py", line 59, in <module>
    donation_amount = get_donation_amount()
  File "/home/inky3/PythonProjects/pidonationtracker/src/pidonationtracker/pidonationtracker.py", line 23, in get_donation_amount
    response.html.render()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 586, in render
    self.browser = self.session.browser  # Automatically create a event loop and browser
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 730, in browser
    self._browser = self.loop.run_until_complete(super().browser)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 714, in browser
    self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args)
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 307, in launch
    return await Launcher(options, **kwargs).launch()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 148, in launch
    self.proc = subprocess.Popen(  # type: ignore
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/home/inky3/.local/share/pyppeteer/local-chromium/1181205/chrome-linux/chrome'

Now I’ve read some older posts (like: OSError when running puppeteer on raspberry pi) that say that the chromium browser is not the correct version for Raspberry Pi’s ARM chipset. A lot of those posts are from 2019 and before, so I’m not sure if their solutions are still accurate.

I’ve tried installing the Chrome driver again from here:
https://sites.google.com/chromium.org/driver/

Still the same error. Probably because Puppeteer is looking elsewhere.

Then I read this post: https://github.com/miyakogi/pyppeteer/issues/250
This says that I should install chromium-codecs-ffmpeg and chromium-browser (from https://launchpad.net/ubuntu/trusty/armhf/chromium-codecs-ffmpeg and https://launchpad.net/ubuntu/trusty/armhf/chromium-browser). This also is 2018 material. But I got to try something.

As soon as I tried installing these with these commands:

wget http://launchpadlibrarian.net/361669486/chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb
wget http://launchpadlibrarian.net/361669485/chromium-browser_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

sudo dpkg -i chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

.. I ran into dependency problems:

Selecting previously unselected package chromium-codecs-ffmpeg.
dpkg: considering removing chromium-codecs-ffmpeg-extra in favour of chromium-codecs-ffmpeg ...
dpkg: no, cannot proceed with removal of chromium-codecs-ffmpeg-extra (--auto-deconfigure will help):
 rpi-chromium-mods depends on chromium-codecs-ffmpeg-extra
  chromium-codecs-ffmpeg-extra is to be removed.

dpkg: regarding chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb containing chromium-codecs-ffmpeg:
 chromium-codecs-ffmpeg conflicts with chromium-codecs-ffmpeg-extra
  chromium-codecs-ffmpeg-extra (version 120.0.6099.102-rpt1) is present and installed.

dpkg: error processing archive chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb (--install):
 conflicting packages - not installing chromium-codecs-ffmpeg
Errors were encountered while processing:
 chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

So here’s a list of things I had to uninstall:

sudo dpkg --remove --force-remove-reinstreq rpi-chromium-mods
sudo dpkg --remove --force-remove-reinstreq chromium-chromedriver
sudo dpkg --remove --force-remove-reinstreq chromium-browser-l10n
sudo dpkg --remove --force-remove-reinstreq chromium-browser

But when I tried to uninstall chromium-browser I got a whole list of dependencies:

dpkg: dependency problems prevent removal of chromium-browser:
 libreoffice-help-en-us depends on firefox-esr | epiphany-browser | konqueror | chromium | firefox; however:
  Package firefox-esr is not installed.
  Package epiphany-browser is not installed.
  Package konqueror is not installed.
  Package chromium is not installed.
  Package chromium-browser which provides chromium is to be removed.
  Package firefox is not installed.
 libreoffice-help-en-gb depends on firefox-esr | epiphany-browser | konqueror | chromium | firefox; however:
  Package firefox-esr is not installed.
  Package epiphany-browser is not installed.
  Package konqueror is not installed.
  Package chromium is not installed.
  Package chromium-browser which provides chromium is to be removed.
  Package firefox is not installed.

Okay this is getting ridiculous. Do I really have to uninstall Firefox, Konqueror and a whole list of other things to be able to install an old version of chromium codec, to have Puppeteer take the correct version of Chromium. Or am I going the wrong direction.

I’ve also tried to use Selenium instead. I you want more details of my failures there I can provide them. They were also Chromium related.

Or is this even possible? It seems such a simple task. Just scrape a float from a webpage..

Filed under: Kiến thức lập trình - @ 19:06

Thẻ: chromiumraspberry-pi4python-requests-html

Thiết kế website giá rẻ

Danh mục

Scraping with requests-html on Raspberry Pi 4