Re: Reading Javascript-heavy webpages from the command line

[Leonid Isaev (ifax)]

On Mon, Aug 30, 2021 at 09:13:57PM +0200, Julius Hamilton wrote:
> I am trying to do as much as possible from the command line.

Not everyhting can be done from the cmdline though...

> I am considering learning Selenium to be able to render text versions of
> those kinds of web pages, i.e., retrieve needed text information from them,
> as well as to be able to interact with them from the command line.

I'm not sure about interaction, but yes, you can use selenium to retrieve
(scrape) text from JS pages. There are other choices as well -- see, e.g. [1].

> Is this the most standard, direct and effective way to try to use and
> interact with JavaScript-heavy webpages from the command line or is there
> an easier way I do not know of, for example, a command-line browser that
> someone has already written for such purposes, or maybe some way to
> manually render the JavaScript myself and extract the text, sort of like
> building scripts that act partially as a web browser themselves?

Once you program support for JS, it doesn't make much sense to keep the browser
text-only. This is why there are none, except elinks. But I wouldn't rely on
its JS engine.

But this has nothing to do with bash (or any shell)... Perhaps you need to do
some research in web scrapers?

[1] 
https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python

HTH,

-- 
Leonid Isaev