I just discovered htmlq (think jq but for HTML not JSON) and it's great!
for url in $(cat ~/tmp/urls.txt)
do
echo $url
curl --silent -H "Accept: text/html" $url | htmlq "meta[property^='og:']"
echo
done
... generates me a nice summary of the OpenGraph tags for each URL in urls.txt.
#shell #cli #html #rust
Duncan Bayne
npub1kq…a7dce
2025-01-07 04:02:14 UTC
Author Public Key
npub1kqky666x2fg0z7nf77vep4jdkadtcjz694ul3qwy2lcnnx379xcs3a7dceSeen on
wss://relay.ditto.pubPublished at
2025-01-07 04:02:14 UTCEvent JSON
{
"id": "b866525bde16388931c70cda014d6085ec62db0181613b94330bbe911c23675b",
"pubkey": "b02c4d6b465250f17a69f79990d64db75abc485a2d79f881c457f1399a3e29b1",
"created_at": 1736222534,
"kind": 1,
"tags": [
[
"t",
"shell"
],
[
"t",
"cli"
],
[
"t",
"html"
],
[
"t",
"rust"
],
[
"proxy",
"https://mastodon.bsd.cafe/users/duncan_bayne/statuses/113785080053301679",
"activitypub"
]
],
"content": "I just discovered htmlq (think jq but for HTML not JSON) and it's great!\n\n for url in $(cat ~/tmp/urls.txt)\n do\n echo $url\n curl --silent -H \"Accept: text/html\" $url | htmlq \"meta[property^='og:']\"\n echo\n done\n\n... generates me a nice summary of the OpenGraph tags for each URL in urls.txt.\n\n#shell #cli #html #rust",
"sig": "a010ce263a36e06d9247231e936c5d55d5d8b9b2aa899411603811382916f91e8c58a9b9853c96b4d04260794c0484620c0a9e7c89188abaf46d4a221893c073"
}