Skip to main content

Posts about tech

Node-Red Website Alerts

Node-RED is a flow-based programming tool, originally developed by IBM’s Emerging Technology Services team and now a part of the JS Foundation. A recent question on /r/selfhosted about what selfhosted service you are missing sparked my interest. A user by the name forthedatahorde mentioned they wanted the ability to monitor arbitrary websites. I figured Node-Red is just the tool to comfortably fill this gap.

I made two options. One using readability.js for a bit of a generic solution and one using css selectors for more targeted watching needs.

The Readability.js Version:

  1. Fetches the site with a HTTP get request.

  2. Runs it through readability.js

  3. Hashes the text

  4. Compares it with an old hash

  5. Emails the change

Required the packages node-red-contrib-md5 and node-red-contrib-readability

image03

[{"id":"c8beb17e.4513c8","type":"http request","z":"1a1be165.968ed7","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"","tls":"","proxy":"","authType":"","x":310,"y":100,"wires":[["8a94f0a4.a1f488"]]},{"id":"ae660213.061188","type":"readability","z":"1a1be165.968ed7","name":"","x":310,"y":180,"wires":[["597a982a.8c9bb"]]},{"id":"8a94f0a4.a1f488","type":"switch","z":"1a1be165.968ed7","name":"","property":"statusCode","propertyType":"msg","rules":[{"t":"eq","v":"200","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":290,"y":140,"wires":[["ae660213.061188"]]},{"id":"c361fc4a.44f6b8","type":"change","z":"1a1be165.968ed7","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.content","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":300,"wires":[["4e96af67.1febe"]]},{"id":"597a982a.8c9bb","type":"md5","z":"1a1be165.968ed7","name":"","fieldToHash":"payload.text","fieldTypeToHash":"msg","hashField":"md5","hashFieldType":"msg","x":470,"y":140,"wires":[["8e8bc713.5693f"]]},{"id":"ed8cd02.da0d03","type":"switch","z":"1a1be165.968ed7","name":"","property":"md5","propertyType":"msg","rules":[{"t":"neq","v":"old_hash","vt":"msg"}],"checkall":"true","repair":false,"outputs":1,"x":610,"y":180,"wires":[["d52d318b.d1ac58"]]},{"id":"5cfa0e17.f15bf","type":"change","z":"1a1be165.968ed7","name":"","rules":[{"t":"set","p":"url","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":60,"wires":[["c8beb17e.4513c8"]]},{"id":"4ab25748.add748","type":"change","z":"1a1be165.968ed7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"url","tot":"msg"},{"t":"change","p":"topic","pt":"msg","from":"^(.*)$","fromt":"re","to":"Update to $1 detected","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":260,"wires":[["c361fc4a.44f6b8"]]},{"id":"4e96af67.1febe","type":"e-mail","z":"1a1be165.968ed7","server":"smtp.gmail.com","port":"465","secure":true,"tls":true,"name":"","dname":"","x":690,"y":300,"wires":[]},{"id":"b7b6adc.1b5c3d","type":"inject","z":"1a1be165.968ed7","name":"rammstein.de","topic":"","payload":"https://www.rammstein.de/de/","payloadType":"str","repeat":"21600","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":60,"wires":[["5cfa0e17.f15bf"]]},{"id":"8e8bc713.5693f","type":"function","z":"1a1be165.968ed7","name":"Get Hash","func":"try{\n    msg.old_hash = flow.get(msg.url);\n} catch(e) {\n    msg.old_hash = \"0\";\n}\nreturn msg;","outputs":1,"noerr":0,"x":620,"y":140,"wires":[["ed8cd02.da0d03"]]},{"id":"d52d318b.d1ac58","type":"function","z":"1a1be165.968ed7","name":"Set Hash","func":"flow.set(msg.url, msg.md5);\nreturn msg;","outputs":1,"noerr":0,"x":480,"y":220,"wires":[["4ab25748.add748"]]}]

The CSS Selector Version:

  1. Fetches the site with a HTTP get request.

  2. Filters the resulting HTML with a css selector

  3. Hashes the html

  4. Compares it with an old hash

  5. Emails the change

Required the packages node-red-contrib-md5

image02

[{"id":"c8beb17e.4513c8","type":"http request","z":"1a1be165.968ed7","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"","tls":"","proxy":"","authType":"","x":350,"y":160,"wires":[["8a94f0a4.a1f488"]]},{"id":"8a94f0a4.a1f488","type":"switch","z":"1a1be165.968ed7","name":"","property":"statusCode","propertyType":"msg","rules":[{"t":"eq","v":"200","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":330,"y":200,"wires":[["28487abf.9cf7be"]]},{"id":"597a982a.8c9bb","type":"md5","z":"1a1be165.968ed7","name":"","fieldToHash":"payload","fieldTypeToHash":"msg","hashField":"md5","hashFieldType":"msg","x":330,"y":280,"wires":[["8e8bc713.5693f"]]},{"id":"ed8cd02.da0d03","type":"switch","z":"1a1be165.968ed7","name":"","property":"md5","propertyType":"msg","rules":[{"t":"neq","v":"old_hash","vt":"msg"}],"checkall":"true","repair":false,"outputs":1,"x":330,"y":360,"wires":[["d52d318b.d1ac58"]]},{"id":"5cfa0e17.f15bf","type":"change","z":"1a1be165.968ed7","name":"","rules":[{"t":"set","p":"url","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":350,"y":120,"wires":[["c8beb17e.4513c8"]]},{"id":"4ab25748.add748","type":"change","z":"1a1be165.968ed7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"url","tot":"msg"},{"t":"change","p":"topic","pt":"msg","from":"^(.*)$","fromt":"re","to":"Update to $1 detected","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":400,"wires":[["9a394414.9956a"]]},{"id":"b7b6adc.1b5c3d","type":"inject","z":"1a1be165.968ed7","name":"rammstein.de","topic":"","payload":"https://www.rammstein.de/de/","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":80,"wires":[["6500331b.56333c"]]},{"id":"8e8bc713.5693f","type":"function","z":"1a1be165.968ed7","name":"Get Hash","func":"try{\n    msg.old_hash = flow.get(msg.url);\n} catch(e) {\n    msg.old_hash = \"0\";\n}\nreturn msg;","outputs":1,"noerr":0,"x":340,"y":320,"wires":[["ed8cd02.da0d03"]]},{"id":"d52d318b.d1ac58","type":"function","z":"1a1be165.968ed7","name":"Set Hash","func":"flow.set(msg.url, msg.md5);\nreturn msg;","outputs":1,"noerr":0,"x":500,"y":360,"wires":[["4ab25748.add748"]]},{"id":"28487abf.9cf7be","type":"html","z":"1a1be165.968ed7","name":"","property":"payload","outproperty":"payload","tag":"","ret":"html","as":"single","x":330,"y":240,"wires":[["b8739927.661e18"]]},{"id":"6500331b.56333c","type":"change","z":"1a1be165.968ed7","name":"Selector css","rules":[{"t":"set","p":"select","pt":"msg","to":".resume-view-news","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":110,"y":120,"wires":[["5cfa0e17.f15bf"]]},{"id":"b8739927.661e18","type":"change","z":"1a1be165.968ed7","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload[0]","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":240,"wires":[["597a982a.8c9bb"]]},{"id":"9a394414.9956a","type":"e-mail","z":"1a1be165.968ed7","server":"smtp.gmail.com","port":"465","secure":true,"tls":true,"name":"","dname":"","x":710,"y":400,"wires":[]}]

I hope someone finds this helpful/interesting. Let me know if you have a better solution in the comments down below.