Webshots and Cookie Analysis

One of the easiest methods to take screen shots is by using Puppeteer, which is a 'Node' library that allows code based control Chrome or Chromium browser.

Watch My Domains SED v5 supports web page screen shots and cookie analysis using Puppeteer. However, to be able to do this you require proper installation of nodejs, a node module called minimist and puppeteer along with all its required libraries. It is difficult to exactly know the dependencies in some cases so you may need to install additional libraries.

All the instructions below require that you do this on the same computer that has Watch My Domains SED installed.

Install Node.js

On Linux systems, the installation of nodejs is simple.

sudo yum install nodejs

or

sudo apt-get install nodejs

On Windows/Mac you can install after downloading the corresponding Installer

Download Node.js

Make sure to get the latest stable version, then follow the instructions and install it.

To check if it is working try

node -v

in the command-line / command-prompt / Terminal

Install Puppeteer

To install Puppeteer, open your command-prompt, create a folder that will hold your images and change to it. Now type

npm install puppeteer

This command will install puppeteer, and download and install the newest compatible version of Chromium.

We will also install another nodejs module called minimist that will help us parse command line arguments.

npm install minimist

Here is an example session

mkdir /home/wmdsed/webdata
cd /home/wmdsed/webdata/
npm install puppeteer
npm install minimist

Check Puppeteer

Copy paste the following code into a text editor and save it to the folder that has puppeteer installed as shot.js

const puppeteer = require('puppeteer');
var argv = require('minimist')(process.argv.slice(2));

(async () => {
	if(argv.hasOwnProperty('u')) {
		var webpageurl = argv['u'];
		try {
			const url = new URL(webpageurl);
			var hostname = url.hostname;
			const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});
			const page = await browser.newPage();
			try {
				await page.goto(webpageurl);
				await page.screenshot({ path: hostname + '.jpg' });
			}
			catch(error) {
				//console.log(error);
				throw Error(error);
			}
			await browser.close();
		}
		catch(error) {
			//console.log(error);
			throw Error(error);
		}
	}
})();

Now type

node shot.js -u https://example.com

If everything is as expected, this will create a screenshot of https://example.com and save it as example.com.jpg.

In some Linux systems you might see error messages related to missing shared libraries. If that happens install the missing libraries. Here are some of the libraries that may need to be installed. This is only an indicative list, install only the ones that are required and reported as missing.

gconf-service, libasound2, libatk1.0-0, libc6, libcairo2, libcups2, 
libdbus-1-3, libexpat1, libfontconfig1, libgcc1, libgconf-2-4, 
libgdk-pixbuf2.0-0, libglib2.0-0, libgtk-3-0, libnspr4, 
libpango-1.0-0, libpangocairo-1.0-0, libstdc++6, libx11-6, 
libx11-xcb1, libxcb1, libxcomposite1, libxcursor1, libxdamage1, 
libxext6, libxfixes3, libxi6, libxrandr2, libxrender1, libxss1, 
libxtst6, ca-certificates, fonts-liberation, libappindicator1, 
libnss3, lsb-release, xdg-utils

For example, on Debian 11 or Ubuntu 20.04, the following will install the required libraries.

Ubuntu 20.04

apt-get -y install nodejs npm libnss3-tools libgbm-dev libasound2 libatk1.0-0 libatk-bridge2.0-0 libcups2 libxkbcommon0 libxdamage1 libpango-1.0-0 libpangocairo-1.0-0

Debian 11

apt-get -y install nodejs npm libnss3-tools libgbm-dev libasound2

CentOS 9 x64

yum install nss-tools at-spi2-atk cups-libs libxkbcommon libXcomposite libXdamage libXrandr libgbm pango alsa-lib

Setup Webshot Folder for Watch My Domains SED

Now that you have the ability to take screenshots, move to the log folder of Watch My Domains SED and create a symbolic link named websites within the logs folder.

You can find the location of your log folder from the verify page of Watch My Domains SED.

https://yourinstalllocation/verify.php

To create the symbolik link, change to the logs folder and then type

ln -s /path/to/the/puppeteer/folder  websites

in Linux / Mac. For Windows, use

mklink /D websites "X:\path\to\the\puppeteer\folder\"

This should make the system ready for webshot generation and cookie analysis. Use the 'Web Screen-shots' option in the 'Settings' module to configure the webshot and cookie analysis.