jupyter-notebooks/webscraping/Splash.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1543948-4781-4f50-849e-1e8b9786b7bb",
   "metadata": {},
   "source": [
    "# Splash"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18805201-1cec-439d-bb09-ac9beba683b2",
   "metadata": {},
   "source": [
    "render.json\n",
    "Return a json-encoded dictionary with information about javascript-rendered webpage. It can include HTML, PNG and other information, based on arguments passed.\n",
    "\n",
    "Arguments:\n",
    "\n",
    "Same as render.jpeg plus the following ones:\n",
    "\n",
    "html : integer : optional\n",
    "Whether to include HTML in output. Possible values are 1 (include) and 0 (exclude). Default is 0.\n",
    "png : integer : optional\n",
    "Whether to include PNG in output. Possible values are 1 (include) and 0 (exclude). Default is 0.\n",
    "jpeg : integer : optional\n",
    "Whether to include JPEG in output. Possible values are 1 (include) and 0 (exclude). Default is 0.\n",
    "iframes : integer : optional\n",
    "Whether to include information about child frames in output. Possible values are 1 (include) and 0 (exclude). Default is 0.\n",
    "script : integer : optional\n",
    "Whether to include the result of the executed javascript final statement in output (see Executing custom Javascript code within page context). Possible values are 1 (include) and 0 (exclude). Default is 0.\n",
    "console : integer : optional\n",
    "Whether to include the executed javascript console messages in output. Possible values are 1 (include) and 0 (exclude). Default is 0.\n",
    "history : integer : optional\n",
    "Whether to include the history of requests/responses for webpage main frame. Possible values are 1 (include) and 0 (exclude). Default is 0.\n",
    "\n",
    "Use it to get HTTP status codes and headers. Only information about “main” requests/responses is returned (i.e. information about related resources like images and AJAX queries is not returned). To get information about all requests and responses use ‘har’ argument.\n",
    "\n",
    "har : integer : optional\n",
    "Whether to include HAR in output. Possible values are 1 (include) and 0 (exclude). Default is 0. If this option is ON the result will contain the same data as render.har provides under ‘har’ key.\n",
    "\n",
    "By default, request and response contents are not included. To enable each, use ‘request_body’ and ‘response_body’ options respectively.\n",
    "\n",
    "request_body : int : optional\n",
    "Possible values are 1 and 0. When request_body=1, request content is included in HAR records. Default is request_body=0. This option has no effect when both ‘har’ and ‘history’ are 0.\n",
    "response_body : int : optiona"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8aaec01c-4b87-4949-8787-e9dec8b2e5dd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-26T03:50:20.355782Z",
     "iopub.status.busy": "2021-10-26T03:50:20.355387Z",
     "iopub.status.idle": "2021-10-26T03:50:20.870039Z",
     "shell.execute_reply": "2021-10-26T03:50:20.855218Z",
     "shell.execute_reply.started": "2021-10-26T03:50:20.355743Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import pprint\n",
    "\n",
    "import requests\n",
    "\n",
    "# script = \"\"\"\n",
    "# splash:go(args.url)\n",
    "# return splash:png()\n",
    "# \"\"\"\n",
    "\n",
    "\n",
    "resp = requests.post(\n",
    "    \"https://kong.donavanaldrich.com/splash\",\n",
    "    json={\n",
    "        \"url\": \"http://example.com\",\n",
    "        \"history\": 1,\n",
    "        \"html\": 1,\n",
    "        \"request_body\": 1,\n",
    "        \"har\": 1,\n",
    "    },\n",
    ")\n",
    "png_data = resp.content\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4, sort_dicts=True)\n",
    "pp.pprint(png_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1630de92-c015-43a3-b747-9d29ca5afba2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-26T04:36:51.911804Z",
     "iopub.status.busy": "2021-10-26T04:36:51.911422Z",
     "iopub.status.idle": "2021-10-26T04:36:55.323679Z",
     "shell.execute_reply": "2021-10-26T04:36:55.322189Z",
     "shell.execute_reply.started": "2021-10-26T04:36:51.911765Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# json: {url: \"http://example.com\", history: 1, html: 1, request_body: 1, har: 1}\n",
    "# !http POST https://kong.donavanaldrich.com/splash < json\n",
    "\n",
    "\n",
    "!http POST https://kong.donavanaldrich.com/splash url=http://example.com har:=1 html:=1 png:=1 history:=1 console:=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc1283f7-19f8-4430-9ae6-19a74c419a91",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}