shittyscrapper
v1.2.2
Published
a JS/TS text extractor
Downloads
3
Readme
shittyscrapper
A JS/TS data extractor for XML
The basic goal of this project is to replace complex and redundant
XPATH queries by an XML pattern that will automatically fetch data where it is stored in the dom.
Install
Package is available as TS project at npmjs registry under shittyscrapper.
$ npm i -S shittyscrapperUsage
1. Variable Naming
To declare a variable, the pattern ${variable.name} is used.
The given key will be added to the result if found.
Example:
${any.variable.name} => result = {any: {variable: {name: "the value"}}}
Variables ending with [] will be considered as arrays:
${any.array} => result = {any: {array: ["value1", "value2", ...]}}
2. Simple Data extraction
A basic usage of this lib would consist in writing the DOM fragments in which the data is and to fetch data.
Example Data:
<html>
<body>
<div class="test" id="test">
<span>test</span>
<span class="t">
test2
</span>
<span class="t">
test3
</span>
</div>
<div class="test" id="test bis">
<span>test bis</span>
<span class="t">
test2 bis
</span>
<span class="t">
test3 bis
</span>
</div>
<div class="people">
<div>
<span class="Name">John</span>
<span class="LastName">Doe</span>
<span>23</span>
<div class="marks">
<span>18</span>
<span>19</span>
</div>
</div>
<div>
<span class="Name">Peter</span>
<span>Parker</span>
<span class="age">18</span>
<div>
<span>20</span>
<span>19</span>
</div>
</div>
</div>
</body>
</html>Let's extract the first span of the first div with class="test"
import SearchNode from "shittyscrapper";
// format for variable fetching is ${valid.variable.name}
const pattern = "<div class=\"test\"><span>${value1}</span></div>"
const searchNode: SearchNode = SearchNode.BuildSearchNode(pattern)
const data: string = "string of the data file";
const result: any = searchNode.MapData(data)
// result = {value1: "test"}Now let's extract the first span with class="t" in the first div
import SearchNode from "shittyscrapper";
const pattern: string = "<div class=\"test\"><span class=\"t\">${value}</span></div>"
const searchNode = SearchNode.BuildSearchNode(pattern)
const data: string = "" // the data
const result = searchNode.MapData(data)
// result = {value: "test2"}3. Complex Types Data extraction
The xml/html tags can be given two additional attributes in pattern.
- The
datatypeattribute that can be either"repeatable","entity"or"block" - The
keyattribute that has to be used alongwithdatatype="entitity"
Note: setting
datatype="block"is equivalent to settingdatatype="entity repeatable"
The datatype Attribute
datatype can be used upon tags to give them different behaviours.
The "repeatable" value is used to say that the given element can be seen multiple times in the DOM and that it should map to the pattern with the key every time an Element in the DOM is matching. Therefore, the variable names in the element should end with []
Let's retrieve all the spans in div.test
import SearchNode from "shittyscrapper";
const pattern = "<div class=\"test\"><span>${spans[]}</span></div>"
const searchNode = SearchNode.BuildSearchNode(pattern)
// const data: string = "<html>...</html>"
const result = searchNode.MapData(data)
// result = {spans:[
// "test","test2",
// "test3", "test bis",
// "test2 bis", "test3 bis"
// ]}The "block" value groups multiple variables under the same object. It has to be used alongwith the
keyattribute. The "block" value acts the same as "repeatable" but creates an additional object
Let's fetch Data for the users.
import SearchNode from "shittyscrapper";
const pattern = `<div class="people">
<div datatype="block" key="people[]">
<span>\${Name}</span>
<span>\${LastName}</span>
<span>\${Age}</span>
<div class="marks">
<span datatype="repeatable">\${Marks[]}</span>
</div>
</div>
</div>`
const searchNode = SearchNode.BuildSearchNode(pattern)
// const data: string = ...
const result = searchNode.MapData(data)
for (let person of result.people) {
console.log(person)
}
// {Name: "John", LastName: "Doe", Age: "23", Marks: ["18", "19"]}
// {Name: "Peter", LastName: "Parker", Age: "18", Marks: ["19", "20"]}4. Data checking
Example Data:
<html>
<body>
<p class="my-text" id="text-number-1">Some text with some data embedded</p>
<p class="my-text" id="text-number-2">Some other text</p>
<p class="my-text" id="text-number-3">Data 1</p>
<p class="my-text" id="text-number-4">Broken Data</p>
<p class="my-text" id="text-number-5">Data 2</p>
</body>
</html>Variables can be embedded everywhere in the text
import SearchNode from "shittyscrapper";
const pattern = `<p class="my-text">Some text with \${data} embedded</p>`
const searchNode = SearchNode.BuildSearchNode(pattern)
// const data: string = ...
const result = searchNode.MapData(data)
// result = {data: "some data"}
// only first one is matched since the others did not match "Some text with ..."Variables can be embedded everywhere in the attributes also:
import SearchNode from "./index";
const pattern = `<p class="my-text" id="text-number-\${index}" datatype="block" key="paragraphs[]">\${data}</p>`
const searchNode = SearchNode.BuildSearchNode(pattern)
// const data: string = ...
const result = searchNode.MapData(data)
for (const parag of result.paragraphs) {
console.log(parag)
}
// {index: "1", data: "Some text with some data embedded"}
// ...
// {index: "5", data: "Data 2"}Some Regex variables can be added to verify some data.
To add a regex checking, add
${/^any\sregex$/}It can be added both before a variable to match a variable or anywhere in the pattern to validate the given data
import SearchNode from "./index";
const pattern = `<p class="my-text" id="text-number-\${/\d$/}\${index}" datatype="block" key="paragraphs[]">\${data}</p>`
const searchNode = SearchNode.BuildSearchNode(pattern)
// const data: string = ...
const result = searchNode.MapData(data)
for (const parag of result.paragraphs) {
console.log(parag)
}
// {index: "3", data: "Data 1"}
// {index: "5", data: "Data 2"}5. Known Limitations
- The pattern is supposed to be a XML element.
- It is not easy and defeating the purpose of this lib to pick a specific element given its position in the parent