Youtube Scraper

| feature | Video | Channel | |--------------------|:------------------------------------:|:--------------:| | title | ✅ | ✅ | | description | ✅ | ✅ | | thumbnail | ✅ | ✅ | | transcripts | ✅ | N / A | | video download url | ✅ | N / A | | other fields | views, duration, viewCount, and more | list of videos |

Also supports YouTube search

Try out this scraper for free on Apify.com

Install

npm install @scrappy-scraper/youtube_scraper    # npm
yarn add @scrappy-scraper/youtube_scraper       # yarn

Environment

Works in Node.js and react-native

Works in Chrome extension when running on youtube.com page.

Does not work in browser due to limited CORS setting from YouTube

Sample Code

/* Get VideoInfo and Download Captions */

import { VideoParser } from '@scrappy-scraper/youtube_scraper';
const videoParser = new VideoParser();
await videoParser.load({videoId: "dQw4w9WgXcQ"}); // Rick Roll

// show the available caption tracks
console.log(videoParser.availableCaptions);     // example: [ { name: 'English', languageCode: 'en', isGenerated: false } ]

// fetch transcripts
await videoParser.fetchTranscripts({languageLimit: 3, preferredLanguages: ['en']})

// show all the data
console.log(JSON.stringify(videoParser.toJSON())) // captions are inside this JSON, along with other data

/* Get ChannelInfo and list of videos */

import { ChannelParser } from '@scrappy-scraper/youtube_scraper';
const channelParser = new ChannelParser();
await channelParser.load({ channelId: "@RickAstleyYT" }); // channelId can be a handle or a regular id

if(channelParser.hasMoreVideos()) await channelParser.fetchMoreVideos(); // can use "while" instead of "if"

console.log(channelParser.toJSON())

/* Perform YouTube Search */

import { SearchHandler } from '@scrappy-scraper/youtube_scraper';
let searchHandler = new SearchHandler();
await searchHandler.search({ query: "Rick Astley" });

if(searchHandler.hasMoreItems()) await searchHandler.fetchMoreItems(); // can use "while" instead of "if"
console.log(searchHandler.toJSON())

proxyUrlGenerator

Please put in Rotating Residential proxy url.

Session is supported by putting in :sessionId into the randomized segment of the url.

For example:

// Provided by Proxy provider:
"http://my_user_name_sid-adsfasdf:[email protected]:port"
"http://my_user_name_sid-qwerqwer:[email protected]:port"
"http://my_user_name_sid-zxcvzxcv:[email protected]:port"

// What you put in:
"http://my_user_name_sid-:sessionId:[email protected]:port"

// create proxyUrlGenerator 
const proxyUrlGenerator = async (sessionId: string|null|undefined): Promise<string> => {
    return "http://username:password@host:port"
        .replace(":sessionId", sessionId || Math.round(Math.random() * 10**6).toString());
}

// instantiate parsers with the proxyUrlGenerator
let videoParser = new VideoParser({ proxyUrlGenerator });
let channelParser = new ChannelParser({ proxyUrlGenerator });
let searchHandler = new SearchHandler({ proxyUrlGenerator });

Local Build and Test

If you want to make modification and test it, look for the test file index.test.ts The main code is in src/index.ts

Clone the Repository

git clone https://github.com/Scrappy-Scraper/YoutubeScraper.git
cd YoutubeScraper
npm install

To Test Run

npm run test       # npm
yarn run test      # yarn

To Build JS File

npm run build       # npm
yarn run build      # yarn