hdfs247

v1.0.0

Published

4 years ago

web HDFS REST API interface.

Downloads

0High
0Medium
0Low

septianw

hdfs webhdfs hadoop

NPM HDFS module

This is node.js interface to communicating with web HDFS interface.

How to use

Installing

Before we can use this module we need to install it first

$ npm install --save hdfs247

Using on Unsecure Cluster

var Hdfs = require('hdfs'),
  localpath = '/tmp/filetest.txt',
  remotepath = '/user/apps/filetest.txt';

var hdfs = new Hdfs({
  protocol: 'http',
  hostname: '192.168.1.225',
  port: 50070
});

hdfs.upload({
  'user.name': 'apps',
  overwrite: true,
  localpath: localpath,
  path: remotepath
}, function(error, response, body) {
  console.log(error);    // Error will be null if upload process is succeed.
  console.log(response); // Raw response from node.js request.
  console.log(body);     // Body of request result.
});

Using on Secure Cluster (Kerberized/SPNego)

var Hdfs = require('hdfs'),
  localpath = '/tmp/filetest.txt',
  remotepath = '/user/apps/filetest.txt';

var hdfs = new Hdfs({
  protocol: 'http',
  hostname: '192.168.1.225',
  port: 50070
});

hdfs.upload({
  'user.name': 'apps',
  overwrite: true,
  localpath: localpath,
  spnego_token: result.token,
  principal: spnego_principal,
  keytab: keytab,
  realm: realm,
  krb_fqdn: krb_fqdn,
  path: remotepath
}, function(error, response, body) {
  console.log(error);    // Error will be null if upload process is succeed.
  console.log(response); // Raw response from node.js request.
  console.log(body);     // Body of request result.
});

Functions

cekpath(param) ⇒ Object

Check the parameter, return path if there is path in parameter, and return / if no path in parameter.

Kind: global function
Returns: Object - Object with only path in it.

| Param | Type | Description | | --- | --- | --- | | param | Object | Parameter from user |

prepvar(param) ⇒ Object

Prepare parameter, separate request option and request parameter, set default option, and explicitly set path.

Kind: global function
Returns: Object - Clean ready to use request parameter.

| Param | Type | Description | | --- | --- | --- | | param | Object | Mixed request parameter. |

Hdfs(data)

Hdfs class constructor

Kind: global function

| Param | Type | Description | | --- | --- | --- | | data | Object | Object data seed. |

Hdfs(data)

hdfs.getfilestatus(param, callback)

Get file status from HDFS Possible properties of param is :

| name | presence | type | default | description | | --- | --- | --- | --- | --- | | path | optional | String | / | Path status that will be taken |

Kind: instance method of Hdfs

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for get file status endpoint | | callback | function | Callback function to return the result |

hdfs.liststatus(param, callback)

List status of file and directory under specified directory

| name | presence | type | default | description | | ---- | -------- | ---- | ------- | ----------- | | path | optional | String | / | Path status that will be taken |

Kind: instance method of Hdfs

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for get list of status file | | callback | function | Callback function to return the result |

hdfs.download(param, callback)

Download file from HDFS with without buffering.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Open_and_Read_a_File

Param properties :

| name | presence | type | default | valid value | description | | ---------- | -------- | ------ | --------- | -------------- | ----------- | | path | optional | String | / | An absolute FileSystem path without scheme and authority. | Path status that will be taken | | offset | optional | Long | 0 | >=0 | The starting byte position | | length | optional | Long | null | >=0 or null | The number of bytes to be processed. null means the entire file | | buffersize | optional | Int | Specified in hadoop configuration | >0 | The size of the buffer used in transferring data. |

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for download file from HDFS | | callback | function | Callback function to return the result |

hdfs.upload(param, callback)

Upload file to HDFS without buffering.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File

Param properties :

| name | presence | type | default | valid value | description | | ----------- | -------- | ------- | ------- | --- | ----------- | | path | optional | String | / | An absolute FileSystem path without scheme and authority. | Path status that will be taken | | overwrite | optional | Boolean | false | true or false | If a file already exists, should it be overwritten? | | blocksize | optional | Long | Specified in hadoop configuration | >0 | The block size of a file. | | replication | optional | Short | Specified in hadoop configuration | >0 | The number of replications of a file. | | permission | optional | octal | 755 | 0 - 1777 | The permission of a file/directory. | | buffersize | optional | Int | Specified in hadoop configuration | >0 | The size of the buffer used in transferring data. |

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for upload file from HDFS | | callback | function | Callback function to return the result |

hdfs.open(param, callback)

Open file from HDFS, this function will buffer content file in memory

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Open_and_Read_a_File

Param properties :

| name | presence | type | default | valid value | description | | ----------- | -------- | ------- | ------- | -------------- | ----------- | | path | optional | String | / | An absolute FileSystem path without scheme and authority. | Path status that will be taken | | offset | optional | Long | 0 | >=0 | The starting byte position | | length | optional | Long | null | >=0 or null | The number of bytes to be processed. null means the entire file | | buffersize | optional | Int | Specified in hadoop configuration | >0 | The size of the buffer used in transferring data. |

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for open file from HDFS | | callback | function | Callback function to return the result |

hdfs.gethomedirectory(param, callback)

Get home directory of current accessing user.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Home_Directory

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for getting home directory. | | callback | function | Callback function to return the result |

hdfs.getcontentsummary(param, callback)

Get summary of content

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Content_Summary_of_a_Directory

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for getting home directory. | | callback | function | Callback function to return the result |

hdfs.getfilechecksum(param, callback)

Get checksum of file

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_File_Checksum

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for get checksum of file. | | callback | function | Callback function to return the result |

hdfs.concat(param, callback)

Concatenating file to another one. Warning : It's still experimental. We warn you.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Concat_Files

| name | presence | type | default | valid value | description | | ----------- | -------- | ------- | ------- | -------------- | ----------- | | path | Mandatory | String | / | An absolute FileSystem path without scheme and authority. | Path status that will be taken | | sources | Mandatory | String | | A list of comma seperated absolute FileSystem paths without scheme and authority. | A list of source paths. |

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for concatenating file. | | callback | function | Callback function to return the result |

hdfs.append(param, callback)

Append data to another file in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Append_to_a_File

Param properties :

| name | presence | type | default | valid value | description | | ----------- | --------- | ------- | ------- | ----------- | ----------- | | path | Mandatory | String | / | An absolute FileSystem path without scheme and authority. | Path status that will be taken | | buffersize | optional | Int | Specified in hadoop configuration | >0 | The size of the buffer used in transferring data. |

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for upload file from HDFS | | callback | function | Callback function to return the result |

hdfs.create(param, callback)

Create file in HDFS.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File

Param properties :

| name | presence | type | default | valid value | description | | ----------- | --------- | ------- | ------- | --- | ----------- | | path | Mandatory | String | / | An absolute FileSystem path without scheme and authority. | Path status that will be taken | | overwrite | optional | Boolean | false | true or false | If a file already exists, should it be overwritten? | | blocksize | optional | Long | Specified in hadoop configuration | >0 | The block size of a file. | | replication | optional | Short | Specified in hadoop configuration | >0 | The number of replications of a file. | | permission | optional | octal | 755 | 0 - 1777 | The permission of a file/directory. | | buffersize | optional | Int | Specified in hadoop configuration | >0 | The size of the buffer used in transferring data. |

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for upload file from HDFS | | callback | function | Callback function to return the result |

hdfs.rename(param, callback)

Rename file or directory in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Rename_a_FileDirectory

Param properties :

| name | presence | type | default | valid value | description | | ----------- | --------- | ------- | ------- | --- | ----------- | | path | Mandatory | String | / | An absolute FileSystem path without scheme and authority. | Path status that will be taken | | destination | Mandatory | String | (an invalid path) | An absolute FileSystem path without scheme and authority. | The destination path. |

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for rename file or directory. | | callback | function | Callback function to return the result |

hdfs.mkdirs(param, callback)

Make directory in Hdfs

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Make_a_Directory

Param properties :

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for make directory in HDFS | | callback | function | Callback function to return the result |

hdfs.setpermission(param, callback)

Set permission of file in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Set_Permission

Param properties :

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for set permission of file and directory in HDFS | | callback | function | Callback function to return the result |

hdfs.setowner(param, callback)

Set owner of file or directory

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Set_Owner

Param properties :

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for set owner in HDFS | | callback | function | Callback function to return the result |

hdfs.setreplication(param, callback)

Set replication factor of file.

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Set_Replication_Factor

Param properties :

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for set replication file. | | callback | function | Callback function to return the result |

hdfs.settimes(param, callback)

Set time and date of file in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Set_Access_or_Modification_Time

Param properties :

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for setting time and date of file. | | callback | function | Callback function to return the result |

hdfs.delete(param, callback)

Delete file or directory in HDFS

Kind: instance method of Hdfs
See: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Delete_a_FileDirectory

Param properties :

| Param | Type | Description | | --- | --- | --- | | param | Object | Request parameter for deleting file and or directory. | | callback | function | Callback function to return the result |

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

NPM HDFS module

How to use

Installing

Using on Unsecure Cluster

Using on Secure Cluster (Kerberized/SPNego)

Functions

cekpath(param) ⇒ Object

prepvar(param) ⇒ Object

Hdfs(data)

hdfs.getfilestatus(param, callback)

hdfs.liststatus(param, callback)

hdfs.download(param, callback)

hdfs.upload(param, callback)

hdfs.open(param, callback)

hdfs.gethomedirectory(param, callback)

hdfs.getcontentsummary(param, callback)

hdfs.getfilechecksum(param, callback)

hdfs.concat(param, callback)

hdfs.append(param, callback)

hdfs.create(param, callback)

hdfs.rename(param, callback)

hdfs.mkdirs(param, callback)

hdfs.setpermission(param, callback)

hdfs.setowner(param, callback)

hdfs.setreplication(param, callback)

hdfs.settimes(param, callback)

hdfs.delete(param, callback)