pyhdfs的安装

1.升级python

将自带python2.6.6升级到python2.7

[root@localhost ~]# wget

http://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz

--2016-12-15 11:44:57--

http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz Resolving www.python.org... 151.101.16.223, 2a04:4e42:4::223 Connecting to www.python.org|151.101.16.223|:80... connected. HTTP request sent, awaiting response... 302 Found Location:

http://219.238.7.71/files/2104000008A9FBE4/mitk.org/download/thirdparty/Python-2.7.3.tgz [following] --2016-12-15 11:45:01--

http://219.238.7.71/files/2104000008A9FBE4/mitk.org/download/thirdparty/Python-2.7.3.tgz

Connecting to 219.238.7.71:80... connected. HTTP request sent, awaiting response... 200 OK Length: 14135620 (13M) [application/octet-stream] Saving to: “Python-2.7.3.tgz”

100%[===============================================================>] 14,135,620 143K/s in 2m 42s

2016-12-15 11:47:44 (85.3 KB/s) - “Python-2.7.3.tgz” saved [14135620/14135620]

进入路径并编译：

[root@localhost ~]# tarzxf Python-2.7.13.tgz

[root@localhost ~]# ls

anaconda-ks.cfg Documents install.log Music Public Python-2.7.3.tgz Videos

Desktop Downloads install.log.syslog Pictures Python-2.7.3 Templates

[root@localhost ~]# cd Python-2.7.13

[root@localhost Python-2.7.13]# ./configure

[root@localhost Python-2.7.13]#make && make install

[root@localhost Python-2.7.13]# mv /usr/bin/python /usr/bin/python_old

[root@localhost Python-2.7.13]# python --version

-bash: /usr/bin/python: No such file or directory

[root@localhost Python-2.7.13]# ln -s /usr/local/bin/python2.7

/usr/bin/python

[root@localhost Python-2.7.13]# python --version Python 2.7.13

更改yum的配置文件（因为升级到python2.7之后，原来的yum就无法使用了）

[root@localhost Python-2.7.13]# vim /usr/bin/yum

[root@localhost Python-2.7.3]# yum install zlib-devel 再重新编译python2.7

[root@localhost Python-2.7.3]# ./configure

[root@localhost Python-2.7.3]#make && make install

easy_install安装

#（上面由于使用了https和ssl的原因，需要使用上面两个参数，否则不能下载，将会提示\）

# tar xzvf setuptools-0.6c11.tar.gz #cd setuptools-0.6c11 #python setup.py build #python setup.py install

pip安装

#yum install openssl

#yum install openssl-devel 再次编译python

[root@hadoop1 Python-2.7.13]# ./configure #make&& make install./

# wget

\.gz#md5=834b2904f92d46aaa333267fb1c922bb\--no-check-certificate

# tar -xzvf pip-1.5.4.tar.gz # cd pip-1.5.4

# python setup.py install

升级setuptools

#pip install -U pip setuptools

Pyhdfs安装

#pip install pyhdfs #pip install hdfs

Successfully installed pyhdfs requests simplejson Cleaning up... 测试

1. 先启动hadoop 2. 编译代码

importpyhdfs

fs = pyhdfs.HdfsClient(hosts=\f = fs.listdir(\ print f

append(path, data, **kwargs)

Append to the given file.

data – bytes or a file-like object ?

Parameters: ? buffersize (int) – The size of the buffer used in

transferring data.

concat(target, sources, **kwargs)

Concat existing files together. Conditions: ? ? The last block in the target file (path) must be full. All blocks must be the same size, except possibly the last block. ?

target – the path to the target destination.

Parameters: ? sources (list) – the paths to the sources to use for

the concatenation.

copy_from_local(localsrc, dest, **kwargs)

Copy a single file from the local file system to dest Takes all arguments that create() takes. copy_to_local(src, localdest, **kwargs)

Copy a single file from src to the local file system Takes all arguments that open() takes. create(path, data, **kwargs)

Create a file at the given path.

data – bytes or a file-like object to upload overwrite (bool) – If a file already exists, should it be overwritten?

? ?

? ?

blocksize (long) – The block size of a file. replication (short) – The number of replications of a file.

permission (octal) – The permission of a

file/directory. Any radix-8 integer (leading zeros may be omitted.)

Parameters:

?

?

buffersize (int) – The size of the buffer used in transferring data.

create_snapshot(path, **kwargs)

Create a snapshot

?

path – The directory where snapshots will be taken

Parameters: ? snapshotname – The name of the snapshot

Returns: the snapshot path

create_symlink(link, destination, **kwargs)

Create a symbolic link at link pointing to destination. ? ?

link – the path to be created that points to target destination – the target of the symbolic link exist, should they be created?

Parameters: ? createParent (bool) – If the parent directories do not

Raises:

HdfsUnsupportedOperationException – This feature

doesn’t actually work, at least on CDH 5.3.0.

delete(path, **kwargs)

Delete a file.

Parameters: recursive (bool) – If path is a directory and set to true,

the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Return type: bool

delete_snapshot(path, snapshotname, **kwargs)

Delete a snapshot of a directory

exists(path, **kwargs)

Return true if the given path exists

get_active_namenode(max_staleness=None)

Return the address of the currently active NameNode.

max_staleness (float) – This function caches the active NameNode. If this age of this cached result is less

Parameters:

than max_staleness seconds, return it. Otherwise, or if this parameter is None, do a lookup.

Raises:

HdfsNoServerException – can’t find an active NameNode

get_content_summary(path, **kwargs)

Return the ContentSummary of a given Path. get_file_checksum(path, **kwargs)

Get the checksum of a file.

Return type: FileChecksumget_file_status(path, **kwargs)

Return a FileStatus object that represents the path. get_home_directory(**kwargs)

Return the current user’s home directory in this filesystem.

get_xattrs(path, xattr_name=None, encoding=u'text', **kwargs)

Get one or more xattr values for a file or directory.

? xattr_name – str to get one attribute, list to get multiple attributes, None to get all attributes.

Parameters:

? encoding – text | hex | base64, defaults to text Returns:

Dictionary mapping xattr name to value. With text encoding, the value will be a unicode string. With hex or base64 encoding, the value will be a byte array.

Return type: dict list_status(path, **kwargs)

List the statuses of the files/directories in the given path if the path is a directory.

Return type: list of FileStatus objects

list_xattrs(path, **kwargs)

Get all of the xattr names for a file or directory.

Return type: list

listdir(path, **kwargs)

Return a list containing names of files in the given path

mkdirs(path, **kwargs)

Create a directory with the provided permission.

The permission of the directory is set to be the provided permission as in setPermission, not permission&~umask.

permission (octal) – The permission of a file/directory. Any radix-8 integer (leading zeros may be omitted.) true if the directory creation succeeds; false otherwise

Parameters: Returns:

Return type: bool

open(path, **kwargs)

Return a file-like object for reading the given HDFS path.

? ?

offset (long) – The starting byte position. length (long) – The number of bytes to be processed. transferring data.

Parameters: ? buffersize (int) – The size of the buffer used in

Return type: file-like object remove_xattr(path, xattr_name, **kwargs)

Remove anxattr of a file or directory.

rename(path, destination, **kwargs)

Renames Path src to Path dst.

Returns:

true if rename is successful

Return type: bool

rename_snapshot(path, oldsnapshotname, snapshotname, **kwargs)

Rename a snapshot

set_owner(path, **kwargs)

Set owner of a path (i.e. a file or a directory).

The parameters owner and group cannot both be null.

?

owner – user

Parameters: ? group – group

set_permission(path, **kwargs)

Set permission of a path.

permission (octal) – The permission of a file/directory. Any radix-8 integer (leading zeros may be omitted.)

Parameters:

set_replication(path, **kwargs)

Set replication for an existing file.

Parameters: replication (short) – new replication

Returns:

true if successful; false if file does not exist or is a directory

Return type: bool

set_times(path, **kwargs)

Set access time of a file.

?

modificationtime (long) – Set the modification time of this file. The number of milliseconds since Jan 1, 1970.

Parameters: ? accesstime (long) – Set the access time of this file.

The number of milliseconds since Jan 1 1970.

set_xattr(path, xattr_name, xattr_value, flag, **kwargs)

Set anxattr of a file or directory.

?

xattr_name – The name must be prefixed with the namespace followed by .. For example, user.attr. Parameters:

?

flag – CREATE or REPLACE walk(top, topdown=True, onerror=None, **kwargs)

See os.walk for documentation

本文来源：https://www.bwwdw.com/article/pbmx.html