@InterfaceAudience.Public @InterfaceStability.Stable public abstract class ChecksumFileSystem extends FilterFileSystem
fs, swapSchemeDEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX| Constructor and Description |
|---|
ChecksumFileSystem(FileSystem fs) |
| Modifier and Type | Method and Description |
|---|---|
FSDataOutputStream |
append(Path f,
int bufferSize,
Progressable progress)
Append to an existing file (optional operation).
|
FSDataOutputStreamBuilder |
appendFile(Path path)
This is overridden to ensure that this class's create() method is
ultimately called.
|
void |
completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target.
|
void |
concat(Path f,
Path[] psrcs)
Concat existing files together.
|
void |
copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is on the local disk.
|
void |
copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
void |
copyToLocalFile(Path src,
Path dst,
boolean copyCrc)
The src file is under FS, and the dst is on the local disk.
|
FSDataOutputStream |
create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
create(Path f,
FsPermission permission,
EnumSet<CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)
Create an FSDataOutputStream at the indicated Path with a custom
checksum option.
|
FSDataOutputStreamBuilder |
createFile(Path path)
This is overridden to ensure that this class's create() method is
ultimately called.
|
FSDataOutputStream |
createNonRecursive(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
createNonRecursive(Path f,
FsPermission permission,
EnumSet<CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
boolean |
delete(Path f,
boolean recursive)
Implement the delete(Path, boolean) in checksum
file system.
|
static double |
getApproxChkSumLength(long size) |
int |
getBytesPerSum()
Return the bytes Per Checksum.
|
Path |
getChecksumFile(Path file)
Return the name of the checksum file associated with a file.
|
long |
getChecksumFileLength(Path file,
long fileSize)
Return the length of the checksum file given the size of the
actual file.
|
static long |
getChecksumLength(long size,
int bytesPerSum)
Calculated the length of the checksum file in bytes.
|
FileSystem |
getRawFileSystem()
get the raw file system
|
boolean |
hasPathCapability(Path path,
String capability)
Disable those operations which the checksummed FS blocks.
|
static boolean |
isChecksumFile(Path file)
Return true if file is a checksum file name.
|
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> |
listLocatedStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
FileStatus[] |
listStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
org.apache.hadoop.fs.RemoteIterator<FileStatus> |
listStatusIterator(Path p)
Return a remote iterator for listing in a directory
|
boolean |
mkdirs(Path f)
Call
FileSystem.mkdirs(Path, FsPermission) with default permission. |
void |
modifyAclEntries(Path src,
List<AclEntry> aclSpec)
Modifies ACL entries of files and directories.
|
FSDataInputStream |
open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
FutureDataInputStreamBuilder |
openFile(Path path)
This is overridden to ensure that this class's
openFileWithOptions(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.impl.OpenFileParameters)() method is called, and so ultimately
its open(Path, int). |
protected CompletableFuture<FSDataInputStream> |
openFileWithOptions(Path path,
org.apache.hadoop.fs.impl.OpenFileParameters parameters)
Open the file as a blocking call to
open(Path, int). |
void |
removeAcl(Path src)
Removes all but the base ACL entries of files and directories.
|
void |
removeAclEntries(Path src,
List<AclEntry> aclSpec)
Removes ACL entries from files and directories.
|
void |
removeDefaultAcl(Path src)
Removes all default ACL entries from files and directories.
|
boolean |
rename(Path src,
Path dst)
Rename files/dirs
|
boolean |
reportChecksumFailure(Path f,
FSDataInputStream in,
long inPos,
FSDataInputStream sums,
long sumsPos)
Report a checksum error to the file system.
|
void |
setAcl(Path src,
List<AclEntry> aclSpec)
Fully replaces ACL of files and directories, discarding all existing
entries.
|
void |
setConf(Configuration conf)
Set the configuration to be used by this object.
|
void |
setOwner(Path src,
String username,
String groupname)
Set owner of a path (i.e.
|
void |
setPermission(Path src,
FsPermission permission)
Set permission of a path.
|
boolean |
setReplication(Path src,
short replication)
Set replication for an existing file.
|
void |
setVerifyChecksum(boolean verifyChecksum)
Set whether to verify checksum.
|
void |
setWriteChecksum(boolean writeChecksum)
Set the write checksum flag.
|
Path |
startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to.
|
boolean |
truncate(Path f,
long newLength)
Truncate the file in the indicated path to the indicated size.
|
access, canonicalizeUri, checkPath, close, copyFromLocalFile, copyFromLocalFile, createPathHandle, createSnapshot, createSymlink, deleteSnapshot, getAclStatus, getAllStoragePolicies, getCanonicalUri, getChildFileSystems, getConf, getDefaultBlockSize, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getServerDefaults, getServerDefaults, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUri, getUsed, getUsed, getWorkingDirectory, getXAttr, getXAttrs, getXAttrs, initialize, listCorruptFileBlocks, listLocatedStatus, listXAttrs, makeQualified, mkdirs, msync, open, openFile, openFileWithOptions, primitiveCreate, primitiveMkdir, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setStoragePolicy, setTimes, setWorkingDirectory, setXAttr, setXAttr, supportsSymlinks, unsetStoragePolicyappend, append, append, append, areSymlinksEnabled, cancelDeleteOnExit, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, createDataOutputStreamBuilder, createMultipartUploader, createNewFile, createNonRecursive, createSnapshot, delete, deleteOnExit, enableSymlinks, exists, fixRelativePart, get, get, get, getAllStatistics, getBlockSize, getCanonicalServiceName, getContentSummary, getDefaultPort, getDefaultUri, getFileBlockLocations, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getLength, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getScheme, getStatistics, getStatistics, getStatus, getStorageStatistics, globStatus, globStatus, isDirectory, isFile, listFiles, listStatus, listStatus, listStatus, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, open, primitiveMkdir, printStatistics, processDeleteOnExit, setDefaultUri, setDefaultUri, setQuota, setQuotaByStorageTypepublic ChecksumFileSystem(FileSystem fs)
public static double getApproxChkSumLength(long size)
public void setConf(Configuration conf)
ConfigurablesetConf in interface ConfigurablesetConf in class Configuredconf - configuration to be usedpublic void setVerifyChecksum(boolean verifyChecksum)
setVerifyChecksum in class FilterFileSystemverifyChecksum - Verify checksum flagpublic void setWriteChecksum(boolean writeChecksum)
FileSystemsetWriteChecksum in class FilterFileSystemwriteChecksum - Write checksum flagpublic FileSystem getRawFileSystem()
getRawFileSystem in class FilterFileSystempublic Path getChecksumFile(Path file)
file - the file path.public static boolean isChecksumFile(Path file)
file - the file path.public long getChecksumFileLength(Path file, long fileSize)
file - the file path.fileSize - file size.public int getBytesPerSum()
public FSDataInputStream open(Path f, int bufferSize) throws IOException
open in class FilterFileSystemf - the file name to openbufferSize - the size of the buffer to be used.IOException - if an I/O error occurs.public FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws IOException
FileSystemappend in class FilterFileSystemf - the existing file to be appended.bufferSize - the size of the buffer to be used.progress - for reporting progress if it is not null.IOException - IO failurepublic boolean truncate(Path f, long newLength) throws IOException
FileSystemtruncate in class FilterFileSystemf - The path to the file to be truncatednewLength - The size the file is to be truncated totrue if the file has been truncated to the desired
newLength and is immediately available to be reused for
write operations such as append, or
false if a background process of adjusting the length of
the last block has been started, and clients should wait for it to
complete before proceeding with further file updates.IOException - IO failurepublic void concat(Path f, Path[] psrcs) throws IOException
FileSystemconcat in class FilterFileSystemf - the path to the target destination.psrcs - the paths to the sources to use for the concatenation.IOException - IO failurepublic static long getChecksumLength(long size,
int bytesPerSum)
size - the length of the data file in bytesbytesPerSum - the number of bytes in a checksum blockpublic FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
FileSystemcreate in class FilterFileSystemf - the file name to openpermission - file permissionoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - block sizeprogress - the progress reporterIOException - IO failureFileSystem.setPermission(Path, FsPermission)public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
FileSystemcreateNonRecursive in class FileSystemf - the file name to openpermission - file permissionoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - block sizeprogress - the progress reporterIOException - IO failureFileSystem.setPermission(Path, FsPermission)public FSDataOutputStream create(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt) throws IOException
FileSystemcreate in class FilterFileSystemf - the file name to openpermission - file permissionflags - CreateFlags to use for this stream.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - block sizeprogress - the progress reporterchecksumOpt - checksum parameter. If null, the values
found in conf will be used.IOException - IO failureFileSystem.setPermission(Path, FsPermission)public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
FileSystemcreateNonRecursive in class FilterFileSystemf - the file name to openpermission - file permissionflags - CreateFlags to use for this stream.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - block sizeprogress - the progress reporterIOException - IO failureFileSystem.setPermission(Path, FsPermission)public void setPermission(Path src, FsPermission permission) throws IOException
FileSystemsetPermission in class FilterFileSystemsrc - The pathpermission - permissionIOException - IO failurepublic void setOwner(Path src, String username, String groupname) throws IOException
FileSystemsetOwner in class FilterFileSystemsrc - The pathusername - If it is null, the original username remains unchanged.groupname - If it is null, the original groupname remains unchanged.IOException - IO failurepublic void setAcl(Path src, List<AclEntry> aclSpec) throws IOException
FileSystemsetAcl in class FilterFileSystemsrc - Path to modifyaclSpec - List describing modifications, which must include entries
for user, group, and others for compatibility with permission bits.IOException - if an ACL could not be modifiedpublic void modifyAclEntries(Path src, List<AclEntry> aclSpec) throws IOException
FileSystemmodifyAclEntries in class FilterFileSystemsrc - Path to modifyaclSpec - List<AclEntry> describing modificationsIOException - if an ACL could not be modifiedpublic void removeAcl(Path src) throws IOException
FileSystemremoveAcl in class FilterFileSystemsrc - Path to modifyIOException - if an ACL could not be removedpublic void removeAclEntries(Path src, List<AclEntry> aclSpec) throws IOException
FileSystemremoveAclEntries in class FilterFileSystemsrc - Path to modifyaclSpec - List describing entries to removeIOException - if an ACL could not be modifiedpublic void removeDefaultAcl(Path src) throws IOException
FileSystemremoveDefaultAcl in class FilterFileSystemsrc - Path to modifyIOException - if an ACL could not be modifiedpublic boolean setReplication(Path src, short replication) throws IOException
setReplication in class FilterFileSystemsrc - file namereplication - new replicationIOException - if an I/O error occurs.public boolean rename(Path src, Path dst) throws IOException
rename in class FilterFileSystemsrc - path to be renameddst - new path after renameIOException - on failurepublic boolean delete(Path f, boolean recursive) throws IOException
delete in class FilterFileSystemf - the path to delete.recursive - if path is a directory and set to
true, the directory is deleted else throws an exception. In
case of a file the recursive can be set to either true or false.IOException - IO failurepublic FileStatus[] listStatus(Path f) throws IOException
listStatus in class FilterFileSystemf - given pathIOException - if an I/O error occurs.public org.apache.hadoop.fs.RemoteIterator<FileStatus> listStatusIterator(Path p) throws IOException
FilterFileSystemlistStatusIterator in class FilterFileSystemp - target pathFileNotFoundException - if p does not existIOException - if any I/O error occurredpublic org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> listLocatedStatus(Path f) throws IOException
listLocatedStatus in class FilterFileSystemf - given pathIOException - if an I/O error occurs.public boolean mkdirs(Path f) throws IOException
FileSystemFileSystem.mkdirs(Path, FsPermission) with default permission.mkdirs in class FilterFileSystemf - pathIOException - IO failurepublic void copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
throws IOException
FilterFileSystemcopyFromLocalFile in class FilterFileSystemdelSrc - whether to delete the srcsrc - pathdst - pathIOException - IO failure.public void copyToLocalFile(boolean delSrc,
Path src,
Path dst)
throws IOException
copyToLocalFile in class FilterFileSystemdelSrc - whether to delete the srcsrc - path src file in the remote filesystemdst - path local destinationIOException - IO failurepublic void copyToLocalFile(Path src, Path dst, boolean copyCrc) throws IOException
src - src path.dst - dst path.copyCrc - copy csc flag.IOException - if an I/O error occurs.public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
FilterFileSystemstartLocalOutput in class FilterFileSystemfsOutputFile - path of output filetmpLocalFile - path of local tmp fileIOException - IO failurepublic void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
FilterFileSystemcompleteLocalOutput in class FilterFileSystemfsOutputFile - path of output filetmpLocalFile - path to local tmp fileIOException - IO failurepublic boolean reportChecksumFailure(Path f, FSDataInputStream in, long inPos, FSDataInputStream sums, long sumsPos)
f - the file name containing the errorin - the stream open on the fileinPos - the position of the beginning of the bad data in the filesums - the stream open on the checksum filesumsPos - the position of the beginning of the bad data in the checksum filepublic FutureDataInputStreamBuilder openFile(Path path) throws IOException, UnsupportedOperationException
openFileWithOptions(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.impl.OpenFileParameters)() method is called, and so ultimately
its open(Path, int).
Open a file for reading through a builder API.
Ultimately calls FileSystem.open(Path, int) unless a subclass
executes the open command differently.
The semantics of this call are therefore the same as that of
FileSystem.open(Path, int) with one special point: it is in
FSDataInputStreamBuilder.build() in which the open operation
takes place -it is there where all preconditions to the operation
are checked.openFile in class FilterFileSystempath - file pathIOException - if some early checks cause IO failures.UnsupportedOperationException - if support is checked early.protected CompletableFuture<FSDataInputStream> openFileWithOptions(Path path, org.apache.hadoop.fs.impl.OpenFileParameters parameters) throws IOException
open(Path, int).
Execute the actual open file operation.
This is invoked from FSDataInputStreamBuilder.build()
and from DelegateToFileSystem and is where
the action of opening the file should begin.
The base implementation performs a blocking
call to FileSystem.open(Path, int) in this call;
the actual outcome is in the returned CompletableFuture.
This avoids having to create some thread pool, while still
setting up the expectation that the get() call
is needed to evaluate the result.openFileWithOptions in class FilterFileSystempath - path to the fileparameters - open file parameters from the builder.IOException - failure to resolve the link.public FSDataOutputStreamBuilder createFile(Path path)
createFile in class FilterFileSystempath - file pathpublic FSDataOutputStreamBuilder appendFile(Path path)
appendFile in class FilterFileSystempath - file path.FSDataOutputStreamBuilder to build file append request.public boolean hasPathCapability(Path path, String capability) throws IOException
true, this instance is explicitly
declaring that the capability is available.
If the function returns false, it can mean one of:
Implementors: PathCapabilitiesSupport
can be used to help implement this method.
hasPathCapability in interface org.apache.hadoop.fs.PathCapabilitieshasPathCapability in class FilterFileSystempath - path to query the capability of.capability - non-null, non-empty string to query the path for support.IOException - this should not be raised, except on problems
resolving paths or relaying the call.Copyright © 2023 Apache Software Foundation. All rights reserved.