| Info | ||
|---|---|---|
| ||
Secure File Transfer Protocol (SFTP) is a network protocol that provides file access, file transfer, and file management over any reliable data stream. |
| Table of Contents |
|---|
Custom Property Recommendation
| Tip | ||
|---|---|---|
| ||
Ingesting many files via the SFTP connection will open and close SSH connections to the server for each file. These connection attempts might be refused with many parallel tasks running, based on the SSH server settings. For that we recommend to set the following custom property:
This allows using the Hadoop FileServer cache on the cluster; with the FileServer cache enabled for SCP/ SFTP scheme, there will be a single SSH connection for each 'das.cluster.hadoop.fs.cache.scheme.whitelist=scp,sftp'. |
Prerequisites
Set up the Authentication Mechanism
...
- Click the + (plus) button and select Connection or right-click in the browser and select Create new > Connection.
- From drop-down list, select SFTP as the connection type.
- Enter the SFTP host name(s), the port number, authentication credentials, ssh key (if needed), and the root path prefix.
Select if the connection to be used for import, export, or both.
- If required, add a description and click Save.
...