Unix: Flexibly moving files with lftp
Moving files between Linux (and other) servers is a lot smarter and more versatile when you discover lftp.
What is lftp?
Some sites -- and even the man page -- claim that "lftp" stands for "sophisticated file transfer program", but I'm still struggling with that one. I might have suggested "limber ftp", "lithe ftp", even "liberal ftp", but someone decided that the "l" could stand for "sophisticated", so who am I to question their logic?
A more important question is "How has it escaped my attention until now?". Maybe because I pretty much stopped using ftp commands years ago in favor of scp. Maybe simply because no one brought this tool to my attention -- until now. But lftp is hardly new (it initially showed up in 1996, but wasn't called lftp until 1997) and its flexibility takes it way beyond what anyone would expect of an ftp program.
Yes, lftp has a lot of interesting features and just got shoved in front of my face when a vendor I deal with changed their download options to be more secure. Using lftp from a Unix command line was one of the choices presented to me going forward. So exactly what is lftp and why it is so much worth using and understanding?
For one thing, lftp doesn't just support ftp. It also supports many access methods including FTP, HTTP, FISH, SFTP, HTTPS and FTPS. And it can download an entire website if you do a recursive get with the tool's mirror command.
The command also has a great handle on reliability. Non-fatal errors are ignored and operations restarted as needed. My first impression was that it's a solid tool and performs very well.
With lftp, you can launch commands in parallel by putting them in the background much as we do on the Unix/Linux command line using & and you can group commands within parentheses and run the whole group in background. To move a running job into the background, just use ^z. You can then bring it back into the foreground with fg. How cool is that? And a jobs command is available to list running jobs.
If some of your jobs haven't completed when you exit lftp, lftp will put itself in nohup (no hangup) mode in the background to complete them.
That's a lot to get from anything that calls itself an ftp program.
What I needed to do was run a series of set commands to prepare my lftp session to work with the server that I needed to grab files from. Fortunately, I quickly determined that I didn't need to type these commands every time I went to the site for updates. Instead, I could drop the needed commands into either a global configuration file (good choice if other accounts on the server would need the same setup) or into a dot file in my personal account (~/.lftprc or ~/.lftp/rc). My ~/.lftprc file now looks like this:
set ftp:ssl-protect-data true set ftps:initial-prot set ftp:ssl-force true set ftp:ssl-protect-data true set ssl:verify-certificate off
This gives me encrypted connections to the server when I type lftp ftp.targetsite.com.
You can launch lftp by typing just lftp and then using an open command to take you to your target site or you can provide the target's name on the same line as lftp like I did.
$ lftp ftp.targetsite.com
$ lftp lftp :~> open ftp.targetsite.com
Once you log in, you can list files and use get, mget, put etc. commands as needed:
$ lftp lftp :~> open ftp.targetsite.com lftp ftp.targetsite.com:~> user sandrahs NotMyRealPassword lftp firstname.lastname@example.org:~> ls
If you ask for help, you're likely to see something like this, showing all the commands that are available to you:
lftp :~> help !<shell-command> (commands) alias [<name> [<value>]] anon bookmark [SUBCMD] cache [SUBCMD] cat [-b] <files> cd <rdir> chmod [OPTS] mode file... close [-a] [re]cls [opts] [path/][pattern] debug [<level>|off] [-o <file>] du [options] <dirs> exit [<code>|bg] get [OPTS] <rfile> [-o <lfile>] glob [OPTS] <cmd> <args> help [<cmd>] jobs [-v] kill all|<job_no> lcd <ldir> lftp [OPTS] <site> ls [<args>] mget [OPTS] <files> mirror [OPTS] [remote [local]] mkdir [-p] <dirs> module name [args] more <files> mput [OPTS] <files> mrm <files> mv <file1> <file2> [re]nlist [<args>] open [OPTS] <site> pget [OPTS] <rfile> [-o <lfile>] put [OPTS] <lfile> [-o <rfile>] pwd [-p] queue [OPTS] [<cmd>] quote <cmd> repeat [OPTS] [delay] [command] rm [-r] [-f] <files> rmdir [-f] <dirs> scache [<session_no>] set [OPT] [<var> [<val>]] site <site_cmd> source <file> user <user|URL> [<pass>] version wait [<jobno>] zcat <files> zmore <files> history -w file|-r file|-c|-l [cnt]
If you want to download an entire directory, you can use the mirror command. Typing mirror images, for example, will copy the images folder from the remote system
and create one in your local directory, complete with images and subdirectories if they exist. Use a second argument if you want to give your local folder a different name.
Once you've logged off, you can look through your .lftp directory and peruse the history (cwd_history) file that was left there showing the sites you visited and the rl_history file to see what commands you issued and files you downloaded.
cd .lftp ls -l -rw------- 1 sandrahs dweebs 43 Sep 1 17:43 cwd_history -rw------- 1 sandrahs dweebs 252 Sep 1 17:43 rl_history -rw------- 1 sandrahs dweebs 18554 Sep 1 17:02 transfer_log
$ cat cwd_history ftp://email@example.com 1378071811:/ $ cat rl_history lftp ftp.targetsite.com user sandrahs NotMyRealPassword mirror images quit $ cat transfer_log 2013-09-01 17:00:43 ftp://sandrahs:NotMyRealPassword@ftp.targetsite.com /images/mugshot.jpg 0-629374 557.7K/s 2013-09-01 17:00:44 ftp://sandrahs:NotMyRealPassword@ftp.targetsite.com /images/myhome.jpg 0-87454 893.1K/s
Moving files around deftly, reliably and securely has just gotten easier. I'm impressed with this limber, lithe, liberal and, ok, sophisticated ftp tool.
Read more of Sandra Henry-Stocker's Unix as a Second Language blog and follow the latest IT news at ITworld, Twitter and Facebook.
flickr / hbp_pix