In this lecture we’ll be discussing the wget command. 1
This is a GNU command line utility for downloading files from HTTP, HTTPS of FTP servers. 2
It provides a number of options, allowing you to download multiple files, resume downloads, limit 3
bandwidth, mirror a website and much more. 4
The wget package is pre-installed on most Linux distributions, but in case you don’t have 5
it, simply install it by running apt install wget on Ubuntu or dnf install 6
wget on Centos. 7
Let’s see the first example and download Kali Linux as an ISO file that can be later installed. Btw Kali is 8
a Linux distribution used for defensive security and ethical hacking. The first step is to get the url of 9
the file. 10
You normally go to the website, find the url, check that it goes to the right file and copy it 11
to the clipboard. 12
So, right click on it and copy the link location. 13
In the terminal,write wget and paste the url from the clipboard. 14
Note that in the terminal, the shortcut is control shift v to paste from the clipboard. I'm hitting 15
enter and it started downloading the file. 16
See how wget is showing the progress bar along side with the filename, file size, download speed 17
and the estimated time to complete the download. Once the download is complete, you will find the 18
downloaded file in the current directory. 19
Now I'm interrupting the download by pressing control and c. If you want to save the file into another 20
directory 21
use the capital P option, for example 22
mkdir kali wget - P the destination and the url 23
I'm pasting the clipboard. 24
OK, I'm interrupting it because it takes some time. 25
If you want to limit the download speed, so that wget does not consume the entire available 26
bandwidth, use the --limit-rate option. By default, wget will consume as much 27
bandwidth as it can. 28
I'm going to limit wget to 100 kilobytes per second. 29
After the commands name, I'm adding the option --limit -reate= 100 K. I'm hitting 30
enter. 31
Take a look here! 32
See how the download speed was limited to 100 kilobytes per second. 33
Use k for kilobytes , m for megabytes and g for gigabytes. 34
Let's go ahead and see how to resume an interrupted download; by default if the connection drops while 35
wget is downloading the file next time when you try to resume it, it will start from the beginning. 36
I'm interrupting the download and start it again without the limit rate option. 37
And when it has already downloaded 100 megabytes, I'll stop it. 38
If I resume it, it'll start it from scratch. 39
I'm resuming the download and it has started from scratch. 40
This behavior is not good for large files; to resume or download, use the -C option. I'm starting 41
it again and at 100 megabytes I'll interrupted. 42
I'm adding the -C option. 43
OK, and I'm resuming it using -c 44
And you see how the download is continuing from where it was left. 45
So from 100 megabytes. 46
If you want to download multiple files at once, you can use the -i option and the path to a file 47
which contains each url on its own line. 48
In this example, I'll download these two images from Kali.org First, I'm opening the image in a 49
new tab. 50
And I'm copying the address. the url/ 51
On Linux, I'm creating a file; for example, images.txt and I'll paste the address of the 52
image. 53
I'm pasting from the clipboard. 54
And they'll do the same for the second image, for this one. 55
I'm saving and exiting the file and I'm running wget- i and the path to the file, it's 56
in the current directory 57
so I'm writing its name and wget has downloaded both images; they are in the current directory, 58
in the user's home directory. 59
For a huge download, like an iso file, you may want to put the download in the background using the option 60
minus B. 61
I'm running again to command that's downloading the iso file and I'm adding the -b option. 62
It is running in the background; you can always check the status of the download using tail - f 63
and the log file. 64
This is the name of the log file. 65
wget- log 66
You see its progress. I'm stopping it pressing control and c; I'm killing wget. Even better, you 67
can use nohup to make it immune to the terminal closing. 68
It will not receive the hang up signal if you close the terminal or your ssh connection drops. 69
So nohub at the beginning. 70
This is even better. 71
Sometimes you want to create an offline copy of a site that you can take in view 72
even without the Internet access. Using wget you can do it easily. 73
Let's see the options wget 74
--mirror 75
--convert-links --adjust-extension 76
--page-requisites 77
--no-parent and the site address, for example, http : // example. 78
org 79
Let me explain to you the options: mirror, among other things, makes the download recursive. 80
Convert links will convert all the links, includingCSS stylesheets, to relative paths, so they will 81
be suitable for offline viewing. 82
adjust-extension will add suitable extensions to the file names, such as html or css depending on 83
on their content type. 84
And page-requisites will also download things like CSS style-sheets and images to properly display the 85
page offline. And no-parent is necessary not to ascend to the parent directory as well. 86
It's useful to restrict the download to only a part of the site. 87
You can also shorten the command and run it as: wget -mkEpnp and the url 88
Note that there are 2 p letters. 89
The first one is for page requisites and np is for no parent.