Curl - What is it ?

Let’s make it basic. Curl is a fast and efficient way to pull the information you need from the internet without using a graphical interface, using only command line. The curl command has the ability to transfer data to or from a server using one of many supported protocols, including HTTP, FTP, SFTP, IMAP, POP3, LDAP, SMB, SMTP, and many more. It’s a very useful tool for the sysadmin. Specially for debugging purpose. It’s a must have to know as a SysAdmin working with Web Application.

Curl and proxy

If you company is using a proxy, you need to tell your OS the address of the proxy

export http_proxy=https://ipproxy:portproxy
export https_proxy=https://10.0.0.1:8080

If you do not need anymore, use the unset command or disconnect as these last command are session related.

unset http_proxy
unset https_proxy

OR you can use the parameter –proxy or -x from Curl

–proxy yourproxy:port

curl -proxy https://ipproxy:portproxy "https://devopshandsonlab.gitlab.io/"
curl -x https://ipproxy:portproxy "https://devopshandsonlab.gitlab.io/"

Get the content from an URL

HTTP/HTTPS

 curl "https://devopshandsonlab.gitlab.io/"

Basically, it’s making a GET request on the URL you provide.

If you want to get the content of multiple URL and use the FTP protocol

curl ftp://ftp.geek.com/amazing[1-20].jpeg

List contents of a web url with curl

curl --list-only "https://devopshandsonlab.gitlab.io/about/"

Authentication

You can provide credentials for your GET command as below :

curl -u superuser:superpassword -O ftp://ftp.geek.com/amazing[1-20].jpeg

:info: the parameter -O downloads the file and saves it with the same name than the URL.

Download a file

The easiest way to download a file is to use the parameter –remote-name with the parameter –output to specify the destination

By example, if you want to download the profile image from my website and output

curl --remote-name "https://devopshandsonlab.gitlab.io/img/intro.png" -output "/tmp/vador.png"
curl -O "https://devopshandsonlab.gitlab.io/img/intro.png"

⚠️ Because some URLs contain special characters that your shell normally interprets, it's safest to surround your URL in quotation marks.

Download a huge file like an iso

If you are downloading a big file like an iso, use a terminal multiplexer like screen or launch the command in background and detach the process from your console with the parameter & and increase the timeout for curl with this parameter –connect-timeout 600

It’s possible to displays the progress of the file with option –progress-bar

curl --remote-name "https://devopshandsonlab.gitlab.io/img/intro.png" -output "/tmp/vador.png" --connect-timeout 600 --progress-bar &

Download URLs From a File

If you have a file with a lot of URL, you can combine xargs and curl

 xargs -n 1 curl -O < listurl.txt

Upload a file on FTP

If you want to upload a file, use the parameter -T . If you want to append to a file, add to this command the parameter -a

curl -u {username}:{password} -T {filename} {FTP_Location}

Troubleshooting

As usual with Linux, the command is coming with verbose parameter -v . It will give you more information/details as by example response header.

curl  "https://devopshandsonlab.gitlab.io/about/" -v

if you want to display only the response header

curl --head "https://devopshandsonlab.gitlab.io/about/"