Linux, FreeBSD, Juniper, Cisco / Network security articles and troubleshooting guides

It is currently Wed Aug 23, 2017 9:02 am

Tips & Tricks, Questions regarding shell scripts, awk, perl, sed and much more.

Author Message
Post  Post subject: Wget: How to test web server gzip compression  |  Posted: Wed Jun 26, 2013 8:39 am
Site Admin

Joined: Mon Aug 03, 2009 8:43 am
Posts: 95


Wget: How to test web server gzip compression

What is wget

Wget and curl are two powerful Linux/Unix tools for transfering files over HTTP and FTP. For HTTP, wget is especially useful when it comes to injecting specific headers into an http request, troubleshooting servers' http response headers or testing if a web server has gzip compression enabled. In order for gzip compression to work it needs two components: 1. the client has to inform the server that it accepts gzip encoding (compression) and 2. The server needs to be compressed with zlib libraries and to have gzip enabled in it's configuration files. For the 1st, the client needs to send the header "Accept-Encoding: gzip" in it's request. For the 2nd, if the server supports gzip compression, it needs to notify the web browser that the content is encoded (compressed) with gzip by sending the attribute "Content-Encoding: gzip" in the response header.

What is HTTP compression

The web is made mostly of text files containing HTML code. HTML is the language spoken by web servers and web clients (browsers). The average size of html page is growing constantly, so transfering a web page from a server located on different continent will load in 1-2 seconds if it is not gzip compressed.
GZIP compression allows webservers to compress the content of the HTTP response (data that comes after the http response header - the html web pages) before it is sent out the wire to the browser. For HTML, gzip compression can have a ratio of up to 80% making the web experience much faster.

Wget: How to test web server gzip compression

By using -S option in wget, we instruct it to print the server's http response to confirm the returned data is gzip compressed. By using --header="accept-encoding: gzip", wget will use the "accept-encoding: gzip" header attribute when sending the http request to the server, letting it know that gzip compression is accepted.
Print the headers sent by HTTP servers and responses sent by FTP servers.
Send header-line along with the rest of the headers in each HTTP request. The supplied header is sent as-is, which means it must contain
name and value separated by colon, and must not contain newlines.

You may define more than one additional header by specifying --header more than once.

wget --header='Accept-Charset: iso-8859-2' \
--header='Accept-Language: hr' \

Specification of an empty string as the header value will clear all previous user-defined headers.

As of Wget 1.10, this option can be used to override headers otherwise generated automatically. This example instructs Wget to connect to
localhost, but to specify in the "Host" header:

wget --header="Host:" http://localhost/

In versions of Wget prior to 1.10 such use of --header caused sending of duplicate headers.

# wget -S --header="accept-encoding: gzip"
--2013-06-26 14:40:42--
Connecting to||:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx
  Date: Wed, 26 Jun 2013 12:37:46 GMT
  Content-Type: text/html; charset=UTF-8
  Connection: close
  Set-Cookie: phpbb_a2srz_u=1; expires=Thu, 27-Jun-2013 12:37:46 GMT; path=/;; HttpOnly
  Set-Cookie: phpbb_a2srz_k=; expires=Thu, 27-Jun-2013 12:37:46 GMT; path=/;; HttpOnly
  Set-Cookie: phpbb_a2srz_sid=0c54fd16187dcb02d441932d93c4b5d7; expires=Thu, 27-Jun-2013 12:37:46 GMT; path=/;; HttpOnly
  Cache-Control: private, no-cache="set-cookie"
  Expires: 0
  Pragma: no-cache
  Content-Encoding: gzip
  Vary: Accept-Encoding
Length: unspecified [text/html]
Saving to: `index.html'

    [ <=>                                                                                                                                    ] 14,864      --.-K/s   in 0.04s   

2013-06-26 14:40:42 (373 KB/s) - `index.html' saved [14864]

# du -sh index.html
16K     index.html

How to enable gzip compression in NGINX

You can globally enable gzip compression for all virtual hosts configured under your nginx server (preferably) by adding the following lines to nginx.conf configuration file under the http section:
    gzip  on;
    gzip_http_version 1.0;
    gzip_comp_level   6;
    gzip_proxied      expired no-cache no-store private auth;
    gzip_buffers      8 16k;
    gzip_min_length   4096;
    gzip_types text/plain text/css text/xml text/javascript application/x-javascript;
    gzip_disable "MSIE [1-6].(?!.*SV1)"; is the best place to start for documentation.
gzip_http_version - Sets the minimum HTTP version for which gzip compression will be enabled.
gzip_proxied - see
gzip_types - NGINX enabled gzip compression only for these mime types. If you or website analysis tool discovers that specific files in your website are not served compressed and you know you have enabled enabled gzip compression, make sure that the mime type of that file in NGINX's http response header is in the line "gzip_types".

VPSie - SSD VPS servers in AMS-IX, LINX, DE-CIX

Display posts from previous:  Sort by  
E-mail friendPrint view

Topics related to - "Wget: How to test web server gzip compression"
 Topics   Author   Replies   Views   Last post 
There are no new unread posts for this topic. Wget error: ERROR: The certificate of is not trusted.




Wed Jan 14, 2015 6:23 am

mandrei99 View the latest post


Who is online
Users browsing this forum: No registered users and 0 guests
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum
Jump to:  
News News Site map Site map SitemapIndex SitemapIndex RSS Feed RSS Feed Channel list Channel list

Delete all board cookies | The team | All times are UTC - 5 hours [ DST ]