Nginx HTTP/2 vs Nginx SPDY/3.1 vs Nginx HTTP/1.1

Previous World Country Flags demo webpagetest.org tests compared Nginx HTTP/2 vs Nginx HTTP/1.1 on Centmin Mod 123.09beta01 branch with added support for Nginx's new HTTP/2 alpha patch [nginx.com]. Now, I'll add Nginx SPDY/3.1 tests into the mix with or without ngx_pagespeed. I used Webpagetest.org's setDns parameter to isolate and point to the IP address of the Nginx SPDY/3.1 server (104.152.214.227) in my cluster which was the same IP address server used for the other 4 tests for HTTP/2 and HTTP/1.1 done previously.

Update: Nginx HTTP/2 version 2 patch fixed some my ngx_pagespeed issues. However, webpagetest.org does still report Error: Timed Out either on First View or Repeat View tests stage. Below results are with Nginx HTTP/2 version 1 with added Nginx HTTP/2 version 2 patch results further below.

Centmin Mod 123.09beta01 Nginx compile options with --with-http_v2_module:

nginx -V
nginx version: nginx/1.9.3
built by clang 3.4.2 (tags/RELEASE_34/dot2-final)
built with LibreSSL 2.2.2
TLS SNI support enabled
configure arguments: --with-ld-opt='-lrt -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -mtune=native -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -Wno-unused-parameter -Wno-unused-const-variable -Wno-conditional-uninitialized -Wno-mismatched-tags -Wno-c++11-extensions -Wno-sometimes-uninitialized -Wno-parentheses-equality -Wno-tautological-compare -Wno-self-assign -Wno-deprecated-register -Wno-deprecated -Wno-invalid-source-encoding -Wno-pointer-sign -Wno-parentheses -Wno-enum-conversion' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_ssl_module --with-http_v2_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module --with-http_secure_link_module --with-http_realip_module --with-http_geoip_module --with-openssl-opt=enable-tlsext --add-module=../ngx-fancyindex-ngx-fancyindex --add-module=../ngx_cache_purge-2.3 --add-module=../nginx-accesskey-2.0.3 --add-module=../nginx-http-concat-master --add-module=../openresty-memc-nginx-module-4f6f78f --add-module=../openresty-srcache-nginx-module-ffa9ab7 --add-module=../ngx_devel_kit-0.2.19 --add-module=../set-misc-nginx-module-0.29 --add-module=../echo-nginx-module-0.58 --add-module=../redis2-nginx-module-0.12 --add-module=../ngx_http_redis-0.3.7 --add-module=../lua-nginx-module-0.9.16 --add-module=../lua-upstream-nginx-module-0.03 --add-module=../lua-upstream-cache-nginx-module-0.1.1 --add-module=../nginx_upstream_check_module-0.3.0 --add-module=../nginx-module-vts --add-module=../headers-more-nginx-module-0.261 --with-openssl=../libressl-2.2.2 --with-libatomic --with-threads --with-stream --with-stream_ssl_module --with-pcre=../pcre-8.37 --with-pcre-jit --add-module=../ngx_pagespeed-release-1.9.32.6-beta

Nginx HTTP/2 World Flags Demo

I've recreated a Nginx HTTP/2 World Flags Demo page so folks can check out and compare both Nginx HTTP/2 vs non-HTTP/2 performance. Note for Nginx HTTP/2 + ngx_pagespeed enabled (which is broken) webpagetest.org test first view results in a timeout eventually, so could be related to ngx_pagespeed not working with Nginx HTTP/2 patch version 1.

6-way WebPageTest.org comparison from Los Angeles location with Chrome 3G Mobile 300ms RTT connection speed and 1920x1200 resolution window. All 3rd party scripts were blocked for testing purposes.

Nginx SPDY/3.1 + ngx_pagespeed was the fastest combination, followed by Nginx HTTP/1.1 + ngx_pagespeed, then Nginx SPDY/3.1 + ngx_pagespeed disabled, then Nginx HTTP/2 + ngx_pagespeed (even if it was broken), then Nginx HTTP/2 and slowest by far was Nginx HTTP/1.1 with ngx_pagespeed disabled.

With Nginx SPDY/3.1 + ngx_pagespeed basically all the page elements were nearly inlined (image data uris) into the html document itself. If you check the html bytes chart you'll see this is the case. Also check the individual result on webpagetest.org. Seems to be a case where resource inlining is still faster with SPDY/3.1 or HTTP/2 ?. Actually, found my answer in Ilya Grigorik's High Performance Browser Networking book in Chapter 12 regarding HTTP/2 Server Push:

In fact, if you have ever inlined a CSS, JavaScript, or any other asset via a data URI (see “Resource Inlining”), then you already have hands-on experience with server push! By manually inlining the resource into the document, we are, in effect, pushing that resource to the client, without waiting for the client to request it. With HTTP/2 we can achieve the same results, but with additional performance benefits:

  • Pushed resources can be cached by the client
  • Pushed resources can be reused across different pages
  • Pushed resources can be multiplexed alongside other resources
  • Pushed resources can be prioritized by the server
  • Pushed resources can be declined by the client

Each pushed resource is a stream that, unlike an inlined resource, allows it to be individually multiplexed, prioritized, and processed by the client. The only security restriction, as enforced by the browser, is that pushed resources must obey the same-origin policy: the server must be authoritative for the provided content.

Nginx HTTP/2 version 2 patch tests

Servers were updated to Nginx HTTP/2 version 2 patch which allowed ngx_pagespeed to work now it seems.

8-way WebPageTest.org comparison from Los Angeles location with Chrome 3G Mobile 300ms RTT connection speed and 1920x1200 resolution window. All 3rd party scripts were blocked for testing purposes.

For SpeedIndex scores, Nginx SPDY/3.1 + ngx_pagespeed was still the fastest combination, followed by Nginx HTTP/1.1 + ngx_pagespeed, Nginx HTTP/2 v2 patch + ngx_pagespeed moved into 3rd place now that ngx_pagespeed is working, then Nginx SPDY/3.1 + ngx_pagespeed disabled, then Nginx HTTP/2 v1 patch + ngx_pagespeed (even if it was broken), then Nginx HTTP/2 v1 patch , then Nginx HTTP/2 v2 patch and slowest by far was Nginx HTTP/1.1 with ngx_pagespeed disabled.